Well, thanks a lot for the kind introduction.
Also, on behalf of my colleagues Bastia Andreas and Avi Udon, I wouldn't hear me.
Well, right after lunch, I hope you're awake.
Had enough coffee to get through this session.
I'm from the German National Library of Science and Technology
or Leibniz Institute for Technical Information in Hanover.
And this is only, let's say, one and a half hours in a train from here.
So it's really around the corner.
And we are a very big library for subjects like chemistry, mathematics, physics, etc.
And we are also a big infrastructure provider for scientific infrastructures.
And especially also for object types, non-textual object types, like research data,
research software, and also for scientific videos.
And that's what my team is doing, the competent center for non-textual materials.
We are developing solutions for those object types.
And we work hand-in-hand with the research group, Visual Analytics,
because they are delivering algorithms and programs, for example,
which are based on artificial intelligence in order to make those object types better searchable.
Scientific videos are, well, the use of those videos is increasing.
Also in higher education, for example, those videos are used in scenarios
like blended learning scenarios, flipped classrooms.
Or sometimes those are the most important, or most of the time,
they are the most important parts of Mesophobe online courses, of course.
But also the use of scientific videos in, like this is Mr. Whitesides,
he's from the University of Harvard, and he's a chemistry professor.
And he says that I increasingly find that the stuff that we do
that has probably the greatest impact in some metrics is done in video format,
and not a paper format.
And he also says that videos are really good if something is too dynamic,
or too complex to only try to explain it in words.
And so the use of video is really increasing, but I have bad news on this point.
The digital preservation coalition says, for example,
that this is an endangered species, videos are endangered species
because of poor management of metadata and poor preservation practices.
And especially if the only copy is on social media platforms,
such as YouTube or Vimeo.
And that's why we thought it would be a good idea.
Our vision was to develop a portal for scientific videos,
a reliable portal, and a reliable infrastructure for objects,
like, for example, conference recordings, lectures, video abstracts,
simulations and animations.
And we developed this portal back in between 2012 and 2014,
together with the Hasselplatner Institute in Potsdam.
And yeah, as you can see, we just celebrated our fifth anniversary in April.
And what we did to...
How did we do this, the user-centered design?
We used a classical user-centered design process.
First, we did some expert interviews to see what kind of technology is around for videos.
And we did user interviews to see what people want to have.
And some requirements were, for example, people want to have a reliable infrastructure,
want to have their videos for at least 10 years preserved,
and they want to cite videos, not only the whole video,
but also a segment of the video, for example.
And lots of other things, like, for example, they wanted to search within the speech.
And so next thing we did was we did personas, and then a sketch, and then wireframes.
And from there, we did, you know, some low fidelity prototype,
and then finally some focus groups, and also the implementation in a high fidelity prototype.
And then used some usability testing and eye tracking to come up with a solution
which would fit all our user groups.
So we did a lot of user-centered analysis and programming here.
The solution was then this sort of portal.
You see the item page here, which is structured,
which gives you first the impression of the object, not the metadata first.
You see the object, and then a structured way of showing the metadata.
And this whole user-centered design process didn't only include the interface design,
but of course also the services which we had to provide for the users,
or which we wanted to provide for the users.
So now we do hosting of the videos in our video repository.
We use standardized metadata description, which is based on data site doubling core
for non-textual materials.
All the videos are assigned by a digital object identifier.
And we also use a technology which is called Media Fragment Identifier,
with which you can really cite a second in the video.
You can see this below the video there.
They have the DOI, and at the end you find the time code,
which you can copy and paste into your journal article, for example,
to really cite a second in the video.
Then you have legal licenses there,
so that people know what they are allowed to do with this object.
And most of the items are under Creative Commons licenses.
And we provide, of course, an embed code,
so that it can be embedded, especially in higher education scenarios,
or in presentations like this, for example.
And something the users wanted to have was also that it's not only about the video, of course,
but the video always comes with a journal article, for example,
or with PowerPoint slides or research data.
And this is also displayed here, and it's linked from, for example,
the article to the video and vice versa.
And we also run a digital preservation system, a Rosetta system,
so that the videos are preserved for at least 10 years.
On top of this approach, we enrich the videos with more metadata,
which come from speech, text, and image recognition,
so that you really can find something in the video,
even if it's not in the metadata, like in the title or in the abstract.
So the process is as follows.
First, we do the scene recognition, and the scene is assigned by a timecode.
And then we do speech recognition, text recognition, and image recognition.
And then from those extracted from the transcripts from OCR and speech recognition,
we map to an underlying knowledge base, so that only the subject-specific terms are linked,
and so that you can navigate in the video with those extracted terms.
So this is the structure again.
First, the video segments.
Then, as you can see, the part of the transcript which is associated with this timecode.
And from there, and the underlying knowledge base,
we have extracted those terms so that you can navigate into the video.
Here, it's the video concept detection.
We detect, for example, a concept like this molecular structure,
and then it's extracted as a term and linked to the knowledge base.
And this linkage to the knowledge base is also good for finding a broader or wider terms,
and also for synonyms.
The visual concept detection was developed in-house by our visual analytics group.
And we trained 73 concepts, which come basically from our subjects, physics, chemistry,
and engineering.
So here on the right side, you see some of these terms,
and it's a classical neural network deep learning approach.
And it's really adopted to our subjects and to the image concepts we find in videos from our subjects.
So this could be completely different for videos from another subject,
and then it needs to have more training.
Oh, here's an example.
For example, this is in German.
I'm sorry, but I think this left is, of course, car, and here it's a chemical experiment.
So those are really typical things you can find in our videos.
As I said, we use a knowledge base in order to only extract the subject-specific terms.
And for the German terms, we use the Gemeinsermannamdertei,
which is a German librarian-structured vocabulary.
And here we find 63,000 terms with which we connect the terms from the video.
And for the cross-lingual mapping, we use the DVpedia,
and also the Library of Congress subject headings.
So I forgot to mention that in the portal, we have maybe two-third English videos
and one-third German videos.
And for our subjects, as I said, those six subjects from science and technology,
this service is completely free of charge.
So if you have good videos, please let me know,
because we would like to have those videos for our portal, of course.
So here are examples on how the users can cite the videos with a DOI.
You can either cite the whole video, the complete video, or also a certain image.
You can cite a certain image, for example, in the middle of the video.
Or you could also cite an equation, which is somewhere in the video.
Or you could also cite a certain quote, which is in the video,
and can use this, as I said, for your papers or for your educational purposes.
As I said in the beginning of this talk, we did a lot of user-centered design.
We used a lot of user-centered design approaches and methodology.
But this, of course, never stops, because we developed this between 2012 and 2014.
And yeah, this whole process never ends, of course.
And we do a lot of user research.
We continuously do a lot of user research.
We use the log files, for example.
And we really track the whole activity of the users here in order to find,
what do people do, where maybe it's something not found or something like that.
But in another project, we did research and new interviews
on what do people do or expect from audio-visual portals.
This was not directly connected to our portal,
but this was done for a project which is called Europeana Media,
where it's about to develop a unified play-out service for this digital library Europeana.
And where we do the user-centered design.
And so we did in-depth interviews in late 2018.
And we did interviews with nine people from education,
six from research and general public too.
And they came from various countries within Europe.
And first of all, we asked them, what do you frequently use as a platform,
which you can find here, which is not really a surprise.
And then we received those answers.
I know it's a lot of text, but it's basically what we already found out
for our audio-visual portal in 2014.
People want in-text search.
They want to add annotations, comments, and not only for the whole video,
but also for the segment.
And they want to have thumbnail navigation.
They want to download videos, embed videos.
They want to display subtitling, create collection playlists,
export bibliographic metadata, and store and record personal information,
add keywords, store keywords, share keywords, and so on.
So this is new for us now.
And as I said, like I introduced, we already have a lot of those features
included in the audio-visual portal.
But now we are thinking about doing an additional front-end maybe
for people who really want to annotate videos and store some personal data,
because we haven't done this already.
And this is basically what we will do now, or what we learned from here.
And then something which is really new, because most of the time after presentations like this,
people ask me, can we use the software?
And now we provide an instance of the front-end
so that you can brand it with your own CI.
This is the solution.
Let's say in between, we can really provide the whole portal for third parties.
What we did is we use a lot of open source software, for example, for OCR.
And we use Tesseract.
And for the speech recognition, we will use a solution based on Caldi.
And the visual concept detection is already open source and on GitLab.
So the whole system is highly modularized
so that we can exchange it if there are better solutions on the market.
And yeah, that's something we will do in the future.
We just started also to provide a video analysis as a service for third parties,
because I think this is especially interesting for big video archives or big photo archives,
so that you could give us those, let's say, 40,000 photos.
And you say, okay, I want to have the audio transcript or the speech or the object detection
or whatever.
And then you receive the enriched metadata.
Also right now, we provide all the metadata, including the enrichment
from the speech text and image recognition as linked open data.
They are available.
You can use this and create your own services on this.
Thanks a lot for your attention.
Thank you.