The fragmentation and wide geographical dispersal
of Holocaust-related sources.
Material related to the Holocaust
is dispersed for many reasons.
There were attempts to destroy a lot of evidence.
It, of course, occurred over a very wide geographical area.
A lot of people, after the war,
migrated to the United States or to Israel,
taking documentation with them.
And so the end result, we find, not just a dispersal,
but a lot of duplication between material located in Israel,
the United States, across Europe,
as archives have, over time, copied material from each other
and as people have moved around.
So one example is the Adler case.
Hans Adler was a Bohemian Jew
who was deported to the terrorism camp
where he began to document his experiences.
He was then later transferred to Auschwitz,
and then, after liberation,
he returned to then Czechoslovakia
for after some years, emigrating to the United Kingdom.
So what we find is that material that he created
and related to him is spread from King's College in London
to the International Tracing Service,
the Jewish Museum of Prague,
New York, Yad Vashem, and these are just
the institutions that are involved with the project.
Excuse me a minute.
Now, what we're concerned with, with areas,
to try and bring some of these collections together
in what we refer to as a virtual observatory.
Now, we're mainly dealing with collections as a unit,
collection-level metadata, such as archival finding aids
and collection descriptions.
And in addition to creating a single place
to browse, search the records that belong
to many different institutions,
we also want to try and enhance that
and allow people to create new connections
between that material.
So we see this as a kind of an exercise
in layering different types of metadata.
The foundation is obviously the metadata
from the archives themselves in the form of finding aids.
But the AV project is also working
to create a multilingual thesaurus, we call it,
which will contain thousands of person names,
organization names, relevant places.
And so that is sort of another,
the basis of our virtual observatory.
In addition to that, we feel there's a need
for attempting to extract automatically
additional metadata in order to integrate
these collections together.
And we also see this automatic extraction of metadata
as a way to sort of bootstrap user-generated metadata,
perhaps involving crowdsourcing to validate
extracted tags and keywords.
And in addition to that, we plan to allow users
to create their own annotations
and create their own virtual collections,
which they can use to hopefully connect these sources
and to share with each other.
So we see this leading to some services
which we want to provide for archives
that participate in the ARRI project.
Now OCR is one thing we think is going to be
an important stepping stone, and at least initially,
a tool to help us get information into our infrastructure.
Although some of the archives that are participating
in this project have a lot of very high-quality
electronic records, perhaps the majority of them
have a great deal of collections that are still on paper,
in particular just finding aids,
like the one I referred to in that earlier slide.
This is a very small one, but typically,
they would run to several hundred words
describing the general content of a collection.
The people who were most prominently referenced in it
in the relevant places.
So we think it will be a useful stepping stone
if instead of having to create these electronic records
from scratch in order to participate in ARRI,
that people can just attach a scanned finding aid.
And we think it will be a useful exercise
to try and extract from that metadata
which people can use to search
those electronic finding aids.
And that's where Named Entity Extraction comes in,
which really led to the question, how feasible was this?
Would the quality of the OCR rule out
being able to extract any useful information from it
for the purposes of searching
and creating search indexes and the like?
So we came at this also from another direction
which was, which fed from a previous project
that we were involved in at King's College,
which gave us a bit of a background in OCR.
And it was all about creating customizable workflows
for the smaller, less moneyed institutions,
shall we say, that would allow them
to more easily create OCR workflows
that were better tailored
to the needs of historical material.
And I'm just gonna attempt to demonstrate
about what we came up with as a prototype
that I hope will give you a bit of an idea of what I mean.
This, oops, this is a web interface,
if I can get it to work.
And this is front end to a way you build workflows
around OCR.
First of all, I'm gonna load in an image here.
There you go, and that you can then navigate
and browse in the web browser.
This image is one of those we're gonna talk about later.
It comes from King's College Archives' Kelly newsletters,
which are the newsletters from a ship,
a Royal Navy ship in the Second World War.
So next thing we're gonna binarize this image.
We've been turning it from color to black and white.
And as you can see, there's a great deal of damage
that's been done to the general definition
of the characters here.
So we need to change a few parameters
in order to get something that's more likely to OCR well.
Whoops, what that was.
Okay, I'm following on from this.
We just add successive steps which perform
new transformations on that image,
removing the borders,
deskewing it so that the lines are more
on the straight plane.
And we can go back and adjust various things in that.
And finally remove anything that's,
well, it doesn't look like text.
Doesn't always work quite well.
And now, once we've got taken that image
from its original to something
that looks more amenable to OCR,
we can then choose from a variety of different
optical character recognizers.
You can use AbiFineRead or Cuniform.
This time we're gonna use Tesseract,
which is a free OCR that Google now manage.
So we've got some text out there.
So the interesting thing is you can go back
through the workflow and see exactly how
each individual step is reflected in the final output text.
So once we got there, we thought, well,
we could do something else with that text.
So it's possible to then put on extra nodes afterwards
to do post-processing.
For example, this one I'm gonna add here
is a Wikipedia annotate node
which calls the DBpedia web service
and gets back a bunch of things that DBpedia thinks
is interesting in that document.
And it's possible to add on a variety
of different named entity recognizers after that.
I'm not gonna go through them all.
There's a mass shape node.
So moving on, this is what we came to
when we wanted to do a sort of more rigorous experiment
to see how well this material would actually work
when we tried to do named entity recognition on it.
And we started with two collections,
one of which you just saw in my screencast,
the HMS Kelly newsletters and another,
which was a set of Holocaust survivor testimonies
from the Veena library.
And they're both backyard to about 93% word accuracy,
which may sound high, but really isn't.
And that's quite noisy as far as OCR goes.
Now we focused on extracting person names,
location names, and organizations,
mainly because we felt these were the kind of entities
most interesting to our users in constructing queries
about this Holocaust data.
And we ran this test on four different tools.
Two of them were web services, OpenCALA and the upcoming API.
And two of them are standalone downloadable tools.
And we ran everything in a completely default configuration.
So there was no training or otherwise tweaking things
to get a bit of result, particularly the web service tools,
you can't really do much to tweak the output anyway.
Now how we did the benchmark was to manually annotate
our collections.
We had a couple of people do it separately,
and then we checked the results to make sure
that there was a reasonable,
you get reasonable reliability
from that manually annotated source.
And then we compared the results
of the named entity recognizers with our manual annotation.
And so these were the results we ended up with,
with emphasis on the F1 score,
which is the combination of the precision and the recall.
I don't know what that's saying.
Let's hope the battery lasts for the next 10 minutes.
So anyway, you'll notice here that the maximum is one.
So the performance on this, of these tools
on our second world war vintage data
was not particularly impressive.
Now, the other thing to notice is that there wasn't
an enormous difference between the quality
of the entity recognition in the corrected text
as opposed to that from the raw OCR output.
The Stanford NLP thing gave us
a rather bigger difference than the others,
but it wasn't hugely notable.
We had much more success with the location data
rather than person names or organization names.
The locations here in the red bars,
and simply just because they're the most stable entity.
From the period in which our documents are from
as to the external sources of knowledge
that the tools use now.
Now for the person names we found,
the main problem was non-standard representations.
In the Kelly newsletters, for example,
there's often semi-anonymized references to people.
In the Vena library material, particularly the finding aids,
there was a very common usage of the surname comma
first name formulation, which wasn't detected
as a name at all by any of our tools as far as I'm aware.
And we also found that none of the tools
had were context aware enough to be able to
disambiguate entities that weren't people,
such as warships, from those that were,
despite things being called the Kelly
and the Caroline and such.
And I think if you had a single sentence as context,
most people wouldn't trouble with that.
Organizations, the performance was very low,
again mostly due to these kind of non-standard representations.
Many of the organizations referred to
no longer exist in the Kelly newsletters, for example,
they were generally very short-lived for periods of the war.
And there's also a very heavy use of jargon
and abbreviations and such like.
Interestingly, we thought the one tool
that tried to sort of use fuzzy matching
and external sources of knowledge to disambiguate things,
ended up getting it wrong more often than not.
As we see with a name here,
we ended up with Clue Gluck's clan.
And we think in the context of Holocaust research,
we would rather not have an entity at all there
than have one that was not relevant
or not especially relevant.
Stanford, named entity recognizer,
gave us the best results most consistently
across the data that we've run so far.
We hope to follow this up with further tests
on different collections.
In particular, we'd like to see if the slight,
the slight, slightly better performance
that Alchemy API gave us on location types
holds up over once we've tested on more material.
Unfortunately, the one tool which has the most open license
and is completely open source and you can use on,
as far as I know, in any way you want
in commercial purposes or whatnot,
is also the least good one for our material.
But it might be something we've been looking at.
Whether we can train it or whether we can improve it,
it's the default models that it comes with.
So, for our conclusion, we mainly felt that
since there wasn't a great difference
between the entity recognition on the raw and corrected text,
then there is potential in using OCR
to feed into a named entity recognizer
for the purposes of building search indexes
and fasted search and the like.
But again, as we find is often the case with OCR,
there's no one magic tool that will solve all your problems.
It's often a case of testing on your material
and choosing on the basis of the strength and weaknesses
of individual tools or a combination of individual tools.
And we'd like to try and maybe work on a system
where we could tell with voting,
so we can combine the output
of several named entity recognizers
and have them validate one another.
So, in the near term, we're looking at sort of
relatively simple, straightforward ways
we can pre-process OCR output
to get better named entity recognizers for us.
And one thing we're going to try is a simple pattern matching
for things like the surname comma first name formulation
to put it in a form that they might more easily understand.
And going forward, we think there's potential
in integrating the output of the named entity recognizers
with the work that he was also doing
on building this thesaurus containing person names,
organization names, and places,
again, to validate the output and improve the,
and reduce the chances of irrelevant entities.
So, that's the end of my talk for now.
Are there any questions?
Hi.
This is more of a,
I guess, question that you used in the virtual reception.