Scandinavian research, which is an institution based in the University of Copenhagen, just to make that clear.
Basically,
my talk is about putting TEI to use, because we were presented with a task
at the Department of Scandinavian Research, where we had to digitize the place names of Denmark, and
publication that's been going on since 1922.
And it's still going on. We're not done.
As it turns out, we lost a war sometimes around the turn of the century, and
we lost a large part of the southern Jutland area, and the King
decreed that we should write down all the place names in that part of the country, in order to lay claim to it at a later time.
An important part of nation-building is to be able to show that all the place names that you have in your
regional area is from your language, and it means stuff for you.
So this has been going on for almost a hundred years. The original
commissariat was introduced in 1903, so it's more than a hundred years.
To date, there is 26 printed volumes in this
publication. We cover about two-thirds of Denmark.
And
like I write, it's just shy of 10,000 pages and approximately 70,000
entries, not necessarily
70,000 place names, because a lot of the entries are
multiple entries that has a lot of place names in one entry.
So the funding is from something called the Diggday Project, and the paper in the abstracts has the website and everything else.
Yes, that's the thing.
So this is basically what it looks like in D.S.,
Denmark's
place names.
It begins with the name of the place, Söneherr, in this case,
and then a lot of gibberish, and then the Söndreherrut is an historical name from
1594.
Same thing with Söndreherritt,
is a place name that is documented in Elie Nogai's
Yordebo, which is a place where they wrote down who owned different pieces of land.
So basically, that's what the researchers have been doing, is they've been putting this
together and researching each and every place name and finding out when was it first mentioned, where was it mentioned, and so on and so forth.
And then there's a
interpretive part that says in this one, it says that
Nordberg is pronounced it Nueherr, or Söneherr, and
the people from the Southland call it the Söneherr.
So, and these were fun to digitize because they're not unicode.
They're just a
simple
mapping of
ASCII, so that they made their computers write a different character.
So,
we have to get from this, once we've done the digitization.
And we chose the TEI-type standard, not so much because it was an incredibly
well-suited format to turn
written entries in a dictionary into database records, but because there's a set contract,
there's an agreed-upon contract between suppliers and
suppliers, and what's the other part of supplier?
Us,
there's a contract that's a standard contract, and as a principle of technology,
I really like standard contracts that has all these things taken care of once and for all, so we could just order them
to make us a TEI-type
digitization of Denmark's place names,
26 volumes, the oldest
printed on paper, so thin that you had to use as a special
equipment to put it in an OCR machine, because otherwise it would tear.
And we would get this thing back, and we knew, once we got it back, that we had a 99.9%,
actually a 99.99% accuracy, we did all the work to make sure of that.
But we didn't just want it like this. We needed to have,
SoonerHair is a place name, so we needed to have that marked up as a place name. The MSK stock is
actually a source.
The C79 is the folio of that source,
and so on, so we had to turn this into an actual database record.
This is a phonetic writing, this has turned around, I hate that. The entry and the source
division, this was a very important part for us to discover. The semicolon between
sources
tells us that another source entry begins, and this was basically the outset.
This is how we came up with the idea.
Let's just Pac-Man this damn thing, because we were thinking we can switch on these topographical,
typographical cues, and know
more or less
precisely where we're at in the text. The interpretation was always,
fortunately, always marked up with a long hyphen, which is, this is the Unicode for a long hyphen,
instead of just a normal hyphen, which meant we had something distinct to switch on. Same thing here with, we have sources and
dates, and this is actually another source, as it turns out. I'll get back to that.
Page numbers, and all these things we wanted to pass so that we would get something like this.
These are Danish tag names, but yeah, I'm thinking you get the drift.
It's Obseis form, it's the place name,
Kiele form is sources, so there's a source, there's the source name.
We have all the historical source names, and the dating to a certain extent.
This is where the dating comes in. This actually turned into be another source.
There was a very distinct lexicography in this work, and
by working with the name researchers, we found out that this was not a dating of the MSK Stockholm.
This was actually a
letter. Every time there was just a single,
every time there would be just a single year or a single date, that would usually mean that they had a hand-written letter somewhere in some collection.
That was actually a source on its own.
This was implicit, obviously, in the text. You had to be a name researcher to actually know this.
But we figured it out, and we could make this into actual database records.
This can be imported into any type of database that you want.
It doesn't matter what, and another big discussion in the project was, what type of database do we want to put the place names in?
I made a big effort in explaining to these people that it really does not matter.
We can keep it in XML, we can put it in a relational database,
we can put it in one of these new fancy database schema that are coming up.
Another important part of the project that made the project succeed was that since we were working,
name researchers and technologists together on this in the very beginning,
we actually knew what we wanted. When we were finding somebody to do the digitization,
we actually was able to be very precise in our specification on what we needed.
The TEI side has this wonderful thing with the standard delivery contract.
It says the delivery is completed when a 5% sample yields the agreed upon accuracy.
So what we basically did was we hired a bunch of student assistants to do that.
Much cheaper than we actually thought, and student assistants would come in,
they would get a delivery proposal from the digitizer, and they would read 5% of it and mark up all the errors.
And then we exactly knew if we were getting what we had paid for.
This to me as a technologist and as a software man, this was wonderful because for once we weren't
discussing if this or that feature was actually what we agreed upon. This was just completely
above board. This is only 99.3% correct, please do it again.
And they couldn't complain about that, they tried, but they couldn't.
So the method that we used was actually, again, it was based in the fact that these things
as it turned out followed quite a stringent lexicography. So the idea, at least, the thesis
that we had, hypothesis that we had was that we could simply eat this front to back, that we could
just make a computer program that like Pac-Man, we're going to eat the place name, we're done
with the place name, but then we put that in the database, then we go on, oh, there's a source,
you know, and it would go on and on and on. Ingesting the entries more or less linearly,
as it turned out, obviously, we had to shortcut that every then again. Interpretations,
which are usually at the end, but which are very distinctly marked up, was just too low-hanging
if it was just too easy to get at. So we would take that out. And also we found out quite early
that there's a lot of entries which has multiple place names in it with their own lexicography,
their own way of distinguishing between several entries in the same entry. Another important
part of the method was that definitely not all the sources and not all the necessary authority
information was presented in the foreworks of the book. So we would successively build up these
authority files. Whenever we found something that we thought might be a new source, then we would
ask the researcher interactively as the program was running, the little pop-up window would come up
and ask the researcher, is this a new source? Is it something entirely different?
The good thing about this method was that in this way, about I think maybe six weeks in after we had
the TEI side, we were done with 80% of the records. And after that, it was merely a question
of how much money do we have? How long can we continue doing this? Because there was this
very, very strong curve of diminishing returns that using this way of doing it immediately yielded
all the low-hanging fruit. And we were left with a lot of stuff that some of it didn't follow,
the lexicography, some of it would all of a sudden have new types of information in it,
and so on and so forth. And we would write programs to deal with that. Having done it this way,
also we knew... I don't know how to go back. I want to go back.
We would keep a lot of processing information in the records themselves. The top, all the...
What are those called? Those are called attributes of the XML entry. We would have a lot of other
attributes on that. That would be like, has this been fully passed? Is there something we don't
understand? Are there uncertainties? And so on and so forth.
The platform that we used for this, I'm ashamed to admit, was a board and paid for platform.
As a university, we have access to all the Microsoft development programs for very little
money. And the one thing that made us settle on C Sharp and Visual Studio was the third line on
this one, Link. It's called Language Integrated Query. If you haven't heard about this before,
check it out. It is wonderful. Basically, what Link is is a technology, it's a Microsoft technology,
that allows you to work with data as if it was code. You can write inline queries
into data architecture of some sort. This is called Link for XML because it can query XML files.
But there's also a link for SQL, and there's a link for objects, and there's a link for all kinds
of stuff. And you can write these queries inline so that while you're doing this work in your
IDE, in your integrated developers environment, then you can actually see if this query is going
to yield any results. And you can see if you just made a crazy query, or a circular query,
or all these problematic things. So Link for XML saved the day because as a link object,
an XML object becomes just a piece of queryable data structures.
The other important thing that we did was the externally configurable. We made this parser so
that the parser could be configured. I will show you the configuration file in just a second.
The core class that we created had one member, something it was called pass, and it took an
object of parameters, and it returned the pass text elements, which was basically two XML files,
one for the XML objects that worked, and one for the ones that failed. So that was wonderful.
Then we could, I think I've written maybe a hundred of these little parser modules,
but as long as they all accepted this interface, then I didn't have to worry about it breaking
other code or doing anything else. And usually what a single parser module would do was to
take all the entries in the data, and do an entry by entry and node by node iterations,
and check, you know, can I pass this would be a usual run through. Then if can't pass,
whoops, put it in the fail file, if I can pass it will go right ahead. And by then we would
gradually ingest this whole thing. The dictate pass XML is the runtime specification so that
as we were building up this library of small parsers that would usually each do a single thing,
as you can, these actually have nice English names, create abbreviation and source lists,
add page numbers, split multiple, and mark up the interpretation, and so on. And these could
just run forever. This was a wonderful thing for the process, because when we had done like maybe
20 or 30 of these, and we discovered that we had done some horrible mistake in the beginning,
we could go back to the point where we knew that things were good, and then we could just
move on from there. Each method has an attribute called has run, and by switching that from false
to true, it wouldn't execute. And then you could just take all the trues and make them false again
up to the point where you knew you had a determinate state, as we call it. That's basically
it. I've tried to keep this not so much as it says in the paper, so that if people like what
they heard, they can look up the paper and read that. And yes, I will be taking questions.