direct analysis with data mining methods.
And so the basic idea that I'd like to present
is that I was able to use a search engine interface
to create an intermediate corpus that I
was able to do data mining on.
So the outline for my talk, I'm going to first talk
about non-extressive use.
And I'm going to talk about how I built this database.
Third, to discuss what the advantages
of this particular method of creating an intermediate corpus
are, and finally, to show you some canned slides
from the open encyclopedia itself, the interface.
So I wasn't quite brave enough to give you a live demo.
So the first thing I want to talk about
is what are the constraints that we're under?
So in my country, in the United States,
and in a lot of other countries, we
have this idea of free speech.
And it's one of our core values.
It's the sort of thing we fight wars for.
But we carve out a section of free speech
to protect creative works.
And that demonstrates how much we value copyright
and how important it is.
So I want to start out by saying that I really like copyright.
But there's some interesting vagaries of law and politics
that have led to an interesting situation.
And I sort of summarized this with,
I don't know if this is, I'm not a lawyer.
I can't say how solid ground this is.
But the consensus in the community
seems to be that anything up to 1923 is good.
And anything after 1924 is not safe.
There's this hard cutoff.
And the reason for that, which is really
distressing to a lot of people, is not
that everything after 1923 is generating huge revenue streams
and is very important to people.
There's this large class of what are called orphan works,
where the problem is not that anyone is actively
protecting the copyright on a work.
It's that determining whether a work is under copyright
is, to use a computer science term, undecidable.
There is no amount of searching that
will guarantee an answer, yes or no.
So the effect of this practically
is that this guy, Warren G. Harding,
who was elected US president in 1920, he's free and clear.
You can do anything you want with him.
But this guy, Calvin Coolidge, who
was mayor of my former hometown of Northampton, Massachusetts,
and became US president after Harding died in 1924,
he is off limits.
You can't touch him.
And so from the perspective of something
like the HathiTrust database, it is
as if we were living in an alternative reality
where shortly after the inauguration of President
Coolidge, the Earth was destroyed by an asteroid,
wiping out all life and all human civilization.
So where are we left with?
What can we do with 20th century books?
And what keeps coming up in the legal circles
is ideas about non-consumptive use.
This always reminds me of La Boheme or something.
A better term that I've heard recently
is non-expressive use, where expression
means the particular sequence of words
that were used to express a work.
So for example, I can describe for you a basketball game.
I can tell you the score.
I can tell you what happened, but I can't show you
the video of the basketball game.
And so a lot of people have identified text mining,
pattern recognition, machine learning, these types
of methods as the best chance that we
have for doing something with the 20th century.
And for me, that's great because I
spent six years in graduate school
developing an expertise in text mining and machine learning.
This is my superpower.
This is thrilling for me if I can use my superpower
to somehow avert the catastrophe of 1923.
So that's my hope, but I'm not going to tear off my shirt.
But the problem is, in order to do data mining, we need data.
And so usually what we do is we take some documents
and we apply a data mining method to them.
But we can't do that here.
Well, there are certain cases.
If you sit down with people, have a beer with them,
convince them that you're a nice person
and you respect the work that they've
done to put into digitization, there
are cases where content hosters will provide direct access
to documents.
But it's a real risk for them.
And increasingly, they're not willing to do it.
What they are willing to do sometimes,
and this is, for example, the JSTOR Data for Research API.
Are people familiar with that?
People used it?
So they have basically all of the periodicals
that have ever been published in academic contexts.
And what they will give you is this word count vector view.
So you can request up to 1,000 of these at a time.
But it's a very restrictive view.
It's just this word occurs this many times in this document.
They will sometimes give you bigrams or up
to trigrams, three word terms.
But it's pretty much the same idea.
And what I was doing, what I was exploring
after talking with the folks at Google about what
they would be OK with, is something
that's a little bit in between.
So rather than getting these word count vectors where
you completely lose word order, we're
going to be using a search engine
to get tiny little snippets, the sort of snippets
that you would see when you type in a Google search.
And it's those tiny little intermediate documents
that I'm going to run my data mining algorithms on.
People OK so far?
So next I'm going to talk about how I actually
built this database.
And here's the big flow chart.
If you start at the top left, I provided Google
with two things.
One was a list of volume IDs from their database
that appeared to have titles similar to books
that were reviewed by the American Journal of Archaeology,
which I had from JSTOR.
The second thing I gave them was a list
of place names, which were the names of archaeological sites
in the Greco-Roman world.
So the Mediterranean, Europe, and the Near East.
They took those volume IDs and applied the queries
that I'd provided to their search interface.
This is all internal.
And gave me back a big file that had the search result
snippets for those queries applied to those volumes.
Does that make sense?
I then took those, did some filtering on it.
I'll discuss that in a moment.
Ran my topic model algorithm on it.
And then collected a bunch of statistics post hoc.
This will all become clear when I show the examples.
And loaded it all into a SQLite database, which
drives the website that I'll give you the URL for.
And that website then links back to Google
in a number of different ways.
So just to summarize, I gave them about 23,000 volumes.
I gave them 50,000 volumes.
They actually had about 23,000 of them.
Around 5,000 site names.
And the result was 4.4 million snippets.
So this is not a trivial corpus.
But even 4.4 million of these snippets
is vastly less than the text in 23,000 volumes.
So this is a very restricted view.
We had to do quite a bit of filtering
to get a useful corpus.
Some of the site names that I'd gotten from the Kazatier
were not useful for a number of different reasons.
Some of them were ambiguous with English words,
like the town of Split in Croatia.
That did not produce useful results.
There's a town called, I'm not sure how to pronounce it,
Die in Gallia Narbonensis.
In this country, I believe you can guess
why that was not useful.
Other more subtle ones, so the original name
or the Hellenistic name of the town of Amman in Jordan
was Philadelphia.
But if you search for Philadelphia
in modern archaeological books, you're not going to get Jordan.
You're going to get something else.
We also had to filter the snippets themselves.
If you just search for a particular word in a book,
a lot of the results that you'll get
are from the back of the book index.
And the snippet that you get from that
looks like one word and a whole bunch of numbers.
That was not useful for data mining.
We split those out.
Tables of content similar.
And finally, the standards or preprocessing
that we do in topic modeling to remove syntactic words
like the and of, et cetera.
So I want to talk a little bit next about the benefit
of this particular means of building an intermediate corpus
relative to a pure engram or a unigram approach.
And to do that, I'm going to show you
what some of the data looks like and identify
some of the problems and why it helps
to have a sequence of text and not just a list of words.
So this is three examples.
And what you can see is each line has three fields.
The first is a volume ID.
So this points to a particular volume.
It's an internal Google ID.
It has the name of the town that was the query.
So in this case, it's Carthage.
And it has the result snippet.
So imagine this 4.4 million times after preprocessing.
And the way that these snippets were gathered
is they each have the search term plus 120 characters
on either side.
So I'd like to point out some of the issues
with this representation.
First of all, it is exactly 120 characters.
So words are cut in half.
It's worse than that.
It's 120 bytes.
So if you have a UTF-8 character in there,
that set me back about a week.
A lot of programs do not like fragments of UTF-8 characters.
More importantly, this collection
of archaeological materials are very multilingual
for a number of different reasons.
So they drop in and out of different languages
with a lot of regularity.
And language identification is one
of those things where having a sequence really doesn't matter.
Anyone worked on OCR books?
Can you guess what this problem is?
Hyphens.
Hyphens are handled terribly by most OCR packages.
So rejoining words like documentation,
it turns out to be a very important thing.
Because as Travis showed you, you
can get a lot of clusters of words that are shun, re-annic.
Those words tend to occur together
because the patterns of how people add hyphens
are fairly regular.
There's also a lot of multi-word terms, hadrianic baths.
That's not quite the same as hadrianic and baths.
It's an entity.
Mid-third century, similarly.
We'd like to handle those.
So just to summarize, why is this representation,
why is this search-based representation
more useful than mere unigrams, mixed languages,
unmarked hyphens?
I didn't even show you any OCR errors in multi-word terms.
But perhaps the most exciting one
is that you have the ability to create a very targeted corpus.
You can specify exactly what you want.
Now, a couple weeks ago, I was presenting at a machine
learning conference on a method where
I was able to do a topic model on a full 1.2 million book
collection.
And that's valuable.
It's good to get a model of an entire corpus.
But it's also kind of vague in places.
Even with 2,000 topics, you don't
get as much granularity as you would from a much more tightly
focused corpus.
So I think this is a good direction for something
like Hathi Trust Research Center.
So finally, I want to show you some examples
of the open encyclopedia.
And just to give you sort of the overriding metaphor of this,
we have three entity types, books, topics, and sites.
And the pages, the interface of this system
is really about the intersection
between these types.
So here's the front page.
If you have very good vision, you can see the URL.
But I'll give you that in larger text later.
And it's basically listing.
These are three of 300 topics.
And I'm pleased to give you a random selection of topics
because it gives you an honest appraisal of how
good some of these are.
The first two, I would say, are fairly good.
One is about second century emperors in the Roman emperor.
The second is more of a general topic about excavations.
The third seems to be Cyrillic characters.
That one I wouldn't see as so useful.
But I've also been able to plot the frequency of this topic
as a proportion of the whole corpus over time.
And so you can see that the excavations topic
becomes more prominent.
I need to put numbers on here, but it's
about late 40s, early 50s.
So we can zoom in on one topic.
This is one about Mycenaeans, especially pottery,
because that's the sort of thing that survives.
And so you can see a summary of the terms
and how frequent they are.
You can get the relative frequencies.
But you can also see that the archaeological sites
or the names of sites that are related to this.
And again, you can get a time series
or the frequency of different sites.
You also see that under Mycenae there's some number of volumes.
I can expand that tab and give you actual access
to the books that have this topic for this site.
So that's that middle intersection at the middle.
Using that Pleiades database of geographic names,
I was also able to plot the frequency of this topic
geographically.
And you can see that Mycenae does
seem to be clustering around Aegean and Cyprus.
Zoom in, and you see a bit more detail.
Somewhere on the west coast of Greece,
there's an interesting city called Pylos.
And if you know anything about archaeology or classics,
then the things that are important about Pylos,
it was mentioned in Homer.
It's the site of a Mycenaean palace.
There were a number of Linear B tablets found there.
And it was also the site of a major battle
in the Peloponnesian War between Athens and Sparta.
And that is fairly well represented in the topics that
were found, the zooming in on the topics that
relate to the site of Pylos.
There's names, tablets, personal names.
And you can see that the colors are a little washed out.
But that first topic explodes shortly
after Michael Ventress decodes Linear B in 1953.
There's also palaces, Mycenae, Athenian War, Troy Homer.
So I think this is a good representation.
I can zoom in on Pylos and the Athenian War
and you can see that I get Thucydides, who
is the main commentator on that war.
I can also do that little search box
and that gets me to back to Google,
where right now this interface is not terribly useful
because they're under a lot of constraints
about what they can do.
But we would hope that in the future
this might be a more rich interface.
Perfect time.