you
what you're looking for. But what happens if you don't? Their corpus is vast and their
entire goal is to capture everything that has ever been written. So where do you start?
If you don't know what you're looking for, where do you start? And that seeded some thoughts
in my mind and that's really the basis of what I'm going to show you today.
I should talk a bit more about the corpus because it's got some pros and it's got some
cons. The OCR quality, they use optical character recognition to try and read the printed material
and that is particularly poor, particularly when you look at the long S. And we have this
kind of issue and I could exemplify it by this. We have the suck and suck problem and
you can imagine that Google's OCR routine doesn't see the first word as suck.
So that's a real issue. In terms of sampling, it is not a particularly well-balanced corpus
whatsoever. It's fairly opportunistic so they've tried to grab material when and where they
can which isn't great. Some of the automatic reading of metadata isn't very good either
so the sort of attributions to particularly years and authors again is a bit circumspect.
And finally from a computational aspect, it's a huge data set, 155 billion words over centuries
worth of language. That's a lot to cope with. So what I created is Diview and I'll show
you this later. I should tell you some of the shortcuts that I took. I'm using the Google
books provides different corpora. I'm using the English 1 million corpus which is the
most curated one I could get my hands on. So in their own words, it is they've removed
the books which have low quality of OCR and serials were removed in particular years.
To avoid this problem with a long S, I'm using 1850 to the present day. So everything that
you'll be seeing is more modern. And that is a cheat. I'm just ignoring that problem.
And when OCR improves and the data set improves, we can go back in time. That still leaves
me with nearly 100 billion tokens worth of data. So this is still a huge, huge corpus.
And to try and speed up some of the processes, I've excluded certain words. So I haven't
necessarily used stop words but I have filtered out infrequently occurring terms. And most
of them or you can see it the other way and see it as including some of the most frequently
used words. Okay. So I wanted to create something, as
you saw there, the original, the first slide, you saw Therangran viewer. I wanted to create
something that's just as quick and easy to use. Something that doesn't suck. Something
that aggregates these 100 billion words into one or two views. So you can really get stuck
in so you can see what you're looking for. See what changes over time. And again, something
that promotes browsing and sort of discovering things that you may not necessarily know or
anticipate in the data. And it was my hypothesis that this would help
identify some cultural trends across 150 years that I'm looking at. And we'll see whether
that's true in a moment. And it does this by trying to identify keywords or salient
terms or just other words which occur and are attributable to particular years or particular
decades. So that's what it's trying to do is pluck out what was important, what was
being written about in particular years. And one issue usually is that you can look
at the data and how can you drill down into that. Can you actually view more information?
Can you go back to the source material? So that was very important for me to provide
through this visualization a way of exploring some more detail and potentially letting the
users go back to the original documents that these came from. So I've done that.
If you look at the website, which you'll get the link to shortly, we can provide in five
year chunks looking at language use as well as individual years. And even though I use
the Google books corpus, it's my intention that this should be corpus agnostic, really.
You should be able to feed any kind of data to it. I'd absolutely love to put in newspapers
for instance over time. I just haven't done that. So briefly, the way I've measured salience
is essentially comparing the proportions of how often a particular term occurs in the
entire corpus, so across 150 years. And then just look and see how many times that occurs
in that particular year that you're interested in or particular five year chunk or decade
that you're interested in. So it's looking at the average and then looking at individual
years that may be overused or underused. And I can scare you a little bit with the maths
here. Very briefly, if we have this fictitious corpus made of two documents, both are 500
words, that makes our corpus 1,000 words. The word and you will know is likely to occur
very often. So in this fictitious example, the word and at the top here, can you see
me pointer? Yep. It occurs 100 times, so 10% of the entire corpus is the word and. If we
look at year one, in our documents in year one, which in this example is 500 words, 45
for those, other word and, which is 10% less than our average, which is 10%. In year two,
the word and is used more often. There are 55 occurrences of that, which is 10% more
than the average. And we can also look at another word. We've got this word sausage,
which maybe it should be brat-vest, which appears here. It only occurs 20 times in our
entire corpus, so it's much less frequent than the word and. But nevertheless, in year
two, in this example, you can see it occurs 16 times, which is 60% more than its average.
So this is the kind of maths that I'm going through, and it makes more sense if I show
you. So essentially, if we're looking for salience, in year one, neither the word and
or the word sausage are particularly overused, in which case we have nothing to say about
that with these two words that we're using as an example. In year two, however, we know
that the word sausage is used 60% more often than it is across the entire corpus, so that
is ranked first. And the word and is overused only by 10%, so that would fall much later
down the list. So if you can imagine doing this for every single word that you have,
every single unique word that you have in the entire corpus, you get a ranking. That's
exactly what we do. So the important point here, unlike the Google corpus, is that word
frequency doesn't necessarily dictate if something's salient. What does is whether it occurs often,
and that is somehow unusual. So the method is to go through the entire corpus year by
year or in decade chunks and try and identify the words which are being overused. These
are then ranked according to their measure of salience, and we color that, we apply some
style to present this, and also create those important links back to the Google engram
viewer. So if somebody's interested in particular where they can click on that and then go back
and see the source material. So I'll show you what it looks like. Here we go. We're
looking from 1850 to more or less the present day as far as the data set will go. I'll zoom
in a little bit and I'll show you what I can see. So unsurprisingly, if you look around 1940 onwards,
you've got an awful lot of times to do with war, aircraft planes, tanks, et cetera, et cetera. That
seems to make sense, okay? That was happening in our culture. What's interesting to me also is if
we look, for example, at 1965 to 1969, we've got the term Negro. But then in the next year,
that turns to black. So I think that's talking about the sort of civil rights movement in America,
predominantly. And again, if we look at more recent years, 1985 to 1989 here, the word computer,
as no surprise, occurs. And by way of sort of looping back to the Google data, if you were to
click on the word computer, you get the Google Engrams viewer chart. And again, I don't know if
you can see that too clearly, but it does peak in those particular years around the mid 1980s. So
one other way of looking at this is that my visualization, what Diview does, is essentially
look for the peaks that you would otherwise find here, except it's searching for every single word
and trying to promote that. So please log on. The URL is at the bottom. And I'm happy to take
questions as we swap over. Thank you.