Only a few of you raised your hands.
But let me tell you, all of you are probably going to be archivists for your own email.
You're going to process it.
You're going to perhaps research it someday, since email is so widespread and touches everyone.
It's a wonderful tool of record for most of us.
And let me just explain my personal motivation for working on this problem, which are these
things.
These are the diaries of my, these are the handwritten diaries of my great grandfather-in-law,
who lived in Southwestern India from 1850 to 1930.
And for the last 50 years of his life, he wrote these diaries.
And they're wonderful documents of record, which I discovered a few years ago and I started
going through them.
And I was struck by how much was both directly recorded in them, as well as implicitly reflected
about the time, about the region, and so on.
And I was just thinking what this meant for us, for our generation, what our grandchildren
would have to look back on our lives.
There's a page from 1884 from that diary.
And the obvious answer was that, well, I don't write diaries anymore.
And I guess many of us are too busy to maintain regular diaries for daily events.
But certainly there's some kind of writing that we all do every day.
And that is in the form of email, because all of us, at least in the wired world, send
and receive so many pieces of email every day.
So implicitly, it captures our thoughts, our feelings, our communication with other
people who we are talking to, where we are, and so on.
So I started looking at email archives a few years ago and realized that, of course, it
reaches every corner of the internet.
There's two billion email users, there's about three billion accounts.
And especially with the online providers providing virtually unlimited storage, there's no reason
to delete anything, never delete anything kind of mindset these days.
It's a very consistent record, that means every single day, pretty much, you are sending
or receiving emails.
So there's some clue to what you were doing on a particular day.
That's not true of other things like photographs or videos and things like that.
Email in particular is a very long-lived and open architecture.
It's been around for 40 years.
I'm going to bet it's going to be around for 40 years more, partly because it's not tied
to any one company.
It's not like AOL messaging or it's not like Facebook messaging or something that it goes
up and down with the fortunes of the company because it's an interoperable format in the
spirit of the internet.
Anyone should be able to message anyone else who's on any service.
And that's one reason why it's so easy to use and so long-lived.
It's also very easy to capture and preserve.
We already have this notion that I own my email, right?
That I can download it, I can create a copy, I can move it to another server, and so on.
So people already have this idea of control over email, which is not true, again, of online
things like even Twitter and Facebook.
And it's surprisingly lightweight in the sense that it's only a few gigabytes, if you think
of your email, which captures your entire life pretty much, or, well, a good fraction
of your online life.
It's only a few gigabytes or something which can be stored for maybe a cent or something
like that these days.
And it's not going to be hard to maybe have many copies of it.
It's likely to survive over a long time.
So maybe your grandchildren will be reading your email or will have access to your email
history someday, and they may be interested in it unless you take steps to wipe it out.
And so this is a really interesting phenomenon.
And it's a phenomenon that we can study today because some people have had email, particularly
in universities, they've had it for maybe 30 years or more.
And I've had people who have all their email from the time they joined, say, Stanford,
32, 33 years ago.
So it's almost an entire career.
It's all captured in their archive.
The question is, what do we do with it?
How do we use it?
What's interesting in it?
And of course, there's so much of it.
So obviously, we need automatic tools to go in and figure out what might be interesting.
And so we built this tool essentially for our own archives, for people to look back
on their own archives.
And it's a tool that downloads to your own computer and runs locally so that there's
no privacy issues and so on.
But then we were contacted by a library saying that, hey, we need these tools to help process
born digital material coming in from various donors.
And a lot of it is in the form of email correspondence today.
So what used to be letters is often email today.
And with every new accession that they make, they're getting all these hard drives, and
sometimes they get entire laptop computers, or they're getting files or emails or whatever.
So we decided to focus particularly on email because it captures communication with other
people.
It's sort of like letters.
And so far, libraries have mostly been capturing and preserving this data.
But there's very little processing and delivery.
It's very hard for an archivist to look at 50,000 pieces of email and decide what's to
be indexed, what's to be put in the finding aid, what's to be redacted, and so on.
So they said, can you help us use these tools for processing these archives and maybe even
delivery?
And in the whole special collections process, there are four main stakeholders.
One is, of course, a donor of the material, whoever owns it.
Sometimes it's the person himself.
Sometimes it could be the family of the person to whom the material belongs.
There are curators who decide what to access and contact the donor and all that.
Archivists take the collection and then they have to process it.
Usually they have very limited time because we have, in Stanford, for example, we have
50,000 linear feet of material in the special collections.
And that grows every year by, I think, about almost 2,000 feet.
And we have only a few archivists who have to look at all this material.
So they cannot spend days and days looking at a particular collection unless there's
special funding for that.
And, of course, researchers come in and they want to look at the collection as well.
So we decided to take one particular corpus that we had access to, which is the papers
of Robert Creeley, who's a very well-known American poet.
And we have, as part of his collection, we got access to his email from, I think he started
using it regularly in 1993 until he died in about 2005 or 2006, I'm not sure.
But we started looking at his data and there was a huge amount of data.
There's about 80,000 messages there, of which there's lots of duplication because accession
was performed over multiple rounds and maybe he himself had categorized emails, put them
in different folders, and so on.
So the challenge of processing email is that it's mostly unstructured.
I mean, there's a little bit of metadata, which is very useful, the timestamp and who
the sender, receiver, all that.
But the rest of it is just, of course, completely unstructured.
There's a very large volume of it, there's a very large diversity.
I mean, you use it all the way from setting up a meeting or asking a one-word question
to somebody, to sending a very emotional and deep letter of condolence or love or something
like that.
And it's very hard to explore when you don't know what's in there.
I mean, you can, of course, maybe search for it, search for something if you're looking
for something specific, but you may not even know what to look for when you have an archive
spanning 30 years.
So we tackle this in four steps, and I'll briefly go over the four steps.
We first try and organize it in some way, then we summarize it, we try and engage people
with the archive, and then build some UIs to help exploration, and I'll run you through
that.
So in short, here's something that you can do today, which is take the email collection,
put it in an email viewer, email client like Outlook or Thunderbird or something like that.
But this doesn't scale too well, because you have to still have to read every message one
at a time.
So can we do something better is what we asked, and that's why we've built this system called
Memories Using Email Muse, and it's online at that URL mentioned over there.
You can download it and run it on your own email if you want, or any other collection
that you have access to.
And it supports formats, standard email formats like mbox files.
You can also convert between PSTs and other email formats to mbox using converters.
Then we have to do various data cleaning steps.
So for example, I mentioned we remove duplicates, we also have to merge identities of people
because people's addresses change over time, even the way they spell their name changes
over time.
So we try and merge that so that there's only one identity for a given individual.
So after we do that, then we can come to the interesting steps, which is how do you
organize the data?
So one of the ways we can organize the data is, of course, by folders.
But folders don't tend to be consistent over a long period of time.
People's folder inhabits change, or they start using different clients, and then they start
doing different things, different ways of filing their email.
So one way we organize is to organize by groups of people.
So we infer groups automatically in the corpus based on who goes with who.
For example, if I email three of you together many times, then that's a signal that those
three people are in the same group.
But of course, you may have groups that are related in complicated ways.
You may have groups that overlap.
You may have groups that are nested and so on.
So we came up with a novel algorithm that handles all those cases and identifies as
many groups as you want.
So typically, we say, OK, take the top 20 groups or something like that.
And you can edit the groups if you want.
But the automatically generated groups are a pretty good starting point.
And once you've done that, you can organize the activity by group.
So you can just see who.
So in this case, this happens to be my own email archive for the last eight years.
And there are various groups here.
They don't make sense to you, but obviously, they make sense to me.
And this graph already tells me stories about when I met a certain person, when I started
interacting, or when I started talking to a particular group, or when a project started,
and so on.
And of course, you can click on this at any point and then go to the messages related
to that particular group.
Another thing we do is summarize the messages using this monthly card technique.
So for every month, we have identified the top 30 named entities in the corpus.
And we have placed them on a card per month.
And the cards are colored by the groups that they are most closely associated with.
So the groups are listed on the left.
So if you wanted to see what this particular poet said with this group of people over 10
years, you could just quickly trace, let's say, all the blue colors, or whatever color
that group is assigned, and quickly see all the terms that were the most prominent.
So at least it gives you a clue as to what topics may be there in the archive.
And this is really useful for researchers in particular.
Before they even come in to see a collection, or even before they raise funding for a collection,
they need to know what might be in there.
At least then they have a hypothesis that they can test out.
So giving a summary overview is really useful.
And I'll mention a small technical trick that we invented here, which is that, so
how do you identify the top terms for a given month?
And one of the ways you could do that is using metrics like TFIDF.
Some of you, I'm sure, are familiar with it.
But one of the problems with TFIDF is that it, so TFIDF is trying to essentially identify
terms that are very frequent in a given month, but not in the other months, or in the other
documents in the corpus, in the general case.
And that's good to identify discriminatory terms in a particular document, or in a particular
month of female in this case.
However, the problem that runs into when you have a long-term corpus is that you may
have a term that emerges at some point in time, let's say in month M term T became prominent.
And then it became, it just is very common across entire corpus.
The TFIDF score is going to actually give you a low, the metric is going to give you
a low score for that term, because it's actually a common term across the corpus, it's there
in half the documents.
And you can imagine term T could be the name of a new family member, or it could be a name
like Barack Obama, that it emerges at some point in time.
It wasn't common before, but after that point it becomes very common.
So you want a metric that emphasizes when a new term emerges without respect to the
rest of the, without respect to what happens in the future, right?
So we did a very simple tweak on the TFIDF metric, and what we do now is that we sort
of replay the messages in time, and then at every time instant, or every month, we extract
the top terms without regard to what's going to happen in the future.
So that lets you highlight, like I said, a baby is born, or a new relationship is established,
things like that, because otherwise that name would not be common, it would be, that name
would not be highlighted, it would be common across the entire archive.
There's another thing we do, we identify, we do sentiment analysis on the text to identify
things that may be deeply emotional that the owner of the archive is deeply connected
to.
And for this we used a personal lexicon that we developed ourselves for things like emotions,
you know, love, anger, fear, things like that, life events or family things, milestones,
travel, vacations, and things like that.
And so we, so we had to develop this ourselves, there are standard lexicons available like
Luke and Senti word net and all that, but we found that those are very general purpose.
For personal communications, you need something that's more person-oriented or human-oriented,
and that's why we developed our own thing.
And it's a very simple technique, but it's also very transparent and editable.
So we have, it's basically a bunch of, maybe about 25 categories with 500 words or phrases
associated with those categories, and it's all, it's also editable.
So if you don't like a particular phrase in a particular category or you use it in a different
way than most other people do, you can just change that.
And one of the uses we found for that was that, you know, when archivists are looking
for processing these collections, they need to identify confidential material, maybe health
records, maybe school records, social security numbers, credit card numbers, things like
that.
So we have, we have these custom built as a sort of sentiment, it's not really sentiment
of course, it's just a category.
And then the archivists can flag those messages and maybe redact them or something like that.
So this is a view of the sentiment graphs, how the sentiments change over time.
And another thing we, and it is just a fun thing we did to engage people with the archive
is to automatically create a crossword puzzle out of the text in the archive.
So the words in the puzzle are, the answers are picked from the monthly summaries that
I mentioned earlier.
So these are important terms that appear frequently.
And the clues are simply sentences picked from the archive with that word blanked out
or that phrase blanked out.
So it's a very simple thing, but it's kind of fun when you pull a sentence out of context
and you present it to a user.
And even though I may have written this in my email five years ago, it's still hard for
me.
Sometimes it's easy, sometimes it's hard, and maybe if I have words filled in in the
grid, then it's, letters filled in in the grid, then it's, it gets a little bit easier.
So it's a fun thing.
So I just told you how you sort of enter or identify interesting messages, but you still
have to go through the trouble of reading them.
And at this point, I think I'll just switch to a little demo so that you can see how it
really works.
So I'm going to show you the, this is how it looks for, in this case, this is a subset
of Robert Creeley's email from 2000, mostly in 2001.
Oh, you don't see it, sorry.
Okay, that's better.
So it's Robert Creeley's email for a subset, and even this often exposes problems like
there's this entire year missing in the data set or something, and then you can go back
to the curator and ask them to get more information.
But you can see that this reflects things like, for example, you can see there's a bump
here right at the bottom, and that is a sentiment category for congratulations.
So suppose you click on that, then you can see that, you know, that's because in this
case, Creeley won the Latin Foundation Award, which was a big award for him.
And you have facets on the left, you can turn on and off facets if you want.
Similarly, you see there's a spike here in things like, there's a category called life
event, births, deaths, and so on, and you can see what the, so here Creeley's actually
talking about his sister who was dying and died around that time.
So this just gives you a clue that there's something related to life events in the corpus
at that time, there's some reason why there's a spike.
And when you choose a particular category, you may still have hundreds of messages that
are related to that category.
So to let you quickly flip through many of them, we have this jog-dial interface which
you can invoke by clicking anywhere on the screen.
The way you use it is like an iPod wheel, you just turn this around and it flips through
all the messages very quickly.
So the inspiration for this is the experience of flipping through a book in a bookstore.
You don't, you know, turn it page by page, you just skim it and maybe you stop when you
see something interesting.
You get it flash by.
And most other things on the web are, you know, you have to, very intentional, you have
to go and click somewhere and things open and you change the tab and this and that.
So instead of that, we want an in-place experience.
And I'll mention one, actually I have two small things to mention.
So another is an attachment view of all the attachments, of all the image attachments
at least.
So this is a 3D wall where you can see all the attachments and just zoom by.
And if you see something interesting, you can click on it and then go to the message
that contains it.
So in 3D's corpus, I think there are about 3,000 attachments of which more than half
are images.
So you can get a quick view of, in some cases, tips he took and things like that.
And one final thing I want to mention is the idea of, so I was, you know, testing out Muse
on this corpus.
And I'm, unfortunately, I'm a CS guy, so I don't really know that much poetry.
And I was looking up Robert Creeley to see what he's known for.
And are these archives any good?
Is things that he's known for, are they deflected in the archives?
So I looked at his Wikipedia page and then I realized that it would be very useful to
just sort of throw this Wikipedia page at the archive and automatically illuminate parts
of it that are connected to the archive.
So what we built for that is a browser extension for Firefox and Chrome.
So what you can do is just click on this thing on a particular page.
And it looks up all the terms, it brute force looks them up in the archive.
And it's now highlighted terms that are in the archive.
So I can say, oh, what was his relation to Allen Ginsberg?
And if I click on it, then I see the messages that are in the archive that mention Allen
Ginsberg.
And then I can, of course, go and see the actual messages and why he's mentioned and
so on.
So we call this experience-infused browsing because you're connecting to the experience
of the owner of the archive in this case.
You're connecting that to your browser.
And you could be browsing just any page or you can even paste in text from some document
into a text pane that we give you and we'll do the same thing for you.
So that's sort of a quick summary of Muse.
And I will just, I know my time is up, I'll just conclude with just the message that email
archives capture very important history.
Archives and archivists can and should capture and preserve it, which I think they're doing
a good job of, but not enough in terms of processing and delivering it to users.
So what we're going to do is put the Creely archive, make it publicly available.
Particularly the summaries and some of the graphs and things like that because we cannot,
for legal reasons, we cannot make the entire archive completely available.
We don't know about copyrights from images and things like that.
But if you come to the reading room, then, of course, you can have access to the entire
thing.
But at least people will get a sense of what's in the archive from anywhere in the world.
And when they come to the library, they have a real tool that they can use to actually
browse it.
And I'm sure there will be more tools like this coming in the coming years.
So we have a bunch of future work.
This is, of course, an ongoing project.
We welcome feedback.
We welcome help, if you want to help, code or whatever.
This is an open source project funded by the USNSF, the Movie Social Laboratory at Stanford
and Stanford Libraries.
So with that, I will, we can have some discussion.
Again, that's the URL, if you want to try it.