but I'm also going to show you some results from those tools.
Both things, I think, would be satisfying to different people,
but if you have more questions about either,
you'll have to ask me afterwards because it's a lot to put
in a 20-minute presentation.
So, many scholars and poets have written
about the remarkable experience
of hearing Gertrude Stein's text read aloud.
Al Kermanis has written an opera based on the making of Americans,
which was nominated for a Pulitzer Prize.
And for 25 years until 2000, well-known poets and artists
in the Paula Cooper Gallery in Soho read the making of Americans
aloud for 50 continuous hours in celebration of the new year.
Recently, New York-based artists, writers, publishers,
scholars, and other collaborators have taken that up again,
gathering at the Triple Canopy, an editorial collective
in Online Magazine in Brooklyn, New York,
and they spent approximately 24 hours
in a marathon reading of that novel.
Despite this enthusiasm for the evocative nature of sound
in Stein's work, very few scholars have had the means
or methodologies to investigate the sound patterns
which may have influenced her writing.
And they've had limited means or methodologies
to consider how Stein's own patterns have influenced others.
First, we have to consider whether sound actually makes meaning.
The debate concerning whether or not sound contributes
to how we interpret written text has a long history,
while French theorists from Saussure to Socrates,
or Socrates to Saussure, actually, and Bart and Derrida,
it's always been something of a conflict.
We actually ascribe to Charles Bernstein's idea
about orality versus orality, talking about how orality can
be understood as a stylistic or even ideological marker
or reading style.
By orality, I mean to emphasize the sounding of the writing
he writes, and to make a sharp contrast with orality
and its emphasis on breath, voice, and speech.
It's very important for the project
that will tell you that what we're discussing is orality
as opposed to orality, or the aspect of sound
that comes with writing rather than speech.
Stein scholars have written extensively
about the influence that German and African-American speech
patterns may have had on Stein's style of writing
during the period at the turn of the previous century
when she wrote Three Lives, a collection of three stories
about three women, Melanchth and African-American,
the good Anna, a servant of solid lower middle class
southern German stock, and that's a quote from the text,
and Gentelena, another German immigrant.
For example, in her reading of Melanchth from Three Lives,
Carla Peterson notices repetitive phrasal patterns
that evoke syncopated rhythms much like the ragtime music that
was popular in Baltimore at the time of its writing.
Peterson points to the deep relationship
that it hears between form and content, style,
and philosophy, but the extent to which
Stein's prosody and rhythm has influenced these artists
goes still mostly undocumented.
Stein, who admits, quote, I don't hear language,
I hear tones of voices and rhythms,
is an excellent case study for understanding
how we hear literary texts.
Large repositories of audio and printed cultural artifacts
exist by which we can study sound elements,
and increasing access to these collections of text and sound
certainly opens up larger questions
about how literary scholars can use such repositories
in their research.
For instance, oral histories and personal memoirs and papers
gathered in African-American and German communities in Baltimore
during the time in which Stein lived there and was writing
Three Lives are part of documenting the American South,
an open access collection of texts.
Hundreds of hours as well of audio performances of works
by Stein, her predecessors, her contemporaries,
and current poems who claim to have been influenced by her
can also be found in archives such as Penn Sound, Ubu
Web, and the Internet Archive.
For the most part, however, scholars
do not have the ability to analyze
the features of text and sound that
correspond to what Charles Bernstein calls
the orality of texts.
Their phonemes and prosodic elements
are the sounding of the writing, much less
the ability to compare these features with similar features
present across large repositories of text and audio
files.
Some scholars have done limited work on hand-coding Stein's
rhythmic and intonation patterns.
What you see here is Stephen Meyer's diagramming
the first paragraph of The Making of Americans.
Linda Desmond and Wendy Salkine recently
composed a performance there at University of Baltimore,
composed a performance of Stein's story, Miss Fur and Miss
Salkine that exemplifies the process by which a composer can
use both computational and musical systems
to sound out or read the text.
Desmond's process of setting Stein's story to music
became a systematic translation in which she transposed
the sounds into computational and musical systems.
First, she transposed the sounds into numbers for graphing.
I interviewed her, and she says, quote,
I went through the whole thing and listened
for the phonemes, the repeating phonemes,
and then I graphed it all of them, numbers of occurrences,
each sentence.
Red is the A sound, blue is the the sound, green is the S sound.
I did it by sentence.
I looked at the average for blue, for the TH sound
across the entire piece, and it goes further or further away
from the average.
So there's a kind of rhythm there for the average,
and for the A sound, the average is a little higher.
But they all come together in the middle, which
is the longest and most complicated paragraph.
That big paragraph 12, quote, the dark heavy men
that Stein talks about, where she totally
changes at that point, everything changes,
and then there's this kind of fight that goes on with the A
and the E getting very close here,
and the T and the H getting very close.
But eventually, the E sound takes over,
and the E comes with brightness and sort of this resolution
at the end of the piece.
The DH also has a history of doing phonetic symbolic research,
going back to 2006 with Mark Plamondin and his tool
Analyze Poems to Pattern Finder, which
was Smolinski and Sokoloff in 2006,
and even Mark Alexander and Christian K's work
here at this conference.
And the project that I'm going to talk about now
is called, well, it has multiple names,
so we'll get to that in a second.
We are working with the surrogate of sound, what we're
defining as a surrogate of sound.
So it's basically the pre-speech promise of sound
as it's exemplified in the syntax of text.
And that sort of theory of what sound is
is very important for how we're actually measuring sound.
And I have an article under review
that goes into depth to the philosophy behind that,
but I'm not going to get into that today,
so feel free to ask me.
The tool that we created is called Prosevis,
and it's actually available online.
And you can use it, and the data sets
that I'll talk about today are there,
and you can play with them.
We're working on documentation, as that
seemed to have come last as we speak.
The project is funded by Mellon,
and it's in collaboration, one of the use cases,
in the Caesar Services Grant.
And Caesar is a software environment
for the advancement of scholarly research
at the University of Illinois.
So on to developing this particular tool.
What we've done is essentially a modular approach.
And the model that we use to actually,
so you can go, if you go to the website that I just showed you,
which is now listed at the bottom of this,
you can upload, right now that's only accepting TEI XML
documents, sorry about that, but yes to some of you.
So it accepts these TEI XML documents,
and it goes through a process, and you get text back
that you can then use in the Prosevis interface to analyze.
And the back end behind this is we
adopted the OpenMary modular architecture for research
on speech synthesis.
And basically what OpenMary does,
and you can go to OpenMary right now, too,
if you only have one small paragraph of text
that you want to analyze, you upload your text,
and it sends you back an XML document with everything
annotated according to these particular aspects of speech.
And we took that application and put it within a data flow
so that we could do some other processes based on the data
that it gets from the text, including looking
at it in Prosevis.
So to go back to OpenMary, it's got a very esteemed history
of research behind it, some of it in Germany,
and using the Carnegie Mellon University pronunciation
dictionary, as well as some machine learning that
comes from the research behind the OpenMary application.
And as I said, it produces an XML document
that includes fields for each word.
So part of speech, accent, the phonetic, breakdown, tone,
break index, which essentially tells you
where the word is in accordance with the phrase, the sentence,
or the paragraph, which tells you also how you say a word,
what intonation you're going to use.
Part of speech, obviously, if I'm asking a question,
it's a little different, right?
So we take all this information, and Mary's best guest
on the text is based on basically the kinds of best
guests that research has shown that human beings use.
We think about part of speech when we're doing intonation.
We think about the formation of the word,
whether it's a verb or a noun, where it is in the sentence,
the sentence type, et cetera.
And this is the XML that Mary spits out.
It's based on phrases.
This is the first phrase from, I think it's from Tender Buttons.
A kind, glass, and a cousin is that first part.
And so each word has actually aligned the T element.
And you can see that it breaks down each word into the sounds.
It tells you what tone it is, what part of speech.
And we take that.
So this is actually the OpenMary module in the CSER data
flow environment.
So as you can see, there's a lot of pre-processing
that goes on, because OpenMary right now won't take anything
larger than a paragraph.
We had a lot of back and forth and development
that poems are actually created of line groups and lines,
and that a line group and a paragraph
is a very, very different thing.
That was a fight I had to fight.
And it's part of this flow environment,
but it's a very important part because it
does the basic analysis of transitioning that text
into phonetic elements.
And it creates this tabular.
We create from the meander flow this tabular output.
And the tabular output basically, as you can imagine,
each line is a sound.
So kind has one line because it's one sound.
But spectacle has four.
Even though it has three syllables,
it is actually spectacle.
So in the OpenMary analysis, S and the P
are actually two different sounds.
So it comes out on multiple lines.
But for each line, you get the sound.
You get where it is in the tone.
You also get, which is very important for the pros of his,
front end.
You get where it is in terms of the paragraph, the phrase,
the sentence, et cetera, so that you can relocate it.
Very important part of this project, to my mind at least,
was that this is far more difficult for me
to do analysis on than this.
This is how I'm used to reading text.
This is not how I'm used to reading text.
So one of the really important things for this project
in terms of our development was to make sure
that we're putting this data back on top of how people are
used to reading information.
It actually helps people we found understand the data
a little bit better and be able to sort of move back and forth
between the two.
So what you're seeing here is Tender Buttons
by Gertrude Stein, colored according to parts of speech.
So you could pull back and see what parts of speech
she uses in particular.
And you can compare between things.
You can actually pick certain colors
if you're only interested in nouns and verbs.
We're working with a group at University of Texas
that want to use this with first year students
to sort of help them figure out the way that parts of speech
are working in text.
So they found that really useful.
And you can pull back and see where those parts of speech are.
You can look at sounds in terms of full sounds,
so the whole sound of a word.
You can look at the beginning sounds of words.
You can look at the vowel sounds of words.
And you can look at the end sounds of words.
What you're seeing here is Matisse and Picasso Stein.
One of the very experimental things that she did
was that she wrote pieces that were supposed to be portraits.
She was very good friends with Picasso.
She was very good friends with Matisse.
They were all buds.
They hung out.
And they had this idea that how do you
you've seen Picasso's pick portraits of people?
Stein wanted to do the same thing with words.
And so she wrote a portrait of Matisse.
And she wrote a portrait of Macasso.
Now, the history behind Stein and Matisse and Picasso
is that she liked Picasso.
She did not like Matisse.
She and Picasso were good friends.
She had an understanding with Matisse
that they weren't good friends, but they had mutual respect.
And that's important, because when you start to look,
these are both the beginning sounds of each of these words.
And they're the same sounds for each one.
So clearly, if you know anything about Stein,
she repeats a lot of things.
So she's using the same patterns in each of these examples.
And the pattern you see in Matisse
is a little more scattered.
The pattern you see in Picasso is this one was one.
So each of those words is wah, wah, wah at the beginning.
So you see this sort of bright red block in the middle.
The other thing you notice, though,
that pattern repeats at the bottom of Matisse.
So while the top of Matisse has this,
the bottom of the Matisse actually starts
to look a lot like the patterns that are in Picasso.
And you can see this kind of at first glance.
And why that's important was,
I'm not gonna read all this right now,
but essentially what she does in the Picasso piece
is that she talks about all the respect
and all the work that he does.
But at the end, she kind of gets frustrated.
And there's a way that maybe this is associated
with this blue period.
She didn't really like that.
For Matisse, at the end, there's this sense of,
even if he frustrated her
and the way that he worked frustrated her,
at the end, his work ethic to her was actually respectable.
So it's interesting that you can see
some of these patterns in the sound itself
that is reflected in the content of these portraits.
So looking at parts of speech, who cares?
But basically what you can do,
here are four pieces by Stein,
and you can see that in the middle two,
she uses similar parts of speech.
And in the two texts on the side,
she uses similar parts of speech.
And it just gives you this kind of glance
at what the words are actually doing in various texts.
So then we decided to do some predictive modeling.
And the predictive modeling we did was to say,
okay, well, now we can see sound in text more or less.
What if we could see what texts sound more like other texts?
So there's a supposition in Stein studies
that Stein's Tender Buttons Sounds Like a Cookbook,
and that she actually wrote that in protest
for the high modernism that was alluding to Greek mythology
and lots of erudite matters.
And Stein wanted to write a book
that was more like a cookbook.
So we did some analysis of 10 texts,
based not on the words at all,
but of part of speech, accent, stress, tone,
and break index.
And we compared Gertrude Stein's Tender Buttons
and all of these texts together.
And you'll see most of the texts are by Stein,
Three Lives, Picasso Portrait,
The Making of Americans, Tender Buttons, Matisse Portrait.
And then we have the New England Cookbook
that was written at that time and was very, thank you,
was very pervasive in a lot of households at the time,
including Stein's, we think.
And we have the Odyssey and the Iliad.
These are the translations that Joyce read
and that Stein most likely will have read.
There's some indication that she read
the same translations that Joyce read,
to see where these would break down
in terms of how they all sounded alike.
Plus, these are freely available.
So this is kind of the backend,
what it looks like under the hood.
I'm not gonna go into how the predictive modeling works
right now because I'd rather show you
some examples of the results.
So what you have here on our vertical axis
is Ulysses is at the top, New England Cookbook.
This is a confusion matrix.
You have all the texts on the left.
You have each color represents a text.
So if you go to the New England Cookbook,
which is the second row, it can predict itself,
60% of the time with some accuracy.
If you give it all 10 texts
and it learns New England Cookbook.
If it learns New England Cookbook
and then you give it all 10 texts,
it can pick out the New England Cookbook as itself.
60% of the time based on a moving window of phrases.
And it picks tender buttons 10% of the time
more than any of the other texts.
Interestingly enough, if you do tender buttons,
it knows itself, what it sounds like,
50% of the time.
And it gets confused with New England Cookbook
14% of the time, which is better
than any of the other Stein texts that were in that set.
One thing that made us feel especially good
was that it was able to pick with almost the same accuracy,
Odyssey and the Iliad for each,
the Odyssey and the Iliad
as things that sound alike each other.
So what does this look like in ProseViz?
In ProseViz, you can pick texts
and the text actually shows up on the phrases
that are most like where they go in certain books.
So the pink in this particular example
is the New England Cookbook.
You're looking at tender buttons.
And as you can see from this particular spot
where the word Alice or alas,
Alice was Stein's partner in life,
sounds this particular section,
a lot of authors have said or scholars have said
that it's because of Alice that it sounds like a cookbook.
Sure enough in this particular section,
it does actually color as if it sounds the most
like a cookbook according to the data underneath.
And then if you actually look at New England Cookbook
and tender buttons and turn themselves off,
the yellow is tender buttons,
which you can see all over New England Cookbook
on the right and New England Cookbook is the pink
that you see all over tender buttons on the left.
So that was one example.
What the confusing things you do get sometimes though
is that a list that's made up of words
that are food words.
And again, we're not using words for this.
So it's just the rhythm of those words.
Half that list will look like the New England Cookbook
and half the list will look like the making of Americans.
And it requires a little bit more insight
and reading into the text to figure out why is this,
you know, or what's going on with the data.
So in fact, the interface actually helps you look
at the data, but it also helps you look at the text.
And it's kind of that interchange back and forth
to figure out, you know, what are we looking at here?
These are more analyses that we did
on various parts of Matisse, Picasso, and Three Lives,
comparing which ones sounded most like each other
and you get these very sort of distinct blocks.
So even if the underlying algorithm is off in some respect,
it's measuring things exactly the same
and it's coming up with the same mistakes all the time,
which even if, and that's actually quite interesting.
We did Shakespeare.
There's a Shakespeare scholar at University of Texas
who's interested in whether Fletcher wrote parts
of Henry VIII and you can see,
I believe Fletcher is green
and he sort of shows,
Fletcher shows up at the very top of Henry VIII there,
you see in that bar.
And then finally, what I wanted to show you was,
was a little bit of Three Lives and the actual tool.
And then I'm going to conclude,
except we shut down my tool in the process.
So I'll just explain to you what you see.
Let's see, let me get back to my slide.
So what you see here is Three Lives by Stein
and we compared it to about 200 oral histories
from documenting the American South.
And what you see on the right are a subset of those.
We took some that matched really well
and then some that didn't match really well
to see if patterns would be different
in each of those circumstances.
And the interesting thing, I don't know if the colors
don't seem as bright up here,
is that what I would have showed you
in the actual live tool was that there's a text called
by Grimball that you can see at the top that I deselected
because if it's selected, then the entire thing is orange,
which shows that, okay, for some reason,
it's picking this one over a bunch of the others.
And Grimball is the journal of Margaret Ann Morris.
So it's the journal of a woman.
And for some reason, it sounds similar to that.
Jacobs, if you turn that one off,
that's the purple in the middle.
So once you turn off Grimball, then the whole thing is purple
because everything's mostly like Jacobs.
And Jacobs is the incident in the life of a slave girl,
Harriet Jacobs.
The story that you're looking at, Three Lives right here,
Melanchtha is the part that you're looking at in the middle.
It's a Native American woman who's having a conversation
with her husband who's a doctor
and the whole text is this conversation back and forth
and people in Stein's time said,
oh, it sounds so much like African American speech.
Isn't she wonderful?
And there's been some debate about that in the meantime.
What I found actually interesting is,
so if you turn off Grimball,
which is a journal of Margaret Ann Morris,
and if you turn off Jacobs,
which is the incidence in the life of his slave girl,
there's a third, and then it all goes yellow.
And that's Pringle.
So you turn off Pringle,
you start to get these differentiations
where you can actually say, okay,
why is this section different from these other sections?
And what you find is that the green on the top is,
I think that's wood, wood,
which is a text from Document of the American South,
which is another journal of a civil war girl.
And then at the bottom,
you get these forget-me-nots by an author named Lee,
who's also a woman.
But this red mark in the middle is Wyeth.
And Wyeth is the only text of this set,
the colors that you see here that was written by a man.
And it's called with saber and scalpel,
the autobiography of a soldier and surgeon
from the civil war.
And interestingly enough, what this maps to,
if you go deep, if you go into the text,
is a turning point, just like Miss Fern, Miss Skeen,
that Stein seems to keep doing.
She has a turning point in the middle
of making Americans as well.
But it's a turning point in the middle of the text
where this conversation between this husband
and this wife turns.
He actually starts to get power over the relationship,
the communication that's going on.
At first, he's chasing her,
and then this middle part in the text
is when the power relation starts to change.
Interestingly enough, and I'm gonna need
to do further research, this is sort of a discovery phase.
Obviously, this character in the text is a doctor.
So it was interesting to see that this red block,
which sort of maps to how he's speaking.
And it's not clear when you read it
that the change in inflections,
that how he's actually speaking in this section
maps the best to an oral, a personal narrative
by a surgeon during the civil war.
And it's the only male one with the colors
that actually comes up on this side.
These are the names of the text that I just showed you.
So the conclusions are vague,
but what I think is interesting about this
is something that Stein writes in Portraits and Repetition.
She says, I began to wonder at about this time
just what one saw when one looked at anything,
really looked at anything.
Did one see sound and what was the relation
between color and sound?
Did it make itself by description,
by a word that meant it,
or did it make itself by a word in itself?
And what I think actually in conclusion
is really interesting about this
is that we're really good as a community,
and I think we've seen that from the panel today,
they were really good at working with text.
We aren't as good at working with other objects,
whether it's sound or video.
And part of that isn't necessarily the tools,
it's also a literacy.
We don't have the language to talk about those things.
It's difficult to show things to students.
It's difficult for students to disseminate
a research paper that's based on sound
without having the words to describe it.
And we do have a lot of archives that have
not only text in them, but sound archives.
And the next project that we'll be working on,
or I'll be working on with the group at Illinois
is to try and look at what we can actually do
with our cultural artifacts that are sound files
in terms of access and visualization.
So thank you.