Okay, good.
All right, we are a mixed group of researchers here
who've carried out this research.
Jackie and I are kind of the more traditional scholars of English,
and Tuoma, who is a graduate student also in my department, also is an engineer,
and he's actually the one who's created the software that you'll be looking at today.
And Tapio Sepinin is a professor of medical engineering
who kind of runs the engineering part of the show.
This work really stemmed out of the fact that I realized that I had an increasing amount of students
who were interested in doing research on TV series,
subtitling of TV series, or movies, or the use of yeah in particular movies, or something like that.
And as many of you realize, the availability of concordance programs
with multimodal support is not very big.
There isn't really an awful lot out there that allows you to do concordances
and then actually access video material from that.
So that was kind of the beginnings of this.
The fact that I'm actually using vampire literature is then just its means to an end more so than anything else.
And I'll really briefly first tell you a little bit about the data that we're using
and how come we actually picked vampire literature.
Then I'll show you the Katma concordance there with the multimodal support
and then the new tool that we've just built that bridges it to the original text.
In other words, the novel that the whole series is based on.
We do actually have the Katma concordance there with the multimodal support that we are giving out freely.
So if you would like a copy of that, please see Tuama.
He's got his DVDs of it with him. So it's perfectly available.
The bridging tool is not because we've just created it and it's still in an early stage.
Okay, now beliefs and legends of undead creatures have existed for years.
I mean, it's a universal idea that some creatures haven't died or somehow spirits and so on and so forth.
The earliest known of writing about this we find in the Assyrian tablets.
So that goes back a long way.
And originally there were spirits and demons, but somehow along the way they turned into blood sucking creatures.
But the common image that we have today derives from Brahm Stoker.
And there you have the idea of the smart, polite, well-dressed and wealthy vampire.
Dracula, who is an old character in certain sense that everybody knows.
But as they've turned from these sort of spirits and demons,
they've also turned more into this Stoker-like character who has some sex appeal.
The more modern characters, of course, have more of that that has been increased.
And that brings us to a sort of turning point because there's been an increase in popularity of the vampire character
ever since Anne Rice, essentially.
And what she did was she turned the vampire from the dark and dreary castles and brought him or her into the city lights.
In Anne Rice's case, of course, it's Lestat who's him, but more the idea of bringing into the city lights.
And focusing on the emotions and anxieties of the vampire.
So you're getting a vampire that is now beautiful, erotic, powerful, quite human-like.
Somebody that you can somehow feel some affinity with because they have human emotions.
But they are still dangerous creatures, but on a certain level they're also loving creatures.
Kind of like a modern day Heathcliff.
And we've seen an exponential growth in both vampire literature and vampire movies since the mid-1990s.
The novels, the films, the TV series, perhaps the idea of the vampire has spread more through the TV series than anything else.
Because they've spread, of course, around the world as well.
And there are masses of them.
But if you look quickly, you can see that coming from 1976 with Anne Rice's interview with the vampire,
you get through to Buffy the Vampire's The Lost Boys, Buffy the Vampire Slayer, the movie version of Bram Stoker's Dracula,
the movie version of Anne Rice in an interview with the vampire.
Then you get the TV series of Buffy the Vampire Slayer's and Blade and Angel and, you know, the spin-offs, etc., etc.
And there's masses of these.
What we'll be looking at now is the Charlayne Harris novels from between 2002-2010 and the True Blood series from 2008 onwards.
And what we're focusing on here is the image of Eric.
He's one of the main characters in the True Blood series.
And just glance himself as being a very easy character or a good character to start looking at and try out what the tool can do.
He's Eric Northman. He's a Viking, supposedly.
He's been a vampire for a thousand years.
And the idea that we sort of started out with was how do other people perceive Eric?
How do the other characters perceive Eric?
How does Eric perceive others?
What kind of a person is he, in other words?
And how does he behave?
Now, in the abstract, we actually said that we would be started out looking at all the adjectives related to Eric.
Well, for various reasons, we didn't get to do that because what you're looking at now is more looking at season one rather than many seasons, which was the original idea.
But what we could do now is look at all the instances of either Eric speaking or other people speaking about Eric.
Okay, so to do this, we started out with Katma, the concordance program, which was created here in Hamburg,
which is really good at doing...
Look, it's like a basic concordance program. It's just it's much nicer in terms of user friendliness than many of the others that are out there.
And of course, it was new.
And then we started talking to the Katma developers and saying, well, can we use this to put multimodal support in it?
So what do you do when you start out with Katma?
And what I've done here is the slide is, unfortunately, you probably can't see very well, but what I've done is opened up a source file.
And in this case, you can see it's an SRT file.
It's the subtitling of the first season of True Blood.
The whole all of the subtitles created into an SRT file and then a structure markup file created on the basis of that.
Here you see edit media.
And what's happened is that we've picked up all the episodes of season one and that tells you also media ranges.
Now, of course, in order to do this, you have to have the subtitles and the actual media files aligned.
But of course, that's not problematic because if you're taking the subtitles off of the media files, then they are actually time stamped.
So you're aligning them by time stamping in effect.
So you can sort of see that you're looking at the text.
You've got the structure of the whole thing and you've got the video kind of pulled in to go with it.
And then you just go and do a normal concordance.
Now, this concordance is done on the SRT files, in other words, on the subtitling of that first season.
And it's called up all instances of Eric.
And in order then to actually see, for example, one of these cases, you find that, okay, you want to look, for example, as I'm not afraid of Eric or something like that.
You want to know, you know, how is this, who is saying this, how are they saying, et cetera.
You then just click on Eric and it will call up the video.
Here's an example of it.
It's Eric. He scanned you twice. He's going to summon us.
So that is here. It's Eric. He scanned you twice.
And there you can see that that's the subtitling just on top of it.
So you've got this movie file coming up and you can actually then go before.
You can also go here from one instance to the next.
So you can find the next instance of Eric or whatever it was that you were looking for.
Okay. So far, so good. You've got the subtitling and you can go into the video file.
Now, we've shown that previously.
But then we realized that what I really want to do is to take a look at how is this same scene portrayed in the original novel.
So Tuomo built this bridging tool, or we're calling it the bridging tool,
which essentially then moves from the novel into the subtitles and therefore also the media.
Now, as you can see here, this is kind of done on purpose.
The media file and the subtitle file go together because, of course, they're time stamped.
So they always go together like this.
But the book, the scene that you're looking for might be earlier in the book or it might be later in the book.
It might not be in book one. It might be in book two.
I mean, you know how it is when people make a TV series or a movie out of a book.
It doesn't necessarily follow exactly the order that it was in the original text.
So what you're looking at is trying to link the textual paragraphs of the novel to the representation on the screen.
But you're using the subtitle track to do that because that's, of course, the only text you have in the actual movie.
The subtitle track, of course, describes and characterizes the content of the film.
Now, in order to do this, in order to attach the time codes to something, whatever it is you've got needs to be divided into sections.
And here what we've done is divided the novel's prose into paragraphs.
The subtitles, of course, are natively divided into utterances.
And the idea is that rare engrams appearing in both the novel and the cross-referenced content of the subtitles are quite good guesses for linkage.
So in practice, what you're seeing here is that it's content-based linkage based on extracted and weighted,
uni, bi, and trigrams of words from the novel's paragraphs and the subtitles' track utterances.
So you're actually searching inside the novel text and then trying to figure out where is it in the video or in the TV series.
And so, for example, the idea is that if you have a trigram of beam me up and it appears only once in the novel and the film's subtitle track, then it's quite likely that it is a match.
Okay, so what do you get?
What we did was we actually then marked up in the novel all the cases where either somebody is talking about Eric or where Eric is actually talking.
And that kind of helped us to do the linking.
So what you're looking at here, for example, is a linking where in the novel I was looking for, he has scanned.
So it's looking for scanned, actually, looking for has scanned.
But it found he's scanned as the lightliest match, and it is exactly a match.
They're talking about the vampire.
Eric, Bill is saying that the vampire over there is handsome.
He has scanned you twice.
And that's essentially what you're getting here.
Now, what you actually find then is that close to this, you get this.
Suki is actually then talking about the vampire he'd indicated was handsome.
In fact, radiant, blonde and blue eyes, tall and broad-shouldered, wearing boots, jeans, and a vest period.
Kind of like the guys on the cover of romance books, he scared me to death.
And this is the image that the viewer is seeing.
So I don't know, you know?
Radiant? I don't think so.
But that was Suki's version of it.
So in some senses, of course, what you're actually seeing as a description of the vampire in the novel itself
is slightly different from what you're getting in the TV series.
And that's, of course, exactly what you're looking for.
You're looking for differences between the two.
Trying to find those and then see what you can make out of that.
In fact, actually, we see that there isn't that much description of Eric.
The viewer's image is largely shaped by the comments made about his behavior, such as the comment,
Eric, you do not humble well, or even I'm not afraid of Eric.
And Eric has a thousand years of practicing deceit.
So the idea or the image that you get of the character sort of more or less stems out of those kinds of comments.
Quite attached is another example.
Are you quite attached to your friend?
So what I searched for was quite attached.
Now, what you see here is that we've also brought in these user-defined markup,
which is the markup that you saw earlier where we had marked up where Eric is speaking or where somebody is speaking about Eric.
And you can see this in Katma here in the background that, you know, I've ticked the box.
Eric and Eric speak so that, you know, whereas Eric speak is underlined in yellow.
That's where he's talking.
And the red ones underlined in red are the ones where somebody's talking about.
That's actually saying Eric the Hunk was asking if I could be borrowed, essentially.
So you can use the Katma's features also to help you find these things.
Okay. Now, that actually leads us into a situation where we realize that the text analysis allows you to find the references in the original text and then link with the TV series.
And as you're doing that, you can also listen to the voice of the character as they're saying things.
And that gives another layer of meaning, of course, because you interpret it on the basis of the type of voice and how people, you know, what are they, how I'm not afraid of Eric, for example.
And then looking at the images adds another layer to the meaning.
And all of these, of course, are crucial to the interpretation.
And the only way you can do this is to have a multimodal tool, one that you can actually then call up all the sound and video as well.
This, of course, brings us to the idea that this is, of course, applicable to any multimodal data.
I mean, okay, we made it for vampire movies, but it can be used for data collected for dialectology, for conversation analysis, for anything really where you're looking at text and sound and possibly video.
And this is the first version of our bridging tool.
As I said, the catma with the multimodal support we are giving away, but the bridging tool, we've just created it and really need to try it out a little bit more.
And one of the problems we're facing is how can we connect the text and the video better when the scene of interest has less commonalities in the two media.
The two examples I showed you, actually, you could find it on the basis of one word, but sometimes, of course, that's not the case.
And we're trying to figure out now how are we going to do that in the future.
Okay, thank you.