and I'm going to tell you about some results of my project,
Automatic Recognition of Speech, Thought and Writing Representation in German Narrative Texts.
So, what's all of this about? No, we don't want to update either.
Yes. So, the project is about exploring automatic methods for recognizing a narrative device,
speech, thought and writing representation.
So, it's about the narrative device, not about intertextuality.
And a possible application would be quantitative studies,
because if you'd be able to recognize this narrative feature automatically in text,
it would be possible to do studies over a large number of texts.
And, for example, see how techniques of speech, thought and writing representation developed over time.
So, during this talk, the focus will be first about the two different methods I used.
Rule-based were this machine learning. Okay. It's kind of slipping.
And those are quite different approaches, and I'm going to show you the different success rates I get.
And also, I'm going to talk about different ways to measure the success.
And all of this will be demonstrated on indirect representations.
So, just one type of speech, thought and writing representation, because otherwise it would have been way too long.
All right. Now, what I did in this project was I compiled a corpus of 13 short German narratives,
which were published between the end of the 18th and the beginning of the 20th century.
And this corpus is about 57,000 tokens.
I annotated this corpus manually, and I also did some preprocessing on it that was mainly sentence splitting
and part of speech tagging, where I used the tree tagger and the RF tagger.
And I used this corpus, first for evaluation of the automatic methods, but also for training for the machine learning.
And I did cross-validation on it.
So, what are we actually trying to recognize here?
I was focusing only on the types of representations in this project.
I was not trying to differentiate between speech and thought, for example.
That would be another research issue.
And I distinguished between direct representation, free indirect representation, indirect representation,
and reported representation.
And you see some examples from the corpus next to the different categories.
And, as I said, we're going to focus on indirect representation today.
So, an example would be, for example, in German, eckbert schlug im voor bei im zu bleiben.
Or in English, which has fortunately quite a similar structure, eckbert suggested that he should stay with him.
So, you have a framing phrase, eckbert suggested.
And then you have a dependent clause where the speech, in this case, that is represented, is located.
Yes. So, I'm going to tell you what I did for the rule-based and the machine learning methods.
And then I'm going to get to the evaluations.
So, the rule-based recognition relies on words which I call speech, thought, or writing words that are verbs or nouns,
which are in some way related or can be related to speech, thought, and writing.
And, as you can see from the examples, there are some very obvious ones, like say, ask, think,
but also some which can only in some contexts be used for pointing to such an event.
And for indirect representation, I used quite simple pattern rules, which always start with one of these words,
and then use some surface features or parts of speech tags to recognize the basic structures.
And basically, the examples you see here are the structures my system can recognize.
Of course, this is not every possible structure of indirect representation, but those are the most common ones,
and we will see how far they will get us.
For the machine learning, the learning task was to differentiate between instances of indirect versus the rest,
and an instance is normally a sentence.
So, I was actually trying to decide whether a sentence contains indirect representation, not the exact boundaries of the indirect representation.
Also, not for all types of representation, but for indirect, I did a different kind of machine learning experiment, too.
And there, I did not use sentences as my instances, but something I call sections,
and those sections are actually sentences split up at certain points, at comma, colon, semicolon, closing quotation mark, and won't, that's and.
And those splits I used because often indirect speech is not, does not span a whole sentence, but is only present in part of a sentence.
So, I was trying, if I could increase the results, if I focus more on the relevant part of the sentences.
All right. I used cost-sensitive learning, that is, if you did not, if the algorithm did not recognize indirect,
it was more costly than if it wrongly recognized the sentence as containing indirect speech.
And my learning algorithm was learning random forest, which is an assembly learner, which is rated quite highly.
Actually, yesterday we had a talk where someone used it to distinguish Dickens' style, and she did quite good with that.
I had 80 attributes for my machine learning, and two groups of attributes.
First, general attributes, which were the length of the instance, and also how often the morphological tags provided by the tree tagger appeared.
And I used those to get an idea for structures which I could not predict that they are important.
But I also used some specialized attributes, which came from ideas I had about what could be important for representations.
For example, first and second pros and pronouns, which I considered important for direct speech, or those speech thought and writing words,
which are actually quite important for indirect speech.
And the results you will see were achieved by a tenfold cross-validation.
So, now we are going over to the evaluation, and I will show you the results of several types of annotations.
First, the rule-based, then the machine learning, machine learning based on sections,
and then also the combination of all the instances found by machine learning and rule-based approach combined.
And when I say machine learning, I usually mean machine learning on sentences, so sections is the special one.
And also the intersection of the machine learning and the rule-based results,
so only those annotations are counted when machine learning and the rule-based methods agree.
And the evaluation is always sentence-based, so it's only checked whether the algorithm recognizes that indirect representation is present in a sentence.
It does not have to find the exact boundaries.
But, as I said, there are also different types of evaluations, so what you are looking at, and I will show you first,
the F1 scores. F1 is the harmonic mean of precision and recall. Precision is how many instances of those you got are the correct ones.
Recall is how many instances that are present you actually found with your automatic method.
And the F1 score weighs both of them equally, so it's the mean of both of them.
And I will give you the total success score for the whole corpus, but I will also show you something about the stability of success.
And for this, I calculated the score for each of my individual texts, and I give you the standard deviation of the results.
And then I will also show you a different kind of measure, that is, I calculate the percentage of sentences containing indirect representation per text.
And then I'm going to compare the curve that is the true one I got from my manual annotation and the one I get with more automatic methods.
And here, again, you can look at the correctness of the estimate, that's the mean absolute error, or on the stability, that's the correlation.
All right, so first for the scores for the whole corpus, I highlighted for you the best result, and this is rule-based.
And you can see that the precision is actually pretty good. Recall is not quite as good, but still you achieve an F1 score of 0.71, even with those pretty rough methods I used.
Machine learning is quite a bit worse, and machine learning on section is even worse, yes.
With the intersection and unions, you can see that precision goes up if you intersect, and recall goes up if you do the union, which should be expected, but the end results are actually not better than using rule-based approach alone.
Now, this is not so surprising because indirect is a type of representation that's actually quite structured.
It's probably the most structured one of all the types I am trying to recognize, so it's not such a big surprise that the rule-based method performs good here.
And now we look at the stability of those F scores, and you can see the scores for each of the texts, and the standard deviation in the legend there.
And again, here you see that the rule-based method is the most successful one. It's the highest scores, and it's also the lowest standard deviation.
And the machine learning one is actually the worst, so it's not stable, and it has the highest standard deviation.
Now we'll look at the different type of evaluation. Here you see the percentages of sentences containing indirect according to my manual annotation.
And keep that line in mind, because in a minute several more lines will appear, and this is the one we want to match as good as possible.
All right, so here I bring in our different automatic methods. Again, the orange line is rule-based, and the blue one is machine learning, and the intersection and the union lines are above and below it.
And you can see that actually first all of them are pretty close, and they are pretty similar actually.
And if you look at how closely they match the line that we want to achieve, they have the same problems.
So here at the burger text, all of them are too low, and also here around the Kafka2Museus, there's this huge drop, and none of them mimics it very well.
So the correlations are actually not extremely good. You can see them here in the legend there.
And now I bring in the machine learning based on sections. Just quite surprising. I was quite surprised because you can see it's widely off, so it overestimates the amount of indirect speech by around 15% all the time.
But it has a higher correlation, so it does actually mimic this peak here with burger, and it also does the slope over there quite well.
So I was pretty surprised about that, because in all the other evaluations it was not so good.
All right, so here you can see the mean absolute errors for the different methods again. That is how much it deviated from the true percentage.
And you can see that the machine learning with sections was by far the worst. Actually, machine learning was the closest, so not rule-based, even though rule-based is not much worse.
Okay, so conclusions. Rule-based methods seem to be best for correctness with indirect representation, but the machine learning leads to better correlations, at least with sections.
And actually, I got a similar trend with my other kinds of representation too. And that is, the rule-based methods tend to be better for the correctness, but for stability and correlation, machine learning seems to be better.
And we can say that the automatic recognition of speech indirect representation is by far not perfect, but it might be feasible depending on what you want to do with it.
For example, if you just want to predict how speech representation developed over time, a good correlation might be enough for what you want to have.
So, this is it for this talk, so thanks for listening and questions.