Good afternoon. My name is Anna Lodjic and I will present the work characterizing authorship
style using linguistic features that was done with Dr. Katrin Blake. So in this project
we propose a new feature for authorship attribution that involves grammatical structures that
surround the named entities or in our case personal names. To our knowledge this feature
hasn't been tested in authorship attribution experiments so far. Before I continue I would
like to say just a few words about authorship attribution methods in general and about
features that have been proposed so far. As many of you know authorship attribution methods have a
long history and many features have been proposed so far. According to Joseph Rudman nearly 1,000
features have been proposed by the late 1990s. And this slide shows the one classification of
existing features as it was proposed by Fstatio Stamatatos. So the features here are divided into
five larger categories and they include lexical and character features such as function words
and character engrams and then we have complex features such as syntactic semantic and application
specific ones. According to Stamatatos complex features represent stronger markers of different
authorials styles because they go a level deeper into the text than surface level features. So our
feature belongs to the category of complex features and not so much of the either it is not either
character or only lexical feature but it belongs to the category of complex features because it
shows relationships between two words in a sentence. So our goal with this project was to explore the
potential of grammatical structures that surround the named entities and again in our case these are
only personal names for authorship disambiguation and to find out if the writing styles of the
authors can be separated based on this sole feature. However I should say that we are not
only interested in predictive performance of this feature but also in characterizing authorship
style with regard to the use of named entities. In other words we would like to find out how
grammatical structures that surround the named entities characterize style and what view or
model of the writing style they can provide us with. And so here are a few examples of sentences
that use references to personal names that come from the collection that we were using. Each of
these three sentences was written by the same author and each of them contains references to
personal names as we can see here. In the context of the of the collection that we were using the
author here is referring to to the actors and characters in the movies. However if we were
dealing with a collection of fiction works we can imagine here references to character names in
the novels and if we were dealing with a collection of news stories we could imagine
here references to politicians or possibly actors and if we were dealing with a collection
of works on history we could imagine here references to historical figures. And here are
a few more examples of references to personal names in the text and again these four sentences
come from our collection that we were using but this time they were written by two reviewers and
again each of these sentences contains references to personal names. However as I said we aren't
interested in personal names per se but in grammatical structures that surround them. And so I will show
now how this feature can be identified in the text and how it can be extracted from the text in an
automatic way. So the next slide will show this last sentence Robert Ryan is the prey he captures
along with the girl Janet Lee it shows the syntactic representation of this sentence and
this was obtained through Stanford lexical parser. So this representation of the sentence shows how
two words in a sentence are related to each other and we first I identify names in the sentence so
Ryan and Lee and then we identify grammatical structures that come before and after the name
and so this is actually the feature in our model not the name itself but the label that comes
immediately before and immediately after the name. So I should say that we do not consider the
relationship between Ryan and Robert and between first and last name that is a compound noun in
this case NN label because only the last name is connected to the dependency structure so we only
focus on last names. And here is another example this is a sentence that was also taken from our
collection it's Arnold's wife Evelyn is also expected to sacrifice for Edward again we see
how two words in the sentence are related to each other and again we focus on the names first we
identify names first and then on these structures that come immediately before and immediately after
the name and again only these labels in square are go into our model not the personal name. I
would like to draw your attention to this POSS possession modifier label between Evelyn and
Arnold. I would like to say that this is counted twice in our model once as possession modifier
after Evelyn and once as possession modifier before Arnold so it is counted twice and now that we
have shown how this feature is identified let me show how this feature can be used in the context
of authorship verification. So our method of creating a model of how an author refers to personal
names includes the following steps. So first we collect and identify sentences in the text in the
entire collection and this was done with the help of natural language tokenizer then we identify
people within each sentence and this was done with the help of Stanford named entity recognizer and
then we generate syntactic dependency trees and this was done with the help of Stanford lexical
parser then we align the person names with grammatical structures and this allows us to
create the author profile by identifying grammatical structures before and after each person. Finally
this number of grammatical structures before and after each person is normalized by dividing by
the number of people referenced in the test or train profile document and this brings me to the
data set that we were using in this experiment. We were using a subset of the IMDB collection of
movie reviews that was harvested from the IMDB website this is internet movie database website
in 2009. This collection that we were using was used in two experiments that were conducted by
Yanir Serousi or is this better by Yanir Serousi and his colleagues in 2010 and 2011. So this subset
of the IMDB collection which is huge includes some 1,000 reviews that were written by 62 different
reviewers and so it has a total of 62,000 movie reviews that are rather short. I should say that
this collection satisfies genre period and topic requirements that is if we consider movies or
films to be an overarching topic of the movies and here is how the experiment process looked like.
So as I said the entire data set includes some 62,000 movie reviews which translates into some
1 million sentences. The first step was to eliminate sentences that contained more than 80 words and
this left us with some 900,000 sentences. The next step was to eliminate sentences that do not
contain references to people because we are only interested in sentences that contain references
to people and nothing else and this left us with some 57,286 reviews or 92% of the original
collection and some 350,000 sentences and I would like to draw your attention here to the fact
that based on these figures it appears that most of the reviews in this subset of the IMDB collection
do contain references to personal names however not all the sentences in the reviews contain
references to personal names so only approximately a third of the original number of sentences
contain references to personal names and then once we had by each author only the sentences
that contain references to personal names throughout their reviews in this collection we
could create a training profile by extracting 95% of the reviews for each of 62 authors and then
also create a test profile which included 5% of the remaining reviews and then we calculated
cosine similarity between test and training documents and rank the correct author results.
I should also say that this process was repeated 10 times each time using a new random sample and
here are some of the results of this experiment. So first I will show how the personal names are
distributed throughout this collection and this histogram shows that approximately that
each of the authors I mean that most of the authors use between 5,000 to 20,000 references
to personal names throughout their reviews however there are two that use less than 5,000
references to personal names and there are two that use more than 32,000 references to personal
names and they represent outliers in this sense. This histogram indicates that references to
personal name are a frequent and common feature of this collection and that they are used by all the
reviewers however with varying frequency and here is the and here is the results of the experiment.
So the x-axis indicates author average rankings through 10 random samples and the x and the y-axis
indicates the percentage of authors that were correctly identified by rank 1, rank 2, rank 3,
etc. If this was a perfect system the average correct author rank would be 1.0 but as we
see here it is 1.84 and as this bar chart indicates for 21 authors in our collection the
model that we came up with gets it right every time or the third of the collection is always
correctly attributed using 10 random samples. I should also mention that this is a conservative
estimate and also because rank 2 means that if for any of these 10 different samples our model
didn't get it right the author would be placed under the rank 2 and we have almost all of the
authors covered by the average rank 4 which is 59 authors and so the results thus far suggest
that this new feature has a good predictive performance and in terms of other results on
the same data set the article by Yanir Serousi and his colleagues reports the accuracy of 25 to
81 percent depending on the number of topics used in the latent Dirichlet allocation model. Our
methods aren't directly comparable and the details of their experiment are included in the article
authorship attribution with latent Dirichlet allocation in the ACL proceedings from 2011. So
as I said the results thus far indicate that this feature has a good predictive performance. However
we were not only interested in the predictive performance of this feature but also in how does
this feature characterize style and I would like to give you some preliminary results on that. So
what were the most prominent features that were found before name and after the name? So frequent
syntactic features that were found before a personal name include nominal subject and object
of a preposition. However we also found with regular frequency opposition direct object passive
nominal subject and frequent patterns after a personal name include punctuation marks and
possession modifier. However we also find we also found opposition nominal subject coordination
relative clause modifier and other patterns after the name. And so in terms of how well do
grammatical features identify and and reviewer we found that each of 62 authors shows certain
preferences when they refer to personal names. So for example the reviewer with the ID 774387
show the preference for using passive nominal subject of before the name such as in this example
George and Susie the lovely Hannah Gordon are newly divorced and have moved to the countryside.
However another reviewer showed the preference for using possession modifier before and
possession modifier after the names such as in the example Wim Wenders his presence did not help
either. So the 29 features that we identify 13 before the name and 16 after the name allows us
to create a fairly complex model of reviewers style when they refer to other people. And in
this way we can get a deeper insight into each other's individual preferences. So this is just
one example. And in in terms of future work and next steps we are still doing some analysis on
this data set and we clearly need to explore other data sets and other genres and see how this feature
will generalize to let's say fiction works history works or you know like news stories.
And this brings me to the final slide. Our goal as I said was to see how well the grammatical
structures can help with authorship attribution and these results suggest that they have the
potential to identify writing styles in an automatic way. And I should point out to that
we are using a collection that has a fairly large number of candidate authors 62. And in the
experiment that was trained on 95 percent of the reviews and tested on 5 percent of the reviews
the average author rank was 1.84 and and the correct author was ranked in the top three for
52 out of 62 authors or 83.9 percent. Our results are in line with earlier predictions in particular
with Stamatatos' assertion that syntactic features have a good potential to separate
writing styles. So thank you for listening and we would like to welcome your questions and comments.