Okay so welcome everyone to the last lecture of statistical methods of language technology today
and this lecture is called best of the rest so we're gonna collect whatever has left is left from
the slide decks that we already started in various occasions. Sorry for this topical break in between
it wasn't possible to do the MT lecture earlier. I hope you enjoyed Meriam who gave the MT lecture
last time. Okay so where Meriam actually stopped was the MT evaluation part and now the question
is we have talked about machine translation systems that transfer language from one language
to the other language in order to translate it and I know you all know these kind of technologies
because they're publicly available on the web. Another question is how do we actually evaluate
the systems because evaluation is crucial in order to improve systems so you want some
automatic evaluation even though it's not perfect in order to have a score which you can automatically
optimize during training in order to know what is good what is bad and of course with MT it's a very
hard question because well there's gazillion ways of translating the content of one language to the
other language because what you want to get right is the concepts and not really the word by word
translation in fact word by word translations are mostly considered bad. So how can we do this? Of
course we can ask people okay so here's the system output how good is this translation and of course
if you once you're on it you basically can also look at the typical errors that systems do and
use these to improve these systems but this of course does not scale you cannot always ask people
for whatever hyper parameter you want to optimize in your neural machine translation system.
So what is a better way of automatizing this maybe is to test MT in some application. So for example
if you had a question answering system that you can ask in German and it answers in English then
of course there's a translation in between and the quantity of the translation will eventually
correlate with the quality of the question answering system. However this is a very indirect
notion so this is like an extrinsic evaluation as we call it we use the thing for doing something
else and measure the something else which is of course higher relevance in say industrial
applications because we really want to know okay what is the impact of a better translation system
is it worth investing in a better translation system or should I rather invest in improving
other parts of my actual goal applications because it's not worth the trouble and yes so
the problem is the signal is very indirect so even if you had a perfect translation it will not result
in perfect question answering score simply because the question answering system is not perfect.
So we are in need for automatic evaluations and we will discuss two one in detail and one a little
bit just showing you the concept of this. So this is BLUE which is the bilingual evaluation
understudy and the other one is YICI which is basically also incorporating semantic elements.
So what's the idea of BLUE? The idea of BLUE same idea as in ROUGE for summarization
is a very simple one so we have a automatic translation say down here from some source
language and we have some reference translation from a human translator which of course costs
money but you can do this once right and then you evaluate your system against this automatically
and the intuition is the more words they share the better is the translation.
Now of course in single words this is not so much fun so maybe we want to do this on
bigrams trigrams and so on and this is actually how BLUE is defined. We measure the overlap of
n grams and there are variants of BLUE, BLUE 1 2 3 4 5 whatever you want for the number is actually
the n as in n grams and also we introduce a brevity penalty so the you could game the
system easily getting a perfect score in the reference translation but just translating
everything into the in English because you can assume that from a certain length on the reference
translation would contain the word the so you get perfect overlap score if you look at how many of
these words you have actually translated and some how many of these words you haven't translated
however this is not a good translation and this is why we have a term that basically ensures that
the length of the machine translation should at least be as long as the reference translation
and yeah this is like a formal definition so for BLUE 4 for example what you do is you multiply
this by one divided by the length times the logarithm of p1 which is the one gram precision
which basically asks for how many of the words that I have in my reference translation actually
are contained in the no how many words of the machine translation are actually contained in
the reference translation and you also do this with a bigram precision and since there are
more possibilities more combinations you divide this by one by two then you can do this for
trigrams so you divide this by one to the power of two to the power of two and you also can do this
divide one through two to the power of three for the four grams and then you
uh yeah subtract the maximum of dividing the words in the reference by the words in the
machine translation and zero which basically gives a penalty uh yeah minus one sorry which gives a
penalty whenever it's too short and this is surprisingly simple but it's actually very hard
to game the system in the sense that on average if you have not only one pair of reference
translation and machine translation but maybe 50 of these then actually the human judgments of
quality of translation highly correlate with blue and this is what you want if you want to
automatically optimize your system on an automatic uh score for the sake of optimizing your
architecture optimizing for learning and so on there are a few variants so one idea is basically
to use multiple reference uh translations and do kind of an average of the blue scores for
the overlap of your machine translation to several translations produced by different translators
and yes this is of course uh a little bit adding more stability because yeah you can criticize blue
heavily because it's no problem to find examples of perfect translations with very low blue score
but this offsets the problem somewhat
yeah and um let's look at some examples so if we start with the input
who speaks french not really a little bit so um reference translation is tomorrow he visits the
his original country to celebrate his mother's birthday and we um yeah suppose that we have
an empty one system that produces he's out in his country to his mother's birthday
which is a kind of a rough translation right it's okay ish but it's not really conveying
everything that is contained here um but actually it has a high overlap so um we put in colors
where the the overlaps are so of course we have country and country too and his mother and his
mother's birthday so there's even a foreground match his mother's um birthday which is very nice
and these are basically the engra matches uh you would have to put in the blue formula and that
gives you a high score as opposed to another translation that says he goes back home to
celebrate the birthday of his mama and actually uh his mama and some more is actually a nicer
translation because it also conveys the personal attachment right uh uh in french you could say
samar which is just a technical mother term and mama is mama i mean you all know mama right uh
but if you look at um this engra matchings for this second system actually there are no
three gram and four gram matches and there are also fewer one gram and two gram matches so this
is one example where actually blue is not so adequate for evaluation and therefore of course
there is development in these sort of measures and i just give you an intuition what is the latest
thing in machine translation evaluation it starts from the question so how well is whatever preserved
in this translation so actually what we want to convey is the concepts whatever we convey is the
facts right so we want to convey who did what to whom for whom when where why how and not really
have word overlap because word overlap is i mean if it's one gram it's a bag of words model right
so you can't even distinguish um that man bites dog or dog bites man right it's perfect overlap
but it's of course a completely reversed semantics behind this and this is the idea between
semantic based evaluation where you actually add more structures so we talked about automatically
processing structures you could do something like okay this is the predicate he goes to his home
tomorrow he has a argument which is the subject which is denoted as argument zero and then to his
home is a location argument tomorrow is a temporal argument and this is a propositional
phrase argument further specifying the going and this is an analysis that comes out of semantic
role labeling which is kind of a semantic processing on top of dependency parsing so you
could as the structure comes more or less from dependency parsing and then you aggregate whatever
you see here into labels that convey the more yeah semantic uh meaning as in who did what to
whom semantics and you also have um another structure down here which you want to compare
with so tomorrow he enters home for mother birthday which is a maybe not so good translation
but you also have this predicate and argument structures and now you can define a system that
would evaluate the overlap not on the basis of engrams alone but on the overlap of how many of
these relations are preserved so um you would distinguish these between predicate and argument
matches so SRL stands for semantic role labeling so of course the main thing must be
right which is what happens so this is mostly the verb so the predicate that actually has a few
slots to fill like who's the subject who's the object what kind of objects and so on
so you could look at how many predicate matches how many argument matches do you have
and now if we look at our two examples here then he's out in his country to his mother's birthday
is and visit is actually different so there is no predicate match and the arguments is also uh
jumbled because the analysis uh of this first sentence is actually showing a different
predicate argument structure as we have in the reference translation whereas in uh he goes back
to home to celebrate uh the birthday of his mama is actually has two predicate matches so
visit and going and the celebration and a lot of arguments per predicate matches so in this case
basically this easy measure um which was sketched here just for you to have an intuition
much better captures the quality of the translation with respect to the semantic level of who to
spot to whom yeah this is basically the remainder of the machine translation lecture
are there any more questions on machine translation or comments for anything you want to share
no so you all know how to build a machine translation system
not quite any more questions to whatever happened before in the slide deck
it doesn't seem to be the case okay so this is all i have for machine translation
and now let's go on with what we talked about two weeks ago which is
disambiguation of lexical ambiguities so words that have different meanings and how we actually
find what is the right meaning in context so what we talked about here maybe a quick
going back so we had this river bank money bank example where bank is the same string but it can
be used in entirely different senses and the question is now how do we find them and there
been a few approaches we looked at uh rather you know top down rule-based approaches from the
principled way of how should it work and we concluded that these actually create too much
work and then a simple thing that we looked at last time was the overlap best approaches so
suppose we have some sort of a sense definition as we would for example find in a machine readable
dictionary or any sort of a lexical resource that defines the meaning of words and also has
sense inventory in the sense that you would look up bank in your monolingual dictionary
of english and it would have bank one with the definition bank two with the definition
and maybe some other definitions uh then we can uh do what uh less has proposed
which is basically assuming that the language of this dictionary is somewhat consistent
and also assuming that the language of the dictionary is maybe a different language from
the language of the text that you want to disambiguate and what uh less proposed there
is basically for every word that has multiple senses in the dictionary we put them available
to the algorithm so say for example you have a sentence where here we have an ambiguous word
of multiple senses here we have an ambiguous word of multiple senses and then we look at
how much overlap is there and for overlap you could use blue score for example or just counting
the number of angrams or maybe something more sophisticated but for now it doesn't matter and
then you would basically see for each of these pairs what is the highest overlap and you would
hope that the same language from the dictionary would actually serve the purpose of connecting
stuff up that is related so maybe one is the animal and one is the um one is the location
where animals roam and then maybe maybe they mention the same words and this is why you can
link them up so this is the original LESC algorithm it's actually much simpler to not assume that the
language of the dictionary and the language of the text is different and to do this directly
with the text you could also basically take your per example ambiguous words you have a multiple
senses stacked on top and then you measure the overlap between each of these sensitive editions
and the actual text so there you don't need multiple ambiguous words per sentence or per
context but you can do this directly for each ambiguous word in a local fashion and this is
what the simplified LESC algorithm is and maybe one very important notion of uh sense disambiguation
it's the most frequent sense uh baseline the most frequent sense assumption so we
talked about power law distributions in word frequencies somewhat and we also have a power law
in the sense frequencies so what this means is that on average if you have a word that has multiple
senses then there is one very frequent sense which covers an average maybe 70 ish percent
of the instances and then we have a second largest sense which maybe has 20 percent and then we have
a third largest which maybe has five percent and then it goes very very small so sense frequency
is actually on average also distributed as a power law which means that a system that would always
pick the most frequent sense and the most frequent sense usually appears first in a dictionary
because people that write the dictionary would also sort it in a way that they think is the most
important thing because this is accessible for people you want to read them you don't want to
start with some obscure sense uh when you read your dictionary and um yeah so choosing the most
frequent sense or the first sense from the dictionary is actually an extremely strong
baseline and it wasn't until a few years ago that actually systems could actually beat this baseline
so we had a long uh span of years where actually the best systems that people would be able to
produce did not beat the most frequent sense baseline of about 70 accuracy yeah and uh last
time i think this was the last slide we had uh so um we already looked at okay so there's details
so how do we define actually these things so if we want to disambiguate dance as in with the music
the dance started with slow movements and we have these three noun definitions for dance there's
also verbs for dance but we assume that we have taking care of this because we have part of speech
taking we know that this is a noun here then uh what which one should it be and now the question
is how do you compute overlap do you want to maybe ignore words like the a off and so on stop
words that actually don't carry do not carry any meaning uh all these kind of things um
yeah so this original algorithm from lesk was uh defined on uh an early machinery dictionary
these days everyone is using wordnet uh back in 86 there wasn't any wordnet they started as in 90
this is probably the reason so what is wordnet has anyone actually uh downloaded or accessed wordnet
yet okay very few people so this is a very very standard semantic resource for english
it also exists for german but it's not as easy downloadable as for english um so what uh it has
is basically a structure of meanings of words and how they relate and wordnet is organized in so
called sunsets so it's a set of terms it doesn't have to be single words it could also be you know
compound nouns or longer expressions um which would actually have the same meaning at least in
some context so something like bank could have a slope because it's kind of slope sloppy land
something because this is uh what a bank is and the river bank example so you have these sets of
words that convey concepts because it contains words or terms that they can actually can refer
to these um concepts and then these concepts or these sunsets are organized with semantic
relations and the most important semantic relation here is the is a or a hyponymy relation
which is basically a taxonomic relation so wordnet is organized as a giant tree where you have the
most generic concept on the top of the tree which is um entity a something right an entity then
goes to physical or abstract entity and it goes from there all the way down to um yeah whatever
concepts are in there like furniture and then you have table and then you have coffee table versus
writing desk versus all these kind of organizations so this is the taxonomic backbone
of it and they also saw some other relations like part of so you could learn that a car has wheels
and these kind of things there are also sometimes instances so there's lists of
american presidents in there under the american president concepts and so on
and it is uh has a high coverage so it covers um yeah a hundred thousand nouns um seven thousand
verbs of 16 000 adjectives four thousand adverbs for a vocabulary of about 130 000 um yeah types
strings and this is actually a lot so with this you cover maybe 95 98 percent of the content
vocabulary of uh general english of course if you have specialized language such as in the medical
or jurisdiction domain then wordnet will not provide the proper coverage and of these of course not all
of them are polysemous meaning not all of them have multiple senses senses so whenever they
occurs is actually when you have a string bank which is occurring in one concept like the slope
of land concept and in the other concept which is financial institution concept so in this case you
would have a polysemous word bank which actually has two senses at least bank one and bank two
and these are the words that are actually polysemous and we learned that of our 100 000
nouns only only 60 16 000 or so are polysemous and uh they have an average about three senses
now sounds like it's not a big problem right but most of the words are not polysemous so why would
you even care well the problem is that the most frequent words are actually the ones that have
the most senses and this is yeah you can explain this with like network dynamics it's like a
typical thing so you want to be short in your conversation so you basically pick short words
in order to refer to things the shorter the words the more frequent they are uh on a global scale
right so you want like one syllable two syllable is much more likely if you have a random process
that events words than longer words and of course the more likely they are the more likely
they come in different situations independently of each other which actually would lead to a
polysemy if you then connect this back to larger speaker groups into larger languages so that's
what happens so basically the average polysemy by frequency is much higher than what we see in this
table here yeah how does this look like so i have an example sin set from magazine that shows how
the structure of wordnet is so in case you want to use wordnet for your purposes which is a good
idea if you want to have some some level of semantics and hack something together in a
meaningful way that's that's a great approach to just download wordnet and use it for that why not
use knowledge that someone has encoded um so there's also a web interface so this is now the
three zero version now we're with three one but the changes are miniscule so you can enter a
word such as magazine and then you can display various things or not and what it comes back with
is in this case six senses of magazine so six senses of magazine marked as magazine one through
magazine six in their respective sin sets so the sin set of where magazine four is contained in
would also contain a card which in its second sense and this refers to a light tight supply
chamber holding the film and supplying it for exposure as required so the magazine of the
camera where the film roll is contained in something that not many people still do these days
using physical film rolls yeah so this is what you get from from wordnet and already you have
these kind of definitions that you could plug into your uh overlap based approaches for example
there's more so this is actually the entire sin set from the definition
then you have lexical members that actually invoke this sin set in these sin sets we always
have a so-called gloss which is a kind of a short definition and sometimes we also have
an example sentence we don't have an example sentence for all of these because the people
that constructed wordnet are psychologists they don't need corpora they come up with these things
on the spot and then they just write it in there and then efforts for basically finding these
things later in real text sometimes find these or they fail and what we see here is
in these little brackets here is the number of examples in a particular corpus
that were actually labeled with this sense manually and you already see this most frequent
sense power law distribution here a little bit so the first sense actually had 13 occurrences
this next one is already down to two then we have one of this one and zero of the others
so these exist of course but they are supposedly rare yeah so this is the sem core
count which gives you an idea about how frequent is the most frequent sense
and i talked about these taxonomic relationships so what we can also get from this is
some level of abstraction so we could just iteratively click through
in order to generate something like this so a magazine too which is defined as the product
consisting of a paperback periodic publication as a physical object so it's the magazine that
you actually touch as opposed to the magazine company that publishes it or to the entire
magazine series that runs over 20 years okay this particular thing is actually something
in a more abstract level it's a product or a production so an artifact that has been created
by someone or some process in order to sell the magazine then which is of course something like
a creation so a creation could also be not a product but something else or the creations
that you don't want to sell for example and a creation is an artifact which is some
countable whole unit kind of thing which is a physical object and a physical object is a
physical entity so it exists basically but physical entities do not have to be tangible
sorts because the physical object has tangible in its definition and then it goes all the way up
or down in this case to entity yeah and then you can utilize also the structure of wordnet in your
word sense disambiguation approaches so yeah we can go back to the last guide here and say okay
this overlap of definitions is because people that write these glosses these definitions
use the same vocabulary so if we have lake in our text and maybe lake only has one sense
which is a body of water surrounded by land and then we have a bank in our context and then we
have two choices for bank one which is the financial institution the other one sloping
land besides a body of water then you have body of water and land and a lot of overlap between
these things this works if it works so basically it's something like a tip of the iceberg approach
so in case you get a high overlap between these kind of things it's a very low chance that you
actually got the sense wrong however the normal situation is that you don't have any overlap
or very few words of overlap and then your algorithm is very unsure so you can actually
construct high precision low recall approaches for this depends on what you want right sometimes
high precision low recall is better than low precision high recall sometimes you really want
high precision high recall because otherwise your method actually does not work that you
want to actually implement so it's it's always like a matter of what you want so for like for
finding good examples to add into a dictionary high precision low recall is perfectly all right
for really using this in in a system maybe not because it's not cherry picking good examples
it's rather performance across arbitrary texts and the problem that we have here is what some
people call the lexical gap basically we have the same or similar meaning but we can express
this in a variety of ways and for most of these valid expressions of the same meaning
the lexical overlap is actually zero and this is where for example we see this in chatbot
development who has ever developed a chatbot okay a few people so with chatbot development
the hard part actually is to anticipate what people will be able to say in order to ask
for a particular thing that you want to implement in your chatbot so you have to basically
anticipate all of these and enumerate them in smarter not so smart ways and this is the exact
same problem it's a lexical gap we have a large variety in like expressing our things
and who has an idea to actually alleviate this lexical gap so the situation is we have
two pieces of text and there's overlap and the problem is most of the time there is not enough
overlap so who has an idea to actually increase the potential overlap anyone
Rami we could use a sense net from webnet or we could also use like co-occurrence graphs
co-occurrence graph this would be basically you pull in other words that co-occur with these
in order to create more lexical items and then if you have more lexical items then you have more
overlap yeah except that of course co-occurrence graphs and undisambiguated right yeah
anyone else an idea how to increase the material of sorts
okay i show you one um so one is uh using the structure of wordnet
so idea is the following if you have a sentence like the bench pronounce the sentence
then the actual correct sentence definitions would be sentence as is as in the penalty
meted out to one of the judged guilty and the bench would be the correct one persons who hear
cases in court of law so it's in judicial language bench and sentence is actually not what you sit
on or what you will say really like kind of sentence and there is no overlap okay but what
you could do is you say okay but i know that sentence is actually something like a final
judgment because i can look this up in wordnet and there's court of law because a court of law
actually pronounces these final judgments and of course you could also walk up on the bench part
and but i don't do this here but suddenly you have an overlap here and there's an overlap score
and in their particular approach bernard jean peterson in 2002 scored n grams
more the longer they are by taking the length to the square so unigrams would count one
bigrams would count four trigrams would count nine so in this case for the extended less proposal
we would have an overlap score of nine because we walk up here and of course there's tons of
things you can vary this maybe you want to walk up many many many steps in the chain and maybe the
more steps you walk up the lower you want to be having the score because you assume that it's
getting too general so you will generate a lot of overlaps or not so this is actually one way of
increasing the amount of text here by extending from the lexical resource if the lexical resource
of course has this structure so dictionaries like the oxford advanced learners does not have this
structure i talked about this most frequent sense and there's one other assumption in
word sense disambiguation that is frequently rolled out which is the one sense per discourse
hypothesis so the idea is if you're talking about financial institutions of sorts and maybe
economics and whatever there's not going to be a lot of river banks in there there's mostly
financial banks or financial institution banks and the assumption of the one sense per discourse
is that in a certain document in a certain wider context we would assume that all
mentions of the same string actually goes to the same sense which is of course not strictly
necessary but you could try this so there is a quite old paper by William Church and Jirovsky
where they actually checked this for a handful of words as given here and they came back with
the accuracy of this assumption being 99.8 percent so there's very rare cases where you actually find
documents where something like tank is used both in the vehicle and in the container sense
makes sense on the other hand how applicable is this so we have the
power law distribution of word frequencies and this actually leads in its conclusion to the
or in its consequence to most content words appearing only once per document
so if this is not the Wikipedia page about tanks then it's rare that we see tank mentioned
twice in a document most documents have only one tank and then there are some that have two
there are even fewer that have three and so on so it's not actually applicable so only 50 percent
of cases which is exactly the consequence of the minus one exponent of the frequency power law
of Sips law is actually cases where this is applicable so in half of the documents we would
expect that we have two ambiguous words per document where our one sense per discourse
assumption actually holds true so this first sounds like a very powerful assumption
and also gives you a hint on how to use this so basically if we want to increase the coverage by
material so in order to see more potential overlaps what we would do here is not only use
this one but also use examples where there are other mentions of sentence in the same document
and this also increases the number of potential overlaps but again this is only happening in half
of the cases so maybe this is not something you could rely on very much in your system
okay so these were early things and then all the semival started so there are formal evaluation
sets of word senses and there are very many of them i just give you like a rough rundown on
what is there so there's semival it's a shared task running since 2001 and they started with
like three yearly then by yearly now we have every year semival with about 15 tasks not all
about word senses but all kind of semantic oriented and they have tons of tasks something
like cross-lingual lexical substitution or temporal relation identification or aspect based
sentiment analysis but two of the core tasks are all worlds all word sense disambiguation
and the lexical sample task and these are the standard data sets where you can measure your
word sense disambiguation performance on so what is the difference the
yeah okay so the difference is on the all words task you have a corpus
like the sem core i mentioned earlier and this corpus wherever there is a word that has multiple
senses in say word net it is labeled with its appropriate sense in context so now if you do
training and testing on this you would basically split your corpus into a training maybe a
development set and then a test set and since this is just you know a sequence of documents
there is no guarantee that you have seen training examples in your training part for
senses that you actually should predict in the test part so you have a problem with supermice
machine learning because this always assumes that you have training examples but here there might be
none so this is why lexical like all words task are usually done using resources only like not
machine learning but kind of smart ways of using for example wordnet so what we just saw with the
lesk algorithm and extended lesk version are examples for algorithms that do not use training
data we just have kind of a prescription kind of an algorithm on how to use the resource lexical
resource and to produce scores that actually lead to a disambiguation but i haven't seen any
you know labeled examples yet right so for all words task you need to do something like this
because you can't assume that you have seen in training and the lexical sample task is actually
different in the sense that you have only a handful or two handful or ten handful of words
and for these target words which have multiple senses you have many training examples
and only this target word per training example is labeled with its sense so there you can assume
that if you split your entire data into training and development and test that you have training
examples for all the senses that you should predict in the test case to give you a feeling
how this worked so semival 2 which is basically the task in 2003 or 2004 or so how well did
actually these approaches work so the best system they had in this semival task had a
precision recall of one score of 40 which is super bad because the most strict consensus already is
like 60 70 depends on the granularity of the inventory this extended less that we discussed
would have come on second place with 34 the second best system actually had only 30
the less algorithm original had like 18 and the random sense basically picking one of the
possible senses is actually uh 14 so something like the average uh level of ambiguity is seven
and then one out of seven is correct yeah so this is of course completely unusable and
this unusable scores have resulted in many ways of changing these things so one of things that
you might have noted here in the word net example from magazine
is okay how many senses of magazines are there yeah if you count them it's six
but the first one is a periodic publication containing pictures blah the second one is the
product consisting of a paper by periodic okay the third one is a business firm that publishes
magazines so these are polysemous but not uh harmonious in the sense that they all refer
to this kind of thing with pages but on different aspects of it it's just like
the regular polysemy of university being a building and a institution it's like a lot
of products actually have associated companies and maybe a whole series of things so maybe this
word net is actually too fine-grained so word net was criticized with being too fine-grained because
drawing these distinctions is actually sometimes not even possible with short contexts given
however it usually is possible for people to distinguish between these first three and the
second three which is similar kind of situation so the magazine as in a chamber holding something
and whether you hold a film or you hold a munition or you hold a munition on the large scale
it refers to sort of a container thing right so these distinctions are maybe possible between
the thing with the pages and the container thingy but not on this lower level and yeah
earlier of these evaluations so if you see these kind of small scores like this this always
is referring to this fine-grained inventory yeah and if you would download these MML data
this is how it would look like so basically you have some sort of a XML-ish markup where you have
a sentence like homeless people not only lack safety and so on and then some of these have
markup so people is actually disambiguated to some D 00 S08 T1 which is actually a key
to the respective word net file where you can find whatever this means and what also has
been noted is that if you ask people on this task they also have problem and distinguish in these
so the upper bound is estimated to be at about 75 percent meaning that if you ask two people
then they agree in average about on only 75 percent of cases because of these you know
fine-grained things then there was developments to actually make this more coarse-grained so
in our example to have magazine one two three and magazine four five six and if you do this then
you get an agreement between people of about 90 percent and this is then also the upper bound
what you can ask your algorithms to perform and actually the performances are approaching
these inter annotated agreements so the top system performances on these old evaluations
were like at 65 as opposed to 75 upper bound and something like 82 or as opposed to 90
um coarse-grained but the coarse-grained most frequent sense baseline is at 78 percent of
course the more coarse-grained it is the higher is the most frequent sense baseline
and it's just a very small margin between a 78 and a 82 given that there's of course like
serious machinery needed in order to produce this little difference here so these are a little
older scores so now with a new neural contextual word embeddings modeling we're actually
approaching even closer the inter annotated agreement upper bounds so this is the all
bridge task and for the lexical sample task it actually looks a little different so here is the
training data for double and doubled so we have a larger context where double appears in and this
is the head and in this one and actually it is a particular sense key which is encoded somehow
in this markup here and then we have the second example for double which has again a sense key
and the third example and so on and typically you have like a couple of hundred examples per word
and only focus on like a low number of words yeah and then you can actually use
yeah any machine learning of sorts representing these examples training a model per word
training one big model for everyone and yeah treat this as a supervised machine learning tasks
now what are features that are actually helpful in this context
of course you're going to look around what is around this word maybe you want to lemmatize
or stem in order to also increase the amount of matches so you don't want to have a mismatch
between mother and mothers it's the same word right okay so we have means of reducing this
with our final stage transducer maybe yeah then yeah part of speech tags you could use
you could also use whatever comes out of these overlap based methods as a feature
and learn these things you could also use topical features for word census immigration
so topical features is has this idea of saying okay this one sense per discourse is maybe
something like a one sense per topic so you remember topic modeling where we actually succeeded in
clustering document collections into various topically coherent parts and yes of course
i mean this is if this is about economy then bank is different than if this is about landscaping
so if you had a landscaping and a banking topic in our topic model and we could use this as a
feature then of course we would succeed in distributing these the problem with this one
is that topic modeling actually produces very coarse topics and some of these are very good
for disambiguating part of the vocabulary but there is also a large body of vocabulary that
doesn't distinguish senses as per the topics that you have induced by your topic model and doesn't
even matter whether you rerun it or whether you recompose your corpus you will always have this
case um i give you one example that is really notorious order so who has an idea of what order
could mean define order one or more of its senses you can order something at the restaurant okay i'm
taking the order fine order of elements for example ordering okay yeah what else do we have
you can instruct some instruction maybe also military order yeah one more
i just had to think of a haircut in the order of phoenix basically yeah yeah this is another one
the order as in uh yeah kind of a group of people yeah yeah but one is actually something
that you wouldn't even regard as the same word maybe which is in order to
part of speech checker produces a noun on in order to correctly right it's in order to is a is a
preposition but it's spelled in three words and it contains a noun and all of the others were
actually topically relevant right like if it's harry potter topic then we know what the order is
maybe maybe not because also these uh witches or wizards give orders maybe just their
elves or i don't know i haven't read it um but you can always use in order too right and and
therefore uh yeah this this can be just a feature but it's actually not the answer to to to everything
and uh here there's a bunch of these methods uh uh that again try to bridge the lexical gap
somehow by not only looking at the actual context but also pulling in the entire topic for example
uh a bunch of methods like uh called topic signatures so topic signatures would be basically
collecting for each word sense a lot of words that are kind of associated with it and this is
kind of your core occurrence idea but you have to like make sure that the core occurrence happens
only uh between any word and my target sense for which i want to collect the core occurrences
then we can use word sense induction features so we talked about word sense induction earlier
with this arm example and maybe we can use whatever was came out of this then lsa or topic models
yeah which is this topical notion and uh yeah we can also use lexical expansion which is basically
using a distribution of the sorrows to generate more of the same content uh context words for
example so i picked this word census immigration in the end because it kind of is an application
which is still hard for computers and where we can actually use a lot of elements that we have
learned from the previous lectures including morphological reduction including part of speech
tagging for knowing is it a verb or is it a noun so which of which are the candidates
uh including dependency parsing maybe if we want to basically characterize our context
in a in a better way than only bag of words uh we can use all of the representation learning
things all of the vector based stuff in order to represent uh the glosses the context whatever in
dense vector representations and not to overlap on these uh on the word level but maybe do
similarity in some embedding space um yeah we can use graph based methods in order to induce word
senses uh or we can use graph based methods for these core occurrences in order to collect more
so this is actually one of these tasks that are still not solved yet in its entirety and where
a lot of nlp techniques culminate and come together in order to to solve it and of course
it's a task that is not an end to itself unless you are a librarian or a um dictionary writer
but of course this is a task that is important for natural language understanding
and would be used as a pre-processing step for for example question answering machine translation
and so on yeah so how a topic signature works is basically we wanted to have more training data
per word sense so maybe we don't have any training data maybe we have to fuel the training data but
we can always use more training data and machine learning and the idea between behind this is the
intuition that uh i think i asked this before so if you enter a word in the search engine and it
comes back with wrong senses what you do you just add another word in order to disambiguate it
and we can also do this automatically using wordnet so what we can actually do is say okay
so we want to have a topic signature for an ambiguous word say bank and then first of all
we go to wordnet and find its senses so maybe it comes back with two senses and then for each
sense we do the following we basically issue query on a search engine or on a large corpus
that we have locally and we combine the word which is bank with some related concepts such as
slope or such as financial like whatever we find in its vicinity and then we look at uh what comes
back and uh get the content words from these and we only we don't do this uh once but basically we
we issue a bunch of queries and we look for words that are frequently reoccurring in these result
sets and this actually gives us something like co-occurring words for hopefully this sense
because of course there's a lot of uh places where this can go wrong uh especially because uh yeah
the search query is not sense aware it just issues bank it could still find the other banks
but if you do this long enough for enough of these things then it's actually a viable approach
in order to grow the material that you want to use for your disambiguation steps
the other one is using word sense induction as features um so just to say my dear as with this
arm uh and uh yeah so what we do is basically we look at the words that co-occur with arm
or that are similar with arm and then then we do our word sense induction that we talked
about last time in order to have uh senses and maybe you get then a lot of arm words for arm
zero a lot of arm words for arm one and a lot of arm words for arm two um which is a ARM as in
financial abbreviation actually um and then you can use these to disambiguate towards
your induced inventory so maybe i can just show this with an example so yeah a question for you
so these are co-occurring words of an ambiguous words clustered with graph clustering
and you should guess what is the word that co-occurs with all of these
so i queried a co-occurrence graph with the word
took the subgraph and clustered it what was the word i used for querying it's not contained here
hard it has three letters
louder the no
first that is h
it's a body part
arm starts with a
what could that be
it's hip hip
hip and we see a typical thing here if you do word sense induction um first of all we have
bigger ones and smaller ones corresponding to more frequent less frequent senses
for example here is about roma and bones and fractured in psychotherapist
then we also have stuff that you as human would probably lump together but still they
come up in different contexts so this is about hip surgery and this is this is about
the dislocation of a hip that eventually leads to the surgery you could argue that this is actually
the same hip right this is hip as in clothing of course it's still the same hip but yeah um
it's a different context so this is what actually where the topical assumption leads to different
induced word senses but it's just maybe it's a regular polysomy maybe something else
uh this is a boxing position this is uh hip hop this is hip hop dj i don't know why it's split
and this is hip hip hooray yeah um but these are co-occurring words so basically if you had
running examples and texts if you want to use this as a feature what you can do is you can say
okay i have a sentence like at his hip he wore a pistol and ancient leather holster
and actually pistol and leather and holster are contained in this
yellow one so actually i can assign a yellow feature on this and uh yeah these are all
a real example so here's also a hip hip hooray example from the british national corpus okay
okay and uh yeah uh is it important that actually these distinctions are correct
um so this is now stuff that was it's like unsupervised learning is kind of like
representation induction but we're doing this not in a neural dense way but in like a graph-based
way so with the advantage of having this inspectable but of course if you have the
same underlying data this is what also you would expect your dense vector representations to model
is it important that you have the correct thing no not at all because this is all about
characterizations this is about features so here we have two different clusterings of of bank
and one is a rather coarse grain clustering and one is a rather fine grain clustering and there
are also problems in there so if for example the bank three right left opposite it is actually
uh the verb to bank if you bank right you bank left maybe with uh sledging or something um
this is a part of speech taker error like in the large corpus uh some of the banks
it's accidentally got tagged as nouns even though they were verbs anyways um is this a problem no
not at all because if you use this kind of thing as a feature representation then you can actually
use multiple clusterings and this is also uh what comes out if you use this as a feature
representations so what uh here is basically this is a lexical sample task and the question is how
much training data do we need in order to reach a particular amount of precision
and this particular plot is called please ignore these red dots these are misplaced um so this is
called a learning curve and the learning curve is actually something that you might want to do
when you do supervised machine learning as well look at how much training data do i actually need
in order to get what kind of performance because a typical learning curve looks like this
you get a lot of gain in the beginning and here you start with the baseline which is 0.7 which
is the most frequent sense baseline and then you get better quality for more training but it's not
linear it's usually like steep here and then it kind of peters out and um what this learning curve
you can use for is also to do this kind of comparisons and saying okay so this is my
previous system uh there are a bunch of other systems and these systems are actually using
one two three up to ten different clusterings as features as we just saw and you could say okay
the difference here is 0.02 is two percent it's not much but you could also and this is where the
red line actually should be um go back here and say okay i can actually do the same precision
as with full training at 35 percent of training and this is a major reduction as you basically
only have to manually label a third of it in order to get the same performance then this is
actually a good feature and yeah learning curves uh this kind of stuff that most people actually
do produce when they publish a paper but they usually don't put it into the paper for lack of
space but it's actually very informative because it tells you also when to stop when to stop
collecting more data uh your effectiveness of features um and so on okay and uh yeah of course
you could use dense vector representations um because what we uh didn't this overlap
approaches is actually a sparse vector space model right so you could encode any context in like a
vector which has length of vocabulary and then you you put numbers there reflecting the number
of times you see the particular words and of course if you use methods like latent semantic
analysis or any of the other dense vector embeddings that we find then the idea is we
fight this kind of sparsity we fight the lexical gap because we assume that same concept is mapped
on the same kind of dimension so similarity of vectors actually reflect semantic closeness
and then maybe we can do the the same thing as we did with the Lask algorithm we're just
on these representations and just take the closest one so uh you would basically compare um vector
representations of say for example your gloss and your current context you measure as usual
the cosine similarity between these on the unit sphere and uh then you would compare the vectors
for each sense with your context and then assign the one that is closest and yeah this works
somewhat so there were papers of pre-vertevec papers with LSA doing this concluding that
the sweet spot on the amount of dimensions is about 1 000 so here actually is advantageous
to have rather long vectors in order to cover or to capture these kind of uh variety of topic
modeling or topic diversity that you would need in order to disemigrate not only one word but many
of these words so what people came back with is the number of dimensions the number of topics so
to say per word you need is something like 20 but if you want to scale this to a large vocabulary
you need at least a thousand two thousand to give you rough ideas about this why does it works
uh well because it generalizes over similar words uh it is also kind of a noise reduction technique
and also it falls in uh knowledge about lexical some similarities that it requires from large
corpora and this is of course another source an additional source and maybe a larger source than
your particular dictionary wordnet of sorts uh alone right so this is how you use these kind
of unsupervised methods or semi-supervised methods in order to um grow the material
for word sense disemigration okay um question
the background corpus must separate the senses no no the idea is basically um
i mean think of it as topic modeling right so in topic modeling you have a bag of representations
of documents and you will assign a topic mixture per document and the topic mixture is actually
used as a feature for machine learning so in your training data you have an example where the sense
is actually annotated and another one with a different sense and you look at the topic
distribution of the first one you look at the topic distribution of the second one this basically
forms you the feature so whenever you find a similar topic distribution of one to these
in your test you can utilize this so no explicit labeling in the background corpus
right and the background corpus like the representation learning the topics uh so to
say or the uh virtual back representations whatever is actually learned on an annotated
background corpus yeah and what is the current latest stuff here um the current latest stuff
here is from word embeddings to sense embeddings so there are a few models that work in a similar
fashion to word2vec but would actually find multiple representations per um word type
that are supposed to be corresponding to senses so it would have a bank one a bank two a bank three
vector that would that would have similarities to slope respective uh
financial institution words for respective something else um this actually you could imagine
it as like a process that goes through and whenever it finds a context that doesn't match
to the context it already knows for these words it opens a new sense roughly speaking there's a
bunch of ways of doing this um yeah and the latest thing which actually works surprisingly
well is uh the contextualized language models so we talked a little bit about the bird model
there's also now xlnet that came out just shortly um before this lecture like
during the lecture actually so two three weeks ago uh and actually what these models do just
to recall you is basically they not produce a representation in like an vocabulary entry where
you have a word and then you can look up the vector for the word this is what word2vec does
but basically it produces a representation a vector-based representation for each word
in context taking the context into account by for example um concatenating the word context
with lstms running from both sides or variants of this and now if you do the following if you
actually take the semival uh two data set that i mentioned earlier lexical sample task so
i don't know 100 training samples per ambiguous word or so and then you look at your test set
and you basically take the you pick the training data example that has the closest contextual
embedding for this particular word and you inherit its label you're actually getting state of the art
results so we're going to publish this next week like we submit this next week it's a small thing
i mean it's nothing very surprising it's not like a top conference something but it actually
shows how far we got with these contextualized word representations that all these mojo-bojo
feature whatever optimizing that i just showed earlier machine learning on these and lexical
resources stacking and topic signature from web corpora whatever it's not as good as just taking
the last four layers of the word representation and using this in a single nearest neighbor
approach on the training data so yeah um time uh goes quickly currently in nlp field is evolving
very fast so maybe next year we're going to have a lecture what can you do with bird i don't think
so but yeah okay are there more um questions remarks on word senses and
um no okay so let's summarize on word senses uh so one thing i haven't discussed so much is
basically is this is like a level of semantics now and the higher we get in the levels of language
processing and these are the levels that are also proposed by lingo is the less clear as the
formalization so i mean people already fight about what are the right amount of part of speech tags
but they kind of still agree that there is such a thing and people still argue about what's the
right amount of dependency relations but people still agree that you know these kind of trees are
more or less not debated but how do you formalize semantics
and this is very like depending on what you actually want to do with it because it's like
the meaning as in your application there's this whole field of semantic web that actually
tries to formalize semantics per se so without any particular application in mind
what you end up with is stuff like wordnet it's basically huge but maybe not so usable
because it draws distinctions that you don't need for your particular application
and it's also like a philosophical debate whether meaning without context exists at all
but okay this is something for late night discussions
we have these extensive semantics resources so i mentioned wordnet there's also babelnet which is
somewhat derived from wikipedia and joined with wordnet but what these resources don't have is
actually a connection to text so sometimes they have sample sentences but it's actually not very
clear on how to connect this static resource to our dynamically evolving language here and
these less algorithm-like overlap things are kind of
straightforward but kind of rough attempts to bridge this gap. What you should remember is
that meaning has many facets so we have coarse-grained finding, distinction, application-specific
distinctions, do you want the distinction or not and but meaning actually manifests itself in
context and this is why it's a question whether there is meaning without context. We talked about
sense inventory based, all words, words and sense immigration we talked about,
no we haven't talked about lexical substitution i kicked this out from the slides for lack of time.
There are also other semantic tasks so in case you wonder how this continues
i mentioned earlier in the MT evaluation that there is semantic row labeling
there's also a frame semantic parsing which is actually not only assigning these subject object
roles to the predicate roles but also labels them as okay this is the instrument this is the buyer
in the buying frame or in the financial transaction frame you have a buyer and you
have a seller and these kind of things which of course gets us as a more formalized way of
modeling semantics about modeling the meaning of text. Yeah okay so that's it basically from the
content. What we have now is a slide about the exam and as usual we have two exams
so the first one will be on 25th of July and the second one is on the 17th of September
who's planning to go to the first one who's planning to go to the second one okay not so many
i hope that no one that went to the first one has also to go to the second one
so usually people pass in our exams sometimes they fail but rarely and yeah locations are
given here and sometimes these change so please check Stina right before we had the situations
where we had to phone people to bring together because the room had changed two days ago
and what is the content of course the lecture of course the exercise is somewhat the reading but
as i said in the beginning i will not ask any questions you can only find the reading material
so reading is basically supplementary for you to help you understand what maybe is too condensed
in the lecture and there is no materials there's no one page whatever you copy down nothing just
bring a pen that's it are there any questions about uh the lecture uh the exam
yep does it allow to use the calculator nope
you don't need it okay you don't need it um so of course we sometimes have log stuff right
but then if something like log of two of course then you just leave it as log of two
don't compute it if you would need to compute it we give you a table uh there's no advanced math
don't worry plus and minus yeah small numbers
more questions on the exam so everyone is very clear what to expect
okay so content is over thank you very much for coming thank you very much for staying
until the very end i think it was uh like a nice group size this year uh steadily growing this
course uh always a few more people let's see what happens next year at least for me it was fun for
me it was a little unusual to get it recorded um funny in this feedback uh last time when it
wasn't recorded you someone mentioned that i was more funny than before it might be so i feel more
free if i'm not recorded maybe i will not record it next year and have more jokes in here um
speaking of which coming to the evaluation uh so by now we have 14 participants thank you very
much for this this is a very important thing for me and i think also for everyone in general
to listen to your feedback and uh yeah let's summarize it so uh i just aggregated these
grades a little bit by the different groups so for the content you gave us a 2.3 for the
course material 2.1 for the lecturer 1.09 and for the practice class a 2.0 which is generally a good
grade and this is also reflected in the overall grade where you give like kind of a Gaussian
distribution here um on the two so it's an average of about two what did you like you liked the
feedback mechanism so maybe this is something to keep so this immediate feedback after the
lecture and actually i also did like it especially now that there are comments um you like the fact
that it's recorded okay i can see that people mentioned it's interesting topics and uh the
exercises are yeah reasonably well aligned with the lectures and uh there's a variety of tools
which some people like and some people don't so especially someone didn't like the fact that you
also had to learn r for one exercise by that yeah okay maybe it's too many tools i know um on the
negative side you didn't find it interactive enough so this is something i'm trying to change
here by asking more questions um maybe not so extreme that i will not continue until someone
asks a question or gives a comment i mean i know people that do that but yeah point well taken so
i like to hear myself talk and if nobody interrupts i just talk okay then uh there is
this two theoretical need for more practical applications this is a fine line to walk um
and this is also i mean i rephrase these right so sometimes like several people said that
there's really a fine line to walk because this is a university's master's course this is not
a applied sciences um i'm gonna work in rnd and really want to know how to actually operate
the modern framework course right so for this you would want to go to the um master project
what we offer or other courses this is a theoretical lecture yeah you might find it too
theoretical that's okay um too many tools here i mentioned this with r not always ending on time
well i'm really trying um sometimes i just run over because i'm too much in the middle of things
so whenever you do this either interrupt me or just vote with your feet and just stand up and go
it's fine then i can just like realize that um and also the balance of all the new methods was
criticized even though i'm constantly like add-in to this lecture but the balance is perceived as
too many old methods well yes and no i still have the um like i'm still in the opinion that
you should get you know the foundational principles of it in order to be able to
understand how people actually could arrive at new mechanisms so for example the recent advance
of xlnet over bird is actually inspired by ibm model 2 which is the reordering of alignments
if you know this reordering of alignment step you immediately understand what xlnet is otherwise
you just pull what's that but it's 80s ideas 80s technologies you have to have the idea basically
to pull this off um so yeah i'm kind of sticking with my concept on this one sorry um
yeah 15 participants sorry and there was one extremely negative feedback uh and this kind
of happens sometimes so there's someone that actually gave me a six on this lecture
and i mean this is a single person this is the outlier okay so whoever crossed the six
you're kind of along with your opinion um i'm just wondering about one thing so of all the
participants only one participant crossed that he or she didn't attend this because of interest
but because of other reasons maybe but this is something i don't understand so if you if you
read just like the lecture it's fine right like it's okay it's a personal thing stuff resonates
with you stuff doesn't like i'm not taking the six personally but if you're really not interested
and you hate the course is not giving me a five but a six which is like super fail right um
why are you here i don't get that point i just don't get it this is not a compulsory course in
any of the um study plans we offer anyways on the more positive and future side
um if you like the course or even if you didn't you want to have something else maybe you'll try
again um what do we offer uh so in the winter we order nop for web this is a lecture which is
less theoretical it's more applied it's more practical uh we're gonna revamp the exercises
so they used to be in java and uema and now we're gonna go python maybe tensor flowish uh modern um
nlp frameworks this is actually where you should go if you really want to apply this in practice
uh and also the um the lecture will be less algorithmical it's more like applied so it's
more like okay what are these applications how would you actually pull these off and as algorithms
we mention them but we treat them more or less like black box because we opened the boxes already
in this lecture or we'll open them next year for these people that attended uh the first time
without having seen this one um we also offer the master project uh which is always a very nice one
i think we also get a very good feedback on this one it's usually set of like five to eight people
so a small course and where you can do like your entire nlp project during one semester and
summer and semester means it ends in september so you can also work on this in the summer if you
want to which might or might not resonate with you depending on your plans and there we basically
look at current technologies and then we build something from scratch which should always include
somehow like a front end and somehow like a back end um so it's called uh
what is it called something with like uh web-based visualizations for natural language processing
or so so so it always has this web component it has a docker component how do we actually
virtualize these applications so this is something where you can learn a lot of skills that you could
use as a use as a data scientist later there's a question about this one what is the language of
the master program english yeah so nlp for web lecture is also in english i stopped offering
lectures in german in the master's level it's just not with it um yeah and of course we also
offer uh other things so maybe you got interested in uh this topic and wanted to do maybe a master's
thesis with us or maybe you look for money and you think i can of course earn maybe more money in
industry but maybe i want to do something uh research related so yeah come talk to me come
talk to uh the others in my group we have always a bunch of opportunities and we also
depending on the project can also pay a little bit more than the usual hebei rates
um and uh what are interesting topics uh for us right now uh many many many so
the best thing is always that you have your particular idea and it involves nlp and you
come by and you want to do the research and we support you and shape this into a master's
thesis this is our favorite mode of operation because then you are super motivated because
it's your topic right um if you need some inspiration um currently i'm interested in
sense change over time and how to visualize this and display this to uh philosophers that
actually look at the digital history of concepts um then uh we uh extensifying our collaboration
with the berseris law school in order to look at the language and the structure of law
so law is something where a lot of computer scientists think why don't we automatize uh
like jurisdiction because it's actually it's like rules right you just have to apply them
it's actually not at all this case there's a lot of interpretation and there's a lot of knowledge
people need to have in order to read all these things because they codified it in the sense that
you need a lawyer to actually understand the law so that's on purpose um this is how they protect
themselves uh but also law has a lot of like cross-references a lot of meaning um related things
a lot of structure uh also beyond uh the the actual paragraphs but you have it's kind of like
a wikipedia there's just linked everywhere um so it's a very interesting uh and upcoming topics
there are not many nlp and law uh researchers and companies still but this is going to be
like a major growth area in the next few years then um also be interested in the project and
web scale argument mining so what we're currently building is like a prototype where you can compare
stuff based on web results and it would tell you why it thinks python is better than java and for
for what for example for web development or for scientific computing and something like this
and web scale means actually that we have a large computing cluster where we have a copy of the webs
called the common crawl where we actually can find arguments in uh real time using a large index if
you like big machines uh we have the biggest machine in the department and uh one thing i
want to go into a little bit more detail because it's like a short-term immediate opportunity
is called the meeting minute bot so what is the meeting minute bot it is actually the idea that
we have a meeting and there's also the bot and the bot actually takes notes of the meeting
using speech recognition and also records who says what maybe by speaker identification
could produce a summary of the meeting could produce a heat map something like
here people talk faster here they interrupt each other okay so this is where the hot discussion is
could maybe produce you know topic distributions or word clouds like this um this is actually
something we are planning to build a demonstrator for and this should be demonstrated in november
and we urgently need people that help us with this task so in case you're in need of money
and you don't know what to do over the summer just talk to me because we can start
uh right away with this one in your in case you're interested so the target audience is actually
uh politicians in the state of hessen and um yeah uh we have all the small elements like we
have something that recognizes speech we will buy some microphones in order to make this happen
properly and we would mainly need help with the front end so the whole bling like basically show
the nice things um but of course there's always opportunity to help in the back end to so in case
you're also into speech uh i haven't talked about this at all this is more like the um the business
of timo gachman in case you visited his lectures uh so he's more about the sound level but we also
do this here so in case you're interested in these kind of things or maybe in chatbot like functionalities
of this thing so maybe you could call the bot to do something kind of like alexa situation yeah
just come talk to us we have a lot of stuff to do we need more hands and with this are there any
more questions comments wishes well then have a nice summer good success for your exams and thank
you very much for staying bye bye