As our group of the Department of Computer Science is working on computer-based methods
of digital manuscript analysis, and what we are doing, I'll give you one example today,
it's about retrieval of writing patterns from these images, and writing patterns can be
characters, they can be parts of characters, they can be ligatures, words, any parts of
words, any kind of that, meaningful or not, and retrieval means finding samples of these
patterns within an image, and they have to be similar based on the visual similarity,
not on the semantic similarity. You can use this for spotting where characters or where
samples, counting them, it may even be useful for the task of writer identification, say
human expert can take a look at several samples of the same character at once, because we
pick them up from the manuscript and place them all on one sheet.
Approaches to retrieval systems have already been done within a group of Professor Srikhari,
the CDAR group, or by the Munk system of Professor Schumacher. They're using different kinds of
descriptors, and some of these systems depend on prior segmentation of these manuscript images in
lines of text or in words, maybe even into single characters. Well, our system does not depend on
any kind of prior segmentation. To give you a quick example of what retrieval can be, we have
a query image, a query writing pattern of this Chinese character on the left, and then our system
could be able to find all samples of this character. Our approach consists of two steps.
The first is determining so-called interest points and local descriptors, fully automatic,
and this is used as a representation of the data. And the next step is finding these patterns
according to an interactively given query pattern by the user and finding all similar occurrences.
So the first, these interesting points, these are points which feature a slope of a brightness. We
call it a gradient. And not only one direction, one direction would mean a representation of an
edge, but in at least two directions. So it features a corner. So these red dots are marking
positions of these corners. We have detected them by using the so-called Heavis Corner Detector.
And we are storing not only the coordinates, but also the type of a corner. I selected 48 different
classes of corner types. I'll come back to this later. It's just now about, there's not much data
to be stored. So within a larger corpus, that means we have determined about 30 points for each
Chinese character. And the computation on this part of a manuscript took on my laptop here about
130 seconds. So you can easily imagine what that means if a modern computer will work overnight
on a larger corpus. These interest points are then stored once and for all. And they serve as
a basis for the representation of the data. Well, having a character like this as a query image,
our approach was able to find 47 from 48 samples. That gives a pretty high rate of precision and
recall. In fact, the more complicated the input image is, the more interest points it features,
the better the results at the moment. This is an example of the codebook, of our current codebook,
of types of corners. We use the structure tensor, that is a representation of the gradient within
an image at the neighborhood of each hairless corner. And we classify, I took samples of each
type of these corners and I fed this into a so-called support vector machine. And they learned
these types of corners. And now this algorithm is able to determine the type of the corner
for each of these hairless corners. So it looks like that.
Just some examples. For the final decision, if within a corpus, within a manuscript,
a certain region is in fact a match of a query image is taken by this, it looks complicated,
but it's rather simple. It's a probabilistic model. And in detail, we take one corner of the
query image, one of these interest points, and we determine all interest points within the data,
which match that certain kind of corner according the type. And then we mark a region around this
corner and the size of nearly the input image. And we try to determine the relation between
all interest points from our query image to this region within the data. So we determine the
probability that, in fact, interest points are there in that region, that the local arrangement
between these interest points match the query image, and that the type of the corner, here
called feature, the class, is also the same. And the combined probabilities here on the left side
is the so-called a posteriori probability, the relationships of two hypotheses, that the target
is a match or the target is not a match, are determined and compared with the a priori
probabilities. And if this test succeeds, we test the query image on the data, and then, again, we
take that region from the data and test it again against the query image. And if both tests succeed,
the algorithm decides that we definitely have a match. So to sum it up, we get quite promising
results on a Chinese script, because Chinese characters feature many kinds of angular positions
of loose ends of script. And we have to refine our algorithm for other writing systems,
which feature less kinds of angular situations. We have to incorporate other more elaborated
methods, like connectivity analysis, for example. And this is still work in progress,
but I hope I could give you some insights about our current status.
So that's it. Thank you for your attention.