I will start with an overview of my presentation, I will talk about the objectives of the project,
I will describe a short experiment related to it and after that some issues about further development and research.
So the project is intending to the construction of an annotated data set of art images which
are actually digitized Byzantine and Orthodox icons using the corresponding textual description
from the painter's manual, text of instruction from 18th century by Dionysus of Forna. The
image segmentation and annotation has as a goal to identify the contiguous areas of interest in
the image and to assign labels coming from the textual description to every area and of course
to publish the results on the web. The experiment used two types of tools to open source Java packages,
GemIdent which is an interactive statistical system for image segmentation and annotation.
First, design for the recognition of types of cell in the medical domain. The other one stands for
parser which was used for parsing sentences in English but can be used for other languages as well.
The segmentation phases included color and phenotype that is types of object training and after that the
classification of the images and the parser was used to extract candidate labels for the classes
of objects in the image and also potentially to derive relations between objects. As an example,
it is an icon on the theme of the baptism of Christ and a corresponding fragment from the
painter's manual. Christ standing in the midst of the Jordan, the forerunner on the bank of the river
to the right of Christ above his heaven and out of it the Holy Spirit in a ray of light descends to
the head of Christ. To the left angels are standing devotedly with their wings outspread over the
garments, fish surround Christ. So we can see the parallel between the pictorial and the textual
context. The first phase was the phase of color training and four colors were chosen, amber, blue,
brown and yellow and the training process consisting in clicking on the relevant area for
every color and what we have on the right side there are small white crosses which is training
points for brown and a small cross which is black cross which is actually a training point for
yellow. There were about 20 training points for every color. The Stanford parser was used to
produce a list of words annotated with part of speech tags and from them nine labels were chosen
for the images for the objects in the image. Angel, bank, Christ, fish, forerunner, Holy Spirit,
Jordan, ray and word from the list of nouns. These labels were used to define nine phenotypes
in the phenotype training phase which consisted in introducing the label in the corresponding field
and also to choose to select training points for every relevant area. We can see the
yellow points on the right are training points for the object ray and the light green
training points for the object Holy Spirit. The software produced on the right side it is
a sort of mask where the blobs are marked with a color for every of the nine objects
and we can see that the shapes are quite accurate but there are still small areas
areas which inside an object which belong actually to another object like the light blue
inside the Christ object or inside angels object. The software also produced separate
files for every object is Christ is Jordan is angel what we see on the right side
with a mask in white and for the rest of the objects in the pictures in black.
Stand for parser was also used to analyze the grammatical relations between a governor element
and a dependent like we can see nominal subject in standing Christ or standing angels or surround
fish direct object like surround Christ or prepositional objects like standing to the left
or standing in the midst of the Jordan and I will come back to this a little later.
Some observations about other tests within the experiment so the idea is that for new images if
a little number of training points were including in the training set the results were less accurate
that we can see for the baptism of Christ another representation of the same theme and
the recognition is less satisfactory for that just a few points were from this image
were were included in the training set. On the other side for the other icon on the
theme of nativity ambiguity seems to be a problem causing systematic errors
for the cases when a color is used in different objects in the image with no distinctive details
like element from the object Mary inside the magic objects for instance.
Further research is of course needed and it is intending to produce new tests with
with new sets of images but also using similar tools like Fiji image image segmentation plugins.
And also to use the parser for translate translation of the painter's manual text
for French and German to annotate for multilingual annotation of the images.
Grammatical relations extracted by the parser can be also used to derive eventually
spatial relations between the objects in the image like to the left to the right above below
and so on or other kinds of scene-based relations like agent action object
recipient fished around Christ or ready sense on Christ.
Other concerns for further research is the publication of the results using semantic
web specifications and some existing vocabularies for art collection like art and architecture
tesoros icon class ulan or other resources like ontologies sumo or lexical semantic database
wordnet. It is also intended to construct an interface which can combine segmentation and
parsing capabilities for the user to re-experiment or reproduce the experiments.
As a conclusion so the project is intending to produce this data set segmented and annotated
digitized art images with potential uses in other applications as a study basis for
content-based image retrieval or learning-based classification or in pedagogical and research
context for the study of text image relationships in art history. The current phase is actually
an exploratory one about the possible approaches and methods tools and formats that can be used
for the project and for the art collection it is a discussion with Valois d'Art Studio and
School in Montreal which agreed in principle to provide the digitized icon actually icon that
they produce in their studio and there are also inquiries on funding matters and collaboration
under consideration. Thank you.