Abstract |
Sign languages are viso-gestual languages, using space and movement to convey
meaning. To be able to transcribe them, SignWriting uses an iconic system of
symbols meaningfully arranged in the page. This two-dimensional system,
however, is very different to traditional writing systems, so its automatic
processing poses a novel challenge for computational linguistics. We identify
as first and fundamental step to overcome this challenge the extraction of
a computational representation of the semantics represented by SignWriting
transcriptions. We propose a data-based modelization of the problem, construed
from real handwritten SignWriting instances. We then propose two solutions
involving state of the art machine learning techniques combined with expert
analysis. The first solution is direct application of an existing deep neural
network. Our second proposal exploits the expert knowledge codified in the
data annotation scheme that we present, in order to craft a system that
improves on the straight-forward solution's accuracy by 30%. This improved
system uses a number of different neural networks to divide the necessary
processing, progressively constructing the prediction in an iterative pipeline
that combines deep learning and domain knowledge in a mixed solution.
|
Citation |
@unpublished{sevilla_automatic_2021,
month = {November},
title = {Automatic SignWriting Recognition},
year = {2021},
keywords = {Sign Language, SignWriting, Deep Learning, Expert Knowledge, Neural Networks, Computer Vision},
url = {https://eprints.ucm.es/id/eprint/69235/},
author = {Sevilla, Antonio F. G. and D{\'i}az Esteban, Alberto and Lahoz-Bengoechea, Jos{\'e} Mar{\'i}a}
}
|