Automatic SignWriting Recognition

Authors Antonio F. G. Sevilla
Alberto Díaz Esteban
Jose María Lahoz-Bengoechea
Keywords Sign Language, SignWriting, Deep Learning, Expert Knowledge, Neural Networks, Computer Vision
Abstract Sign languages are viso-gestual languages, using space and movement to convey meaning. To be able to transcribe them, SignWriting uses an iconic system of symbols meaningfully arranged in the page. This two-dimensional system, however, is very different to traditional writing systems, so its automatic processing poses a novel challenge for computational linguistics. We identify as first and fundamental step to overcome this challenge the extraction of a computational representation of the semantics represented by SignWriting transcriptions. We propose a data-based modelization of the problem, construed from real handwritten SignWriting instances. We then propose two solutions involving state of the art machine learning techniques combined with expert analysis. The first solution is direct application of an existing deep neural network. Our second proposal exploits the expert knowledge codified in the data annotation scheme that we present, in order to craft a system that improves on the straight-forward solution's accuracy by 30%. This improved system uses a number of different neural networks to divide the necessary processing, progressively constructing the prediction in an iterative pipeline that combines deep learning and domain knowledge in a mixed solution.
Citation
@unpublished{sevilla_automatic_2021,
  month = {November},
  title = {Automatic SignWriting Recognition},
  year = {2021},
  keywords = {Sign Language, SignWriting, Deep Learning, Expert Knowledge, Neural Networks, Computer Vision},
  url = {https://eprints.ucm.es/id/eprint/69235/},
  author = {Sevilla, Antonio F. G. and D{\'i}az Esteban, Alberto and Lahoz-Bengoechea, Jos{\'e} Mar{\'i}a}
}
Links Preprint version