9.1 Introduction

The human language capacity is flexible and very powerful. It gives us not only the usual languages that are the focus of linguistics, i.e. natural, oral languages, but also artificial systems for communication that have some or most of the features of languages and can therefore be studied and processed with linguistic tools.

An example of these systems are graphical languages, which rely on images for communication. Musical notation (Figure 9.1), Feynman diagrams, elementary arithmetic notation, or the Unified Modeling Language (UML, Technical Committee: ISO/IEC JTC 1 Information technology (2012), Figure 9.2) are systems which share some of the important characteristics of languages: signs with a signifier-signified nature, syntactic rules for combining them, and meaning which is compositional, a whole resulting from the consideration of the symbols but also their context and relative arrangement.

However, graphical languages have an important characteristic which is not common in natural languages: they are visual, and exploit the two dimensions of the page as a fundamental feature for codifying meaning. Location of symbols in the two dimensions and their relative arrangement is key to the correct understanding of graphical languages, but it is a problem beyond the tools habitual to natural language processing.

If a graphical language is not stored in an abstract, semantic representation, but rather in its graphic realization, it is a challenge to process it automatically. It is necessary to locate symbols within the page, storing their relative locations since these are meaningful data, as well as finding the meaning intended for each symbol, meaning which can be encoded with overlapping graphical features such as size, rotation, color, etc.

In this article we present a software tool we have developed to deal with the annotation and automatic processing of images of graphical languages: Quevedo.

Quevedo is an open source python library and application with both command line and web interfaces. It can create, organize and manage sets of images containing samples of a graphical language. The web application provides a way of visually inspecting and exploring the datasets, as well as featuring an extensive annotation interface. It can be run locally, or deployed as a server for access or annotation by a team.

Processing of the images and their annotations can then be performed, or they can be used in other programs by importing Quevedo as a library. The processing allowed by Quevedo includes training and testing computer vision neural networks for the recognition of the target graphical language. These networks are configured using a declarative format, and can then be organized into a pipeline, allowing more complex processing and automatic recognition to be performed.

We strive for Quevedo to be a complete solution for the computational processing of graphical languages, guided by our own efforts in that domain. However, we are aware that every task has its own quirks, so Quevedo is very configurable, and extensible with user scripts. The organization of the datasets uses standard file formats and is straightforward and accessible, so the source data, annotations, and neural networks can also be consumed externally or enhanced with other tools.

In our research, we use Quevedo for managing SignWriting data. SignWriting (Sutton y Frost 2008) is a graphical system for transcribing sign languages, using the two dimensional possibilities of the page to encode the movement and use of space of sign languages. In Figure 9.3, an example SignWriting transcription can be seen, and in Section 9.5 we give more detail into this use case.

Before that, in Section 9.2 we examine some related work that is relevant, Section 9.3 gives an overview of the different features in Quevedo, and Section 9.4 gives some notes into how to use Quevedo as a library and application. After we have presented our SignWriting case study in Section 9.5, Section 9.6 summarizes our conclusions with Quevedo and the research it has enabled, and Section 9.7 explains our future development and ideas for how we want to improve it and expand the range of problems it can solve.