Tradução automática de línguas de sinais: do sinal para a escrita

Abstract

Automatic recognition of signs of a Sign Language by a computer is still a challenge. So far, there is no automatic conversion signals for sign language written in the notation in SignWriting (SW). These factors were that motivated the development of this work in order to find a solution to this problem that had not been solved in the literature. Thus, in this work we recognized language signals in images and converted them into written signals known as SW. More precisely, we focused on the parameter of hand shape, which is important to define the hand shape that the hand will assume during the execution of a signal. For this, we trained Convolutional Neural Network (CNN) that is a mechanism of Machine Learning (ML) which provided the signal recognition. To capture images, we used the Kinect motion sensor together with an developed application that does the preprocessing of the images. We did training the CNN with 16 different hand shapes and have achieved the average accuracy of 87.5%.