The primary goal of sign language recognition is to develop methods and algorithms to accurately identify a series of produced signs and to discern their meaning. The majority of studies have focused on recognizing those features and methods that can properly identify a sign out of a given set of possible signs. However, such methods can only be used on a particular set of signs and, thus, a specific sign language, which makes it harder to study the relationships between and evolution of various sign languages.

An additional motivation behind Sign Language Recognition (SLR) is to build automatic sign language to speech or text translation systems to assist the communication between the deaf and hearing community [Fang et al. 2004]. Moreover, SLR plays an important role in developing gesture-based human–computer interaction systems [Kelly et al. 2009]. Sign language linguists have used such systems to facilitate the annotation process of sign language corpora in order to discern the different signs in a video recording and further study the linguistic phenomena presented.

There are numerous studies dealing with the automated recognition of sign languages as clearly presented by Cooper et al. [Cooper and Bowden 2007], in their review study on the state-of-the-art in sign language recognition. However, the experiments presented in most of these studies are either hard to replicate, or they pose limitations as far as their applicability is concerned. For instance, most of these studies use depth sensors, most notably MS Kinect, to capture 3D images of the environment [Aitpayev et al. 2016] [Pigou et al. 2015] [Zhang et al. 2013]. As a result, using the frameworks developed in these studies requires a machine with similar features as the one used for testing and most probably will only work for that sign language on which they have been trained.

Recently, computer vision techniques have been applied to sign language recognition to overcome the aforementioned limitations. Roussos et al. [Roussos et al. 2012] created a skin color probabilistic model to detect and track the hands of a signer on a video, while Cooper et al. [Cooper and Bowden 2007] use this model to segment the hands and apply a classifier based on Markov models. However, systems based on skin color [Buehler et al. 2009] [Cooper and Bowden 2007] [Farhadi et al. 2007] [Starner et al. 1998] are prone to errors and have difficulties on tracking the hands and the signer’s features against cluttered backgrounds and in noisy conditions in general. Also, they do not work in videos with multiple signers.

Human pose estimation has been extensively studied due to its numerous applications on a number of different fields [Moselund et al. 2006]. Due to low computational complexity during inference, pictorial structures have been commonly used [Felzenszwalb and Huttenlocher 2005] [Ramanan et al. 2007] [Sivic et al. 2006] to model human pose. Recently, studies have focused on improving the appearance models used in these structures by modelling the individual body parts [Eichner and Ferrari 2009] [Eichner et al. 2012] [Johnson and Everingham 2009] [Sapp et al. 2010]. Felzenszwalb and Huttenlocher [Felzenszwalb and Huttenlocher 2005], relying on the pictorial structure framework recommended a deformable part-based model. Additionally, Yang and Ramanan [Yang and Ramanan 2011] showed that a tree-structured model using a combination of deformable parts can be used in order to achieve accurate pose estimation. Furthermore, Charles et al. [Charles et al. 2014] showed that human body joint positions can be predicted using a random forest regressor based on a co-segmentation process over all video frames.

In general, most of the vision-based approaches developed for sign language recognition tasks utilizing pose estimation, have used the RWTH-PHOENIX-Weather data set [Forster 2012] to validate their models. This data set consists of weather forecast airings from the German public tv-station PHOENIX along with transcribed gloss annotations. However, it is a question to what extent such systems tested in this data set can be replicated with real-life conditions in the corpora. It is often the case that sign language and gestural corpora, especially the ones filmed outside of studio conditions, have bad quality, low brightness and often contain more than one person in the frame. These characteristics create an additional challenge to the tracking and prediction task.

A data set of 7,805 frames in total (approximately 4 minutes) labeled as signing or not signing has been compiled for the first part of the study. The dimensions of the frames were 352 by 288 pixels and were extracted from the Adamorobe and Berbey Sign Language corpora [Nyst 2012] [Nyst et al. 2012]. These corpora portray an additional challenge as the signers have been filmed in and around their homes, in natural conditions, outside of a studio, with strongly varying brightness and background noise. Furthermore, they may contain signing from one and two people at the same time. As a result, they can be considered as one of the hardest corpora to perform classification tasks. It is arguable that if the methods developed in this study can perform reasonably well on corpora of such poor conditions, then they can be applied to any sign language corpus under better settings.

Additional videos from YouTube with higher quality have been selected for testing purposes too. For the first task of this study, the original data set was split into a training and testing set of 6,150 and 1,655 frames respectively and the labels were one hot encoded (i.e. signing as 1 and not-signing as 0).

After a successful training of the first prediction model, the tool was applied on a different part of the corpora. The predicted signing sequences were manually labeled as one- or two-handed signs. Together with randomly selected not-signing sequences (as predicted by the first tool), they formed a second data set. The size of this data set was slightly larger than the previous one: 10,120 frames in total.

Using OpenPose, the pixel coordinates of the hands, elbows, shoulders and head were extracted from each frame. In the case of the handshape recognition module, the fingers joints coordinates were additionally extracted. We avoided using the finger extraction module of OpenPose on the first two parts of the study as that would have increased the computational time significantly. The positions of the rest of the body joints were disregarded as most of the time they were out of the frame bounds. Although the quality of the frames was poor, it created an advantage for the pose estimation framework, reducing the computational time to a reasonable level.

The first tool is a temporal segmentation method to predict the begin and end frames of a sign sequence in a video sample. Thus, it is important to compare the performance of multiple different machine learning algorithms consistently. Four classification methods were used, namely: Support Vector Machines (SVM), Random Forests (RF), Artificial Neural Networks (ANN) and Extreme Gradient Boosting (XGBoost). The majority of these algorithms have been extensively used in machine learning studies as well as in sign language applications [Agha et al. 2018]. Performance was measured using the Area Under the Receiver Operating Characteristics (AUC) to validate each model. The AUC is a performance measurement specifically designed for binary (i.e. two class) classification problems. In general, it expresses how well a model is capable of distinguishing between classes, for example whether someone is signing in a video fragment or not. A model that makes random predictions will have an AUC of 0.5, a perfect model will have an AUC of 1. AUC stands for “Area under the Receiver Operating Characteristic (ROC) Curve”, the curve of True Positive Rate (probability of detection) versus False Positive Rate (probability of false alarms). It is better than just accuracy, i.e. percentage of correct predictions by the model, because it is not dependent on the relative amount of positives, i.e. percentage of total videos with signs in our case. We searched for the optimal setting of the various classification method parameters by exhaustive testing of the possible parameter settings and testing the performance on a validation set (“grid search”, searching the “grid” of possible parameter values).

The second tool’s goal is to predict not only if a person is signing or not, but also to identify the number of hands involved (one- or two-handed). We hypothesized that this task is more complex than before, thus we considered it as a time-series problem. By using a sliding window technique, the original data set was parsed to form new training sets, where different possible frame intervals (1,2,3,5 and 10) were tested. Furthermore, similar (to some extend) classification methods with Tool 1 have been used [1].

Moreover, recent studies in the sign language recognition field suggest that the use of Long-Short-Term-Memory (LSTM) networks can yield accurate results. LSTM is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks (like the one tested in Tool 1) LSTM has feedback connections. It can not only process single data points, but also understand patterns in entire sequences of data, by combining its internal state resulting from previous input with a new input data item. In our case, instead of predicting whether a specific pose belongs into a class, we investigate whether a sequence of poses can be used for the same purpose. In this part of the study an LSTM network with different layer units as well as sliding window intervals has also been tested and compared with the above traditional machine learning classifiers. The overall architecture and technical details of the LSTM network can be found in Appendix A.

The handshape recognition module was considered a so-called unsupervised learning problem as no ground truth information regarding this feature was available prior to the experiment, i.e. in contrast to the previous two problems we did not know what classes (handshapes) to detect. Such an unsupervised learning method can be useful in other newly compiled sign language or gestural corpora where there is no information regarding the different handshapes presented by the signers in the video. Additionally, an unsupervised learning method can be useful in other newly compiled sign language or gestural corpora where there is no information regarding the different handshapes presented by the signers in the video. We approached this as a clustering task: can we find groups of signs that were similar. Two different clustering methods have been tested: K-means and DBSCAN. The first clustering method was chosen for its simplicity as well as its fast implementation on the Python library that was utilized (namely scikit-learn). However, as the complexity of the data is unknown and it is case sensitive, it was decided to employ Density-Based Spatial Clustering of Applications with Noise (DBSCAN) as an alternative option. Given a set of points in some space, DBSCAN groups together points that are closely packed together, marking as outliers the ones that lie alone in low-density regions. This clustering method is one of the most common clustering algorithms.

Determining the optimal number of clusters (i.e. total number of expected handshapes) is a crucial issue in clustering methods such as K-means, which requires the user to specify the number of clusters k to be generated. The definition of clusters is done so that the total within-cluster sum of square (WSS) is minimized, hence, in this study the elbow method was utilized to estimate the number of clusters.

All classifiers performed adequately well, apart from the Support Vector Machines (AUC: 0.80) (Table 1). Extreme Gradient Boosting (XGBoost) showed the highest AUC score at 0.92[2]. Figure 2 presents the ROC curve after a 10-fold cross-validation. The Artificial Neural Network was found to perform sufficiently well (AUC: 0.88). By exploring the importance of each feature on the prediction of the model we observe that the y and x pixel coordinates of the dominant (i.e. right) hand are on the top two positions (Table 2).

The fact that the Artificial Neural Network turned out to be a less efficient approach than the XGBoost can be accounted to the small training data set. Typically, Neural Networks require a lot more training data than traditional machine learning algorithms. Additionally, designing a network that correctly encodes a domain specific problem is challenging. In most cases, competent architectures are only reached when a whole research community is working on those problems, without short-term time constraints. Fine-tuning such a network would require time and effort that reach beyond the scope of this study.

To account for multiple people signing in one frame, an extra module was added. This module creates bounding boxes around each person recognized by OpenPose, normalizes the positions of the body joints and runs the classifier. This process makes it possible to classify sign occurrences for multiple people irrespective of their positions in a frame (Figure 4).

Once all the frames have been classified, the "cleaning up" and annotation phase starts. A sign occurrence is annotated only if at least 12 consecutive frames have been classified as "signing". That way we account for the false positive errors. This sets the stage for the annotation step. Using the PyMpi python library [Lubbers and Torreira 2013] the classifications are translated into annotations that can be imported directly to ELAN, a standard audio and video annotation tool [Sloetjes and Wittenburg 2008]. Figure 3 shows the result of the overall outcome.

The second tool is responsible for not only recognizing whether a person in a video is signing but also if the sign is one or two-handed. We have previously hypothesized that this is a more complex task than the previous binary classification. Results on the accuracy of all the classifiers suggest that it is not as intricate as initially thought of; the higher the sliding window interval, the lower the accuracy of the model. As seen in Figure 5 of all classifiers tested, Random forest had the highest accuracy at the sliding window interval of 1 frame at a time. Similarly to the previous experiment, a frame-to-frame prediction can produce the highest results.

Furthermore, the results regarding the Long-Short-Term-Memory networks (Figure 6) suggest that the highest accuracy can be achieved at a sliding window interval of 56 frames and at a hidden layer size of 8 units. However, such a high window interval contains more than one sign, as the average length of a sign is approximately 14 frames. This discrepancy can be caused due to the architectural properties of the LSTM network. The average length of the signs is too small for the network to converge. The LSTM units needed more timesteps in order to prevent overfitting to the data. This property in addition to the small dataset used to train the network caused this anomaly.

Although the tool performs well on predicting whether a sign is one- or two-handed (using a Random Forest classifier) there are cases were the output is not as expected. In particular, cases where there is a two-handed symmetrical sign produced, the tool fails to accurately predict the correct class. It is likely that such signs were under-presented in our data set, thus resulting in poor classification.

In order to understand the distribution of the different handshapes presented in a video, Principal Component Analysis (PCA) was utilized on all the normalized finger joint coordinates for all the frames at once (Figure 7a). This process allows us to reduce the dimensionality of the data while retaining as much as possible of the variance in the dataset. Each multidimensional array of the extracted finger joints positions, for each frame, has been reduced to a single x,y coordinate. The result already suggests that there are regions dense enough to be considered different clusters. The utilized elbow method suggested that at k=5 the highest classification could be achieved (Figure 7b). On the video sample used in our study that number seemed to reflect the proper amount of discerned handshapes. However, as OpenPose captures all the finger configurations in each frame it is at the linguist’s discretion to decide on when a handshape is significantly different from another. Additionally, experiments to optimize the hyperparameters (eta, min samples and leaf size) for the DBSCAN failed to create an accurate clustering (Figure 7c). Subsequently, the module creates annotation slots for the different handshapes in the video and adds an overlay containing the number of the predicted cluster on each frame.

However, special consideration must be given to the overall handshape recognition module. Although the hand normalization process prepares the finger joints adequately enough to be used in the clustering methods, it fails to account for hands perpendicular to the camera’s point of view. Additionally, handshapes that are similar to each other but are rotated towards or outwards of the signer’s body will most probably clustered differently. Some of these limitations can be solved by manually editing the cluster numbers prior to the annotation process.

In its current form, this method can already be used to either fully annotate the handshapes in a video sample or be used in different samples and treated as weakly annotated data in order to be used in other handshape classifiers similarly to Koller’s et al. study [Koller et al. 2016].