Given the central place of the embodied mind in experientialist approaches to language, co-verbal gestures have become a valuable data source in cognitive, functional and anthropological linguistics (e.g. [Sweetser 2007]). While there exist various views on embodiment, the core idea is that human higher cognitive abilities are shaped by the morphology of our bodies and the way we interact with the material, spatial and social environment (e.g. [Gibbs 2006]; [Johnson 1987]). Drawing on these premises, some gesture scholars stress that gestures are conditioned by the forms and affordances of their material habitat as well as the speakers’ interactive and collaborative practices (e.g. [Enfield 2009]; [Streeck 2011]). Pioneering work done by Kendon (e.g. [Kendon 1972], [Kendon 2004]), McNeill (e.g. [McNeill 1985], [McNeill 1992], [McNeill 2005]) and Müller [Müller 1998] has shown that manual gestures are an integral part of utterance formation and communicative interaction. The state-of-the-art of research in the growing interdisciplinary field of gesture studies has recently been presented in the International Handbook on Multimodality in Human Interaction ([Müller 2013], Vol. 1 2013, [Müller 2014] Vol. 2 2014). One quintessence to be drawn from this large body of work is that language, whether spoken or signed, is embodied, dynamic, multimodal and intersubjective (see also [Duncan 2007]; [Gibbs 2006]; [Jäger 2004]; [Mittelberg 2013]; [Müller 2008]).

Indeed, human communication typically involves multiple modalities such as vocalizations, spoken or signed discourse, manual gestures, eye gaze, body posture and facial expressions. In face-to-face communication, manual gestures play an important role by conveying meaningful information and guiding the interlocutors’ attention to objects and persons talked about. Gestures here are understood as spontaneously emerging, dynamic configurations and movements of the speakers’ hands and arms that contribute to the communicative content and partake in the interactive organization of a spoken dialogue situation (e.g. [Bavelas 1992]; [Kendon 2004]; [McNeill 1992]; [Müller 1998]; [Mittelberg 2016]). The contribution of gestures to multimodal interaction may consist in, e.g., deictic reference to locations, ideas, persons or things (both abstract and concrete); they may also fulfill metalinguistic functions, e.g., in referring to citations of other speakers or outlining the structure of their argumentation. Gestures may also provide schematic, iconic portrayals of actions, things or spatial constellations [Mitteberg 2014]; [Rieser 2012]. As we use the term here, gestures are not to be confused with emblems (e.g. the victory sign), which have a culturally defined form-meaning correlation in the same sense that words may have a fixed meaning [McNeill 1992]. So gestures are always to be investigated in view of the co-occurring speech with which they jointly create (new) semiotic material and support speakers in organizing their thoughts or drawing connections to their social and physical environment [Streeck 2011]; [Mittelberg 2016].

Drawing on this large body of gesture research across various fields of the humanities and social sciences, the interdisciplinary approach presented here aims at identifying and visualizing patterns of gestural behavior with the help of custom-tailored computational tools and methods. Although co-speech gestures tend to be regarded as highly idiosyncratic in respect to their spontaneous individual articulation by speakers in spoken dialogue situations, it is safe to assume that there are recurring forms of dynamic hand configurations and movement patterns which are performed by speakers sharing the same cultural background. From this assumption follows the hypothesis that, on the one hand, a general degree of similarity between gestural forms may be presumed – trivially – due to the shared morphology of the human body (e.g. [Tomasello 1999]). On the other hand, the specific cultural context plays an important role in both language acquisition and the adoption of culture-specific behavioral patterns (see, e.g. [Bourdieu 1987] on Habitus and Hexis). Moreover, gesture has also been ascribed a constitutive role regarding the human capacity for language both ontogenetically and phylogenetically (e.g. [Goldin-Meadow 2003]; [McNeill 2012]). Previous empirical gesture research indeed shows that co-verbal gestures exhibit recurrent form features and movement patterns, as well as recurring form-meaning pairings: see, for instance, Bressem [Bressem 2013] on form features; Fricke [Fricke 2010] on “kinaesthemes”; Kendon [Kendon 2004] on “locution clusters”, McNeill [McNeill 1992], [McNeill 2000] on cross-linguistic path and manner imagery portraying motion events; McNeill [McNeill 2005] on “catchments”; Ladewig [Ladewig 2011] and Müller [Müller 2010] on “recurring gestures”, and Cienki [Cienki 2005] and Mittelberg [Mittelberg 2010] on image-schematic patterns. From this perspective, a question central to gesture research concerns the factors that may motivate and lend a certain systematicity to forms and functions of human communicative behaviors not constituting an independent sign system in the Saussurian sense as spoken and signed languages do. Manual gesture is a semiotically versatile medium, for it may, depending on a given context, assume more or less language-like functions: from accompanying spoken discourse in the form of pointing, accentuating beats, schematic iconicity or managing social interaction to carrying the full load of language (e.g. [Goldin-Meadow 2007]).

In this paper, we will focus on certain kinds of co-verbal gestures, i.e. specific image-schematic gestalts, e.g. spirals, circles, and straight paths [Cienki 2013], [Mittelberg 2010]. Figure 1 shows visualizations of several different gestural traces belonging to these movement types. In what follows, we will present a novel method designed to identify, search and cluster in an automatized fashion gestural movement patterns throughout our data set, and potentially also any other motion capture data set, e.g. to recognize and group those traces that are similar regarding their formal features and the way in which they unfold in gesture space.

Whereas the gesture research discussed above mostly relies on observational methods and qualitative video analyses, our aim is to add to the catalogue of methods for empirical linguistics and gesture studies by outlining a computational, quantitative and comparative 3D-model driven approach in gesture research. While there is a trend to combine qualitative with quantitative as well as experimental methods in multimodal communication research [Gonzalez-Marquez 2007]; [McNeill 2001]; [Müller 2013], standardized and widely applicable tools and methods still need to be developed. In order to derive statistically significant patterns from aligned linguistic and behavioral data, some recent research initiatives have started to compile and work with larger-scale corpora, e.g. drawing on technological advances in data management and automated analyses (e.g. the Little Red Hen Project, [Steen 2013]). While using audio and video technology to record co-speech gestures remains the dominant way to construct multimodal corpora, some research teams have begun to employ 3D motion capture technology to overcome the limits of 2D video, e.g. Lu and Huenerfauth [Lu 2010] for signed language, and Beskow et al. [Beskow 2011] and Pfeiffer et al. [Pfeiffer 2013a] for spoken discourse (see also [Pfeiffer 2013] for an overview). Our contribution is to obtain some kind of numerical instrument for graduating gestural similarity for measuring gesture similarity in sets of recorded behavioral data. While this instrument, in its current state, in no way addresses the problem of the meaning of certain gestural forms, it is the first step towards a model of measuring similarity between recurring dynamic gesture form patterns. Our goal is to first establish a robust, flexible and automated methodology which allows us to determine

1. whether there are shared or common reoccurring gestural movement patterns in a given set of 3D recorded, behavioral communication data,
2. exactly which forms there are, and
3. the extent to which they occur,

and then to apply this methodology to the recorded 3D numerical MoCap data of a group of participants.

Both the alignment of gestures with the co-occurring speech, and the semantic comparison of the established (formally) sufficiently similar gesture-speech constructions, still have to be done manually by human gesture researchers, through semiotic analyses of the multimodal, speech and behavioral data corpora. The primary aim of developing an automated indicator of gesture similarity is to identify recurrent movement patterns of interest from the recorded 3D corpus data computationally, and thus to enable human gesture researchers to handle these data sets in a more efficient manner. In order to make gesture similarity automatically accessible, we propose a distance-based similarity model for gestures arising in three-dimensional motion capture data streams. In comparison to two-dimensional video capture technology, working with numerical three-dimensional motion capture technology has the advantage of measuring and visualizing the temporal and spatial dynamics of otherwise invisible movement traces with the highest possible accuracy. We aim at maintaining this accuracy by aggregating movement traces, also called trajectories, into a gesture signature [Beecks 2015]. This gesture signature has the ability of weighting trajectories according to their relevance. Based on this lossless feature representation, we propose to measure similarity by means of distance-based approaches [Beecks 2013], [Beecks 2010]. We particularly investigate the Earth Mover's Distance [Rubner 2000] and the Signature Quadratic Form Distance [Beecks 2010] for the comparison of two gesture signatures on real conversational motion capture data.

From a philosophy of science point of view, before being able to apply computational algorithms to naturalistic real-world gestures, there must be a translation from the real-world dialogue situations, involving people speaking and gesturing, from which data are captured, to a computable set of data. For this purpose, a marker-based Vicon Motion Capture system was used in this study. Participants wear a series of markers attached to predetermined body parts of interest (fingers, wrists, elbows, neck, head, etc.). The Vicon system automatically generates a chart of numerical 4-tuples of Euclidean space-time coordinates for each marker attached to these points on the participants’ bodies. The movement of the markers is tracked by 14 Vicon infrared cameras, and the physical trajectories of the markers are represented in a chart of space-time coordinates. These space-time charts form the data sets that are investigated algorithmically, relieving the gesture analyst of the difficult, and subjective, task of manually examining highly ephemeral real-world dialogue situations. But what are the crucial features that such a numerical representation must have in order to enable researchers to not only investigate a model but also to finally derive statements and theories about a modeled real-world situation? We address the following research questions: Which logical features of the model are essential if one wants to investigate the real world by investigating a model? And secondly, what are the epistemic benefits of investigating models instead of real-world situations?

The most important feature is that the model represents its representandum. The representandum in question is a set of relevant features and relations of a given part of reality, namely the change in space-time location of certain body parts caused by the participants’ kinetic movement. The representing model, on the other hand, is the virtual, computational 3D recording of the real-world kinetic movement, mapped onto Euclidean space. Representation itself is the relation which holds between the model and its representandum. Representation, as we understand the term here, is a non-reflexive and non-symmetric relation, which simply means that an a does not necessarily represent a, and that if a represents b, then b does not necessarily represent a. We further assume that representation depends on transitive relations, such as identity between some complex relational features of the entities to be represented in a model and the entities that represent/model them. In short, this means that if there are entities x, y, z, there must be at least one mutual complex relational feature Fr(x,y,z), which these entities have in common. The transitive relation R, then, is the “identity” relation, in that two entities have an identical relation to a third entity – a frame of reference, or a tertium comparationis.

Definition: Transitivity

For a binary relation R over a set A consisting of x, y, z, the following statement holds true:

The transitivity of R is so important here because, in terms of modeling, it is the crucial feature that R must have. The different relata involved are: movements of body parts (x), movements of markers (y), and computational trajectories (z). The relation R which holds between these relata is the identity of their curve with respect to space – either to a given virtual Euclidian space, or the physical space (which functions as a reference frame). The identity of the movement curve of body parts and the movement curve of markers simply stems from their physical attachment/ conjunction in physical space. The identity of marker movements and the computational trajectories is a result of metering the markers’ light reflections, by means of the 14 Vicon infrared cameras, and numerically mapping the outcome onto Euclidean space coordinates. But if one differentiates between physical and Euclidean space, there is hardly any identity one could honestly speak of in this case. In what sense could physical and virtual movement curves ever be identical? Only in the sense that we identify physical and Euclidean space by conventions of scientific modeling and metering practices; the term curve is itself an indication of how familiar that convention is. Since Euclidean space is a conventionalized and well-accepted geometrical model of physical space – we are well accustomed to talking about physical movement in terms of, distances, trajectories, vectors, miles, kilometres and change in space/time coordinates etc. – one can say that Euclidean space is our familiar standard model for describing our perception of movement in physical space, both in everyday conversation and scientific discussion. Thus, it is justified to speak of the identity of the spatial curves of trajectories in Euclidean space and those of moving physical objects such as Motion Capture markers, only if we accept this convention. Regarding representation, what does this imply? If representation depends on the transitivity of relation R holding between the entities x, y, z, and the identity of that relation depends on conventions of metering and modeling, representation additionally depends on following conventions. Given a 4-tuple of coordinates of a MoCap marker, the movement of this marker is modeled by a vector which points from tuple-1 to tuple-2 to tuple-n. The crucial feature of this kind of modeling is that the movement of a single marker a is represented as a dynamic space-time trajectory which aggregates the consecutively changing coordinates of a over a given time frame.

Regarding the above-mentioned definition of transitivity, let our variables take the following values:

x = movement of body part from position a to b;
y = movement of marker M from position a to b;
z = trajectory of marker M

Given these values, we outline the transitivity relation as follows:

This means that if x, y, z obtain a transitive relation R in the above sense, then x, y, and z are to be regarded as homomorphous abstract concepts that all denote the same event of spatiotemporal movement, and R is an equivalence relation. So the extensional equivalence of these concepts is a necessary and sufficient condition that allows us to investigate reality by investigating the model: If a language and its translation are equivalent, it should be equally valid to investigate one or the other. However, since the translation of the concept “movement of marker” into “aggregated marker coordinates” fails to be an intensionally adequate translation, i.e. the concepts do not mean the same (event vs. aggregated states of affairs), it could at first seem odd to describe real movement in terms of trajectories. But, since the concepts are at least phenomenologically and extensionally equivalent, this basically remains a question of the interpreter’s ontology (see [Quine 1980]) and how the final research result is to be formulated. If we decide to treat “event” and “aggregated states of affairs” as being synonymous, the problem completely disappears. Otherwise, we have to re-translate the problematic concept into one which suits our needs. In terms of epistemic benefits, one major advantage of the proposed distance-based gesture-similarity model (see the following section), i.e. the combination of gesture signatures with signature-based distance functions, is its applicability to any type of gestural pattern and to data sets of any size. In fact, distance-based similarity models can be utilized in order to model similarity between gestural patterns whose movement types are well known and between gestural patterns whose inherent structures are completely unknown. In this way, they provide an unsupervised way of modeling gesture similarity. This flexibility is attributable to the fact that the proposed approaches are model independent, i.e. no complex gesture model has to be learned in a comprehensive training phase prior to indexing and query processing. Another advantage of the proposed distance-based gesture-similarity model is the possibility of efficient query processing. Although calculating the distance between two gesture signatures is a computationally expensive task, which results in at least a quadratic computation time complexity with respect to the number of relevant trajectories, many approaches such as the independent minimization lower bound of the Earth Mover's Distance on feature signatures [Uysal 2014] and metric indexing [Beecks 2011], as well as the Ptolemaic indexing [Hetland 2013] of the Signature Quadratic Form Distance, are available for efficient query processing and, thus, for assessing gesture similarity in a larger quantitative way.

In this section, we present a distance-based similarity model for the comparison of gestures within three-dimensional motion capture data streams. To this end, we first introduce gesture signatures as a formal model of gestures arising in motion capture data streams. Since gesture signatures comprise multiple three-dimensional trajectories, we continue with outlining distance functions for trajectories before we investigate distance functions applicable to gesture signatures.

Evaluating the performance of distance-based similarity models is a highly empirical discipline. It is nearly unforeseeable which approach will provide the best retrieval performance in terms of accuracy. To this end, we qualitatively evaluated the proposed distance-based approaches to gesture similarity by using a natural media corpus of motion capture data collected for this project. This dataset comprises three-dimensional motion capture data streams arising from eight participants during a guided conversation. The participants were equipped with a multitude of reflective markers which were attached to the body and in particular to the hands. The motion of the markers was tracked optically via cameras at a frequency of 100 Hz. In the scope of this work, we used the right wrist marker and two markers attached to the right thumb and right index finger each. The gestures arising within the conversation were classified by domain experts according to the following types of movement: spiral, circle, and straight. Example gestures of these movement types are sketched in Figure 1. A total of 20 gesture signatures containing five trajectories each was obtained from the motion capture data streams. The trajectories of the gesture signatures have been normalized to the interval [0,1]3∈R3 in order to maintain translation invariance.

The resulting distance matrices between all gesture signatures with respect to the Earth Mover's Distance and the Signature Quadratic Form Distance are shown in Figure 2 and Figure 3, respectively. As described in the previous Section, we utilized the Dynamic Time Warping Distance based on Euclidean Distance as trajectory distance for the Earth Mover's Distance and converted this trajectory distance by means of the power kernel [Schölkopf 2001] with parameter α=1 into a trajectory similarity function for the Signature Quadratic Form Distance. Since weighting of relevant trajectories by motion distance and motion variance, approximately shows a similar behavior, we include the results regarding motion variance gesture signatures only. We depict small and large distance values by bluish and reddish colors in order to visually indicate the performance of our proposal: gesture signatures from the same movement type should result in bluish colors while gesture signatures from different movement types should result in reddish colors.

As can be seen in Figure 2 and Figure 3, both Earth Mover's Distance and Signature Quadratic Form Distance show the same tendency in terms of gestural dissimilarity. Although distance values computed through the aforementioned distance functions have different orders of magnitude, both gesture signature distance functions are generally able to distinguish gesture signatures from different movement types. On average, gesture signatures belonging to the same movement type are less dissimilar to each other than gesture signatures from different movement types. We further observed that the distinction between gesture signatures from the movement types spiral and straight are most challenging. This is caused by a similar sequence of movement of these two gestural types. While gesture signatures belonging to the movement type straight follow a certain direction, e.g., movement on the horizontal axis, gesture signatures from the movement type spiral additionally oscillate with respect to a certain direction. Since this oscillation can be dominated by the movement direction, the underlying trajectory distance functions are often unable to distinguish oscillating from non-oscillating trajectories and thus gesture signature of movement type spiral from those of movement type straight.

Apart from the quality of accuracy, efficiency is another important aspect when evaluating the performance of gesture similarity models. For this purpose, we measured the computation times needed to perform single distance computations on a single-core 3.4 GHz machine. We implemented the proposed distance-based approaches in Java 1.7. The Earth Mover's Distance, which needs on average 148.6 milliseconds for a single distance computation, is approximately three times faster than the Signature Quadratic Form Distance, which needs on average 479.8 milliseconds for a single distance computation. In spite of the theoretically exponential and empirically super-cubic computation time complexity of the Earth Mover's Distance [Uysal 2014], this distance is able to outperform the Signature Quadratic Form Distance. The reason for this is the high number of computationally expensive trajectory distance computations. While the computation of the Earth Mover's Distance is carried out on the trajectory distances between the two gesture signatures, the computation of the Signature Quadratic Form Distance additionally takes into account the trajectory distances within both gesture signatures. Therefore the number of trajectory distance computations is significantly higher for the Signature Quadratic Form Distance than for the Earth Mover's Distance.

To sum up, the experimental evaluation reveals that the proposed distance-based approaches are able to model gesture similarity in a flexible and model-independent way. Without the need for a preceding training phase, the Earth Mover's Distance and the Signature Quadratic Form Distance are able to provide similarity models for searching similar gestures which are formalized through gesture signatures.

In this paper, we have investigated distance-based approaches to measure similarity between gestures arising in three-dimensional motion capture data streams. To this end, we have explicated gesture signatures as a way of aggregating the inherent characteristics of spontaneously produced co-speech gestures and signature-based distance functions such as the Earth Mover's Distance and the Signature Quadratic Form Distance in order to quantify dissimilarity between gesture signatures. The experiments conducted on real data are evidence of the appropriateness in terms of accuracy and efficiency of the proposal.

In future work, we intend to extend our research on gesture similarity towards indexing and efficient query processing. While the focus of the present paper lies on dissimilarity between pairs of gestures, we further plan to quantitatively analyze motion capture data streams in a query-driven way in order to support the domain experts' qualitative analyses of gestural patterns within multi-media contexts. The overall goal of this research is to contribute to the advancement of automated methods of pattern recognition in gesture research by enhancing qualitative analyses of complex multimodal data in the humanities and social sciences. While this paper focuses on formal features of the gestural movements, further steps will entail examining the semantic and pragmatic dimensions of these patterns in light of the cultural contexts and embodied semiotic practices they emerge from.

This work is partially funded by the Excellence Initiative of the German federal and state governments and DFG grant SE 1039/7-1. This work extends [Beecks 2015].