The practice-led research collaboration Ancient Dance in Modern Dancers (ADMD) was set up in 2013 with the aim of translating the verbal and iconographic evidence for Graeco-Roman dance back into movement. Our initial hypothesis was that applying this evidence in the formulation of physical performances would provide an alternative mechanism for understanding dance from the performer’s standpoint, as a kinaesthetic process, rather than its usual treatment as spectacle or ocularcentric product. By drawing on the expertise of professionally-trained dancers, we have been able to produce a range of conjectures for the representation of selected scenarios, and at the same time to extrapolate a number of key principles for the art-form.

The dance genre we have concentrated on is tragoedia saltata, otherwise known as Graeco-Roman tragic pantomime, or simply orchēsis. This was a form of solo storytelling through dance and gesture popular between the first and fifth centuries CE across the Roman empire, especially in the Greek East. Much excellent scholarship over the last decade or so has established the attributes of orchēsis, which is now recognised as a sophisticated cultural discourse and an important vehicle for the late antique transmission of tragedy [Lada-Richards 2007] [Hall and Wyles 2008] [Webb 2008] [Macintosh 2010] [Garelli 2007] [Zanobi 2013]. Orchēsis scenarios could be comic or tragic, erotic or philosophical; a single masked dancer, the orchēstēs, played multiple roles, indicating character, setting and plot through a combination of hand gestures, choreograpic sequences, and iconic poses. He (usually he, although some female orchēstes are also known)[4] wore a flowing robe and cape, and was accompanied by musicians and a singer or singers who performed a libretto alongside.

No extant libretti have been definitively identified,[5] but one very probable Latin source is Ovid’s poetic compendium of myth, the Metamorphoses [Ingleheart 2008] [Lada-Richards 2013].[6] Transformation between characters, or from one state to another, was one of the skills for which orchēstes were celebrated;[7] moreover, the themes of orchēsis as summarised by the second-century satirist Lucian, “all of the mythical metamorphoses (muthikas metamorphōseis), those who have changed into trees and beasts and birds… and above all, the love-affairs of [the gods], even Zeus himself”, closely resemble the subject matter of Ovid’s poem.[8] For the current phase of ADMD, the two passages selected as our libretti were one episode in which sea-nymph Thetis attempts to evade a rapist by transforming herself into multiple creatures (Met. 11.229-65), and one in which the impious king Pentheus is torn apart by worshippers of the god Dionysus (Met. 3.699-729) (click here to listen; both texts may be found in the Appendix to this article).

Hitherto, ADMD has experimented only with live performance. Digitization has added new variables and raised a new set of questions regarding the relationship between the body of the dancer in the present, ever-mutable moment and the absent figure of the ancient orchēstēs, for whom the live dancer acts as a sēma, a marker. Live dance practice offered us insights into the vital symbiosis between the orchēstēs and the libretto, the orchēstēs and the musician, the orchēstēs and costume, character, emotions, mask/s and space; the additional layer of digitization placed more acute emphasis on the fundamental process of embodiment. Rather than the proprioceptive (internal sensorimotor) feedback which dominated the dancers’ choices in the development of their live performance pieces, the activation of an external avatar required ongoing interaction with this distorted mirror-image, this not-quite-self which both resisted and responded to the dancers’ manipulation. In setting up this experiment in motion capture, we shifted the point of reception from the human body to its virtual analogue.

The objective of the pilot workshop run with ADMD was to examine how the dancers’ kinaesthetic translation of ancient data pertaining to orchēsis would be affected by the additional factor of digital interaction. For this purpose we used Motion Capture in order to create 3D avatars of the dancers. Once animated, the resulting videos will also enable each dance piece to be analysed from any angle in the context of a virtual ancient theatre environment.

Over the last few decades motion capture engines have been mostly enhancing our understanding of normal and pathological human movement and have been used in the context of medical practice and education, gaming and cinema.[9] Motion capture (or motion tracking) generally refers to the process of recording human actor or object movement and this information is used to animate digital character models in 2D or 3D user animation. Early techniques used images from multiple cameras to calculate 3D positions and assemble them to reconstruct the movement but not the actor’s visual appearance. Improved knowledge of locomotion drove the invention of new methods of observation. Currently, the most common method for the capture of three-dimensional human movement requires a laboratory environment inclusive of markers, fixtures or sensors. The movement of the markers is typically used to capture movement between two adjacent segments (e.g. knee joint, ankles, elbows etc) with the goal of near-precisely recording the movement of the joint (figures 1 & 2). For ADMD’s purposes, the crucial difference between 3D tracking and video recording is that video fixes a static viewpoint and coerces prospective users into a spectatorial role, with all the binary dynamics implied by this relationship.[10] In contrast, the interactive user plays an active role in determining the order in which already-generated elements are accessed. This is the simplest kind of interactivity, and it allows for variability of interpretations, as opposed to mere spectatorship.[11] In order to bring the user into the performance space, as it were, alongside the dancer, we found the polyfocal coverage of 3D capture to afford a less restricted medium.

For the purpose of recording the dancers we used the “Black Box”, an isolated motion capture laboratory space in the ground floor of the Sliperiet facility at Umeå University. In a space of approximately 20m2 (4x5m) we positioned a wooden flooring (3x3m), and initially twelve cameras on different angles to record the dancers. The hardware we used for the motion capture was an OptiTrack 13W system and the interface software was Motive. On the first day, we began warming up one by one the twelve motion capture cameras in by using the “wanding option” OptiTrack CW-500 (B= 250mm). We then tested the sensors with an OptiTrack three point sensor triangle (CS 200). The 90 degree triangle was rigid but served as a testing point to observe and to capture the two axes for the 3D capture. The software we used (Motive) offered several different interfaces including:

1. Avatar, skeleton (no gender) that could be modified with default models that are male or female (figures 3–4).
2. Motion tracking avatars with strings that display the direction and the velocity of the motion capture (figures 5–7).

Numerous problems were encountered while trying to set up and use the tracking system. The dancers’ feet could not be “captured” at first due to reflection from the concrete floor. The system would not pick up anything lower than 15cm, so the dancers’ feet remained undetectable. While the system was ready, the software was still not reading all the markers properly; it was picking up all 37 out of 37 sensor points, but informing us that it required more markers. This was solved by a combination of adjustments. We raised the floor surface to a height of about 3cm, and used non-reflective wooden boards. We also altered the angle of the cameras and moved the capture area so cameras could be attached to fixed points as opposed to being placed on moveable stands.

All this resulted in wasting some valuable time. By midday on the first day of laboratory work we still could not capture the whole skeleton as one, and we had to move on to test it with the dancers. While we got the 37 sensor points to show, there were further glitches in the system. For example, we would temporarily lose all movements of the arms and legs (because they had frozen or disappeared), but then manage to detect all the points again after a few moves. We were informed by Humlab and Sliperiet senior technician Jim Robertsson that glitches of that sort may be rectified with post-production animation. Another issue was the fact that we could not capture movement in detail – for example hands and feet. While there was no need to capture facial expressions given that historically pantomime dancers normally wore masks, there was absolutely no possibility to capture the movement of fingers or hands, or small, subtle movement. That put some constraints on how the dancers would perform.

The first question asked by the participating dancers was whether they should modify their performances in any way to compensate for these technological constraints. Many movements in the “Thetis” piece, for example, took place on the ground; but crouching, kneeling, rolling, or reclining brought the dancer perilously close to the plane below which the cameras could not reliably track. We instructed them not to make any initial modifications, but to wait until after reviewing the footage in order to identify problem areas. Feet proved troublesome, placing much of the footwork, the essential pedēmata of orchēsis, out of range, creating glitches in which they popped up reversed at the ankles or vanished altogether from the figure on the interface. Although our dancers typically performed their pieces barefoot, we found that wearing shoes under the suit’s felt slippers tightened the fabric and hence improved the consistency of capture. Another common issue was the concealment of bent limbs, particularly in some of these floor-level positions. Rapid or complex sequences were easier to track if executed in the centre of the performance area, where more of the cameras could track each marker. We added more markers, taking the total suit count up to the maximum of 50. It seems that for dance, and for this genre of dance in particular, more flexibility is needed in the options for positioning the markers on the body. The mask eliminates any need for capturing facial expressions, but the corollary is that a need for greater accuracy accrues to the hands, feet, and torso. (Click here to see an extract from one of the dances and the resulting avatar pre-animation).

Another of the issues we encountered had to do with veils, shawls and loose clothing. These costume items are integral to orchēsis [Wyles 2008] (figures 8–10) but could not be recorded in any possible way. We were advised to use small triangular “rigid body” markers attached to the edges of the cloth, and attached these rigid bodies onto a veil made of semi-transparent gauzy fabric. The rigid bodies we used had sensory properties but also made the thin veil the dancers wore significantly heavier. Moreover, they would create further problems for animators in post-production as they only marked the extremity of the cloth. The inability of the motion capture system to track the movement of the veil, however, presented a creative opportunity as well as a problem. Even if the material object could not be tracked by the sensors, its presence could still be perceived in the quality of the dancers’ movements, as they had to adapt to the constraints the prop imposed on them.

The use of masks in itself has a profound effect on the way the dancers move. When facial expressions are hidden behind a mask, the dancers can only rely on body language to create meaning through their movements. This also allows them to embody with more ease a wider range of characters or other entities such as landscapes or elemental substances, as their individual self is made less visible. The dancers are aware that the focus of the audience shifts from the face to the whole body, and that they have to adapt their choreography accordingly. In the absence of any expression on the face to read, the effect of very subtle movements, such as those of a shoulder, a finger, or an inclination of the head, is magnified, thereby acquiring the potential to become very powerful and highly significant. Consequently, each movement has to be very precise and neat, as the smallest imprecisions are more likely to catch the audience’s attention and thus to blur the dancer’s intention. Working with a veil (a pallium) equally pushes the dancers to make clearer and more precise propositions. It also forces them to make more ample movements when dancing with it.

Working with the avatar turned out to have similar effects on the dancers’ choreography to those triggered by the use of the mask and of the pallium, and indeed actually amplified them. On the one hand, the avatar appears to become an extension of the mask, as it conceals the dancers’ own identity one step further. It promotes dissociation between the dancer and his performance. The body is perceived to be a kind of raw material that can be shaped at will to produce the desired effects, and is thus less emotionally invested. On the other hand, working with an avatar also imposes constraints which resemble those created by the use of the pallium. It requires that the dancers focus more on the smoothness and details of their movement, and less on their levels of energy and emotions. The movements have to be bigger in order to translate on the screen. Consequently, the dancers have to rely more on their technical skills as well as on the visual cues provided by the avatar, and less on their inner feelings. This also applies to the dancers’ energy levels. How much energy they put into each movement no longer reflects their emotional state, but instead is largely dependent on how much of it is needed to convey the desired effect through the avatar.

The performer of orchēsis is already in possession or occupation of dual, even multiple body-images, and the introduction of the on-screen avatar further complicates this perceptual relationship. Firstly, the dancer is both subject and object, skilled manipulator of the instrument that is her body. She is both spectacle and impresario. Second, the notorious plasticity of the orchēstēs means that by altering factors such as bearing, rhythm, tension, and speed, this instrumental body becomes the vehicle for a multitude of characters.[12] The articulate body which consists of a visible, exterior manifestation of motion – the shared sign-system of pantomimic gesture – coexists with the subjective, kinaesthetic evolution of the moving body-schema as experienced from within (on which distinction in dance more generally, see [Cohen Bull 1997]).

In antiquity, orchēstes had no means of seeing the whole of this outer body except as reflected in the movement of a trainer or rival performer. Their sensory feedback was purely proprioceptive. In the studio, our collaborating dancers make use of the mirror; previous ADMD research has embedded the use of video footage in the development process of pieces for performance. The 3D feedback of the screen extends this strategy of separating the visible (exterior) from the kinaesthetic (interior) body. The screen interface was available to our participants throughout, and although they did not typically refer to it during the execution of their performances, it was used in much the same way as a mirror when preparing. This mirror-image, however, was distorted, producing an appreciable disconnect between the motion of the dancer and its on-screen realisation. (A very basic example: because the markers were on the backs of the hands, they remained some 5cm apart on the avatar when the dancer clapped; in order for the avatar to clap, the dancer’s hands must cross.)

This raises the matter of translation. This has been central to ADMD from the beginning, as we define the activity of participating dancers as a form of intermedial translation, or translation across media:[13] performances witnessed in antiquity have been translated into the written texts and static images which are our source-material, and these we then translate back into movement using the kinetic discourses of twenty-first-century dance training and practice. One primary challenge addressed by ADMD has been that of making comprehensible a language of gesture that both references the extant data on orchēsis and resonates for present-day spectators. In the project’s initial phase, it was sufficient that participants invented an idiolect, a private language accessible to other members of the group. Subsequent phases involved public performance and consequently demanded expansion of the gestural vocabulary as well as refinement of its utterance: elements had to be at once precise, repeatable, and recognisable. Instead of addressing human spectators, on the other hand, the Umeå participants were addressing their performances towards the impersonal and inorganic but no less partial and interpretive gaze of the motion-capture system. As when a performance is transferred from an intimate venue to a large stage – which may certainly have been the case for Roman orchēstes –[14] gestures suited to the studio had to be adapted in scale and orientation if their effect was to be comparable.

Ironically, the space in which the dancers were working (3m x 3m, as mentioned above) had shrunk to a fraction of their rehearsal room. In a similar fashion to the multiplication of bodies, the introduction of a screen environment added another dimension to the already-complex spatial dynamics operational in orchēsis. As well as representing character, the orchēstēs is responsible for establishing his setting: the sea-cave where Thetis is assaulted, the mountainside where Pentheus is caught spying. Without scenery or backdrop, space is transformed imaginatively through the interaction of movement and libretto. The musical soundscape is vital, as it fills up the space like liquid with the mood which the orchēstēs crystallises and channels. The other fictional setting, however, simultaneously being defined by the movements of our participants was the on-screen grid. In one sense featureless, in that it consists of nothing but the geometric axes that contain the stick-figure or “skeleton”, this anonymous screen-space in fact makes a powerful intervention into the conceptual location of the dance. No longer constitutively identified with the live dancing body and its material context, the dance is transposed into an intermediary, purgatorial zone where it is perceptibly neither the actions performed by the present-day dancer opposite, nor those of an ancient orchēstēs. A participant may be animating (dancing the role of) Pentheus on Mt Cithaeron, but he is at the same time animating (dancing the role of) the skeleton in the grid.

From Humlab’s perspective and that of Digital Humanities in general, the collaboration with ADMD offered new insights. There is no comprehensive research about how the collaboration between artists, technicians, and academics may approach and appropriate technology as a critical tool, how technology changes conditions for participation in artistic (re)creation, and what is the value of technology as a medium and a reflective tool for knowledge processes within academic settings. Through the process of capturing the movements of Roman pantomime, we attempted to give at least partial answer to these questions. We examined first and foremost how the entanglements between technology, scholarship and performance art enable new knowledge production in multiple dimensions. Technology acts as a lens that affects the way we conduct empirical research. It brings new conditions and demands for research.

While the team of researchers, technicians and dancers initially saw this as a non-teleological experiment, the final implementation deliverable was to create a Virtual Reality prototype of a pantomime dancing avatar within a Roman theatre setting. With the aid of Humlab-based 3D artist Mattis Lindmark a simple, neutral avatar was first placed within a test space without a specific theatre background: (http://cultumea.com/testzone/MocapHistoryWeb/index.html last accessed 27.09.2017). At a later stage, the team borrowed a Virtual Reality environment of the theatre at Pompeii.[15] (https://www.youtube.com/watch?v=gApcaxSGqh0&feature=youtu.be last accessed 27.09.2017 see also figure 7). The avatar used was a synthesis of disparate gender and age features: a conventional female face on a young/early teenager male body with an olive skin complexion and African hair. The reason for the combination of these aesthetic elements was to reflect that professional dance training started from a young age in antiquity (Libanius, Oration 64). The final virtual reality prototype is intended primarily for HTC VIVE headmounts, while there are currently discussions for its incorporation within higher education as a study tool for the intangible cultural heritage of the Roman empire.

There is an advocacy for a gradually stronger integration of technology within arts and humanities research, and that is manifested in academic infrastructure, recognized by scholars as “inevitable change”  [Burdick et al. 2012]. In spite of being a relatively young direction, still on a stage of self-determination and struggling to situate itself in a bigger context [Crane, Seales, and Terras 2009], the digitalization of arts and humanities infrastructures has become an increasingly popular topic in academic spheres. Digital and computational tools promise new relationships between individuals and the knowledge production processes, the collapse of the long-standing opposition between the natural sciences and the humanities as well as freeing the humanities from anxious attempts to advocate for their worth and relevance in contemporary society [Nygren, Foka, and Buckland 2014] [Foka 2016]. We further argue in this article that immersion dictates that movement, auditory components and other forms of expression enable an interactive appreciation of materiality beyond the visual, thus enabling a simulation of what UNESCO defines as “Intangible Cultural Heritage”. Typically intended for education or for cultural heritage purposes, virtual worlds are gradually becoming popularized beyond the cultural or the creative heritage industry and have become tools for research, education and dissemination of the past (on mixed reality and CH see [Kolsouzoglou and Veneris 2006]). The complexity of building digital immersive visualizations, as well as the power of experiencing the past emotionally and physically ought to be addressed.

This discourse, however, is not unproblematic. First, although recent scholarship reflects the notion that humanistic study should correspond to the contemporary reality of the human experience, there are no relevant discussions for performance arts in relationship to its gradual reorientation toward technology. Second, there are no comprehensive studies about how new technological competences may critically relate historical performances to our current post-human, technologically intertwined condition. As a consequence of this digitization of disciplines that are (incorrectly) considered impractical and valueless, new directions have arisen: the importance of practical engagement, hands-on work, thinking-through-making alongside developments in digital software and hardware [Burdick et al. 2012] [Gold 2012]. A third issue that arises is that these directions remain equally under-studied and elusive. Against this backdrop, digital infrastructure for academia, recently reiterated as the material presence of technology that may bring praxis and theory together, is only discussed at the level of potentials of settings [Smithies 2014] as a core component for new, digital forms of knowledge production. The intersections of art with traditional humanistic research and the role of technology as infrastructure have only been studied in their theoretical potential, and not necessarily addressing organizational tensions related to the installation and use of digital research infrastructures [Foka et al. 2017].

While for ADMD technology as a medium facilitated interdisciplinary research and brokered collaboration between the artists, academics and technicians, there are some further, practical observations that need to be made. One issue was that the system required a lot of extra calibration and time to test cameras, sensor points and interface response to begin with. In the reality of academic experiments as such that require specific time and resources, it is good, in the future, to allow extra time so as to optimize the empirical part of the process. One striking example was the recalibration early in the workshop that lasted for a total of four hours. The further, practical engagement of scholars and practitioners benefitted greatly from keeping a diary of events, achievements, and problems.

In order to optimize the process of motion capture to record movement more accurately, there is not much to be done at the moment beyond post-production development of the captured avatars. We suggest that developers of motion capture may in the future consider how to ameliorate the issues identified above: glitches between sensor points and interface response, and the lack of ability to record items of clothing or motion intensity. Although early and preliminary, our very engagement with artists, text, motion, and captured image gave us new insights into both technological and humanistic inquiry. Humanities laboratories should ideally facilitate empirical engagements with text, accessed through the full range of sensory modalities, and slowly open up to recreating immersive portals to conduct humanistic research through crafting, experiencing and reflecting. Digitalized infrastructures for Arts and Humanities gain meaning and become innovative through equality of the socio-technical components and strong collaborative efforts between technology, art, and the humanities.

This collaboration between practitioners, scholars, technicians, and indeed with the technology itself can be defined as an instance of what we term here distributed reception. According to theories of distributed cognition more generally, cognitive (intellectual) activity takes place not exclusively within the human brain but rather within a more extensive, networked field of interactions incorporating both biological matter and inanimate objects (e.g. [Clark 2008 (1998)], [Damasio 1999], [Rowlands 2010], [Shapiro 2004], [Noë 2004], [Bennett 2010], [Sheets-Johnstone 2011], [Gallagher 2005]). In this instance, we relocated the cognitive act of processing movement from the locus of a human being to a (distributed) technological recipient. In translating ancient dance for a post-human world, for a cultural context in which digital media have become integral to everyday activities, the human actor-interpreter may not necessarily remain the sole target audience of ancient texts. Dance, it could be argued, is so intrinsically embodied an activity that it occurs only within the closed circuit of the human body; but we feel on the contrary that the technological can be incorporated productively into the terpsichoreal.

Performances, we found, had to be modified in order to convert the electrical firing of muscular innervation into the electrical signals of the motion-capture system and transmit a version of orchēsis meaningful to our many-eyed nonhuman spectator. A feedback loop was created as the dancers’ movements were informed in turn by the reactions of the figure on the screen. Subtle movements, for example, had to be rendered in a more explicit fashion if the system was to register them; hands must overlap to clap; a hidden foot glitches. The dancer dances their avatar like a marionette, but the marionette’s responses also affect the dancer. Agency was thus likewise distributed (as in [Bennett 2010]). Further modification and refinement of the performance could of course be deferred to post-production, but another option is to develop effective translation techniques for meeting the medium halfway and adapting this dance form to make it comprehensible not just to human viewers but also to the alternative sensory faculties of a machine. Dispensing with the unattainable goal of ever more isomorphic capture, we might instead exploit the given conditions of this unique partnership, creatively accommodating the demands of this obedient yet exacting collaborator.

In doing so, the dancer’s relationship to the antiquity s/he represents becomes manifest. When the dance is live, it is possible to elide the present-day performer with the absent ancient dancer, to mistake the translation for the source-text and to forget its duplicity, or (less pejoratively) its doubleness; but when, on screen, the skeleton simultaneously appears in all its inhuman smoothness and misapprehended disjointedness, we are reminded of how these performances are inescapably haunted by their predecessors. If and when the avatar appears in its fully reanimated Graeco-Roman guise, the effect will be even more pronounced. The figure in the grid, the imagined orchēstēs, comes into being only as the ghostly body-double of the dancer on the floor: visible as an external analogue of the kinaesthetic metamorphosis experienced within, its separateness supplies an imagined point of origin, a parallel text created in the act of reception.