First, and most common, is the ubiquitous digital analogy for print or manuscript most obviously represented by documents prepared using word processing software. This approach imitates the visual appearance of print or manuscript: the default setting of a word-processor's window in the English-speaking world, for example, yields a two-dimensional image of black letters arranged sequentially in lines from left to right and top to bottom on a white rectangle; the relationship between words and background can be set to represent the relative proportions of various fonts and margins on various sizes of paper. Often the digital analogy for paper is enhanced with navigational tools (such as hyperlinks, search functions, and zoom features), and sometimes non-print media are embedded into paper-analogue frames. YouTube is an obvious example, in which windows containing time-based media appear in a two-dimensional, static white background populated with black text; Sophie, a software system for creating, “reading”, and annotating multimedia digital “books” demonstrates a more sophisticated version of the same principle (it similarly uses a static, two dimensional background with designated areas for other kinds of media, including hyperlinking and collaborative authoring tools); Wysiwyg Web Builder provides a 'what-you-see-is-what-you-get' system for creating web pages that can also accommodate forms, PayPal, and RSS functions. But these features are enhancements to print format rather than violations of it, operating like traditional tables of contents, indexes, annotations, or figures. Digital editions of theatrical works such as the Internet Shakespeare Editions' annotated As You Like It [Bevington, ed. 2007] and the Canadian Adaptations of Shakespeare Project's multi-media Interactive Folio edition of Romeo and Juliet [Fischlin 2007] take advantage of opportunities like these to illustrate textual variants, provide historical context, or even present performance records from a range of productions. The Reading View of Watching the Script also operates in this way. It divides a text into passages of manageable length on a readable scale, minimally interfering with the integrity of the text as a whole in order to make it more accessible to readers, and in doing so maintains print's typographical visualization of characters speaking in turns.

A second approach to digital text visualization deconstructs and re-assembles texts to reveal patterns not visible within the confines of a single page or window and then communicates the emergent patterns metaphorically in the form of graphs, images, or concordances. This is essentially an analytic approach to text visualization, described by Brent Nelson [Nelson 2008] as the digital evolution of the disassembly and reassembly of material artifacts that has been historically typical of humanities research. An example from a theatre-related discipline is ScriptGeist, a film pre-production software system that extracts mentions of props, costumes, locations, and other material and logistical details from scripts and re-assembles them into lists for distribution to various production departments [McKie 2007]. More innovative digital visualizations communicate textual patterns in intuitive, challenging, and aesthetically pleasing ways. Mandala, for example, a rich prospect browser[1] for exploring text collections more intuitively than traditional interfaces can, allows users to upload texts (including theatrical texts), weight criteria for word-searches, and view results in circular patterns [Sinclair, et al. 2009]. But the analytic approach nonetheless maintains the analogy to print and manuscript formats. Like those formats, its essential paradigm is the arrangement of information in two dimensional space. It deconstructs the arrangement of words on a two dimensional page and re-arranges the same words into new patterns in similar two-dimensional spaces, patterns familiar from print and manuscript such as tables, figures, graphs, two-dimensional images, and so on. Although the goal of re-arranging the original text is to offer an alternative representation of the same content, emphasizing patterns not readily discernible but nonetheless present in the original text, the result of the process is an entirely new text, often non-narrative or even non-syntactical in nature.

Watching the Script takes an analytic approach to visualization in its Overview panel, its features for associating character and colour, and its features for highlighting or excerpting certain kinds of text such as all text belonging to one character. While there are clear benefits to this approach, the casualties to the intended technological function of a theatrical text are especially apparent in this last case: if the actor playing Romeo has only his own lines for the Capulet ball scene to study, he doesn’t know, for example, that he and the actor playing Juliet are together speaking a sonnet, an elevated poetic form which signals that this new romantic connection extends beyond the whimsical romantic intrigues of Romeo’s past (1.4.206-223). Analytic visualization reduces rather than enhances the intended functionality of a theatrical text as a theatre technology.

A third common approach to digital visualization of both theatrical and non-theatrical texts participates in the long tradition of literalizing images described in or suggested by a text. Sometimes the elements illustrated are individual characters, objects, or events, as in Michael Lahanas' interpretive drawing of the shield of Achilles posted with a translation of Homer’s description of it from book 18 of the Iliad [Lahanas 2006]. Sometimes literal illustrations are symbolic or emblematic schemas synthesizing a variety of moments or ideas in a single two-dimensional frame, visual “confections”, to use Edward Tufte’s term [Tufte 1998], such as the promotional posters for Baz Luhrmann's Romeo + Juliet [Bazmark Films 1996]. A third strategy selects and extracts a sequence of events or movements of characters or objects as described or implied by a text, and plots them through two-dimensional space over time, providing a kind of kinetic summary of a textual work. Homer’s Trojan Theatre, for example, allows users to watch icons representing ships and armies move through the of battles of the Trojan war on a simplified map almost in ground-plan [Clay and Kenan 2007]. Since they re-organize their originals, we might even think of these latter two forms of literal illustration as literal, illustrative analytics. Multi-media illustrations -- movie trailers, for example -- often combine literal illustrative techniques to create analytic, kinetic confections. Most software designed to assist in motion picture pre-visualization or storyboarding operates as illustration: images representing shooting locations defined in the equivalent of stage directions, and sometimes also the arrangement of characters against these backgrounds, are organized in visual sequences analogous to the traditional storyboards used to plan the visual progression of a film. One example of such software is Celtx [Celtx 2010]; FrameForge Previz Studio [Schafer 2010] is a more sophisticated version, incorporating 3-D images embedded in two-dimensional frames; and the movie-making simulation game Moviestorm [Moviestorm Ltd. 2012] operates on similar principles.

The Stage View in Watching the Script attempts a limited stylized literalization of textual images in its association of character dots with characters’ speeches. In supplanting the typographical arrangement of the text it represents, it, like other literal visualizations, distorts its source text at the same time as enriching and augmenting it. Magnificent though a drawing of Achilles' shield can be, it cannot capture the lines and letters that indicate the rhythms and other sound-patterns of Homer's verse and through them the connotative visceral and intellectual meanings essential to the work as a whole. Nor, taken out of context, can a drawing of the shield effect the sudden shift in a reader's imaginative experience from syntagmatic progress through the epic to paradigmatic expansion of the lyric moment in which the symbolic object is encountered. On the other hand, the visualization adds specificity and detail not necessarily included in the original, and helps a novice reader understand the detail that is included in it. Similarly, animated maps and movie trailers do not necessarily aim to create rich epic or theatrical experiences of the inexorable accumulation of horrors – visitations from gods, violations of the bodies of heroes, illicit sexual encounters, murders of cousins, concealment in tombs, resurrections from the dead; instead, their concerns are traffic patterns and release dates.

Any literal, illustrative approach to text visualization can represent a part or an analysis or an interpretation of a printed textual work, demonstrably faithful to something in the original work, whether that be its material appearance (type, page, layout) or its contents (word, commentary, image, event, pattern), or both. No method of literal illustration of a theatrical text can faithfully represent a work of theatre, however, neither its material appearance nor its contents. Theatrical text is clearly intended to lead to a visualization of some kind, a material manifestation of something suggested by the text, but not to a literal visualization of the world of the play. Rather, a theatrical text is intended to lead to another material manifestation of the work, a theatrical production in which characters embodied by living actors move around on a three-dimensional stage in real time and space before an audience. That visualization is in turn intended to suggest the world and action of the play, making the work of theatre imaginatively present in the minds of an audience. Since theatrical texts rarely literally describe stages, but instead describe the fictions that staging is in turn intended to describe, a literal illustration of a theatrical texts skips over its performativity altogether; it treats text not as a performance technology, but as another, non-technological form of literature.[2] To the extent that theatrical texts are intended for performance, they, unlike other kinds of texts, require the addition of representational elements other than those that can be supplied imaginatively by readers. Even a theatrical work whose primary mode of expression is verbal must necessarily incorporate elements other than text. Literal illustration alters theatrical text at the ontological level.

In response to this challenge, modern print and digital editions take a fourth approach to visualizing theatrical text, incorporating images of theatrical production rather than literal illustrations of the text. Jill L. Levenson's introduction to the recent Oxford edition of Romeo and Juliet [Shakespeare 2000], for example, includes eight illustrations of stage productions and two of films [Zeffirelli 1968], [Lurhmann 1996]. Other print series such as Cambridge's Shakespeare in Production interpolate notes on and images of stage production into the text itself (see [Shakespeare 2002]). The online Internet Shakespeare Editions provides a broader collection of records in a searchable database embedded in a separate section of the site; it functions somewhat like an appendix to the text [Prescott 2009]. A number of Peter S. Donaldson's projects, including the Shakespeare Electronic Archive, Shakespeare in Asia, Hamlet on the Ramparts, and his Cross-Media Annotation System (XMAS) explore new ways of linking and integrating performance records and texts [Donaldson 2011a], [Donaldson 2011b], [MIT Shakespeare Project]. By representing a variety of productions in relation to an individual text, these approaches to visualization implicitly acknowledge that no individual illustration of an individual production should be privileged. Accepting the premise that different productions of Romeo and Juliet can represent the same work, they provide a variety of alternative visualizations for comparison.

Nonetheless, no group of contrasting production records can be said to visualize the work any more than any individual production record does. Production records participate in the same conventions as other material media, and perhaps more importantly, by providing a set of alternative illustrations for the same single text, groups of contrasting production records privilege the printed text as the locus of the work. Even editions that include extended representations of alternate texts (see Levenson's inclusion of the entire first Quarto of Romeo and Juliet [Shakespeare 2000], for example, or the Internet Shakespeare Editions' inclusion of early variants of the plays [Best 2005]) treat texts as coherent entities derived from print sources. Production records, on the other hand, are fragmented excerpts, arguing that coherence lies outside or between those records rather than in any individual production that may be represented by one or another of them. Visualized in this way, the theatrical work is still, ultimately, conceived as a complete and stable textual entity.

There are some precedents for a technological approach to visualizing theatrical text in digital media. Michael Best, Stewart Arneil, and Martin Holmes' Scenario [Best, Arneil and Holmes 2002] enables students to create blocking tableaux of scenes from Shakespeare's plays by placing characters in fixed positions on a two-dimensional perspective-drawing of a stage similar to the Globe theatre. Each frame can be annotated with explanatory notes. Larry Friedlander's TheatreGame [Friedlander 1991] allows students to make audio recordings of scenes, program synchronized stage action in a 2.5D graphical interface (a pseudo 3D or ¾ perspective such as that used commonly in first-person shooter computer games), and play back the virtual 'scene' created. Both systems acknowledge the spatial medium of theatrical performance by offering a visual analogue for three-dimensional space; Scenario invests more fully in the specificity of the performance space, whereas TheatreGame acknowledges time as a performance medium. Both also engage users in activities that imitate a theatre artist's evolving use of theatrical space through a rehearsal process, Scenario emphasizing the analytical and interpretive activities of a director in its annotation function, and TheatreGame emphasizing the actor's work by having students record audio performances of scenes. Credo Interactive's DanceForms system for choreographing dance implies a relationship between tools and technologies of performance by presenting a view of a choreographic record in the same interface as a view of avatars executing the choreography [Credo Interactive 2009]. However, none of these systems makes an explicit visual argument about the nature of the relationship between theatrical text and performance in its interface design.

Our new prototype adopts some of the same principles as these systems, but offers a different range of functions in response to our thinking about the nature of theatrical text. The Simulated Environment for Theatre (SET) is shown in Figure 2.

SET offers a centrally-placed visualization of a line of action in the form of a horizontal image analogous to a timeline. The Line of Action links a Reading View in the lower half of the screen to a Stage View consisting of a three-dimensional scale model of an actual venue and set populated by abstracted character avatars in the upper half of the screen. The user’s agency is emphasized in that users can upload, edit, and annotate all aspects of the Stage View, Line of Action, and Reading View, including the texts and stage and set models used, and the movements of “characters” on the stage. The Line of Action can be broken down by users into smaller units of action, ranging in size from acts to scenes to French scenes (delineated by the entrance or exit of a character) to speeches to individual “beats” of dramatic action. Users define divisions between units of action by associating them with words, stage directions, or intervals between words and stage directions in the Reading View, or with character movements in the Stage View. All three views can be edited in the main interface; edits made in one view affect the others; and all three views scroll simultaneously during playback. In other words, the system treats units of action, rather than individual characters' speeches, as the basic units of the works visualized.

In each individual view, additional features serve the new paradigm. The Reading View shifts primary ontological integrity to the line of action rather than the text. The text in the Reading View can be cut, augmented with stage business, expanded with textual additions, and annotated; but any and all of these alterations are linked to (and hence metaphorically justified by) actions in the continuous Line of Action. The Reading View also emphasizes the embodied nature of text by placing it next to a Character Panel, which visualizes the presence or absence of characters during the passage shown in the Text Panel, whether they are speaking or silent. Our revised Stage View attempts to visualize the materialization of a theatrical text in fuller and more concrete terms than our first model did. Instead of offering a map-like aerial view of hypothetical traffic-patterns, our new Stage View uses three-dimensional models of actual theatre spaces.[3] Models are populated by abstract, three-dimensional avatars representing five classes of beings: adults, children, babies, large animals, and small animals, as shown in Figure 3. Each has a “nose” to indicate the direction in which it is facing. The avatars are distinguished from one another by colours keyed to names in the Character Panel, and their user-defined movements are optionally visualized in colour-coded “paths” showing each character's progress and stopping-points [Ruecker, Grotkowski, et al. 2013]. Speeches also appear in the Stage View, in colour-coded bubbles, visually associating characters' bodies and voices. Alterations to the Stage View are, like alterations to the Reading View, linked to the Line of Action.

An additional advantage of the new model is that it permits users to move through three-dimensional space to view the stage or the audience from a variety of perspectives, including actors' perspectives, as demonstrated in Figures 4, 5 and 6. So the system can represent the multiple perspectives of audience members in different parts of the theatre, a phenomenon that must be accounted for both in planning and in analyzing theatrical production. The Stage Map in the upper right corner can be used to define predetermined viewpoints, for ease of navigation.

Finally, the visual emphasis on any individual element of the interface is determined by the user, by adjusting the sizes and positions of panes, as shown in Figure 7.

There are four main areas of focus to this new phase of development. For theatre researchers, we have added an annotation function that permits users to transparently acknowledge and document their research sources, as illustrated in Figure 8.

Because our research-oriented usage case makes non-textual performance records explicit, we have come to think of it as a historiographical iteration of SET [Roberts-Smith 2013]. For students of theatre, by contrast, we are conscious of the need to embed pedagogical guidance into the design of the system, so that students’ experience as users is a learning activity in itself. In our first tentative exploration of a pedagogical iteration of SET, we have analyzed the ways in which our current design implicitly guides users through the process of answering a theatre-historical research question [Roberts-Smith and Harvey 2013], by first presenting users with an empty environment to which theatre-historical records must be added before the records can be interpreted in a visualization like the one shown in Figure 8 above. This step-by-step process is illustrated in Figures 9, 10, and 11.

We hope to develop our pedagogical usage case into an iteration that can guide students through a process of theatrical text and performance analysis, as well as theatre-historical research. We are beginning to develop our third, creation-oriented usage case by re-imagining our avatars as sketching tools for theatre directors to use in preparation for rehearsal. This process is taking the form of an open-ended, iterative movement back and forth between an analysis of preparatory sketches in Richard Rose’s rehearsal scripts [Rose 2011] and the SET team’s collaborative generation and review of a range of potential approaches to visualizing the creative functions of Rose’s sketches. We have come to think of this design methodology as a “research discourse of alterity”  [Gabriele and Ruecker 2013]; it is typical of our development process to date. Lastly, the fourth strand of our current exploration of use cases and usability addresses the more intuitive interface that was indicated as a priority by participants in our pilot user study. To that end, we have begun to explore the potential applications of game interface design to our system [DeSouza-Coehlo and Roberts-Smith 2012], and we are seeking responses to our initial re-conceptions of the interface, along with our approaches to research annotation, pedagogical organization, and creation-oriented avatar design, in our user study.

In addition to these four main areas of focus, however, we are also exploring an interesting theoretical question that arose during the first stage of SET development, namely: can non-textual records of production be visualized as elements of a technological process, like theatrical texts, or are their modalities different enough to warrant a different approach to visualization? It is obvious, for example, that some records – such as a set design like William Chesney’s in Figure 8 — are intended to directly to lead to material manifestations that we might think of as literal illustrations of their contents. Since they are not necessarily used literally, however (for example, the set actually built for the University of Waterloo production of Julius Caesar made a significant change to the horizon line indicated in the set model), they may arguably become technological in the process of their application. Other production records – such as Richard Rose’s pre-rehearsal sketches – are designed to be technological, since they record the deployment of material production elements whose spatial and temporal semiotics can be analyzed for theatrical action [Rose 2011]. Still other records operate primarily as production illustrations, as do production photographs that record moments in a live performance event (though these need to be carefully distinguished from publicity photographs, which are often artificially set up during dedicated photo shoots). Lastly, a final category of production records seem to intend a hybrid of literal, technological, and illustrative/documentary functions. Our curiosity about this final category of record has led to some tentative experimentation inside the SET system with the well-known map appended to the only surviving manuscript copy of the medieval morality play, The Castle of Perseverance [Castle of Perseverance], shown in Figure 12. It records a set design, which apparently is not only intended for future literal materialization, but also describes possible alternative future materializations, as well as seeming to record how the set was built in a particular past performance. The map’s stage directions seem similarly intended for future technological application while simultaneously recording how the play has already been staged.

Figures 13 and 14 show our early experiments in visualizing the multiple modalities of this text in the SET system.

These initial, incomplete experiments have opened a new avenue of inquiry for us, namely: might it be possible to conceive of an entirely new approach to editing plays as digital performance editions? By performance edition, we mean a visualization that takes on the traditional responsibilities of communicating the transmission and interpretive histories of a text, as well as acknowledging its extant material witnesses, without sacrificing access to its technological functions. How might we acknowledge the multiple modalities of our research sources, their historical development, and their continuing present technological instability, all in the same virtual space?