It has long been a premise in the study of media that multiple senses are in play when we view movies, watch television, listen to live or recorded music, read or hear poems read aloud, and consume advertising, whether in print or on screens [Benjamin 1939] [McLuhan 1964] [Hansen 2004]. Since the late twentieth century, the interplay of seeing and hearing has yielded richly variegated writing and thinking about, on the one hand, vision and visuality and, on the other, the acoustic, the audible, and the aural. In this essay, we pursue this interplay into the digital humanities. More specifically, we advance the concept of audiovisualities in order to describe that interplay in the context of digital annotation of time-based media. In these media, seeing and hearing are inseparable and our goal is to understand how processes of digital annotation can help scholars and students investigate this entanglement. To do this, we describe a platform we have named Mediate: An Annotation Tool for Audiovisual Media, which was developed with the Digital Scholarship Lab at the University of Rochester and has been used in undergraduate courses across the humanities and social sciences there. In these settings, Mediate has enabled our students and ourselves to see and hear the data of our eyes and ears in reflexively collective and recursively interdisciplinary ways. This collective process in a variety of classrooms is what has brought us — three faculty members (two tenure track, one instructional track), two students (an undergraduate and a graduate student), and two staff members from the library (the programmer and the Director of the Digital Scholarship Lab) — to the concept of audiovisualities we explore in this essay.

That concept cannot be disaggregated from visual studies and sound studies, two fields that have developed alongside their more disciplinary counterparts in art history, film studies, musicology, and music theory. Broadly conceived, visual studies offers a means of understanding the expansive domain of the visual beyond what disciplines such as art history or film studies allow us to see. Wildly diverse in the directions it has taken since emerging in the late twentieth century, one of the central legacies of visual studies has been the concept of "visuality," or "sight as a social fact," which cannot be disaggregated from the act of "vision," or "sight as a physical operation" [Foster 1988]. Adapting W. J. T. Mitchell, we can think of visuality as a "dialectical concept" in which the study of "visual culture cannot rest content with a definition of its object as the social construction of the visual field, but must insist on exploring the chiastic reversal of this proposition, the visual construction of the social field. It is not just that we see the way we do because we are social animals, but also that our social arrangements take the forms they do because we are seeing animals" [Mitchell 2002].

As in visual studies, scholars working in sound studies, which took identifiable shape in the early twenty-first century, refuse to be content with a definition of its object as solely the social construction of the sonic field, but also account for the sonic construction of the social field [Bull 2013] [Novak and Sakakeeny 2015] [Pinch and Bijsterveld 2012] [Sterne 2012]. Pushing beyond logocentric and ocularcentric theoretical frameworks in various established disciplines, sound studies treats the audible and the aural, as Jonathan Sterne has put it, as "an artifact of the messy and political human sphere" [Sterne 2003]. Pondering that "artifact," three researchers exploring what they call "digital sound studies," including one of the authors of this article, have posed a question about the assumed modes in scholarship itself. "How," they ask, "can scholars write about sound in sound?" [Lingold et al. 2018].

A key effect that sound studies has had on visual studies has been to remind those working in the latter that the sights we see often go hand in hand with the sounds we hear. While Sterne's work in The Audible Past has systematized and historicized this sight-sound relation, film theorists such as Michel Chion [Chion 1994] and Kaja Silverman [Silverman 1988] and media archeologists such as Siegfried Zielinski [Zielinski 1999] [Zielinski 2006] and Bernard Stiegler [Stiegler 2014] have generatively tracked the dialectic of the visual and the audible in their work. We think of this dialectic as producing the audiovisualities — the physically and culturally interpenetrating modes of audiovisual experience and audiovisual inscription where hearing and seeing remediate one another for all of us as sensory and social subjects — that this essay aims to chart in relation to digital annotation in Mediate.

At the University of Rochester, students have used Mediate to annotate cinematic, televisual, musical, literary, and commercial media in courses housed in the Film and Media Studies Program, the Musicology Department, and the Department of Linguistics. In these courses, the audiovisual specificities of a given medium become radically legible to students in the data they yield by annotating in Mediate. Further, as we will show in this article, in exploring the medium-specific qualities of film or music or advertising in their unique material forms, cultural contexts, and social functions, we have unexpectedly ended up in a broader concept of audiovisuality that cuts across disciplinary differences. To riff on Mitchell one last time, Mediate allows us to examine the audiovisual construction of the social field as much as the social construction of the audiovisual field. Through the annotation it supports, Mediate provides a platform in which that field is no longer immediately intuited through our senses, but turned into an object of analysis — an audible and visible "exteriority" [Sterne 2003] that allows us to grasp the interplay of seeing and hearing beyond the often self-contained ways in which we process sensory data internally and individually.

Mediate arose out of Joel Burges's desire to have a digital tool that would enable the collection of large amounts of data about how time works on television. Originally working with Nora Dimmock, Jeff Suszczynski, and Joshua Romphf by experimenting with digital humanities projects in the classes "The Poetics of Television," "Film History, 1989-Present," and "Clocks and Computers: Visualizing Cultural Time" between 2012 and 2016, this desire gave way to the still ongoing project of developing a digital annotation tool for audiovisual media that would be of more general use. As it did, we moved from combining software such as Jubler, DaVinci Resolve, and Adobe Encore to building our own platform, primarily through the labor of Romphf [Burges et al. 2016]. Mediate is a web-based platform that allows users to upload audiovisual media; produce real-time notes; generate automated and manual annotations (which we also call markers) on the basis of customized schema; preserve the annotations as data that can be queried; and export the data in CSV and JSON formats for further exploration, interpretation, and visualization. The platform is built in Python and JavaScript, and it makes use of several open source libraries, including Django, OpenCV, FFMPEG, and React. Through websockets tied together by a REST API, Mediate supports concurrent updates of annotations added by multiple users. Mediate provides a real-time system for annotating and analyzing myriad genres that yield audiovisualities in which sight and sound come into chiastic interplay in medium-specific ways.

There are other tools like Mediate, including ELAN and NVivo, the subjects of a recent study of the audiovisual state of digital annotation [Melgar et al. 2017]. The build of ELAN and NVivo, however, make them technologically and methodologically distinct from Mediate. NVivo is not open source, while ELAN is — we hope Mediate will be open source and widely accessible in the future. Both ELAN and NVivo are desktop-based programs and have limited collaborative capabilities, whereas synchronous and asynchronous collaboration are foregrounded in Mediate. ELAN projects can only contain one media file whereas NVivo has a more multimodal approach and supports a variety of file formats, similar to Mediate. ELAN provides tiers (like the schema we use in Mediate, described below) with controlled vocabularies (akin to markers that make up schema in Mediate, again described below), whereas the categorization in NVivo is done after annotating through a code book approach common in the social sciences. Neither tool offers, as Mediate does, automated shot detection, and the ease of querying and exporting data across projects, media, and/or schema. Furthermore, they both feature high learning curves. Mediate adopts a more streamlined approach to consuming media in the design of its interface, which echoes familiar interfaces from our historical moment.

When a user logs into the Mediate website (username: mediate_guest password: mediate2019!), they encounter the User Interface that displays their research groups. Each group includes the initials of the members along with the media assigned to the group. The media thumbnails include a count in the upper right-hand corner for annotations already generated.

Selecting, for example, I Love Lucy, the user enters the Annotation Interface. Here they can watch the media in an interface akin to familiar streaming services, but with the addition of a vertical column of time coded annotations that scroll as the media plays.

Clicking on the blue marker button in the upper left-hand corner reveals the Schema Pane. Here, users select markers and set short-cuts, which enables the marking process once the user returns to the Annotation Interface. This process can take weeks, especially if a group is marking across multiple objects. The repetitive work of collaborative marking not only helps students comprehend the forms they are analyzing, but also reveals the necessary judgments involved in deciding how and when to mark. As a result, it emerges that each marker and the concepts it represents become legible as a formal construct instead of a natural given.

At a high level of frequency across seven years and twelve classes, Armoskaite, Burges, and Mueller have observed that Mediate encourages a process of learning that slows down the rapid-fire consumption of everyday media, upending the seemingly intuitive and immediate dimensions of the audiovisual field.[2] Recursive and reflexive, digital annotation in Mediate has made our students tune into the audiovisualities that construct them as seeing and hearing subjects through a range of material forms that operate in different cultural contexts and with differing social functions. The digital annotation in Mediate align with what Liliana Melgar Estrada et al. [Melgar et al. 2017] conclude is "the most significant methodological impact" of using ELAN and NVivo: "making the analytic procedures more explicit," can generate "more self-reflection about scholarly work." The authors suggest that digital annotation's greatest strength is its reflexive ability to draw users' attention to various units of analysis that they might not otherwise notice. Similarly, Mediate allows users to slowly comprehend what makes a film or poem or composition what it is as a mediated genre unto itself, but also to grasp how this medium-specificity turns on "units of analysis" [Melgar et al. 2017] that are subjectively chosen in the first place. Just as all data are capta [Drucker 2011], the unit of analysis by which a datum is captured when digitally annotating audiovisual media is itself invented. The invented dimension of these units are revealed whenever students discuss and debate marker definitions, as we have seen in all of our classes.

The collaborative process of digital annotation enabled by Mediate, starts to address one of the concerns raised by Melgar et al. in their analysis of the current state of digital annotation: that more "collaborative" and "systematic" efforts might allow scholars to transcend "small scale" analyses easily replicated on paper [Melgar et al. 2017]. In this, they acknowledge what has long been a both celebrated and critiqued feature of the digital humanities: its problematically neoliberal stress on teamwork that we hope might be rescued as a project of collective reading and collaborative curriculum [Burges et al. 2016] [Sebok 2014]. Significant as the debate over the political economy and research efficacy of the digital humanities is [Allington et al. 2016] [Da 2019a] [Da 2019b], we nonetheless want to stress that in our courses we have found that scaling up — collectivizing and collaborating on digital annotation through groups of students producing and sharing data with one another about audiovisual media — has allowed those students to arrive at analytic findings that go beyond what they are able or willing to do otherwise.

To arrive here, we must not assume that our students are "digital natives" who are naturally better at studying and thinking with computational devices than with pen and paper. None of us believes this lazy assumption; some of us actively work against it in our classes. As we will show, we have nonetheless seen outcomes that are educationally remarkable, especially due to the collaborative learning that Mediate enables, when it comes to how digital annotation in Mediate fuels our students' grasp of a range of audiovisual media in medium-specific ways. Often such medium-specificity is tethered to the discipline-specific approaches that we use in our courses, as we chart in the next section. But something interdisciplinary has arisen across our courses too: the cross-disciplinary concept of audiovisualities that this essay advances.

In our courses, we position individual mediums as possessing unique material forms that exist in cultural contexts and have some social function, as is reflected in our schemas. These schemas emerge from discipline-specific frameworks, with some of us stressing material form, cultural context, or social function more when teaching with Mediate. In this section we provide three case studies. The first draws on a number of film and media studies classes where Burges wanted students to understand how a range of audiovisual media — television, poetry, and pop songs in the two cases discussed here — work formally such that they can be materially differentiated from another medium, even if they share certain properties. While Mueller and Armoskaite share this concern to varying degrees, their expertise has driven them to underscore questions of cultural context and social function more prominently in the study of audiovisual media. In his history class for music majors, Mueller asks students to interpret how a range of historical contingencies influences the creation and performance of specific musical sounds. Here, Mediate is a prompt to redirect assumptions about music that students already think they "know." Bringing together questions of material form and cultural context, Armoskaite uses Mediate to spark students to delve into how language in advertising itself is audiovisual — or more precisely, how language has a social function in commercial media that turns on how it activates the interplay of hearing and seeing vis-à-vis linguistic and discursive content meant to induce an action in someone.

Across our individual classes, we have seen our students more fully enter the study of audiovisual media as they are defined by material form, cultural context, and social function within our respective disciplinary frameworks. In sharing these experiences with one another across our disciplines, we have been reminded that, when it comes to the audiovisual field, a film and media studies scholar should sometimes see and hear that field like a music historian who should sometimes see and hear it like a linguist. In noticing that we should see and hear like each other more, even as we explored medium-specific matters with our students, we have arrived at the cross-disciplinary concept of audiovisualities. Pedagogically and intellectually segregated from another due to the division of labor that organizes the modern research university, this concept allows us to think about the interplay of sight and sound more promiscuously and productively, overcoming the binaries that too often divide the audible and the visual and the divides that splinter disciplines from one another institutionally. In working together the last few years on digital annotation, we have learned to think more comprehensively across our respective fields about the (re)mediated sites where the physical and cultural operations of audiovisual experience converge. It is these locations of convergence that construct not only sensory and social subjectivities grounded in seeing and hearing, but also material forms and collective technics that set the conditions of possibility to see and hear to begin with — in short, that construct a manifold of audiovisualities. Our work on Mediate has helped us to estrange, even to alienate, the "natural order" that has been imposed on our experience of that manifold, or what Michel Chion describes as the "audiovisual contract" [Chion 1994, 9].

As this gesture to Chion telegraphs, we are not the first group of scholars to explore such conditions. The cross-disciplinary concept of audiovisualities on which we have landed already has a genealogy of thinkers — many of them cited at the outset of this essay — associated with visual studies and sound studies, not to mention film and media studies, behind it. Indebted to them, we nonetheless think the practice of digital annotation that Mediate provides contributes a collaborative model of learning through collective reading that allows our students to conceptualize audiovisualities beyond their individual selves (and our individual disciplines). It may do this, as well, for any scholars that take it up, especially, if not only in a collective and collaborative form. The collectivity of digital annotation can take that which feels intuitive and internal and remake it as unfamiliar and external; the collective act of exteriorizing that occurs in Mediate brings a new awareness to the qualities and characteristics of a given audiovisual medium.

"History is nothing but exteriorities," writes Jonathan Sterne in The Audible Past, by which he means that we can only know the "sonic world" of the past through its "efforts, expressions, and reactions" [Sterne 2003]. Mediate embraces this point of view. Digital annotation in Mediate asks students to exteriorize their reactions to audiovisual media not only by slowing down their consumption of them, but also by turning what feels subjectively intuitive, immediate, and internal (listening to and even playing music, taking in a poem, consuming an advertisement, watching a TV show) into a mediated object to be analyzed collaboratively and collectively beyond oneself. Vivid examples of this process of exteriorizing abound in our case studies. In "Experiments at the Edges of 20th Century Music," students produce digital "notations," so to speak, through their use of Mediate, thus resituating the ocularcentric primacy of musical notation through careful listening and historicizing. Similarly, in "Language and Advertising," the interface renders commercials a constellation of elements that act on us linguistically, visually, and musically in ways students can tangibly analyze. And the experimental visualizations of two pop songs for "Introduction to Media Studies" draw on data produced collectively through digital annotation about the aural experience of intensity to visually represent the material form of that intensity.

Mediate therefore enables a defamiliarized perception of audiovisialities, first and foremost, by challenging the consumption of media as an individual and discrete act atomized from others. The work of collaborative annotation, which sets Mediate apart from platforms such as ELAN and NVivo, reveals not only the potential for different experiences of the same mediums, but that the criteria through which we name and identify media — and indeed, our respective disciplines — can, and perhaps should, be subject to the scrutiny made possible by collective re-examination. Our respective fields are built upon often now unspoken agreements about what constitutes film or poetry or music or television or advertising or language. Mediate shows how "agreeing to disagree" on a given medium's properties remains a necessary move within and across disciplines, especially if we are to take into account critiques of both collaboration and computation leveled at the digital humanities. The reverse of the earlier claim, in other words, is also true. Our students often debate what a unit analysis means when marking, mobilizing the differences amongst themselves in collaborating on digital annotation. Similarly, when it comes to the cross-disciplinary concept of audiovisualities, a film and media studies scholar should see and hear the interplay of sight unlike a music historian who should sometimes see and hear it unlike a linguist as much as we should see and hear it like each other.

The case studies recounted in this article reflect this collaborative process of disagreement — the self-aware reflection upon "units of analysis" — as a pedagogically necessary exercise in understanding the audiovisual world we inhabit in the present. However, such a practice is not limited to the undergraduate classroom alone. The collaborative nature of digital annotation breaks down a process that scholars, at all levels, often take for granted: the terms and tools through which we analyze media, especially within our respective fields. By making us be both like and unlike each other, Mediate has allowed us to take hold of those terms and tools anew, discovering audiovisualities out of annotation as a concept that unsettles what we do with the interplay of sight and sound inscribed everywhere into experience at present.