Taylor Arnold is Associate Professor at the University of Richmond.
Stefania is a Research Associate at the Luxembourg Centre for Contemporary and Digital History (C2DH).
Lauren Tilton is Assistant Professor of Digital Humanities at the University of Richmond. She directs the Distant Viewing Lab.
Jasmijn Van Gorp is Assistant Professor of Audiovisual Data and Digital Culture at Utrecht University.
This is the source
Our special issue explores audio and visual (AV) data as form, method, and practice in the digital humanities. Spurred by recent advances in computing alongside disciplinary expansions of what counts as evidence, audio and visual ways of knowing are enjoying a more prominent place in the field. Whether the creation, analysis, and sharing of audiovisual data or audiovisual ways of communicating scholarly knowledge, scholars are building compelling avenues of inquiry that are changing how we know, what we know, and why we know in the digital humanities (DH). These epistemological shifts not only challenge existing methodological and theoretical pathways within the field of audiovisual studies, but most importantly defy existing knowledge hierarchies within the entire field of DH.
This is the introduction to the special issue on AudioVisual data in DH.
Many scholars have repeatedly demonstrated how expanding our areas of inquiry builds new
routes for the field
One of those developments is shifts in technology. The ability for computers to create,
"read", and store AV data followed by advances in areas such as machine learning have
augmented computational image and sound analysis. Pioneering approaches such as
cinemetrics that once relied on hand coding and text-based annotations can now be
automated. Within DH, this has led to new theories and methods such as cultural analytics,
distant listening, and distant viewing
Another critical development is digital access to AV materials. As DH scholars
increasingly think of their sources as data, they have benefited from large-scale
digitization of audiovisual collections.
Finally, we turn to institutional developments.
New pathways are bringing about exciting opportunities in DH as exemplified by the articles in this issue. They model how AV research can be the subject of analysis (e.g. film) or result of analysis (e.g. podcast). They highlight less visible humanities disciplines in the DH, model the importance of interdisciplinary collaboration across institutional structures, demonstrate how cutting edge scholarship comes from a plethora of positions, and offer new questions that the field is only beginning to grapple with. The contributors' model paths for constructing entry points, building bridges, or adding intersections for engaging with audiovisual in the digital humanities. Amplifying the work of scholars with a range of disciplinary, institutional, and political commitments, the special issue constructs a more capacious configuration of DH.
The special issue is organized into five sections. The first focuses on annotation of AV material as method and theory. The second explores analyzing (meta)data, which often includes annotation, to build and analyze AV corpora. The third focuses on creative and liberatory ways to remix AV (meta)data as a way to innovate pedagogically and methodologically while furthering discipline specific interventions. The fourth dives further into computational methods, particularly machine learning, in turn demonstrating how DH is reconfiguring these methods and ways of knowing to answer humanities questions. The special issue ends with a focus on how AV forms such as podcasts and film can also be the form of scholarly knowledge in the field, highlighting how form and argument can be mutually constituted.
Next, we turn to the contributions that comprise each section. The first explores
annotation as a powerful way to add context and analyze AV data. A particularly prominent
area of such work has been in film studies. Therefore, the first three articles offer a
snapshot of different approaches and tools for film annotation.
Cooper, Nascimento and Francis
present their KinoLab and discuss the opportunities and challenges of Omeka for
narrative film language analysis, including the challenges related to copyrights. They
argue for a universally accepted data model for film analysis. We then turn to a new
annotation platform called
The next section focuses on how (meta)data can open up analytical possibilities. Using metadata to reunite AV materials, Sapienza et al. describe the process of reuniting radio and text files virtually that belong together but ended up at different institutions. After discussing why audiovisual collections in general are heavily under described, they describe how virtual reunification and integrated access was realized through the use of linked data, minimal computing, and synced transcripts. Next, Hoyt et al. discuss the analytical possibilities afforded by metadata. Focused on podcasting, they discuss three different methods for studying RSS feeds and podcast metadata, and point at the specificities of methods for born-digital media vis-à-vis digitized media. Carrivé et al. then focus on the first development phase of their ANTRACT project for the transdisciplinary content analysis of 1262 newsreels containing more than 20,000 French news reports. They discuss how they dealt with the project's main technological challenge to process data and build tools to familiarize historians with the automated research of large audiovisual corpora in order to then use the data to pursue inquiry about Les Actualités Françaises news reports. Finally, Gienapp et al. show what can be done when data is brought together from different sources to analyze music collaboration. They demonstrate how network analysis can reveal the contours of collaboration among musicians.
In the third section, the authors creatively (re)mix AV data with a focus on audio data. Using audio data, Tyechia and Carrera demonstrate how centering Afrofuturism in DH pedagogy through mixtapes can not only realize the goals of an undergraduate composition course but realize a liberatory DH praxis. Bonnett et al. combine data art, landscape art and augmented reality in their DataScapes Project. Departing from the premise that data can be translated into visual and sonic forms, they use protein data and texts from the bible, turn them into sequences, and translate these into visualisations and compositions. Kramer then asks what if we listened to images. Building off of previous work on "image sonification", he argues that transforming the visual into audio opens up new ways of seeing and hearing the past. Next, Have and Enevoldsen demonstrate how toggling from close to distant listening offers insights about the longue durée of Danish radio content by scrutinizing what is audible with the human ear and searching for patterns using AI. Next, we turn to work that makes field specific interventions. Martin constructs a new path for listening to gentrification in Washington DC by combining ethnography, passive acoustic recording, and computational sound studies. The work also demonstrates how centering Black DH offers new ways to understand the relationship between embodied and computational audio analysis in DH, in turn forging new liberatory possibilities for the field.
The next section continues with the application and reconfiguration of computational techniques, particularly machine learning, to conduct data analysis on large collections of AV data. Looking at a large collection of artwork showcasing musical instruments, Sabatelli et al. introduce the usage of computer vision techniques to automatically locate musical instruments in images. They investigate the algorithmic properties of their analysis and show how it leads to innovative scholarship in music iconography. A paper by Lupker and Turkel illustrates the potential of investigating novel intersections between research in the humanities by using musical theory to guide the training and usage of machine-learning algorithms applied to a large corpus of digitized music. A born-digital collection of K-pop dance videos hosted on YouTube is analyzed using state-of-the-art computer vision techniques in a paper by Broadwell and Tangherlini. Their article develops a typography for describing and analyzing poses and choreography to facilitate the data-driven analysis of time-based media. Oyallon-Koloski et al. present a different approach to the study of movement in space by showing how motion capture technology can be used within film, dance, and movement studies. As with the Lupker and Turkel article, Oyallon-Koloski et al. illustrate the novel integration of theoretical frames from the humanities – in their case, Laban Movement Analysis and Bartenieff Fundamentals and Movement Studies – and the usage of computational techniques. In another take on the detection of bodies moving in space, Fragkiadakis et al. introduce an automatic system and taxonomy for tagging and describing digital videos of sign-language usage. Together, these articles illustrate the potential for work in AV DH to infuse machine learning with analytical commitments from the humanities.
Finally, we turn to articles that use DH to push the boundaries of form for scholarly knowledge. In order to demonstrate how the podcast format expands our definitions of text, Edwards and Hershkowitz reveal how podcasts can realize intersectional feminist approaches to DH. Along with demonstrating and discussing the creation of the Books Aren't Dead (BAD) podcast in the article, they discuss the process in a podcast for this special issue. Kim then explores how motion caption and virtual reality can be used to record and visualize movement histories as a form of cultural heritage preservation. Through these forms, one can then use visual storytelling, she argues, to demonstrate how movement, dance, and ritual cannot be separated from a person's personal narration of the experience. Finally, Mittel's contribution showcases twenty audiovisual deformations of the classic musical "Singin in the Rain" in still image, GIF, and video formats. The essay considers both what each new deformation reveals about the film and the way we engage with the by algorithmic practices derived object as a product of the "deformed humanities."
The invitation of the authors in this special issue to think about the relationship between form and argument is one we also embraced. The publication needs, some of which weren't possible, of our special issue pushed the boundaries of the form and format of DHQ as a journal that is catered for linear reading of articles as written texts in XML. As a result, the articles in this special issue include 5 sound files, 52 embedded videos, and 176 gifs and images. These AV components are key elements of the authors' argumentation. The special issue attempts to more closely mirror how scholars of AV materials in DH actually produce and create scholarship. In this context we take as a guide other pioneering initiatives in this field such as Scalar, VIEW journal, Audiovisualcy, and [in]transition that question the relation between the affordances of a publication platform and the interactivity and multimodality of scholarship that increasingly embraces creative modes of production. We hope that these incremental steps within DHQ can forge exciting new possibilities for the field.
Finally, our decision to partner together to co-edit was driven by features of AV work in DH. First, our own areas of expertise – statistics and digital images, history and audio, american studies and photography, and media studies, television & film – reflect a range of audio and visual scholarship that animates DH. Second, the inclusion of a colleague housed in a Math & Computer Science department, Tayor Arnold, demonstrates how digital humanities scholarship often requires working with and giving proper credit to experts trained in computational fields. Third, we wanted to build collaborations across geopolitical boundaries and languages that might help us think critically and beyond the particular configurations of DH that shape our local, regional, or national context. We recognize that our positionalities as White able-bodied scholars living in the global west and north also brings limits. As a part of our efforts, we paid special attention to circulating the CFP beyond our immediate DH circles with particular attention to reaching beyond the US and Western Europe. However, there is still more work to do. Yet, we do hope that the issue in aggregate reveals how thinking across disciplinary, cultural, and spatial boundaries enables a more capacious configuration of the field than currently articulated.
As we look toward the future, we are enthused about the possibilities and realistic about
the challenges. Along with the work featured in this special issue, areas such as 3D,
AR/VR, and game studies are forging exciting paths. As disciplines (albeit slowly) adopt
more capacious guidelines for what counts, forms of scholarship such as films, multimodal
digital projects, podcasts, and software are receiving well overdue credit. Because of the
teamwork and expertise often required to access and work with AV data, this area of DH
also pushes us to work across ossified divisions such as the "Humanities" and "Sciences",
"faculty" and "staff", and "university" and "cultural institution" in ways that can help
us realize a more collaborative, equitable, and generous configuration of the
field.
At the same time, challenges remain. There are major obstacles to working with AV. For
example, digitized images have significantly larger file sizes than textual data making
them hard to transfer and process even in light of recent technological advances
Finally, we want to thank the contributors, reviewers, and DHQ, specifically Managing Editor Cassandra Cloutier, for their work. Even under what were once "normal" conditions, writing an essay for publication is demanding. The challenges quickly mounted as authors revised amidst a global pandemic that disrupted everyone's daily lives and affected communities unequally due to structural inequalities. As authors and our colleagues at DHQ tried to balance caregiving, jobs, and their own health, among other duties, they still carved out time to make this issue possible. This is quite an achievement, and for which we are grateful. Finally, we want to leave with an invitation. We encourage readers interested in continuing to further engage with AVinDH to join the SIG. We look forward to all that lies ahead.