DHQ: Digital Humanities Quarterly
Volume 15 Number 1
2021 15.1  |  XMLPDFPrint

The Media Ecology Project: Collaborative DH Synergies to Produce New Research in Visual Culture History


This essay details the development and current NEH-funded research goals of The Media Ecology Project (MEP), directed by Prof. Mark Williams and designed by Dr. John Bell at Dartmouth. The virtuous cycle of access, research, and preservation that MEP realizes is built upon a foundation of technological advance (software development) plus large-scale partnership networks with scholars, students, and institutions of historical memory such as moving image archives. The development of our Onomy vocabulary tool and NEH-funded Semantic Annotation Tool (SAT) are detailed, including their application in two advancement grants from the NEH regarding 1) early cinema history, and 2) television newsfilm that covered the civil rights movement in the U.S.

MEP is fundamentally 1) a sustainability project that 2) develops literacies of moving image and visual culture history, and 3) functions as a collaborative incubator that fosters new research questions and methods ranging from traditional Arts and Humanities close-textual analysis to computational distant reading. New research questions in relation to these workflows will literally transform the value of media archives and support the development of interdisciplinary research and pedagogy/curricular goals (e.g., media literacy) regarding the study of visual culture history and its legacies in the 21st century.

This essay is dedicated with respect to the legion of significant Film and Media Studies scholars who passed away during the time it was written: Eileen Bowser, Edward Branigan, Thomas Elsaesser, Jonathan Kahana, Paul Spehr, Bernard Stiegler, Peter Wollen.


Our moving image heritage is at enormous risk. Moving image archivists and digital repository advocates are developing solutions to these problems, but we cannot sustain interest in “preservation” without a better sense of the historical value of these materials. “Access” is not enough; new knowledge production is required in order to connect archival materials with audiences and accelerate preservation efforts. The Digital Humanities must move concertedly forward to engage visual culture with the same dedication and technological ingenuity it has brought to the study of word culture.
Figure 1. 
Logo of MEP
The Media Ecology Project (MEP) is a digital resource at Dartmouth directed by Prof. Mark Williams that enables researchers across disciplines to access moving image collections online for scholarly use. Dr. John Bell (Dartmouth ITC) has designed and built the overall technical architecture for MEP. MEP promotes the study of archival moving image collections, enhances discovery of relevant corpora within these archives, and develops cross-disciplinary research methods. These efforts help ensure the survival of these collections via new published scholarship, plus contributions of metadata and research on studied corpora back to the archival community. The virtuous cycle of access, research, and preservation that MEP realizes is built upon a foundation of technological advance (software development) plus large-scale partnership networks that result in new practical applications of digital tools. This article will demonstrate the steady progress toward these MEP goals and designs as a DH project, present reflections about the significant emergence of visual culture DH, and posit certain directions forward.
With internal support at Dartmouth and especially support from the National Endowment for the Humanities, MEP has developed several digital tools that support and sustain the creation of new networked scholarship and pedagogy about archival moving image materials. These include:
  • The Semantic Annotation Tool (SAT), which enables the creation of time-based annotations for specific geometric regions of the motion picture frame.
  • Onomy.org, which is a vocabulary-building tool that helps to grow and refine shared vocabularies for tags applied to time-based annotations.
Together, these two tools support close textual analysis of moving pictures based on time-based annotations Annotations denote a start time and stop time for a subclip, a description and tags related to that clip, and attribution for its creator. This granular approach to media literacy and scholarly annotation is flexible enough to be applied to many types of research and analysis.
MEP is fundamentally 1) a sustainability project that 2) develops literacies of moving image and visual culture history, and 3) functions as a collaborative incubator that fosters new research questions and methods ranging from traditional Arts and Humanities close-textual analysis to computational distant reading. The deployment of close textual analysis is a critical aspect of MEP in developing media literacies within DH. It realizes a practical response to concerns about the acceleration of contemporary culture, the related speed-read dynamics of many audiences, and vacancies of historical and aesthetic insight as a factor of modern consumerist behaviors [Virilio 1997] [Rosa and Scheuerman 2009] [Rockhill 2019].­­­ Enabling a spectrum of purposeful and reflective considerations of the mediated past is keenly recognized to be pressing and necessary.
At the other end of the methodological spectrum, MEP's work with computer scientists has produced new tools supporting machine-reading of moving images, which produce an expansion of time-based annotations that require lucid and informed evaluation. One direction of this research produces feature extraction (isolating specific formal and aesthetic features of moving images), while another uses deep learning approaches employing convolutional neural networks to identify objects and actions in motion pictures. Data from these tools can be critically assessed and collated with the “manual” (human-produced) annotation tools mentioned above to create synthetic and iterative research workflows that “learn” across the disciplines. SAT enables real-time playback of all annotations.
While developing MEP as a rather distinctive Digital Humanities project, we have learned first-hand several key lessons about this important and emerging field. Because we are building MEP from an Arts and Humanities perspective, we recognize that our goals must always be framed to raise awareness about the significance of cultural-critical perspectives within the various institutions that we have engaged (archives, libraries, universities, grant resources, etc.).
Like many in DH, we underscore the need for collegiality and connectedness in pursuing collaborative work that depends upon openness and mutual respect as well as a balanced critical eye. This corollary of the MEP profile as a virtuous cycle is echoed in Kathleen Fitzpatrick's recently published call for “generous thinking” [Fitzpatrick 2019]. Everyone who engages in MEP is at some level working outside their comfort zones: across disciplines, across expertise, across vocabularies. In a very real sense we are engaged in “translation” work, the great benefit of which can be experimentation regarding methodologies of study but also in infrastructural designs of work-flow and output.
New research questions in relation to these workflows will literally transform the value of media archives and support the development of interdisciplinary research and curricular goals (e.g., media literacy) regarding the study of visual culture history and its legacies in the 21st century. These goals have grown to be especially timely during the publication process of this essay: the conceptual and ethical significance of re-imagining our collective purchase on historical imagination has been axiomatic to the socio-political demonstrations of both outrage and engagement that are iconic to 2020.

Situating MEP in relation to transnational media archives

MEP was conceived by Prof. Williams at an early meeting of the ambitious Project Bamboo DH initiative. Bamboo is no more, but its basic principles still inform those of MEP: most digital tools have been built for the sciences, resulting in a crying need for DH resources in the Arts and Humanities. But if every institution assumes it must fulfill all recognized needs, our goals will be doomed; we must work collaboratively and in a series of progressive arcs forward to develop both traditional and emergent methodologies of DH scholarship.
The more significant early institutional affiliation for MEP was Prof. Williams' inaugural presentation to The Association of Moving Image Archivists (AMIA), which generated immediate and enduring collaborative synergies. We are poised to realize new research trajectories regarding large moving image collections as “big data,” and are developing institutional ties to vast digital collections of historical moving image materials. The notion of ecology is central to the project in several ways. Those of us who work on media history recognize all too well that the materiality of historical media is fated. These historic materials simply will not endure, but for the work to preserve and archive them. This work is especially important and timely in our contemporary media environment. Most media audiences and publics simply do not recognize the dilemma that contemporary archivists face. With the rise of social media, many people know that there are thousands of videos posted per hour on sites such as YouTube, and do not imagine that moving image culture is deeply imperiled, since it seems to be ubiquitous and unquenchable. Such an impression effaces the true condition of most historical media, which archivists are vigilantly working to preserve.
The specific platforms we initially engaged and began to bridge are 1) Mediathread, a classroom platform developed at Columbia University, that we helped develop as a research platform that supports publication of time-based annotation metadata and integration of external controlled vocabularies for tagging; 2) Scalar, a digital publishing platform developed at The University of Southern California, expanded to support import of time-based annotations and controlled vocabularies for tagging; and 3) Onomy.org, specifically designed for MEP to facilitate the collaborative creation and sharing of controlled vocabularies for annotating online media files.[1] The Media Ecology Project has been developed to sit in between and in relation to these platforms and media collections, navigating the import, export, and production of metadata across participating archival content that has been engaged by a scholar or team of scholars. In this way we can propel capacities for search and discovery across these media, and develop capacities to realize new forms of research, scholarship, and publication.
Figure 2. 
MEP Archival screening poster of the 2013 founding symposium at Dartmouth
We have enjoyed the participation of multiple renowned archives in several key pilot projects essential to honing the developmental vision for MEP.[2] One of the goals in each pilot study is the scholarly development of taxonomies or controlled vocabularies that can be deployed regarding the assignment of tags and other metadata to specific media content areas. The application of these vocabularies will enhance the functional discoverability of archival content and augment efforts to produce new forms of digital scholarship.
MEP archival connections are being built on public standards such as the Open Archive Initiative and the W3C Web Annotation format. Use of these widely available standards is key to making an ecology of applications that encourage bidirectional communication and share information as peers, treating archives as not just a source of raw materials but also a consumer of new analysis and scholarship. MEP has received funding from a variety of internal sources at Dartmouth College[3], which has supported software development and metadata generation but also conference travel and stakeholder meetings at Dartmouth.

Tools to Build MEP: Key Early Grants

In addition to internal support within Dartmouth, Prof. Williams has been fortunate to share three significant start-up grants that have been formative to MEP development.

1. NEH Digital Humanities Start-Up Grant: The ACTION Toolbox (with 1st PI Prof. Michael Casey at Dartmouth, 2011)

ACTION (Audio-visual Cinematic Toolbox for Interaction, Organization, and Navigation) is an open source platform that supports the computational analysis of film and other audiovisual materials. ACTION features extraction and multi-feature pattern analysis and machine learning tools. These tools include color features, motion features, structural segmentations, audio features, and analyses based on automatic labeling of the data via machine learning. ACTION provides a work bench to study such features in combination with machine learning methods to yield latent stylistic patterns distributed among films and directors.[4] As such, ACTION is a platform for researching new methodologies in the study of film and media history.[5]

2. NEH Tier 1 Research and Development Grant: Semantic Annotation Tool for The Media Ecology Project (with John Bell at Dartmouth, 2015)

The Semantic Annotation Tool (SAT) is an open-source drop-in module that facilitates the creation and sharing of time-based media annotations on the Web by researchers, students, and educators. SAT is composed of two parts: first, a jQuery plugin that wraps an existing media player to provide an intuitive authoring and presentation environment for time-based video annotations; and second, a linked-data-compliant Annotation Server that communicates with the plugin to collect and disseminate user-generated comments and tags using the W3C Web Annotation specification.
Figure 3. 
Semantic Annotation Tool graphic
The goal of building this system was to create an end-to-end open source video annotation workflow that can be used as either an off-the-shelf or customizable solution for a wide variety of applications. Potential uses include collaborative close reading of video for humanities research, simplified coding of time-based documentation in social science studies, enhancing impaired vision accessibility for media clips on web sites, and many others.[6]
As is typical for linked data-compliant systems, annotations created by SAT are structured using a combination of multiple, type-specific data standards.[7] A SAT annotation consists of:
  1. A Media URI describing the location (source) of the object being annotated
  2. Basic identifying metadata for the source object (e.g., title, author) when available
  3. Provenance information for the annotation
  4. A textual annotation body and multiple tags that apply to the delimited media fragment. SAT annotations create relationships between these components and the media object being annotated using several standards, including subsets of Friend of a Friend (FOAF), Simple Knowledge Organization System (SKOS) W3C Open Annotations (OA) and Dublin Core (DC). Annotations are encoded for transmission using JSON-LD.
Statler is the server half of the Semantic Annotation Tool. Built on a Ruby on Rails framework, Statler is a standalone linked data server that allows persistent annotations to be added to media files with minimal changes to the host platform. Statler's public face is an API that serves W3C Web Annotation11-compliant metadata describing arbitrary media URLs.
Waldorf.js is the client half of the Semantic Annotation Tool. It is a jQuery plugin that can be added to any HTML page with only a few lines of code. Once installed it searches the page for HTML5 media tags and dynamically wraps them in an interface that supports annotation of time-and geometrically-delimited media fragments. Waldorf.js was developed in collaboration with VEMI Lab[8] to ensure that accessibility was forefront in its development and that playback of annotations is compatible with screen reader software.[9]

3. Expanding SAT via Knight News Challenge: “Unlocking Film Libraries for Discovery and Search” (with Prof. Lorenzo Torresani and John Bell at Dartmouth, The Internet Archive, VEMI Lab at UMaine, 2016)

This 6-month Knight grant successfully demonstrated the potential for the Semantic Annotation Tool to help make troves of film/video housed in thousands of libraries searchable and discoverable. Working with Dartmouth College's Visual Learning Group (directed by Prof. Lorenzo Torresani), we collaborated to apply machine vision tools already being developed for object, action, and speech recognition to a collection of one hundred educational films held at the Internet Archive.[10] The goal was to set the stage for future full-scale integration by examining the output of these tools and comparing them to one another as well as to human-generated annotations.
Part of the significance of our project was to enable essential first steps in object and action recognition for historical formats of film/video, thereby providing incentives for the field of computer vision to develop research capacities regarding film/video from prior eras.[11] Typical library cataloguing practices provide only basic information for such content: title, subject, synopsis. Libraries have made great strides in unlocking word culture texts through optical character recognition; they are opening up audio items with voice-to-text transcription. But thus far, libraries have not found ways to unlock moving images and annotate them at scale. Developing steps to realize an automated solution to producing high-quality metadata about such historical film and video content will be critical to allowing libraries and archives of all sizes to make the collections they own available for public use. The data generated by this prototype grant provided a significant model for first steps toward developing such a system (see Figure 4). The deep learning output was not error-free, but the success rate outperformed expectations.
Figure 4. 
Machine Vision Search overview
The prototype also demonstrated the utility of The Semantic Annotation Tool as the back end of such a research protocol, robust enough to host the entire iterative annotation cycle: to enable the creation of manual (curated) time-based annotations by knowledgeable scholars and scientists, and to host the subsequent machine-learning output of many times more time-based annotations. The fulfillment of the research process will allow the content curators (manual annotators) to quickly evaluate the machine-learning results, which will produce a new enlarged and sweetened training set for the algorithms, etc. The resultant iterative cycle of excellence would indeed produce game-changing results for moving image libraries and archives everywhere. The ideal interface would operate by translating in real-time the text-queries provided by users into content-based classifiers that recognize speech, audio, objects, locations, and actions in the video, in order to identify the desired segments in the film. When implicitly validated by users (by viewing), the search results and the original text queries would be fed into SAT which will add these annotations to each film for permanent semantic browsing and search.
With talented undergraduate students at the DALI Lab at Dartmouth, we were able to produce a Machine Vision Search prototype website that illustrates the research process and also demonstrate key research results: SAT was used to display tags generated by machine vision analysis of films as time-based annotations of those films. MVS demonstrates the flexibility of SAT by significantly changing its presentation interface, eliminating the annotation bodies and instead displaying only tags. In addition, Waldorf.js was extended to add new functions like flagging and deleting tags/annotations that the machine vision system identified incorrectly. These new functions additionally demonstrated SAT's flexibility, because they required no changes to the annotation server itself. Dartmouth students were able to create the custom MVS interface on a very short development timeline due to SAT's architecture and simplicity.

Applications: MEP Advanced NEH Grant Projects

In 2018 Prof. Williams and Dr. Bell were honored to receive two advancement grants from The National Endowment for the Humanities for The Media Ecology Project. These grant projects had each been initially developed as demo pilots for MEP and are now poised to realize significant advances in Digital Humanities scholarship via further developments of SAT, both technologically and conceptually.

1. The Paper Print/Biograph Linked Data Compendium: Understanding Visual Culture Through Silent Film Collections (2018-2020)

One of our inaugural pilot projects was in conjunction with the Library of Congress regarding their early silent film era materials, with an emphasis on the historically significant Paper Print collection [Williams 2016].[12] The Paper Print collection is, especially in the U.S. context, roughly the equivalent of the Rosetta Stone for those who study moving image history in relation to visual culture: a vast and inspiring series of historical objects that is unique in film history. As motion pictures were invented and experimented with, their producers applied for copyright of each film by placing a positive print of the film materials on ribbons of photosensitive paper for deposit at the Library of Congress. This has resulted in a record of the literal development of early cinema practices that no other archive can duplicate. We are extremely proud that the Library of Congress has promised to digitize the entire corpus of Paper Print titles in relation to the partnership forged by MEP and the esteemed early cinema and pre-cinema study organization DOMITOR. Early research in the pilot was represented as the plenary panel of the 2017 Women and the Silent Screen conference in Shanghai.
This recent advanced NEH grant project will produce a digital compendium of over 400 select films from the silent cinema era documenting the aesthetic practices of early cinema, with attention to the transition of visual culture from stage to screen. The Compendium will afford many new questions regarding historical visual culture that span the extraordinary history of early cinema from attractions to narrative, from the natural world to vaudevillian theatrics, from abstraction to realism. It will combine highly-influential and rare works archived at the Library of Congress with materials preserved at the Eye Filmmuseum in Amsterdam, The British Film Institute (BFI), and The Museum of Modern Art (MoMA) to create a digital resource for film scholars around the world. Many early films from Eye will be digitized from prints derived from films shot in the original Biograph format of 68mm, prints that offer better raw material for machine vision analysis than Paper Print versions, for example. Taking an innovative approach to annotating the digitized films with diverse types of scholarly description, archival metadata, and machine-generated annotation, the compendium will present visitors with a variety of analytic lenses embedded in a single, simple interface.
Figure 5. 
Poster for NEH grant Early Cinema Linked Data Compendium
The late Paul Spehr's meticulous chronological production logs of American Mutoscope & Biograph films, derived from various historical collections over many decades of research, will serve as a backbone for the Compendium to provide a framing infrastructure for all of the Compendium films and foreground the 68mm films especially as neglected marvels of early cinema, ripe for rediscovery and counter-history. The corpus will also feature new digital access to the collection of Biograph exhibitor catalogs at The Museum of Modern Art, a resource that features three keyframes from each motion picture title described — rich historical information and extremely rare kernels of visual culture, in many cases for films that are otherwise considered lost. The compendium will frame each film and its historical record as a resource for rediscovery and fresh methodological interventions, central to the advancement of the digital humanities in relation to visual culture.
To create the compendium, we will integrate MEP's SAT with software developed by the Alliance for Networking Visual Culture (ANVC). ANVC's Scalar is a web publishing platform designed to present text, media, and data using integrated, flexible interfaces that was an early integration target for MEP when the project built data exchange tools connecting it with Mediathread. Integration of SAT into Scalar is an evolution of those earlier efforts: rather than attempting to move data between annotation tools that do not follow common standards, the new project would take the standards-compliant SAT module and drop it directly into the Scalar platform. Both SAT and Scalar are built on semantic web principles that make it easy to gather and merge diverse data from across the web using linked data, making the integration a natural fit.
Several types of data will illustrate the compendium's wide range of methods to assess and study these valuable works of cultural history (see Figure 5). These data include:
  • The films themselves, as streamed[13] from the Library of Congress and Eye Filmmuseum
  • Basic metadata and select annotations identifying creative personnel and genre
  • An extensive database of film production information
  • Descriptions of performance styles and gestures of some films, encoded using Laban Movement Analysis (LMA)
  • Algorithmically-generated analysis, such as optical flow visualizations, to highlight formal characteristics of some films
The compendium will also serve as an evolving source of information and scholarly possibility. Because of its open software framework, interested scholars can contribute their own analyses based on interpretative contexts of their own devising. This MEP linked data compendium, then, will not only unite a wide and growing variety of data, but offer the chance for scholars to gather and trade ideas with one another, creating fertile territory both for discussion and the sparking of new knowledge about these essential works of cinema. The Compendium will contain data describing many different aspects of films in the corpus.
For example, one key area of emphasis that evolved in the MEP pilot study on the Paper Print Collection is the analysis of performance styles. One of the characteristics of the era is the transition from heavily codified theatrical performance styles derived from late 19th century theater, toward the uneven development of more “cinematic” performance styles that evolved in relation to the proximity of the motion picture camera. An ideal case study emerged regarding the career of Florence Lawrence who, though uncredited (as were most all performers of the pre-Nickelodeon era), came to be known to audiences as “The Biograph Girl.” In this study, primarily developed by Prof. Jenny Oyallon-Koloski, time-based clips of Lawrence's onscreen appearances were demarcated via brief description and tagged according to a simplified protocol of Laban Movement Analysis (LMA). Mediathread provided the capacity to codify her performance style (gestures, facial expressions, other aspects of the expressive body) and potentially contrast her performance style with those of other Biograph actresses of the era such as Mary Pickford.
In MEP's pilot studies, Mediathread allowed for the documentation of delimited written descriptions and metadata relevant to each title among the archival films accessed to that point. The annotation of Lawrence's performances, for example, were applied via full-frame time-based annotations. Using the Semantic Annotation Tool in the Compendium will enhance the precision of our already granular annotation methodology by adding geometric targets within a frame and real-time playback of annotations with sub-second resolution. These innovations will enable the creation of time-based annotations that reference films with greater specificity, a key enhancement given the speed with which performance modalities shift.
Traditional cataloging techniques depend on key concepts like normalizing metadata into standardized sets of descriptive fields and ensuring consistent minimum coverage across a collection. The Compendium will instead organize annotations produced via networked scholarship using linked data concepts drawn from the semantic web. Linked data was designed to distribute data across a decentralized network: the entire Internet. Like the Internet itself, it was designed for heterogeneity and fault tolerance. Using linked data as the basis for research annotations will allow the Compendium to use highly specific data models for each type of inquiry that a scholar wishes to pursue, rather than try to force data into pre-approved ontologies. Since it contains no expectation of complete coverage, researchers can choose to annotate as many or as few films as they need for their research–new data simply adds to what is already known about a film.[14] The “linked” concept means that disparate data types can be connected to one another using either simple relationships–two annotations may refer to the same timecode in a video–or semantically rich relationships–for example, cause and effect.
In addition to manual annotations, we are working to apply another type of data in the Compendium: optical flow tracking based on computer vision analysis of the films.[15] SAT will allow us to pinpoint where in the frame the annotated movement takes place, which will facilitate the isolation of gestures and smaller movements and allow us to visualize actors' movement pathways through the frame. These improved annotation strategies will strengthen our ability to document the movement patterns we are observing, will aid in our communication of these findings to research collaborators, and will enhance the complementarity between our manual annotations and computer vision analysis.
The films represented in this Compendium will designate a galaxy of new research inquiries, especially when placed into linked data relations with one another and with the new textual and contextual metadata we will provide. The Compendium will in a sense re-animate early cinema history, phenomenologically and conceptually, especially for audiences and users new to this material. Contemporary media theorist Bernard Stiegler's preferred phrase is to “re-enchant” our sense of history and the world [Stiegler 2014], a necessary tonic to the information bloat and hollow exploitation of much digital media engagement today.
But we also will cross a new set of thresholds in a digital humanities context, critical to this grant opportunity, by re-articulating the dialectic described in the very notion of digital humanities. The tension that exists between the traditional Humanities tenets of close textual analysis versus the demand for distant reading and analysis at scale in the computational sciences will be both visualized and progressively informed by this linked data Compendium.[16] The use of optical flow visualizations and metadata in the Compendium project (previously utilized in the ACTION toolset) will hopefully contribute to both the existing data pool of optical flow research and to the fundamental experiential distinction between manually-generated granular performance annotations and machine-generated cinemetrics. Both types of data will be shown in context with one another within the Compendium, inviting new relationships between close and distant viewing. Scientists, scholars, and artists alike will be in a position to imagine and explore unique ways to further interrogate and mobilize this new experience and pursue new research questions and representational innovations.
Though it will contain several finished essays, the Compendium will also exist as a first draft of complex research on these films with a large multi-archive body of films and related metadata to be iteratively added. But it will also be an engine for new and previously unconsidered research questions and methods, a first draft of varied directions of inter-disciplinary DH pursuits that can directly engage the arts, historical and cultural studies, and computational analysis.

2. The Accessible Civil Rights Heritage Project (2018-2020): Expanding the Goals of Access

Our MEP pilot on historical news materials (newsreels, news telecasts, newsfilm, and other associated footage) was dedicated to new scholarship on news materials from multiple archives. We gradually honed this topic and its participants into a focused address to Civil Rights content,[17] with a double purpose of developing new scholarship plus a dedicated line of research and development to enable access of these semantically rich and complex materials for blind and visually impaired (BVI) users.
Figure 6. 
Poster for NEH grant Civil Rights ACRH NEH grant
The term “newsfilm” has evolved in relation to historical changes in media technology and media formats. Television newsfilm evolved after decades of motion picture newsreel and news magazine production that was intended for exhibition in motion picture theaters on a weekly or bi-weekly basis. Local television newsfilm–often shot on site by local television station news crews that only broadcast a fraction of what they recorded–is a largely untapped source of local and national history that captured powerful moments throughout the emotionally and politically charged American civil rights era. Television newsfilm was produced in a different media industry context and was intended for “exhibition” to domestic audiences on an almost daily basis in the U.S. for many decades. It was regularly produced by both local television stations and network television news divisions.
A key context significant to most television newsfilm collections is local television itself, which is a conspicuous, persistently ignored aspect of U.S. media history, even though the local station is the backbone and the condition of possibility for the dynamics of U.S. network television. Local television history is a common site of disavowal regarding many media histories, especially work on U.S. media history. As such, it can be the site of significant resultant capacities for historiographic depth and complexity. What is often truly compelling about sophisticated historical research is the relationship between the already-understood and what we think we know, versus the capacity to interrupt given history, perhaps even to intervene in that history. Local and network television newsfilm features the full spectrum of these historiographic capacities.
Television newsfilm was a primary form of extended news coverage for roughly forty years, from the evolution of television in the 1940s to the gradual adaptation to video formats in the 1980s. Much of the footage to be considered for this study is what is sometimes termed “raw” newsfilm, the footage that existed in-camera before it was edited and repurposed by the news professionals at a station or network. “Newsfilm” also describes local and network “finished” news stories and news programs and documentaries that utilized footage from multiple sources, aired for domestic audiences and sometimes distributed via film prints to local, national, and international markets. Collections of television newsfilm are being preserved and curated today in many archival collections across the U.S. and around the world. Most of these collections are as yet unavailable for critical and historical consideration.
It is important to underscore that historical newsfilm from different eras and industrial contexts have become digitized and available for study and time-based annotation only very recently. Many of the newsfilm clips and raw footage materials engaged for this study will be seen by scholars for the first time. This is a burgeoning area for both scholarly and public interest, a site for critical awareness about the (mediated) past. Also distinctive to this project is that much of the newsfilm to be studied was never screened publicly. That is, in many instances this newsfilm footage has never before been part of the public sphere, and thus has never been considered by even casual historians in any field. It has existed outside of critical inquiry and scholarship that is devoted to social history and media history. From a cultural perspective, these are indexical materials that have not yet been in a position to be remembered, let alone forgotten.
The conditionally absented or fugitive aspects of these civil rights materials inspire awakenings of the historical imaginary, and we expect this material will become especially relevant within our very contemporary 2020 context of global demonstrations and calls for social justice in response to the brutal murder of George Floyd and other African Americans in the U.S. The “Black Lives Matter” outcry in the weeks and months prior to the publication of this essay has been widespread and sustained, and seems to represent what Raymond Williams referred to as a major shift in “structures of feeling” [Williams 1977]: affective relations between consciousness and social institutions that are strongly emergent and inflect palpable pressure on commonplace rationalizations and actions.
The historical and historiographic potential of these materials is both vast and substantial. The Accessible Civil Rights Heritage (ACRH) project (see Figure 6) will help to expand the discoverability of these historical materials for critical consideration, by developing scholarly practices in relation to archival practices that will enhance searchable access to these historically rich items that would otherwise continue to be isolated in archival and data silos and virtually unavailable for search of any kind.
The hundreds of newsfilm elements made available for study in this project will serve as a representative sample of the ocean of television newsfilm collected in archives and historical societies across the U.S.[18] The study of the newsfilm era and subsequent eras of news coverage (i.e., post-celluloid eras) will be significantly enabled by the development of workflows, protocols, scholarly methods, and augmented vocabularies/ontologies to be developed in this proposed study.[19]
In order to consolidate these diverse materials we have engaged archivist Becca Bender (Rhode Island Historical Society) to advise on the creation of a common metadata spreadsheet format, and veteran professional moving image cataloguers Kathy Christensen and Laura Treat to help develop an Onomy vocabulary specific to the purpose of annotating civil rights news footage. These new cataloging and access procedures will assist in the parallel development of innovating high-quality, meaningful experiences of the collection to BVI users.
Given the special concerns of close textual analysis and its importance to humanities researchers, it is critical that any toolset designed to support humanities research be developed with that specific application in mind. However, any existing collection of materials and scholarship would carry with it the limitations of the tools that were originally used to create it–vocabulary and metadata that was designed to fit into a particular schema, as discussed by Owens [Owens 2014].
For the purposes of ACRH research, annotations featuring close reading analysis of civil rights newsfilm will be merged with extant metadata from the contributing archives and scholarly essays that feature newly-generated time-based descriptions. The result will be a re-animation of sorts for these historical media documents where specific events and images are contextualized in relation to known descriptors and vocabularies. ACRH's research into articulating the hermeneutics of moving images using annotations will result in a synthetic process that engages archivists, scholars and students to share their experience of these key cultural heritage texts with others–even those who cannot see the original texts.
ACRH will repurpose a selection of assorted newsfilm to produce a corpus of material that is uniquely challenging to describe: historically charged footage laden with contextual meaning but limited extant metadata. This sub-corpus is being repurposed for research into adaptive technology for BVI users, a fundamental example of cross-disciplinary opportunities that MEP is designed to enable and investigate. Although the potential historical value of newsfilm materials for BVI access is evident, accessible delivery of online video is a challenge that higher education has struggled to meet, leading to thousands of hours of video instruction being taken offline because the schools that created it could not provide equal access to all users.
The state of BVI accessibility on the web is, in short, disastrous. Web browsers in general are riddled with inconsistent implementations of reference specifications and vendor-exclusive features. Adding accessibility features that are often poorly understood and costly to implement to that unstable environment has resulted in–at best–inconsistent efforts to ensure web content meets accessibility guidelines, e.g. [Clossen and Proces 2017]. The type of content that ACRH is targeting, online videos with time-based annotations, is so new that accessibility has not yet been thoroughly considered in this context. By researching key guidelines and technologies, ACRH has an opportunity to direct the accessibility conversation about time-based annotations in positive directions.
As the online market has matured, the penalties for organizations that fail to make content accessible online have grown. In 2015 the Department of Justice settled with online education giant EdX for failure to comply with the Americans with Disabilities Act and forced EdX to implement a number of accessibility standards including WCAG 2.0 and WAI-ARIA.[20] UC Berkeley decided to remove thousands of hours of open educational audio and video content because it did not have the resources needed to make it ADA compliant,[21] touching off a heated back and forth between university administrators and faculty.[22] Accessibility problems are not limited to higher education either, as in the case of a 2014 lawsuit brought against Seattle School District 1 that resulted in a consent decree that was estimated to cost the district in the range of three-quarters of a million dollars.[23]
BVI students in particular may have trouble fully understanding primary video sources because the text or audio descriptions associated with them rarely convey the full meaning and context of the images on screen. BVI users cannot see content filled with small clues that may be critical to its interpretation. Humanities scholars pore over information-dense resources like video to closely read it as a primary historic text at a level of detail that goes far beyond the ability of traditional accessibility adaptations like captioning to capture. ACRH proposes that time-based annotation techniques[24] can provide support for humanistic interpretation of video far better than existing adaptive technology. Beyond the BVI community, though, researching best practices for time-based annotation will provide scholars with a new perspective on how to integrate data-centric digital heuristics with deeply cultural hermeneutics.
Existing accessibility guidelines for online video usually focus on creating secondary audio or caption tracks that synchronize playback with the video itself.[25] The closest these recommendations come to SAT's methodology is Mozilla/A11y's recommendation to embed a timed text track into Ogg video. Setting aside the browser restrictions introduced by using Ogg video, timed captions have a number of drawbacks in an educational setting when compared to full annotations: they are only delimited by time, not geometric space in the frame; they do not carry additional metadata like tags that are useful for cataloging and search; and they do not include authorship information that is important to convey in a scholarly context. Additionally, SAT's separation of annotation data from the video file provides opportunities to readily query that data using external tools–a key feature that streamlines the workflow of digital humanities scholars.[26]
ACRH will study how to write video annotations that convey the rich content of evocative videos, and create adaptive technology that supports playing back those annotations audibly. The resulting guidelines and technology will be published as an open resource that all schools, museums, and archives can use to make their own video collections more accessible to BVI users. The grant brings together scholars, archivists, cataloging experts, and cognitive neuroscientists to research best practices for these requirements. The result will be an evidence-based set of guidelines for creating accessible video annotations, documentation on how to implement those guidelines using open-source software, and a demonstration corpus of civil rights newsfilm showing humanities scholars how to apply these guidelines to their own research. Just as there is a concept of resources that are “born-digital,” ACRH proposes to build a humanities corpus that includes video, annotation, and metadata so it is, as a body, “born-accessible.”
Composing annotations of moving image culture that are meant to assist blind and low vision viewers redefines certain basic assumptions about visual culture among the sighted, and demands careful attention to details otherwise taken for granted. It is surprisingly difficult to capture the basic information of a shot. Our methods are experimental, in that we are near the completion of compiling a sufficiently large data set to provide our colleagues at VEMI for their qualitative social science research.
The participating archives have generously provided core descriptive metadata for hundreds of civil rights newsfilm clips, and assisted in selecting the dozens of clips for which we will provide more extensive time-based annotations via SAT. The methodology, process, and culminating metadata will be published and made available for open access and use. Much of the archival media will also be available for scholarly and public access from the participating archives, dependent upon archival protocols and online capacities.
A culminating symposium for the ACRH Project will bring together archivists, technicians, and especially scholars from across academic disciplines to critically engage and assess the results of the project and imagine next best steps to develop the ARCH Project research materials.[27]

Visual Culture DH: Reflections on the Way(s) Ahead

As the article has shown, MEP has a history of supporting projects exploring intersections between different methodologies of study and critical approaches to varied archival content. We have found time-based annotations and tags to be a key enabling technology that allows scholars the freedom to engage with digital media texts from multiple perspectives that can then be programmatically synthesized into a cohesive assemblage.
Among the methodological comfort zones to be negotiated in Digital Humanities, we are committed to the development of Visual Culture Studies in DH, which can produce tension with legacy approaches to DH that primarily focus on word culture alone. In addition, the field of Film and Media Studies often features attention to research methods that address and engage audiences and the reception of media texts, emphases that are less prominent in the historical study of word culture. Most important is the prominent DH tension between the traditions of “close reading” that are central to the Arts and Humanities versus the goals and practices of reading at scale that are crucial to computational approaches to vast corpora of texts under analysis. Taylor Arnold and Lauren Tilton revise the terminology regarding visual culture to “distant viewing”, which takes into account the implicit interpretive quality of acts of viewing [Arnold and Tilton 2019].
Recognizing sites of potential dissonance can help to reformulate them as sites of new inquiry and critical intervention in the pursuit of Visual Culture Studies, to produce instead a growth area for both productive research inquiry and rigorous critical evaluation. The close and distant reading/viewing dissonance of DH can be seen to work within the motivated, intentional bi-play of manual and automated annotations in MEP as a defining and iterative dialectic which allows us to better recognize an always-already evolving spiral of creative and critical exchange across manual and automated realms. Realizing a full awareness of this dynamic model may present a fundamental perspective toward progress in the emerging interdisciplinary space that is DH.
One signal theoretical turn to recommend in this realization of an ongoing and dynamic dialectic toward computer vision and machine-reading excellence is the historic approach to early cinema theory provided by Dziga Vertov's Kino-Eye or kinoglaz theory.[28] Vertov's theory famously consists of two major tenets, related yet distinct from one another. In tenet one, “kino-eye” is engaged with his great enthusiasm and devoted pursuit of new technologies of enhanced vision (e.g., the motion picture camera and related lens technologies), which represented material advances beyond human vision alone that seemed capable of both futurist and constructivist goals for enhancing society and culture. But equally if not more important is the second major tenet, “kino-edit”, which requires the rigorous study and understanding of the world and its historical processes by the artist/scientist, in order to actively interrogate, differentiate, and recontextualize what may be recognized to be merely positivist technological outputs of the Kino-Eye (e.g., racial and gender bias in facial recognition software, etc.) [Nakamura 2008] [Noble 2018] [Benjamin 2019] [Ng et al. 2020].
Vertov's theory is widely recognized in its motivated call for critical and aesthetic discernment and socio-political insight. Like many aspects of visual culture historiography, it is surprisingly applicable to not only early cinema but also to the rise of digital culture and its related epistemologies. We can anticipate that the development of new DH tools and platforms for Visual Culture Studies will necessitate sophisticated theoretical frameworks that will prove essential to 21st research and scholarship. The two tenets/steps of Kino-Eye theory are directly applicable to the MEP advanced NEH grants described above, especially in relation to the iterative workflows of manual close-textual annotations, vast machine-reading expansions in the number of those annotations, and manual curated evaluations of the machine-reading results (that then afford a new training set for another iteration of the cycle). Also significant to recognize is that these annotations are sometimes frame grabs, but primarily time-based sub-clips, the study of which can judiciously contribute to a new and contemporary re-understanding and elaboration of Vertov's elusive, unfolding, yet core concept of the moving image “interval”, tied to the organization and elaboration of movement [Heftberger 2018, 220–221].
There is more to say about the conceptual and phenomenological value of manual time-based annotations. As suggested earlier, manual annotation of visual culture is a contemplative, iterative, and essential task that mirrors more innovative annotation practices across Digital Humanities endeavors. For example, annotation platforms such as hypothes.is have inspired new scholarship that promotes the generative aspects of close reading practices in relation to networked scholarship when annotating word culture texts in their platform [Zamora 2016] [Rheingold 2016] [Bali 2016].
Indeed, part of the spirit of intervention within MEP and SAT is the process of manual annotation itself, which necessarily slows down the apprehension and understanding of moving images as aesthetically expressive media. This produces a palpable countervailing force against many contemporary viewing practices of moving image culture. For example, professors and other teachers of media history and arts can attest almost uniformly to a gradual change across generations of their students in the craft of “reading” moving image culture beyond a stencil or scaffold of factual and narrative “content”.
For media historians and artists, this mode of reception can seem as though many audiences today have been trained as pattern recognition filters, impatient for the next plainly evident delta change of attractions in the image or soundtrack, and fundamentally inattentive to basic aesthetic information provided as signal contributions to the expressive registers of the text. This results in an implicit or even explicit audience “demand” or drive to accelerate perceptual stimuli (“I've got it, move on; I've got it, move on…”) that eschews many of the fundamental temporal and spatial registers of visual culture, and moving images in particular.
This presumptive need for speed may be directly related to changing ecologies in the expansive amount of available quality mediated content, but also inter-medial changes in the patterns of consumption of time-based media (e.g., scrubbing through video, binge-watching); emergent formal practices within games and social media; and perhaps even a resultant competitive pressure for the use of one's time within and across mediascapes, e.g. [Guo 2016]. Even though these contemporary reading practices and competencies may ultimately prove to be valuable in specific contexts,[29] there is indisputable value in also learning to deepen one's capacity for better attending to the historical aesthetic practices in moving image craft and art (across the full range of expressivity), and also to better understand the value of such an investment.[30] The rise of the global “slow cinema” movement is a key index of truly widespread cultural resistance to accelerated media culture, and exists in part as a contemplative critical response to the aesthetics and underlying political economy of mediated subjectivity based upon expediency, speed, presentism, and the exploitation of a delimited attention economy, e.g. [de Luca et al. 2016].
A broader but related perspective regarding digital culture writ large, and perhaps especially the debated value of the rise of artificial intelligence (machine-reading), comes from the technological imperatives derived by Bernard Stiegler [Stiegler 2014], who has been developing a series of incisive theoretical tropes concerning the rise of digital culture as a pharmakon: at once a powerful and enticing possible remedy for specific needs and demands in a socio-political system, but also a poison if used unknowingly or improperly.
Stiegler's most important intervention regarding DH resulted from his career-changing participation in the artist-scientist collaborative Ars Industrialis in 2005, which led to his committed politics of critique that makes a concerted call to “re-enchant” the world via new and rigorous attention to history and the arts, imbued with critical literacy about them. Central to his increasingly complex and refined theories is a renewed attention to the significance of what he terms tertiary memory (e.g., the archive), an externalized and technical extension of “internal memory”, plus a call to return to an intentional and motivated anamnesis, a refusal to forget the vital importance of critical engagement with these memory deposits in the interest of sustainable cultural environments. He positions this essential work of critical engagement in relation to what Plato termed “self-care”, but jettisons Plato's dismissal of the technological. Contemporary society, deeply imbued by technology and mediation, is now, perhaps for this very reason, threatened to become history-less and therefore uncritical about itself as a control Society [Stiegler 2014] [Barker 2012] [Abbinnett 2018].[31] Conscientious and motivated engagement with the archive is an essential component of balancing and even responding to the pharmakon of drives and controls associated with accelerated digital culture and AI. The call for both a renewed ethic of self-care about societal memory and a rigorous set of practices that enable such motivated and critical engagements are directly parallel with the goals and practices of MEP.
One purposeful area of related debate in the contemporary moving image archive world circulates around the recent proliferation of high-definition “video upgrades” (4K, 60fps) achieved via deep learning methods and often applied to early silent film era footage, see e.g Denis Shiryaev's youtube channel and Simon (2020). Passions can run high when archivists rightly insist that these media entities are separate and derivative “objects”: not archival acts of preservation or restoration, not grounded in meticulous curatorial insights and not dedicated to the indexical materiality of historical photo-chemical film prints. But there may be potential to realize a Stieglerian teachable moment in relation to these experiments that “push” the capacities of digital tools in order to change the experience and affect of watching historical “cinema” texts. Wide interest about these materials in online communities may indeed represent a re-enchantment of public imagination regarding early cinema, a literal sense of re-newed awareness and interest about history and moving images--and therefore an opportunity for archivists and scholars to direct attention toward an assortment of knowledges about early cinema (including the essential work to preserve and respect its history). In other words, the “drive” (compulsion?) within sectors of the AI industry to transform historical media artifacts into dramatically enhanced localized forms of “attractions” is itself an overdetermined project regarding desire for/within the historical imagination that is worthy of much further consideration. Without immediately casting aspersions on these considerable technological efforts, the experiential enthusiasm they inspire in audiences might be recognized as a digital framework to build upon. If the “upgrade” aesthetic dynamics border on a potential for mere spectacle of the hyper-real,[32] this potential may represent a techno-cultural pharmakon that is ripe to be more fully understood and historically grounded. How best to channel the many implicit investments in history of the visual arts embedded in these twin dynamics of digital production and reception, to engage these investments and further develop them in edified and conceptually insightful contexts? There is a compelling relationship to the MEP Early Cinema NEH grant, in that the great anticipation within the archival community to experience screenings of the newly restored 68mm films shot with the Biograph camera during the earliest years of cinema (restored at 4K and 8K at the BFI and the Eye Filmmuseum) offer a substantial counter-example to deep-learning “video upgrade” productions, and may set the table for dialectical Stieglerian discourse called for here[33].

Conclusion: Emerging Contexts for MEP

The time for this critical engagement has never been more necessary and opportune. The moving image archive world is keenly engaged in efforts to delineate and address the imperiled status of their collections [Casey 2015]. Access to archival content is becoming more foregrounded as a goal of preservation,[34] and as Giovanna Fossati points out in her standard-setting book From Grain to Pixel: The Archival Life of Film in Transition, DH is gaining momentum as scholars work to bridge the gap between digital methods, film and media studies, and media archives [Fossati 2018, 334].
Figure 7. 
MEP mind map: 2019 overview of the project
The connections of MEP to existing trends in DH research are evident (see Figure 7). Formal stylometric analysis grounded in new capacities for annotation are emerging in several international centers of DH study [Junior Research Group 2018], and are conversant with the fragmentation aesthetics of many artists working with large media corpora
We have already contributed to the conversation about artists engaging with archival content initiated by our colleagues at the Eye Filmmuseum: The Sensory Moving Image Archive (SEMIA). We anticipate and welcome opportunities to build bridges toward, for example, the WJAN BOT at The Eye and also Brian Foo's “Moving Images” video project [Foo 2019]. At the same time we are inspired by and look forward to collaborating with more traditional filmmakers who work primarily with archival footage[35], and also more directly cinephilic endeavors such as scholar artists involved with the burgeoning Video Essay component of Film and Media Studies [Keathley et al. 2019].
The analytic, annotation, and presentation tools engaged to work with audiovisual materials in DH are themselves becoming more collaborative. MEP is a member of the Video Annotation Interoperability (VAINT) group that includes the makers of such tools as the CLARIAH Web Annotation tool, ELAN, Frametrail, and VIAN. While each of these tools was developed for a specific purpose, their core data all indexes back to time-based annotations. VAINT is developing a standard that will allow the tools to exchange data so scholars can, for example, directly export results from VIAN's color analysis tools into SAT and present them alongside textual commentary using SAT's integration with Scalar. Also currently under consideration by the group is a further integration with IIIF-AV, which would open up data exchange with the large set of IIIF-compliant presentation tools. This approach of common data exchange, rather than consolidating functions into a single tool, is a match for MEP's philosophy that tools addressing specific intellectual concerns can be put into conversation with one another to support synthetic, interdisciplinary scholarship.[36]
Prof. Williams and Dr. Bell have been invited to participate in a working group of the FIAF (International Federation of Film Archives) Cataloguing and Documentation Committee Task Force dedicated to Linked Open Data, in order to further investigate the sharing of FIAF Glossary of Filmographic Terms in an RDF structure. A related area of development is an endeavor to initiate a multi-lingual dictionary of film terms.[37] We have initiated work with the American Film Institute and the Women Film Pioneers Project to expand the recognition of women filmmakers in digital and online resources. Very recently MEP has been funded to work with the Distant Viewing team on developing a prototype digital resource at Dartmouth regarding the prestigious James Nachtwey collection of photographic journalism.
In continuing to realize a virtuous cycle of engagement with media art and history, we will work collaboratively and intentionally to be poised toward a spirit of critical inquiry that is engaged with innovative work at other academic and cultural institutions around the world. Our efforts to engage via DH more access to and scholarship about visual culture and moving image history presents innovative approaches to fundamental historiographic questions about media and history, and also address the danger of further losses to that history in both practical and theoretical terms. We intend the scholarship that we conduct and inspire to avoid a gloss of mere positivism in its pursuit of new research questions, and to invoke issues of the missing, the fragmentary, the occluded, the repressed, and the fugitive regarding this history.


[1] At Dartmouth we were able to convene an extremely productive symposium in May 2013, which brought together representatives from multiple participating archives and institutions (http://mediaecology.dartmouth.edu/wp/news/page/2). The symposium was successful in producing a series of agreements about the future of the project. One key outcome was a unanimous call to develop a metadata server and attendant middleware that could help to facilitate and maintain quality metadata produced in relation to archival elements. The symposium also generated significant interest outside of Dartmouth in addition to the invited members of the archival community. MEP has been significantly featured at numerous national and international conferences and symposia ever since. This includes multiple conferences of the Association of Moving Image Archivists, ADHO Digital Humanities, the Society for Cinema and Media Studies, The Orphans Film Symposium, The American Studies Association, EUScreen, the Expanded Semantic Web Conference, Open Repositories, and the International Association for Media and History.
[2] These include a pilot in conjunction with the UCLA Film and Television Archive researching a ground-breaking public television program “In the Life” that assayed gay and lesbian experience in the U.S.; an inaugural international pilot, researching informational and documentary films produced at Films Division in India; and two pilots that have matured into advanced NEH-funded grant projects to be detailed later in this essay (Paper Print collection at The Library of Congress, and historical newsfilm materials across multiple archives).
[3] Principally from The Neukom Institute for Computational Science, The Leslie Center for the Humanities, The Dean of the Faculty and Associate Deans, Office of Information, Technology, and Consulting, and The Dartmouth College Library. The Dean of the Faculty Award to Prof. Williams for Scholarly Innovation and Advancement in 2014 contributed a considerable stimulus for The Media Ecology Project.
[4] Interest in the use of the ACTION Toolbox was especially marked at the Workshop on Computational Methods and Film Style in Potsdam in May, 2018.
[5] The platform allows such features as access to and analysis of low-level frame-by-frame data, automated segmentation and clustering of this data, audio analysis for soundtracks, and other content analysis tools. The histogram extractor class can be used to analyze streams of images or video files. The histogram class steps through movie frames and extracts two kinds of histogram for each frame. The first is a histogram of the entire image. The second is a set of sixteen histograms, each describing a region of the image evenly arranged in a four-by-four non-overlapping grid. Histogram values describe distributions of color values in each region on the screen and can be used for analysis of both full-frame and 4-by-4 subframe grids in L*a*b* colorspace. The Optical Flow class generates analysis data of general motion on screen. Optical flow data is extracted using an implementation of the Lucas-Kanade algorithm, operating tracked features (corner detector) of monochrome image data, tracking salient points of image data across consecutive frames. Feature extractor classes allow access to audio frame-rate data including spectral and timbral features. In addition to developing the toolkit, we collected metadata for over 200 films from across the full history of cinema, focusing on a representative chronological selection of 22 films by Alfred Hitchcock in relation to works by other prominent directors across the cinematic aesthetic spectrum, from mainstream Classical Hollywood to Maya Deren, Jean-Luc Godard, Chantal Akerman, and David Lynch.
[6] The SAT has been developed in response to feedback from the scholars and researchers participating in MEP pilot studies, especially regarding the generation of annotation metadata to describe media files. This feedback highlighted several shortcomings in existing time-based annotation toolsets, most notably a lack of interoperability and a set of divergent interfaces that can be difficult to learn or use collaboratively at scale. Development of a drop-in annotation tool like SAT that addresses these needs is a natural supplement to MEP's previous work promoting interconnected systems, and helps ensure that MEP's research, access, and collection development goals can be met.
[7] The Annotation Server and jQuery Annotation Plugin were developed in separate but related workflows with the Virtual Environments and Multimodal Interactions (VEMI) Lab at the University of Maine. As part of the initial submission for review, VEMI released the test version of both server and client components of the SAT.
[8] The VEMI Lab at the University of Maine researches accessibility and adaptive technology with a focus on blind and low vision users. MEP and VEMI collaborated on the development of SAT. They are a primary partner in the second advanced NEH grant discussed below.
[9] Though the Annotation Server was originally intended to be a Hydra/Fedora (renamed Samvera) application, VEMI Lab developers found Hydra/Fedora to be difficult to install and maintain. We removed Hydra/Fedora from the SAT software stack and replaced it with a simple Ruby on Rails application. Statler implements the same W3C Web Annotation-compliant public interfaces that Hydra/Fedora would include, but greatly reduces the server overhead necessary to implement the system.
[10] Prof. Williams and a colleague at The Internet Archive produced manual time-based annotations that subdivided 20 of the 100 films into several sequences of short clips, about 10 seconds each. The Machine Vision Search team utilized Google image searches generated from manual search terms to partially train a neural network. The software leveraged image recognition algorithms to enable content-based search and metadata generation across the entire video collection. Once trained, the network was demonstrated to find additional short clips of the concepts and tag them in the larger collection.
[11] It was important to develop data specific to the study of archival film, because the content of a typical film library is much different than the types of video generally used in the development of machine vision software. Most of the progress being made in applying this software is delegated to very contemporary, hi-definition video formats such as videos taken via new cell phones.
[12] Our pilot project was conducted with the participation of the renowned DOMITOR research society, and engaged Prof. Tami Williams (University of Wisconsin at Milwaukee) plus scholars who utilized the pilot study materials in their courses on silent cinema, including Prof. Frank Kessler (Utrecht University), Prof. Laura Horak (Carleton University), and Prof. Amy Lawrence (Dartmouth).
[13] Streaming media is used both as a technical and legal solution. The Compendium will not need to host large amounts of streaming media or worry about intellectual property constraints since the media is already publicly available.
[14] In December 2014, the W3C produced its first public working draft of a standardized data model for Open Annotations (OA). The model encapsulates text comments, tags, and external links as annotations that can be applied to a variety of assets embedded in web pages. Critically, it also provides for annotations that apply to “fragments” of assets, such as a geometric selection area within a video frame or a timecode-delimited subclip of a media file. Using a standard like W3C WA makes new applications for time-based annotations more feasible and available to a wider variety of scholars. Linked data concepts, which permit new types of structured data and incomplete coverage of a collection, will allow scholars to easily create new data facets that are specific enough to support advanced analysis in the Compendium. While that functionality had previously been available in proprietary or vendor-defined formats, OA is the first time a standards body with the weight of W3C endorsed an annotation data model that developers can depend upon for stability and interoperability. The OA model has now become part of the broader W3C Web Annotation (W3C WA) spec.
[15] Initial attempts to apply optical flow to Paper Print copies of early films have been uneven at best, due to the poor resolution and almost ubiquitous appearance of analog “noise” in the images. We have not yet made tests on the early 68mm materials.
[16] For example, research in convolutional neural networks theory has suggested research questions informed by what is called the two-streams hypothesis: the difference that exists in the human visual cortex between two pathways of understanding, between the ventral stream tied to object recognition and the dorsal stream tied to the recognition of motion [Simonyan and Zisserman 2014].
[17] Prof. Williams' introduction to the genre of “newsfilm” was courtesy of his participation for many years in The Association of Moving Image Archivists and especially a program about UCLA archive's 1970s newsfilm from Los Angeles television station KTLA at the Orphans West Coast symposium in 2011. An especially poignant example depicts a 1979 public protest by dozens of concerned African-American women at Parker Center in Los Angeles after the shooting death by police of Eula Love, a recently widowed 39-year-old African-American mother. The killing of Eula Love resulted from a dispute over an unpaid gas bill, a tragic landmark in the notorious racialized encounters by the LAPD and citizens of color. The protest event captured by the KTLA newsfilm footage provides an indelible memorialization of that tragedy. It is not known if any of the footage ever aired on television, but the power and salience of the imagery is deeply instructive today regarding the value of historical newsfilm. The women collectively and elegantly performed a public demonstration of the question “Can we speak back to power?” For additional analysis of this footage see [Williams 2018b]. For an inter-medial relationship of this event to the coterminous rise of New Black Cinema in Los Angeles, see [Field et al. 2015].
[18] Newsfilm content will be selected from archives across the United States, including The University of Georgia (local television newsfilm plus The Peabody Archives), The Mississippi Department of Archives and History, The University of Arkansas Pryor Center, The Wolfson Archive at Miami Dade College, The Bay Area Television Archive, Media Burn Archive (Chicago), Washington University Archive (St. Louis), The National Museum of African American History and Culture (Smithsonian), The MIRC Collection at The University of South Carolina, The Minnesota Historical Society, Southern Methodist University, The American Archive of Public Broadcasting, The UCLA Film and Television Archive, WGBH, The Boston TV News Digital Library, The National Archives, and The Library of Congress. Note that archival footage used in the BVI corpus will be drawn from publicly available collections.
[19] The team of consultants enlisted for this project are renowned experts in media studies and issues of racial and ethnic representation. Their varied personal and professional backgrounds will help create annotations highlighting the significance of the selected corpus of newsfilm and evaluate the value of those annotations to scholarly study. Participating scholars include Jacqueline Stewart (U Chicago) and Desirée Garcia (Dartmouth). Research into specific adaptive strategies will be led by Nicholas A. Giudice and Richard R. Corey of the Virtual Environments and Multimodal Interaction Lab (VEMI Lab) at the University of Maine. Technologies used include MEP's tools SAT and Onomy, ANVC's Scalar, and the Distant Viewing Lab's Distant Viewing Toolkit (DVT). Creating a pathway for DVT's machine-generated annotations to support and inform human-generated annotations written in SAT is an important step that extends the merged distant-and close-reading methods developed for the machine vision search project into more nuanced forms of scholarly commentary and analysis.
[24] For the purposes of ACRH, an annotation consists of a reference to a specific time and geometric region of a video, a textual body describing the content in that region, a set of tags associated with the textual body, and additional provenance metadata as needed to attribute an annotation to an author.
[25] Increasing accessibility also improves the experience for other users [Schmutz et al. 2017].
[26] WebVTT is also worth mentioning in this context, but it has limitations similar to A11y's Ogg recommendation except the data is not stored within the Ogg container file.
[27] In light of the present pandemic conditions, the symposium is likely to be virtual and online.
[28] Vertov's films have been a point of primary focus and inspiration for landmark books about digital humanities methods such as [Manovich 2001] [Heftberger 2018]. The emphasis here is on the historiographic significance of Vertov's theory, especially as it may be applied to contemporary and emergent issues regarding computer vision and machine learning (AI).
[29] Robert Samuels articulates a strategic refusal to denounce generational differences via the construction of a hyper-binary about media consumption [Samuels 2008].
[30] Anecdotally, students participating in the creation of ACRH annotations for use in BVI research report that this work has clearly enhanced their appreciation of and capacity to read for visual culture aesthetics.
[31] Stiegler passed away suddenly on August 5, 2020. See https://www.theguardian.com/world/2020/aug/18/bernard-stiegler-obituary.
[32] See [Williams 2018a] for background about the inter-medial history of electronic culture, and historiographic details about uneven development toward capacities (and demand) for a digitally mediated subjunctive “now” in practices of representation.
[33] In August, 2020, the online “release” of a significant 68mm film restored by The Museum of Modern Art led to viral social media enthusiasm among the archival and film fan communities: “The Flying Train” (1902)
[34] The Eye Filmmuseum hosts an online channel devoted to their restoration efforts: Restoration at Eye.
[35] Recent films include Dawson City: Frozen Time (Morrison, 2016), Apollo 11 (Miller, 2019) and Recorder: The Marion Stokes Project (Wolf, 2019). For more on this tradition see [Northrup 2019].
[36]  A related new pedagogical project utilizing SAT has been initiated at Dartmouth: enhancing the use value of archival motion pictures by utilizing them in basic language instruction. This project was inspired by the innovative teaching methods of Prof. Hua-Yuan Mowry at Dartmouth, who uses graphics and motion pictures in her classes to teach Mandarin. The pilot for this project uses SAT to illustrate written language as it is spoken in a famous Chinese animated short, Three Monks by Jingd Xu (1980). The SAT interface “plays” in real time several sequential annotations of transcribed Mandarin characters at the same time that the associated Mandarin words are spoken in the film. This basic SAT schema could enhance instruction across many languages and potentially even dynamically toggle or alternate subtitle languages as students watch foreign language films.
[37] In response to a generous bequest by the Taiwan Film Institute Archive, who gifted to Prof. Williams their unique and considerable bi-lingual dictionary of film terms (English and Mandarin), the Dartmouth Library worked with MEP to OCR and transcribe this resource in order to make it available as an Onomy spreadsheet. The work to achieve this complex final spreadsheet (that features three varieties of Mandarin script) was completed by gifted Dartmouth student Janine Sun. This has inspired the internal funding of several more undergraduates at Dartmouth to help develop additional bilingual Onomy spreadsheets that will contribute to a larger multi-lingual dictionary goal.

Works Cited

Abbinnett 2018 Abbinnett, R. The Thought of Bernard Stiegler: Capitalism, Technology, and the Politics of Spirit. Routledge, New York (2018).
Arnold and Tilton 2019 Arnold, T. and Tilton, L. “Distant Viewing: Analyzing Large Visual Corpora” Digital Scholarship in the Humanities (2019).
Bali 2016 Bali, M. “What I Like About Hypothes.is” Chronicle of Higher Education ProfHacker, Jan 13 (2016).
Barker 2012 Barker, S. “Enchantment, Disenchantment, Re-Enchantment: Toward a Critical Politics of Re-Individuation” New Formations 77: Bernard Stiegler: Technics, Politics, Individuation (Autumn 2012): 21-43.
Benjamin 2019 Benjamin, R. Race After Technology: Abolitionist Tools for the New Jim Code. Polity Press, Medford, MA (2019).
Casey 2015 Casey, M. “Why Media Preservation Can't Wait: The Gathering Storm.” International Association of Sound & Audiovisual Archives Journal, 44 (January 2015): 14–22.
Clossen and Proces 2017 Clossen, A. and Proces, P. “Rating the Accessibility of Library Tutorials from Leading Research Universities.” portal: Libraries and the Academy, 17.4 (2017): 803-825.
Dowd 2018 Dowd, D. B. Stick Figures: Drawing as Human Practice. Spartan Holiday Books, St. Louis (2018).
Field et al. 2015 Field, A., Horak, J. and Stewart, J. N. (eds.) L.A. Rebellion: Creating a New Black Cinema. University of California Press, Berkeley (2015).
Fitzpatrick 2019 Fitzpatrick, K. Generous Thinking: A Radical Approach to Saving the University. Johns Hopkins University Press, Baltimore (2019).
Fossati 2018 Fossati, G. From Grain to Pixel: The Archival Life of Film in Transition, Third Revised Edition. Amsterdam University Press, Amsterdam (2018).
Frick 2011 Frick, C. Saving Cinema: The Politics of Preservation. Oxford University Press, London (2011).
Guo 2016 Guo, J. “I have found a new way to watch TV, and it changes everything” The Washington Post (June 22, 2016), https://www.washingtonpost.com/news/wonk/wp/2016/06/22/i-havefound-a-new-way-to-watch-tv-and-it-changes-everything/.
Heftberger 2018 Heftberger, A. Digital Humanities and Film Studies: Visualizing Dziga Vertov's Work. Springer Nature, Cham (2018).
Junior Research Group 2018 Junior Research Group, Audio-Visual Rhetorics of Affect, Freie University of Berlin “Videoannotation — Relating Precisely to Audio-Visual Images” (2018), http://www.ada.cinepoetics.fu-berlin.de/en/Methoden/Videoannotation/index.html.
Keathley et al. 2019 Keathley, C., Mittell, J. and Grant, C. The Videographic Essay: Practice and Pedagogy, 2019, http://videographicessay.org.
Manovich 2001 Manovich, L. The Language of New Media. MIT Press, Cambridge, Mass (2001).
Nakamura 2008 Nakamura, L. Digitizing Race. University of Minnesota Press, Minneapolis (2008).
Ng et al. 2020 Ng, E., White, K. and Saha, A. “#CommunicationSoWhite: Race and Power in the Academy and Beyond” Special issue of Communication, Culture & Critique, 13.2 (2020).
Noble 2018 Noble, S. U. Algorithms of Oppression: How Search Engines Reinforce Racism. New York University Press, New York (2018).
Northrup 2019 Northrup, A. “Honoring the Art of the Archival Film” Hyperallergic website (2019) https://hyperallergic.com/532479/idfa-2019-re-releasinghistory/?fbclid=IwAR1h1IATzR_w6485jJt9hj3oOu9WsBjE19foUnhkcUUuzEJIhFdlH_C7oOk.
Owens 2014 
Popple 2011 Popple, S. “‘It's Not Really Our Content’: The Moving Image and Media History in the Digital Archive Age” in D. W. Park, N. W. Jankowski, and S. Jones (eds.) The Long History of New Media: Technology, Historiography, and Contextualizing Newness. New York: Peter Lang Digital Formations series, 2011: pp. 317-332.
Rheingold 2016 Rheingold, H. “Annotation, Rap Genius, and Education: Howard Rheingold Interviews Jeremy Dean” Connected Learning Alliance (Feb 8, 2016) https://clalliance.org/blog/annotation-rap-genius-and-education/.
Rockhill 2019 Rockhill, G. “Temporal Economies and the Prison of the Present: From the Crisis of the Now to Liberation Time” Diacritics, 47.1 (2019): 16-29.
Rosa and Scheuerman 2009 Rosa, H. and Scheuerman, W. eds. High-Speed Society: Social Acceleration, Power, and Modernity. Pennsylvania State University Press, University Park (2009).
Samuels 2008 Samuels, R. “Auto-Modernity after Postmodernism: Autonomy and Automation in Culture, Technology, and Education” in T. McPherson (ed.) Digital Youth, Innovation, and the Unexpected. The MIT Press, Cambridge (2008),: 219-240.
Schmutz et al. 2017 Schmutz, S., Sonderegger, A., and Sauer, J. “Implementing Recommendations From Web Accessibility Guidelines: A Comparative Study of Nondisabled Users and Users With Visual Impairments,” Human Factors, 59.6 (2017): 956 – 972.
Simon 2020 Simon, M. “AI Magic Makes Century-Old Films Look New” Wired (August 12, 2020).
Simonyan and Zisserman 2014 Simonyan, K. and Zisserman, A. “Two-Stream Convolutional Networks for Action Recognition in Videos” (2014) https://arxiv.org/abs/1406.2199.
Stiegler 2014 Stiegler, B. The Re-Enchantment of the World: The Value of Spirit Against Industrial Populism. Bloomsbury, London (2014).
Virilio 1997 Virilio, P. Open Sky. Verso, New York (1997).
Williams 1977 Williams, R. “Structures of Feeling” in Marxism and Literature. Oxford, Oxford University Press, 1977: pp. 128-135.
Williams 2016 Williams, M. “The Media Ecology Project: Library of Congress Paper Print Pilot” The Moving Image: The Journal of The Association of Moving Image Archivists, 16.1 (Spring, 2016): 148-151.
Williams 2018a Williams, M. “From ‘Live’ to Real Time: On Future Television Studies” in J. Sayers (ed.) The Routledge Companion to Media Studies and the Digital Humanities. Routledge, New York (2018a), pp. 283-291.
Williams 2018b Williams, M. “Archives of Liveness: Television Newsfilm Reconsidered” in M. G. Cooper, S. B. Levavy, R. Melnick, and M. Williams (eds.) Rediscovering U.S. Newsfilm: Cinema, Television, and the Archive Routledge AFI Film Reader series, New York (2018b), pp. 288-309.
Zamora 2016 Zamora, M. “Reading as a Social Act” Connected Learning Alliance (March 24, 2016) https://clalliance.org/blog/reading-social-act/.
de Luca et al. 2016 de Luca, T. and Jorge, N. B. eds. Slow Cinema (Traditions in World Cinema). Edinburgh University Press, Edinburgh (2016).
2021 15.1  |  XMLPDFPrint