Our moving image heritage is at enormous risk. Moving image archivists and digital repository advocates are developing solutions to these problems, but we cannot sustain interest in “preservation” without a better sense of the historical value of these materials. “Access” is not enough; new knowledge production is required in order to connect archival materials with audiences and accelerate preservation efforts. The Digital Humanities must move concertedly forward to engage visual culture with the same dedication and technological ingenuity it has brought to the study of word culture.

The Media Ecology Project (MEP) is a digital resource at Dartmouth directed by Prof. Mark Williams that enables researchers across disciplines to access moving image collections online for scholarly use. Dr. John Bell (Dartmouth ITC) has designed and built the overall technical architecture for MEP. MEP promotes the study of archival moving image collections, enhances discovery of relevant corpora within these archives, and develops cross-disciplinary research methods. These efforts help ensure the survival of these collections via new published scholarship, plus contributions of metadata and research on studied corpora back to the archival community. The virtuous cycle of access, research, and preservation that MEP realizes is built upon a foundation of technological advance (software development) plus large-scale partnership networks that result in new practical applications of digital tools. This article will demonstrate the steady progress toward these MEP goals and designs as a DH project, present reflections about the significant emergence of visual culture DH, and posit certain directions forward.

With internal support at Dartmouth and especially support from the National Endowment for the Humanities, MEP has developed several digital tools that support and sustain the creation of new networked scholarship and pedagogy about archival moving image materials. These include:

Together, these two tools support close textual analysis of moving pictures based on time-based annotations Annotations denote a start time and stop time for a subclip, a description and tags related to that clip, and attribution for its creator. This granular approach to media literacy and scholarly annotation is flexible enough to be applied to many types of research and analysis.

MEP is fundamentally 1) a sustainability project that 2) develops literacies of moving image and visual culture history, and 3) functions as a collaborative incubator that fosters new research questions and methods ranging from traditional Arts and Humanities close-textual analysis to computational distant reading. The deployment of close textual analysis is a critical aspect of MEP in developing media literacies within DH. It realizes a practical response to concerns about the acceleration of contemporary culture, the related speed-read dynamics of many audiences, and vacancies of historical and aesthetic insight as a factor of modern consumerist behaviors [Virilio 1997] [Rosa and Scheuerman 2009] [Rockhill 2019].­­­ Enabling a spectrum of purposeful and reflective considerations of the mediated past is keenly recognized to be pressing and necessary.

At the other end of the methodological spectrum, MEP's work with computer scientists has produced new tools supporting machine-reading of moving images, which produce an expansion of time-based annotations that require lucid and informed evaluation. One direction of this research produces feature extraction (isolating specific formal and aesthetic features of moving images), while another uses deep learning approaches employing convolutional neural networks to identify objects and actions in motion pictures. Data from these tools can be critically assessed and collated with the “manual” (human-produced) annotation tools mentioned above to create synthetic and iterative research workflows that “learn” across the disciplines. SAT enables real-time playback of all annotations.

While developing MEP as a rather distinctive Digital Humanities project, we have learned first-hand several key lessons about this important and emerging field. Because we are building MEP from an Arts and Humanities perspective, we recognize that our goals must always be framed to raise awareness about the significance of cultural-critical perspectives within the various institutions that we have engaged (archives, libraries, universities, grant resources, etc.).

Like many in DH, we underscore the need for collegiality and connectedness in pursuing collaborative work that depends upon openness and mutual respect as well as a balanced critical eye. This corollary of the MEP profile as a virtuous cycle is echoed in Kathleen Fitzpatrick's recently published call for “generous thinking” [Fitzpatrick 2019]. Everyone who engages in MEP is at some level working outside their comfort zones: across disciplines, across expertise, across vocabularies. In a very real sense we are engaged in “translation” work, the great benefit of which can be experimentation regarding methodologies of study but also in infrastructural designs of work-flow and output.

New research questions in relation to these workflows will literally transform the value of media archives and support the development of interdisciplinary research and curricular goals (e.g., media literacy) regarding the study of visual culture history and its legacies in the 21st century. These goals have grown to be especially timely during the publication process of this essay: the conceptual and ethical significance of re-imagining our collective purchase on historical imagination has been axiomatic to the socio-political demonstrations of both outrage and engagement that are iconic to 2020.

MEP was conceived by Prof. Williams at an early meeting of the ambitious Project Bamboo DH initiative. Bamboo is no more, but its basic principles still inform those of MEP: most digital tools have been built for the sciences, resulting in a crying need for DH resources in the Arts and Humanities. But if every institution assumes it must fulfill all recognized needs, our goals will be doomed; we must work collaboratively and in a series of progressive arcs forward to develop both traditional and emergent methodologies of DH scholarship.

The more significant early institutional affiliation for MEP was Prof. Williams' inaugural presentation to The Association of Moving Image Archivists (AMIA), which generated immediate and enduring collaborative synergies. We are poised to realize new research trajectories regarding large moving image collections as “big data,” and are developing institutional ties to vast digital collections of historical moving image materials. The notion of ecology is central to the project in several ways. Those of us who work on media history recognize all too well that the materiality of historical media is fated. These historic materials simply will not endure, but for the work to preserve and archive them. This work is especially important and timely in our contemporary media environment. Most media audiences and publics simply do not recognize the dilemma that contemporary archivists face. With the rise of social media, many people know that there are thousands of videos posted per hour on sites such as YouTube, and do not imagine that moving image culture is deeply imperiled, since it seems to be ubiquitous and unquenchable. Such an impression effaces the true condition of most historical media, which archivists are vigilantly working to preserve.

The specific platforms we initially engaged and began to bridge are 1) Mediathread, a classroom platform developed at Columbia University, that we helped develop as a research platform that supports publication of time-based annotation metadata and integration of external controlled vocabularies for tagging; 2) Scalar, a digital publishing platform developed at The University of Southern California, expanded to support import of time-based annotations and controlled vocabularies for tagging; and 3) Onomy.org, specifically designed for MEP to facilitate the collaborative creation and sharing of controlled vocabularies for annotating online media files.[1] The Media Ecology Project has been developed to sit in between and in relation to these platforms and media collections, navigating the import, export, and production of metadata across participating archival content that has been engaged by a scholar or team of scholars. In this way we can propel capacities for search and discovery across these media, and develop capacities to realize new forms of research, scholarship, and publication.

We have enjoyed the participation of multiple renowned archives in several key pilot projects essential to honing the developmental vision for MEP.[2] One of the goals in each pilot study is the scholarly development of taxonomies or controlled vocabularies that can be deployed regarding the assignment of tags and other metadata to specific media content areas. The application of these vocabularies will enhance the functional discoverability of archival content and augment efforts to produce new forms of digital scholarship.

MEP archival connections are being built on public standards such as the Open Archive Initiative and the W3C Web Annotation format. Use of these widely available standards is key to making an ecology of applications that encourage bidirectional communication and share information as peers, treating archives as not just a source of raw materials but also a consumer of new analysis and scholarship. MEP has received funding from a variety of internal sources at Dartmouth College[3], which has supported software development and metadata generation but also conference travel and stakeholder meetings at Dartmouth.

In addition to internal support within Dartmouth, Prof. Williams has been fortunate to share three significant start-up grants that have been formative to MEP development.

In 2018 Prof. Williams and Dr. Bell were honored to receive two advancement grants from The National Endowment for the Humanities for The Media Ecology Project. These grant projects had each been initially developed as demo pilots for MEP and are now poised to realize significant advances in Digital Humanities scholarship via further developments of SAT, both technologically and conceptually.

As the article has shown, MEP has a history of supporting projects exploring intersections between different methodologies of study and critical approaches to varied archival content. We have found time-based annotations and tags to be a key enabling technology that allows scholars the freedom to engage with digital media texts from multiple perspectives that can then be programmatically synthesized into a cohesive assemblage.

Among the methodological comfort zones to be negotiated in Digital Humanities, we are committed to the development of Visual Culture Studies in DH, which can produce tension with legacy approaches to DH that primarily focus on word culture alone. In addition, the field of Film and Media Studies often features attention to research methods that address and engage audiences and the reception of media texts, emphases that are less prominent in the historical study of word culture. Most important is the prominent DH tension between the traditions of “close reading” that are central to the Arts and Humanities versus the goals and practices of reading at scale that are crucial to computational approaches to vast corpora of texts under analysis. Taylor Arnold and Lauren Tilton revise the terminology regarding visual culture to “distant viewing”, which takes into account the implicit interpretive quality of acts of viewing [Arnold and Tilton 2019].

Recognizing sites of potential dissonance can help to reformulate them as sites of new inquiry and critical intervention in the pursuit of Visual Culture Studies, to produce instead a growth area for both productive research inquiry and rigorous critical evaluation. The close and distant reading/viewing dissonance of DH can be seen to work within the motivated, intentional bi-play of manual and automated annotations in MEP as a defining and iterative dialectic which allows us to better recognize an always-already evolving spiral of creative and critical exchange across manual and automated realms. Realizing a full awareness of this dynamic model may present a fundamental perspective toward progress in the emerging interdisciplinary space that is DH.

One signal theoretical turn to recommend in this realization of an ongoing and dynamic dialectic toward computer vision and machine-reading excellence is the historic approach to early cinema theory provided by Dziga Vertov's Kino-Eye or kinoglaz theory.[28] Vertov's theory famously consists of two major tenets, related yet distinct from one another. In tenet one, “kino-eye” is engaged with his great enthusiasm and devoted pursuit of new technologies of enhanced vision (e.g., the motion picture camera and related lens technologies), which represented material advances beyond human vision alone that seemed capable of both futurist and constructivist goals for enhancing society and culture. But equally if not more important is the second major tenet, “kino-edit”, which requires the rigorous study and understanding of the world and its historical processes by the artist/scientist, in order to actively interrogate, differentiate, and recontextualize what may be recognized to be merely positivist technological outputs of the Kino-Eye (e.g., racial and gender bias in facial recognition software, etc.) [Nakamura 2008] [Noble 2018] [Benjamin 2019] [Ng et al. 2020].

Vertov's theory is widely recognized in its motivated call for critical and aesthetic discernment and socio-political insight. Like many aspects of visual culture historiography, it is surprisingly applicable to not only early cinema but also to the rise of digital culture and its related epistemologies. We can anticipate that the development of new DH tools and platforms for Visual Culture Studies will necessitate sophisticated theoretical frameworks that will prove essential to 21st research and scholarship. The two tenets/steps of Kino-Eye theory are directly applicable to the MEP advanced NEH grants described above, especially in relation to the iterative workflows of manual close-textual annotations, vast machine-reading expansions in the number of those annotations, and manual curated evaluations of the machine-reading results (that then afford a new training set for another iteration of the cycle). Also significant to recognize is that these annotations are sometimes frame grabs, but primarily time-based sub-clips, the study of which can judiciously contribute to a new and contemporary re-understanding and elaboration of Vertov's elusive, unfolding, yet core concept of the moving image “interval”, tied to the organization and elaboration of movement [Heftberger 2018, 220–221].

There is more to say about the conceptual and phenomenological value of manual time-based annotations. As suggested earlier, manual annotation of visual culture is a contemplative, iterative, and essential task that mirrors more innovative annotation practices across Digital Humanities endeavors. For example, annotation platforms such as hypothes.is have inspired new scholarship that promotes the generative aspects of close reading practices in relation to networked scholarship when annotating word culture texts in their platform [Zamora 2016] [Rheingold 2016] [Bali 2016].

Indeed, part of the spirit of intervention within MEP and SAT is the process of manual annotation itself, which necessarily slows down the apprehension and understanding of moving images as aesthetically expressive media. This produces a palpable countervailing force against many contemporary viewing practices of moving image culture. For example, professors and other teachers of media history and arts can attest almost uniformly to a gradual change across generations of their students in the craft of “reading” moving image culture beyond a stencil or scaffold of factual and narrative “content”.

For media historians and artists, this mode of reception can seem as though many audiences today have been trained as pattern recognition filters, impatient for the next plainly evident delta change of attractions in the image or soundtrack, and fundamentally inattentive to basic aesthetic information provided as signal contributions to the expressive registers of the text. This results in an implicit or even explicit audience “demand” or drive to accelerate perceptual stimuli (“I've got it, move on; I've got it, move on…”) that eschews many of the fundamental temporal and spatial registers of visual culture, and moving images in particular.

This presumptive need for speed may be directly related to changing ecologies in the expansive amount of available quality mediated content, but also inter-medial changes in the patterns of consumption of time-based media (e.g., scrubbing through video, binge-watching); emergent formal practices within games and social media; and perhaps even a resultant competitive pressure for the use of one's time within and across mediascapes, e.g. [Guo 2016]. Even though these contemporary reading practices and competencies may ultimately prove to be valuable in specific contexts,[29] there is indisputable value in also learning to deepen one's capacity for better attending to the historical aesthetic practices in moving image craft and art (across the full range of expressivity), and also to better understand the value of such an investment.[30] The rise of the global “slow cinema” movement is a key index of truly widespread cultural resistance to accelerated media culture, and exists in part as a contemplative critical response to the aesthetics and underlying political economy of mediated subjectivity based upon expediency, speed, presentism, and the exploitation of a delimited attention economy, e.g. [de Luca et al. 2016].

A broader but related perspective regarding digital culture writ large, and perhaps especially the debated value of the rise of artificial intelligence (machine-reading), comes from the technological imperatives derived by Bernard Stiegler [Stiegler 2014], who has been developing a series of incisive theoretical tropes concerning the rise of digital culture as a pharmakon: at once a powerful and enticing possible remedy for specific needs and demands in a socio-political system, but also a poison if used unknowingly or improperly.

Stiegler's most important intervention regarding DH resulted from his career-changing participation in the artist-scientist collaborative Ars Industrialis in 2005, which led to his committed politics of critique that makes a concerted call to “re-enchant” the world via new and rigorous attention to history and the arts, imbued with critical literacy about them. Central to his increasingly complex and refined theories is a renewed attention to the significance of what he terms tertiary memory (e.g., the archive), an externalized and technical extension of “internal memory”, plus a call to return to an intentional and motivated anamnesis, a refusal to forget the vital importance of critical engagement with these memory deposits in the interest of sustainable cultural environments. He positions this essential work of critical engagement in relation to what Plato termed “self-care”, but jettisons Plato's dismissal of the technological. Contemporary society, deeply imbued by technology and mediation, is now, perhaps for this very reason, threatened to become history-less and therefore uncritical about itself as a control Society [Stiegler 2014] [Barker 2012] [Abbinnett 2018].[31] Conscientious and motivated engagement with the archive is an essential component of balancing and even responding to the pharmakon of drives and controls associated with accelerated digital culture and AI. The call for both a renewed ethic of self-care about societal memory and a rigorous set of practices that enable such motivated and critical engagements are directly parallel with the goals and practices of MEP.

One purposeful area of related debate in the contemporary moving image archive world circulates around the recent proliferation of high-definition “video upgrades” (4K, 60fps) achieved via deep learning methods and often applied to early silent film era footage, see e.g Denis Shiryaev's youtube channel and Simon (2020). Passions can run high when archivists rightly insist that these media entities are separate and derivative “objects”: not archival acts of preservation or restoration, not grounded in meticulous curatorial insights and not dedicated to the indexical materiality of historical photo-chemical film prints. But there may be potential to realize a Stieglerian teachable moment in relation to these experiments that “push” the capacities of digital tools in order to change the experience and affect of watching historical “cinema” texts. Wide interest about these materials in online communities may indeed represent a re-enchantment of public imagination regarding early cinema, a literal sense of re-newed awareness and interest about history and moving images--and therefore an opportunity for archivists and scholars to direct attention toward an assortment of knowledges about early cinema (including the essential work to preserve and respect its history). In other words, the “drive” (compulsion?) within sectors of the AI industry to transform historical media artifacts into dramatically enhanced localized forms of “attractions” is itself an overdetermined project regarding desire for/within the historical imagination that is worthy of much further consideration. Without immediately casting aspersions on these considerable technological efforts, the experiential enthusiasm they inspire in audiences might be recognized as a digital framework to build upon. If the “upgrade” aesthetic dynamics border on a potential for mere spectacle of the hyper-real,[32] this potential may represent a techno-cultural pharmakon that is ripe to be more fully understood and historically grounded. How best to channel the many implicit investments in history of the visual arts embedded in these twin dynamics of digital production and reception, to engage these investments and further develop them in edified and conceptually insightful contexts? There is a compelling relationship to the MEP Early Cinema NEH grant, in that the great anticipation within the archival community to experience screenings of the newly restored 68mm films shot with the Biograph camera during the earliest years of cinema (restored at 4K and 8K at the BFI and the Eye Filmmuseum) offer a substantial counter-example to deep-learning “video upgrade” productions, and may set the table for dialectical Stieglerian discourse called for here[33].

The time for this critical engagement has never been more necessary and opportune. The moving image archive world is keenly engaged in efforts to delineate and address the imperiled status of their collections [Casey 2015]. Access to archival content is becoming more foregrounded as a goal of preservation,[34] and as Giovanna Fossati points out in her standard-setting book From Grain to Pixel: The Archival Life of Film in Transition, DH is gaining momentum as scholars work to bridge the gap between digital methods, film and media studies, and media archives [Fossati 2018, 334].

The connections of MEP to existing trends in DH research are evident (see Figure 7). Formal stylometric analysis grounded in new capacities for annotation are emerging in several international centers of DH study [Junior Research Group 2018], and are conversant with the fragmentation aesthetics of many artists working with large media corpora

We have already contributed to the conversation about artists engaging with archival content initiated by our colleagues at the Eye Filmmuseum: The Sensory Moving Image Archive (SEMIA). We anticipate and welcome opportunities to build bridges toward, for example, the WJAN BOT at The Eye and also Brian Foo's “Moving Images” video project [Foo 2019]. At the same time we are inspired by and look forward to collaborating with more traditional filmmakers who work primarily with archival footage[35], and also more directly cinephilic endeavors such as scholar artists involved with the burgeoning Video Essay component of Film and Media Studies [Keathley et al. 2019].

The analytic, annotation, and presentation tools engaged to work with audiovisual materials in DH are themselves becoming more collaborative. MEP is a member of the Video Annotation Interoperability (VAINT) group that includes the makers of such tools as the CLARIAH Web Annotation tool, ELAN, Frametrail, and VIAN. While each of these tools was developed for a specific purpose, their core data all indexes back to time-based annotations. VAINT is developing a standard that will allow the tools to exchange data so scholars can, for example, directly export results from VIAN's color analysis tools into SAT and present them alongside textual commentary using SAT's integration with Scalar. Also currently under consideration by the group is a further integration with IIIF-AV, which would open up data exchange with the large set of IIIF-compliant presentation tools. This approach of common data exchange, rather than consolidating functions into a single tool, is a match for MEP's philosophy that tools addressing specific intellectual concerns can be put into conversation with one another to support synthetic, interdisciplinary scholarship.[36]

Prof. Williams and Dr. Bell have been invited to participate in a working group of the FIAF (International Federation of Film Archives) Cataloguing and Documentation Committee Task Force dedicated to Linked Open Data, in order to further investigate the sharing of FIAF Glossary of Filmographic Terms in an RDF structure. A related area of development is an endeavor to initiate a multi-lingual dictionary of film terms.[37] We have initiated work with the American Film Institute and the Women Film Pioneers Project to expand the recognition of women filmmakers in digital and online resources. Very recently MEP has been funded to work with the Distant Viewing team on developing a prototype digital resource at Dartmouth regarding the prestigious James Nachtwey collection of photographic journalism.

In continuing to realize a virtuous cycle of engagement with media art and history, we will work collaboratively and intentionally to be poised toward a spirit of critical inquiry that is engaged with innovative work at other academic and cultural institutions around the world. Our efforts to engage via DH more access to and scholarship about visual culture and moving image history presents innovative approaches to fundamental historiographic questions about media and history, and also address the danger of further losses to that history in both practical and theoretical terms. We intend the scholarship that we conduct and inspire to avoid a gloss of mere positivism in its pursuit of new research questions, and to invoke issues of the missing, the fragmentary, the occluded, the repressed, and the fugitive regarding this history.