CBIR focuses on the retrieval of images in an image archive or database from a given query, which can be either textual or provided as an example image (query-by-example). The first content-based retrieval systems came up in the 1990s and included systems like QBIC [Flickner et al. 1995] and VIRAGE [Gupta and Jain 1997], which were able to query a multimedia database by example images. Aside from finding similar images to a query, typical CBIR problems are the retrieval of objects, concepts and scene properties [Zhao et al. 2017] as well as image classification [Druzhkov and Kustikova 2016]. CBIR is thus strongly interlinked with the fields of computer vision and machine learning. Research in CBIR focuses on the retrieval of patterns from various types of images, including medical images, remote sensing imagery and user-generated content (e.g. in personal photo collections and in social media). Broad surveys on the field of CBIR are provided by Datta et al. (2008) and more recently by Zhou et al. (2017). Automatic film analysis makes intensive use of content-based image analysis. Even though the study object are films and videos, many visual analysis problems in film analysis (e.g. object detection in movies [Flückiger 2019], visual composition analysis [Mitrovic et al. 2001] and the recognition of shot size and shot type [Zaharieva et al. 2010]) can be performed on individual images without taking the temporal dimension into account.

Closely related to CBIR is Content-Based Video Retrieval (CBVR). Typical problems in CBVR are the retrieval of similar videos to a query as well as temporal segmentation of a video in shots and scenes [Cotsaces et al. 2006], action and activity recognition [Cheng et al. 2015] event recognition [Tzelepis et al. 2016], motion retrieval [Zeppelzauer et al. 2011a], video summarization [Del Molino et al. 2016] and video browsing [Schöffmann et al. 2010]. Content-based video retrieval integrates many techniques from CBIR and extends them with temporal analysis (e.g. motion analysis or object tracking). As a first step in CBVR, the video stream is usually partitioned into smaller units, such as key frames, shots and scenes [Rui et al. 1998]. Next, information at the frame as well as the shot level are extracted and aggregated for retrieval and classification. Typical types of video data include sports videos, news videos and surveillance videos. Beyond these types of motion images, there is a large corpus of literature on the automatic analysis of movies, documentaries and archive film [Ewerth et al. 2009] [Zaharieva et al. 2010], which has developed in parallel to visual research in the digital humanities. A frequent pattern that can be observed from the literature is that a broad range of filmic concepts has been investigated already in the computer science, such as film rhythm, tempo, style and narrative structures [Adams et al. 2001] [Adams et al. 2002] [Choudhary et al. 2019] but with rather simplified definitions of these concepts.

CBAR has developed similarly to CBIR in the early 1990s. Pioneering work is presented by Wold et al. (1996), with their development of an audio retrieval system called “Muscle Fish” that was able to automatically distinguish between different types of sounds (see Pfeiffer et al. [1996]). Since then, there has been rapid development in different directions, such as audio segmentation, music information retrieval and environmental sound retrieval. Furthermore, the analysis and recognition of speech, which has developed separately, starting already in the 1950s, can be considered one sub-branch of CBAR [Juang and Rabiner 2005]. Compared to the visual modality, the acoustic modality is under-represented in automatic film analysis. One reason is that the acoustic modality alone is often not sufficient for film analysis. Acoustic information is, however, highly useful when combined with visual information in a multimodal approach [Snoek and Worring 2005]. Information from soundtracks can significantly help to classify film genre, scene types and film style [Choudhary et al. 2019]. Furthermore, the analysis of the affective dimension of film requires acoustic and visual clues [Soleymani et al. 2009]. Finally, film montage may exhibit correlated patterns between the acoustic and visual track [Zeppelzauer et al. 2011b].

From the brief review of the field of MMIR, we can observe that a large body of automatic analysis approaches exists that would (at least in theory) be applicable to filmic content investigated by film scholars. Especially in the early years of MMIR, however, these methods were not powerful enough to model the complex semantic concepts investigated by film scholars. Today, automatic visual (and audio-visual) analysis is well advanced. Developments in machine learning and especially in deep learning in the last decade have made significant progress in the extraction of information from audio-visual media and in modeling semantic relationships. Certain problems which appeared unsolvable 15 years ago can now be solved automatically with almost human-level performance [He et al. 2015]. This has therefore changed the situation for the digital humanities fundamentally. MMIR has reached a level of maturity that can provide a real added value for film scholars and the digital humanities in general. Many previously unanswered research questions from film scholars are waiting to be (re)investigated by the support of automatic film and video analysis methods.

Using the plural form in histories is more than just acknowledging the democratic attitude that writing history enforces. It is a reaction to the fact that it even lacks an agreement about the same temporal dimension when researchers of the field try to systematize the history of the use of digital technologies in film and media studies. In Germany, for instance, Julian Sittel (2017) states that film studies lags behind when it comes to the adoption of digital humanities methods, while Patrick Vonderau (2017) suggests that digital humanities is the continuity of a certain way to use digital technologies that has long been superseded by other usage types. This claim is illustrated by an analysis of the use of digital technologies in film and media studies that distinguishes three consecutive phases between 1985 and 2005: The first phase produced tools to manually annotate and segment movies. The second phase introduced database-driven quantification of annotated and classified units in moving images. The third phase gives testimony of a repurposing of the goal of databases themselves. They now shift from facilitating statistical analysis to becoming a metaphor and facilitator for the situational and explorative organization of content of any kind in research in the logic of the collage. Databases are more conceived of as an expression of an abstract notion of media practice instead of just being a tool to query and count [Böhme et al. 2012].

In Vonderau's perspective, digital humanities is just an attempt to perfect narrow use of digital technology that was predominant in the second phase. It lacks a broad theoretical framing for digital transformation and technology as such and thus misses new emerging paradigms of technology use. In Sittel's line of argument, however, digital humanities are the clear sign of a more fundamental shift; a shift that reconfigures the entanglement between theory, technology and research – with respect to the first paragraph the media/message relationship – itself and that, accordingly, cannot be integrated into film and media studies appropriately. Claims such as those about “the end of theory” [Anderson 2008] or the “fourth paradigm” [Hey et al. 2009] come to mind here. In summary, different notions of technology instigate different historical narratives as well as different methodological portfolios and needs. Each of these configurations are, furthermore, linked to struggles for disciplinary autonomy but also to debates about the internal structure of the field itself. This is most obvious when Vonderau declares that actually the main object of film studies – the analysis of movies – mostly disappeared in film studies with the arrival of digital technologies, while Chavéz-Heras (2012) announces that now, with digital tools, film analysis is reborn.

Such media-reflexive preoccupations have caused a lot of alienation between film and media studies on the one hand and digital humanities on the other.[2] It would, however, be misleading if digital humanities just considered it an internal issue of film and media studies. In fact, the digital humanities have cultivated their own version of the above-mentioned tension, a tension that recently created a schism within in the field itself. Since 2014, concepts such as the post-digital humanities [Berry 2014] or the critical digital humanities [Grimshaw 2018] [Dobson 2019] emerged within debates on digital humanities. The former relates to the methodological inconsistencies behind the term digital methods, while the latter stresses cultural conditions and impacts of algorithmic meaning making processes. In opposition to these branches, in 2019, the computational humanities research community spun off the field of digital humanities, preceeded by debates that digital humanities had lost its core – humanities computing – and that rather “techie” topics are not sufficiently represented [Elwert 2019].

This example is just the entertaining expression of an open space of entanglements between digital technologies and doing research. In film and media studies these include approaches such as artistic research [Ruszev 2018], speculative computing [Drucker 2009], statistics and stylometry [Casey and Williams 2014] [Baxter et al. 2017], digital surrealism [Ferguson 2015] [Ferguson 2017], or multimodal storytelling and videographic criticism [Erlend 2012] [Keathley et al. 2019]. The case of videographic criticism or the video-essay, as it is sometimes called, is especially interesting because it demonstrates how an object of research becomes the means of analysis itself. It investigates movies by producing short movies and follows the media reflexive idea that it might be best to represent facts of a certain modality within that same modality. Although not computational in a strict digital humanities sense, the video essay clearly is interpreted as a product of digital technologies and an established method of film studies research [McWhirter 2015]. Together with the above-mentioned dispute about the obsolescence or the revival of the analysis of movies, the whole episode demonstrates very well that questions raised by digital technologies are not just questions of new methods, and even less questions of computational or statistical methods. They are, with similar importance, questions about the changing status of research objects within a changing, partially digital, environment. Accordingly, the challenge is not just to define better or new methods, but also to develop a new sense of research objects and research ecologies – all at the same time.

With this in mind, it is necessary to critically re-approach the notion that film and media studies lag behind. The holistic angle on digital research that not only considers new methods for known objects but also new cultural embeddings for known as well as unknown objects, suggests another kind of endeavor. In this respect it is a meaningful observation that in the history of digital humanities, fields with clearly definable and established research entities – the edition, the textual source and the artifact among others – were the most influential. It goes without saying that the analysis of our partially digital environment and its objects enforces to also apply digital humanities methods in a stricter sense of the term and cannot be carried out comprehensively without them. In consequence, film and media studies researchers that might not have welcomed these methods, due to methodological reasons, in the first place are, nevertheless, beginning to open up as has been mentioned before. The challenge, both for the digital humanities and for film and media studies, remains to integrate perspectives of method, object and ecology without creating a hierarchy among them; be it methods for the digital humanities or objects and ecology for film and media studies.

It was a striking feature of Vonderau’s periodization that it ended around 2005, which is more or less the same time in which the digital humanities began to aspire. Both histories, the one that ended in the article's narrative and the one that started with a companion, were motivated by the intent to separate one research field from the other. The first history tried to show that the newly emerging DH was in some sense always already part of the established field of film and media studies. The second history presupposes that only by creating a new field, the old will actually become the new. Comparing the history of digital humanities methods in film and media studies with the history of film and media studies topics in the digital humanities clearly shows that both lines of arguments get it wrong, because in the growing number of encounters between film and media studies and digital humanities, a new reality emerges that is both post-humanistic and post-digital [Cramer 2014].

Certainly, film and media studies have to engage more profoundly with the methodological part of statistical and algorithmic approaches. This has slowly begun to happen throughout the last years, as projects such as the large-scale video analytics project [Kuhn et al. 2015], activities of the scientific networks Digital Cinema Studies[3] or New Directions in Film Historiography[4] or projects like FilmColors [Flückiger 2017] and Kinepoetics[5] suggest. At the same time, it appears to be equally important for the digital humanities to reflect on their rather broad conception of technology, in order to continue to make substantial contributions to the reconfiguration of research by digital technologies. A fuzzy reference to the “big tent” [Terras 2013] or the “big humanities” [Svensson 2016] will not suffice in this respect.

This branch of DH is mostly concerned with the digitization of cultural artifacts. This also includes the modeling of analog artifacts, i.e. the formal representation of a cultural object. This formalization is an important prerequisite for digital preservation and dissemination strategies, but also for computational analyses of the material. For the formal representation of film, the fundamental question is: how can we model a time-based and multimodal medium such as film to be represented in discrete categories? As this is obviously a demanding and complex task, it is important for scholars to be clear in advance – i.e. before the actual modeling – about what they want to investigate: Are setting and scene boundaries relevant for later analyses? Are they interested in constellations of characters or the use of leitmotifs? Or do they want to take a closer look at the dialog level? Due to the modeling effort involved, it is hardly possible in practice to model everything pro forma, so a prior research question is indispensable for the digitization and modeling of film and video.

As a consequence, it is rather challenging to create film and video resources that can be reused by a variety of scholars, as the research questions (and modeling implications, cf. Unsworth [2002]) vary widely and as there are hardly any established annotation standards as known for other types of media, for instance the Text Encoding Initiative (TEI) or the Music Encoding Initiative (MEI).[6] Further challenges are raised by born digital videos, for example YouTube videos or Twitch.tv streams, as specific concepts for offline storage and potential long-term preservation are needed that lead to open copyright issues and new requirements for digital infrastructure. In this special issue, Eva Hielscher contributes to the topic of modeling and representing films by presenting an extensive case study on video annotation for a corpus of “City Symphonies” in her article titled “The Phenomenon of Interwar City Symphonies: A Combined Methodology of Digital Tools and Traditional Film Analysis Methods to Study Visual Motifs and Structural Patterns of Experimental-Documentary City Films”.[7] Another article in this special issue is on “Matching Computational Analysis and Human Experience. Performative Arts and the Digital Humanities”.[8] Here, Jan-Hendrik Bakels et al. investigate the combination and mediation of experience-based and data-driven analytical procedures in film studies. They transfer common film analytical vocabulary into a machine-readable ontology to set up a systematic annotation routine for films that can be visualized and queried.

Digitization of films and videos is also important for film heritage institutions, as it helps to collect filmographic information from diverse sources by means of specialized search functions, OCR or image recognition among others. This can also support digital archives with “mining” their objects, for example finding certain subjects/topics/people etc. Therefore, archives are excellent agents for advancing the standardization of filmographic metadata, for instance by benefitting from experiences with metadata standards such as EN 15907 (see Heftberger [2014]: 143). Another key task here is the presentation of collections (or parts thereof) online in an appealing, sensible and effective way. Institutions are fully aware that building good interfaces is critical, but there seem to be many obstacles still in their way. Insufficient financial resources might be one of those obstacles, being stuck in building traditional entry points to collections may be another one. Masson et al. provide a deep and comprehensive discussion of traditional and new paradigms in this area in their article titled “Exploring Digitised Moving Image Collections: The SEMIA Project, Visual Analysis and the Turn to Abstraction”.[9] Additionally, they are able to substantiate this discussion by introducing the SEMIA platform, a convincing example for how adequate interfaces might be designed.

In the past film archives, for the most part, differentiated between technical expertise and work with content. However, a greater permeability between the areas of responsibility is required in the future in order to comply with the abovementioned challenges. It is already the case that many tasks are no longer addressable by traditional roles. Digital data management, for instance, cannot be restricted to just one area of material (e.g. films, photos or documents) and metadata must be managed across the boundaries of collections in order to use technical infrastructures and personnel resources in the best possible way among other reasons. The strategy must therefore be twofold in the future: Teaming up with researchers and artists as well as staffing their places with technically savvy people who at the same time understand archival objects and are interested in exhibition. Digital humanities as digitized humanities have organized around such entanglements for years and may, therefor, provide valuable points of reference for the field of digital film studies.

This second category is oftentimes also referred to as computational humanities, highlighting the aspect of empirical analyses by means of statistical procedures and machine learning techniques. In recent years, we have seen a gradual extension of the established formal-stylistic analyses [Salt 1974] [Tsivian 2009], which mostly relied on the lengths and frequencies of shots within a movie. Going well “beyond shot lengths” [Burghardt et al. 2016], a whole series of studies have been carried out recently, for example to examine the color dimension [Burghardt et al. 2017] [Flückiger 2017] [Pause and Walkowski 2018] [Pause and Walkowski (forthcoming)] or the dialogs of films [Bednarek 2018] [Hołobut et al. 2016]. Another branch of work in this category can be found in the area of visual analytics [Burghardt et al. 2018] [Hoyt et al. 2014] [Kurzhals et al. 2016], as visualizations can greatly facilitate the analysis of visual time-based media such as film and video [Manovich 2013].

This special issue features two numerical humanities studies that investigate the textual dimension of series and movies. In their article on “The Stylometry of Film Dialogue: Pros and Pitfalls”,[10] Agata Hołobut and Jan Rybicki analyze 300 film dialogs by means of stylometry and sentiment analysis to detect patterns of similarity and difference between screenwriters and/or a priori IMDB-defined genres. Joanna Byszuk goes in a similar direction with her article called “The Voices of Doctor Who – How Stylometry Can be Useful in Revealing New Information About TV Series”.[11] More concretely, she uses stylometry to examine the changes driving the development of the "Doctor Who" series from a character-oriented to a showrunner-oriented series.

As part of the numerical humanities, deep learning approaches are also increasingly being used in the sense of “deep watching” [Howanitz et al. 2019] or “distant viewing” [Arnold and Tilton 2019], in order to automatically segment films and identify objects or people. In this special issue, Barbara Flückiger and Gaudenz Halter contribute an article on “Methods and Advanced Tools for the Analysis of Film Colors in Digital Humanities”,[12] which provides an excellent example for numerical humanities, showcasing a sophisticated deep learning approach to investigate color usage in film. Another deep learning example is provided in the article “Automated Visual Content Analysis for Film Studies: Current Status and Challenges”,[13] by Ralph Ewerth et al. The authors provide a thorough survey of approaches and tools for automated video analysis to support quantitative and qualitative analysis in film studies.

For digital film studies, the digital humanities as numerical humanities mostly represent methods that try to automate the identification of established entities and units within film studies research. By doing so, they mimic analytical methods of the field, but on a much larger scale than would be possible by means of qualitative analyses of – typically – just a few films. In recent time, however, there are also new approaches that aim at a semi-automatic and, thus, more interactive analysis which seem to prioritize exploration instead of extraction. This re-evaluation of priorities may well open-up the perspective for contributions to a more diverse understanding of numerical and computational approaches to the study of films.

This category describes humanities scholars’ broader interest for the inclusion of phenomena that emerge from the “digital culture.” In the field of film studies, this might, for instance, entail the abovementioned investigation of born digital videos, which very often also brings with it a dimension of mass communication and thus big data: either, because a multitude of videos are available, or because there is additional mass communication channel available, e.g. in the form of comments about a video (reception aesthetics). It also includes an examination of digital aesthetics as well as of new forms of production and reception in an increasingly digitized, post-cinema film culture. Such changes inevitably raise theoretical questions that, at the same time, are necessarily linked to new research practices. Due to the sheer volume of digitally available material, a serious opening of the humanities towards digital culture obviously requires an increased integration of digital methods into scholarly practices as well.

From the perspective of film studies, the central challenge humanities of the digital imposes, is the need for a holistic instead of a pragmatic integration of digital methods, not the least because these methods have become methods of production and analysis (film essay) at the same time. A central task in this context is to reflect on the changing relationship between research instrument and research subject, to bring together quantitative and qualitative methods, to find ways to mediate between metrization and interpretation. It is important to demonstrate that approaches of data-driven research can be particularly productive in the field of film and media studies when they are conceived of as tools within the framework of a diverse practice (cf. Antonijevic [2015]) or mixed methods [Pereira 2019]. In this setup, digital methods are part of a practice, but not its overarching theme. One example for such a more praxeologically oriented approach to the integration of digital methods in film studies has been recently systematized under the term scalable viewing [Burghardt et al. 2018] [Pause and Walkowski 2020].

In this context, the “Humanities of the Digital” provide theoretical foundations for new notions of the digital that have the potential to make digital methods more accessible for film studies research. Such notions root deeper in the tradition of the humanities, while at the same time highlighting the necessity and added value of algorithmic strategies. In this special issue, Sarah-Mai Dang contributes to this third sub-field of digital humanities by presenting an article on “Unknowable Facts and Digital Databases: Reflections on the Women Film Pioneers Project and Women in Film History”.[14] More concretely, she discusses the epistemological function and the scholarly implications of databases in film and media studies.