Introduction to the Special Issue: Using Visual AI Applied to Digital Archives

As more and more museums and archives are making their image collections digitally available, new forms of knowledge and exploration of this growing mass of pictures from the past are needed. Computational techniques, which can process and help visualise tens of thousands of images, will be key to unlocking these digital archives. This Special Issue focuses on the theme of  Using Visual AI Applied to Digital  Archives. It seeks to improve the discoverability, accessibility, and use of digitised cultural archives by working at the crossroads between the humanities (including visual studies, history, and ethics), computer science, and other fields (including information and archival studies). It addresses key challenges and solutions in preservation, cataloguing, and discoverability, as well as cutting-edge digital technologies that are being applied to make visual records more accessible within the cultural heritage sector. These technologies include artificial intelligence (AI) and its subsets: machine learning, computer vision, and automated information retrieval, extraction, and labelling.

There are multiple ways in which AI can help unlock digitised (i.e., paper to digital) and born-digital images. AI can be used to automatically create metadata when metadata is otherwise missing or relies on outdated and potentially harmful language (e.g., racist language in the case of images produced during the colonial period). In addition, computer vision can identify individuals and objects across huge numbers of visual images. This includes the identification of sensitive and violent content, which can be useful when flagging such material and adding content warnings.

At the time when libraries, archives, and museums are striving to make their collections more discoverable and accessible to diverse audiences, AI therefore offers unprecedented opportunities to process and analyse collections at scale — but it also carries risks. In particular, AI can magnify existing issues in collections. In the case of archives with potentially harmful language used in metadata, the risk is that an AI system will reproduce and amplify the contentious data used for its training. Computer vision can also be viewed as unethical when applied to photographs obtained without the informed consent of those they represent (e.g., medical images of patients with diseases and physical deformities). Keeping humans in the loop is therefore essential to apply responsible AI to visual archives.

This Special Issue is born of the EyCon (Early Conflict Photography 1890-1918 and Visual AI) project[1], funded by AHRC in the UK and LABEX in France. Using visual AI technologies, the EyCon project aims to unlock the potential of digital archives, focusing particularly on images of war and colonial violence. AI applications to sensitive historical archives raise ethical questions regarding data ownership, consent, and cultural sensitivity. Furthermore, transparency in how AI algorithms are trained and used within archival contexts is essential for accountability. This includes consideration of the sources of data, the methodologies used in AI applications, and the potential limitations or biases inherent in these systems. Balancing these sometimes-conflicting imperatives will continue to pose a challenge for researchers, users, and professionals in the GLAM (galleries, libraries, archives, and museums) sector.

This collection of articles offers an interdisciplinary forum to explore innovative technologies, methodologies, and practices applying visual AI to digital archives — and the digital humanities and social sciences more broadly. With the advancements in text-recognition technologies applied to digitised written materials, image archives are facing increasing pressures from users to return similarly accessible results. However, the technologies to enable this accessibility and discoverability are still at an experimental stage.  More collaborations between institutions and disciplines are urgently needed to unlock these visual digital archives.

In their article “Augmenting Access to Embodied Knowledge Archives: A Computational Framework”, Giacomo Alliata, Yumeng Hou, and Sarah Kenderdine demonstrate that interdisciplinarity and collaborative work between archivists, computer scientists, artists, and other stakeholders can reveal novel strategies for archival exploration. In their work, they employ a computational framework incorporating machine intelligence, archival science, and digital museology to examine the link between living heritage archives and embodied knowledge experience. Through their examination of two cases (the Prix de Lausanne archive, a collection of 50 years of video recordings of dance performances, and the Hong Kong Martial Arts Living Archive), the authors conclude that computational capacities can make archives easier to discover and analyse. Indeed, algorithmic tools can be used to extract archive-specific features and create meaningful representations of human bodies.

While AI offers unprecedented opportunities to unlock digital archives, it also brings new challenges. Jonathan Dentler, Lise Jaillant, Daniel Foliard, and Julien Schuh explore the ethical dimensions of digitising sensitive war archives from colonial pasts in their article “Sensitivity and Access: Unlocking the Colonial Visual Archive with Artificial Intelligence”. From a large database of sensitive visual materials from colonial conflicts gathered within the EyCon Project (Early Conflict Photography 1890-1918 and Visual AI), they propose an experimental multi-modal computer vision tool for analysing such sensitive archives. Dentler et al. contend that critical and transparent multimodal AI can improve responsible access to colonial archives for researchers and the public. Their proposition demonstrates the importance of developing research tools that check biases, particularly around sensitive historical events such as colonial expansion and violence.

Lise Jaillant and Katherine Aske continue to explore the ethical dimensions of AI in the archives sector in the next article, “AI and Medical Images: Addressing Ethical Challenges to Provide Responsible Access to Historical Medical Illustrations”. Like colonial images, medical pictures are often very sensitive — for example, when they show images of people with diseases who were photographed without informed consent. On the one hand, AI can improve access to digital images while also safeguarding fragile materials in print format. On the other hand, AI also raises concerns about privacy and ethical use when dealing with potentially harmful content. Through interviews with 10 archivists, librarians, and researchers in the UK and US, the authors demonstrate that improved access to medical illustrations is essential to producing new knowledge in the humanities and medical research and to bridge the gap between historical and modern understandings of the human body. They also highlight the importance of appropriate metadata, which can be enhanced with AI tools to improve discoverability and facilitate access to these archives in an ethical way. The article concludes by making recommendations for cultural heritage institutions to help them find the right balance between providing access for research and education and protecting vulnerable audiences from potentially traumatic encounters with such sensitive images.

Captions often enhance the storytelling experience by providing additional context or insights into illustrations or photographs. In their article “Capturing Captions: Using AI to Identify and Analyse Image Captions in a Large Dataset of Historical Book Illustrations”, Julia Thomas and Irene Testini take the less-trodden path of examining the significance of captions from historical books published between the sixteenth century and early twentieth century. Their dataset consists of over a million illustrations from 68,000 volumes in the British Library's collection, which were digitised by Microsoft. By interrogating the captions of historical book illustrations at such a large scale, the authors aim to uncover how captions relate to the content of the pictures that they accompany. They also explore how the wider dialogue between word and image characterises illustration as a mode of representation. The authors propose “indeterminacy” — what is not clearly known, defined, or fixed — as an analytical concept for examining the relationship between word and image. Indeed, the words of a caption are neither part of the image, nor do they fully belong to the body of the text: something we rarely notice in our everyday encounters with captions.

In the final article, “Deep Learning for Historical Cadastral Maps and Satellite Imagery Analysis: Insights from Styria's Franciscean Cadastre”, Wolfgang Thomas Göderle, Fabian Rampetsreiter, Christian Macher, Katrin Mauthner, and Oliver Pimas examine the significance of AI for enhancing the interpretation of nineteenth-century historical cadastral maps of the Franciscean Cadastre in the province of Styria in Austria. This article analyses outcomes of a 2022 Austrian Science Fund (FWF) project, RePaSE (Reading the Past from the Surface of the Earth). Cadastral maps offer fascinating insight into land use, property boundaries, and, more broadly, economic and social histories. AI can be useful to fully exploit the potential of these historical records, which have not been sufficiently exploited.

With the rapid deployment of AI in the GLAM sector, these articles offer the opportunity to pause and reflect on the opportunities and challenges offered by artificial intelligence. This special issue is therefore an invitation to work collaboratively, across disciplines and sectors, to address these challenges and fully embrace the potentialities of technology to make visual archives more accessible in an ethical way. Future avenues of research include the application of generative AI (GenAI) to archives. Commercially developed tools such as ChatGPT often do not work well on historical texts, largely because they have been trained using contemporary data. Developing better GenAI tools requires collaborations between humanities scholars, archivists, and computer scientists. Applying GenAI to visual archives rather than text is another exciting research avenue. In May 2024, Google announced the launch of Ask Photos, an experimental add-on feature relying on its GenAI tool Gemini. Instead of scrolling through pages on Google Photos, or using keywords, users can ask for what they are looking for using natural language. Similar technologies could be trained on historical visual archives and modified to take into account the unique features of these collections. AI will continue to transform the archive sector. As the authors in this special issue collectively show, humanities scholars and GLAM sector professionals should not be left aside by this revolution.