Imagine that you are a researcher starting a Ph.D. project or a new book. In addition to identifying a topic, conducting a literature review, and fine-tuning your research question, you will also need to navigate an extensive archive of millions of documents, large parts of which are yet to be cataloged; some collections have been opened to researchers only in recent decades and the digitization of the archive is still at its earliest stages, making a keyword searchable corpus unavailable.[1] This situation is the reality of historical research in the Ottoman archives at the time of writing this article. In addition to the challenges of accessibility of the archive, there is also a fundamental question of script literacy; Ottoman Turkish (OT) is written in Arabic script that most Turkish speakers today cannot read. To what scholarly apparatus would historians of differing skill levels turn for doing their work — annotating and transcribing for recall, or simply combing through the most relevant sections of the archive — when these documents are printed in a writing system that is foreign to their contemporary script habitus? It might seem that we are speaking about a many century's old historical situation, but in the case of OT, archival documents created as little as one hundred years ago are not accessible to speakers of modern Turkish who have not received a specialized education in OT.

In Istanbul, where the central archive of the Ottoman State and several of the most important research libraries in the field of Ottoman studies are located, there is an informal market of hireable transcribers to meet the demand for transcription of OT historical documents to Latin script modern Turkish. Consisting mainly of students and graduates of departments of history and Turkish literature, the informal labor of this market offers its services to scholars who do not have the time required to transcribe their documents. In the absence of extensive digitization of archives, these transcribers also provide on-site research services for their individual clients, many of whom do not live, or work regularly, in Istanbul.[2]

As the digitization of Ottoman-era archives is underway,[3] our article concerns one approach to increasing searchable (and computational) access to them. We discuss experiments that we carried out by automatically transcribing a small amount of the holdings of the larger Ottoman archive: two Ottoman Turkish periodical series from the Hakkı Tarık Us (HTU) Digital Repository. With some 1,500 periodicals and an estimated number of 400,000 pages, the HTU online archive is perhaps the most comprehensive digital collection of the Ottoman Turkish press from the 1840s to the 1920s. Originally the private library of Mr. Hakkı Tarık Us (1889-1956), the digitization (2003-2010) of this collection was a collaboration between Beyazıt State Library in Istanbul and Tokyo University of Foreign Studies. Despite being online for a little more than a decade, searchability of the HTU documents is still unavailable.

Whereas some newspaper mass digitization projects in the world were “launched under circumstances in which searchability was not a priority” [Salmi 2021, 45],[4] it is not known to us if this was the case with the HTU corpus or whether OCR-based text creation was simply technically impossible at the time, or both. In the mid- to late-2000s, the HTU platform digitized images in DjVu format — a format that offers a corresponding text layer for each image — the choice of which presumably was to leverage the linguistic skills of literate Ottomanists to transcribe the documents into modern Turkish.[5] Unfortunately, this format was already outdated at the time [Baykal 2011, 206–8]. Although the file format would have allowed for the creation of a text layer for the HTU as early as 2010, the promise of a text searchable OT corpus remains to this day an unfulfilled one.[6]

You do not need to be an Ottomanist, of course, to confront such a problem with unsearchable digitized archives; the same could be said of other digital collections in many other world languages: medieval manuscripts, letter collections, miscellanea archives. It is important to note that both digitization and OCR are historical processes with differing degrees of technical development and variable success rates across world languages. In the case of mass digitization of periodicals in the nineteenth century US context, we have been encouraged to “[take] dirty OCR seriously” [Cordell 2017]. This is not only because early digitization and OCR methods did not produce high quality text and it is unlikely that either process will be carried out again, but also because no matter what methods of studying imperfect corpora are devised in the future, understanding the layered histories of digitization and text creation remains a key element of digital historical inquiry.

For languages written in Arabic script from right to left, OCR is a complex process in which research has made progress, but is yet to become fully efficient [Romanov et al. 2017].[7] As OCR is an algorithmic process in computer vision that transforms images of letters and words — by many iterations of guessing, ranking, and even human intervention and correction — into digital text, the more intervention that is required the more labor-intensive the process of text creation becomes. Additionally, the more the features of the language that one works with differ from the Latin-based scripts on which OCR was first developed, the more development and research into new systems are required. At the time of writing this article, however, the situation of historical OCR has already started to change radically. This development is mainly thanks to the increased accessibility of user-centered neural systems for transcribing handwritten archives, and by extension for printed cursive scripts, such as Arabic or Ottoman Turkish, in which it can be difficult for a computer to determine where one letter ends and another begins. Some of the current generation neural systems’ text recognition abilities go beyond simple character recognition, which has been the main challenge OCR faces with handwritten texts and cursive scripts such as Arabic — due to the fact that some letters are connected to each other — to allow for larger chunks of the text, such as words or whole lines, to be recognized as units. In this paper, we adopt one such system for automating the transcription of printed OT texts, the platform known as Transkribus, and we argue that not only are new approaches to OCR beneficial in opening up historical archives, but also that with those approaches comes an evolving role for human effort and new, concomitant challenges for working with such machine-created text.

Our choice to apply handwritten text recognition (HTR) technologies to non-Latin script digitized historical periodicals is an intentional one, with the purpose of creating a workable access point to Ottoman historical materials. The long term goal of our research is to generate general HTR models for text creation from this periodicals collection that will facilitate its use in computational research. In this article we discuss some of the initial challenges we have faced, as well as some of the promising results that we have achieved.

As interest turns to a multilingual DH, the accessibility of archives and modes of participatory text creation, we find the Ottoman Turkish example to be a thought-provoking case study since it links problems which the GLAM sector has traditionally confronted with other problems not typically found in Western language archives: text directionality and script, in particular.

In our paper we have discussed ongoing research into text creation via automatic transcription in order to bring OT into dialogue with analytical modes of the textual digital humanities. The limited development and application of OCR methods compatible with Arabic script languages has no doubt been one of the rate determining steps, not only in the development of textual corpora, but also in the attendant language technologies that support the use of such corpora. To wish — in 2022 — for basic keyword search capacity within digitized media of the nineteenth and early twentieth century might seem to some as a rather old fashioned request, and yet for Ottomanists, it is an actual one. Limited access to digitized versions of the archive, a lack of language- and script-specific OCR in addition to a lack of scholarly infrastructure in the OT situation have meant that scholars have been slow to adopt digital methods.

The landscape of Arabic script languages is changing rapidly, especially with neural automatic transcription systems, as they can accommodate cursive-like scripts and typefaces. In order for these systems to perform well, however, they will require training on many more domains and spatio-temporal variants of the language and handwriting styles. In the case of printed materials, training on different typefaces and printing conventions will also be necessary.

A move to new transcription methods for OT will not mean the total automatization of the transcription process, removing it from scholarly labor; instead it will require a reorganization of those efforts from informal to somewhat organized one. We do not expect that people will stop transcribing small amounts by hand for specific purposes, but we do suspect that it will change the way that large text creation projects work, as well as how archives create finding aids as digitized material becomes available.

In the specific case of printed periodicals in OT, in order to achieve the goal of enabling keyword searching and more complex forms of modeling and analysis of OT textual content via HTR produced text, a multi-pronged approach will be required. First, a team-based approach to transcription for the purposes of the creation of ground truth will be required (or even some crowdsourced, community-based approaches that have yet to be defined). Such a community effort might be achieved by sharing the model generated for the HTU print collection publicly, although how the endeavor would scale remains to be seen. Second, even though the platform-as-service model of the READ COOP may provide scholars with a variety of public models, the computing resources that allow these to run might exclude scholars and archives without the resources to buy in. It may be that only open models provide a more sustainable solution. Third, research is needed not only to recognize the Arabic script that is used in the specific genre discussed in this article — periodicals — but also, as in many other language situations, for structure recognition and image extraction that would allow for formal modeling of the resultant periodicals as research objects.

We are not experts in the histories of all writing systems, but we believe our results may be applied to other language situations that have changed, or have used multiple, scripts over time: Aljamiado, Azeri, Bosnian, Crimean, Judeo-Spanish, Malay and Wolof, just to name a few. Our research with automatic transcription of periodicals printed in the Ottoman Empire during the mid nineteenth to early twentieth century has important implications for multi-script archives of nation-states located in the borderlands of empires and religions. After all, language reforms forged new modernized versions of languages such as Hebrew and Arabic while maintaining their original scripts, but in other places writing systems were changed altogether. As we described in depth above, in Turkey the Arabic script of OT was abandoned in favor of a modified Latin alphabet. The political disintegration of states at other moments of the twentieth century, in particular the former Yugoslavia and Soviet Union, led to fragmentation of similar languages into script-divergent varieties that both straddle current political borders and co-exist with older variance found in archives. If script change comes about in moments of radical political change enacting ruptures in archives, it also seems to map onto a slow globalizing of the Latin script — and not without significant political debate — in different countries around the world around the turn of the twenty-first century. Turkmenistan adopted a Latin alphabet in the late 1990s as well as Kazakhstan as recently as 2017. This geopolitical context raises the question of what will become of digitized archives in such locales — printed and handwritten — and adds totally new contours to the notion of the “great unread” in nations already cut off from the scripts of their past, as well as those that may be facing a phasing out of old alphabets. Future developments in digitization and text creation from borderland archives — if they are to happen — need to take into account not only the automatic transcription of written language, but also the various writing systems in which those languages will be expressed.

There are a number of terms that are intrinsically related, and some of which we use interchangeably in this paper, that are central to our automated recognition of OT project.

Transliteration is the substitution of the characters of a script by those of another alphabet. In an ideal transliteration, each character of the source script will be represented with only one and always the same symbol in the target writing system, creating a one-to-one mapping, or graphemic correspondence, between scripts.

Transcription, on the other hand, is the representation of the sounds of an alphabet with the characters of another script. A transcription attempts to offer an accurate phonetic rendering of the source language even when this violates the graphemic correspondence between scripts. This means that, depending on the textual context, a polyphonic character- a letter that represents more than one sounds- may be substituted by several different letters in the target alphabet. In some cases, the reverse might be true. In the instance of OT transcription to modern Turkish, transcription also involves representing sounds, vowels to be more specific, which are only implied but are not written in the original text. Precisely because of this, the transliteration of OT inevitably becomes transcription.

A popular, or “loose,” transcription, is an approximation of the conventional orthography and popular pronunciation of a word in a different script. It will produce a transcription that is easy to follow by contemporary audiences while inevitably forgoing important linguistic information in the original text.

Finally, romanization is the substitution of the characters of a non-Roman script by those of the Roman alphabet [Halpern 2007].