Printed books still remain persistent in the workflow of scholars even though there is a plethora of digital options available that afford great power and flexibility to the user. Word processors and other applications have been completely integrated into people’s daily lives and have started to replace pen and paper as a modality for interacting with the written word; when it comes to books, the affordances offered by digital platforms such as search and copy are considered paradigm shifting additions to the act of reading. But, even though these tools exist, scholars still write on paper and still have books on their bookshelves. There is a tension that exists between these new digital formats and our history. We often create digital tools to mimic the affordances of books, but while they improve steadily, the weight, smell, and sounds of a book are still unique to bound paper and ink. It is important to note that it is still not known how these affordances affect cognition, and for literary scholarship, interpretation. Mehta et al. studied fourteen literary critics and found that each had idiosyncratic methods of marking up a literary document, but most importantly, all of them engaged in some form of annotation when working with poetic texts [Mehta 2017]. Our own domain expert, a modernist literary critic, confirmed the findings of that study and presented the idea to us for a system that allows scholars to quickly digitize a document for augmentation but retain the look and feel of the original.

Beyond the affordances of physical texts, many older texts used for research often do not have reliable digital versions, and many corpora are still digitized as images such as Early English Books Online [EEBO 2003]. The solution we offer is a combination of techniques that bring together the paper and ink history of our past, with the digital affordances of our present. To demonstrate, we present a web-based interface, geared toward researchers, that allows for the quick digitization and augmentation of paper documents. By using any web-based device with a camera or uploading an existing image, Textension allows the user to create interactive digital objects from analog texts that retain the look and feel of the originals.

To start this project, we asked ourselves the question: How can we allow users the ability to interact with both analog and digital text at the same time? Not knowing exactly what we lose when we digitize a text in terms of interpretation or cognition is a much larger problem, and we wanted to address how to interact with these texts without completely discarding the originals. While there are great efforts to digitize the world’s books, such as the Google books projects [Google 2017], large-scale OCR projects are difficult to implement. With the growth of digitized text repositories that leverage these technologies, two problems are still outstanding: digitally supporting books that have not yet been digitized, and enabling better use of books that have been digitized as images and are not currently interactive. The flexibility and freedom of digital writing and reading are leading to increasing pressure to digitize texts. However, most of these solutions are costly, time-consuming, and never seem to reach the document of current interest. There is a need for a quick, direct and simple way to gain these freedoms with the document you currently have in hand — whether it is a hand-written letter, an old book, or the newspaper. What we have laid out here is a method for achieving this goal, while still having and keeping the original text close at hand.

In this paper, we present a framework that extends the power of the digital to physical books in near real-time. Our contribution is bringing together ideas studied in digital document spaces and existing word-scale visualizations to demonstrate how these known quantities can be leveraged to bridge analog and digital reading and writing. Our framework is informed by previous results describing document spaces and the different ways they are used with analog documents. We outline each of the document spaces and describe how they can be used in tool design, and we implement a prototype to demonstrate the robustness of this framework (see Figure 1).

Through Textension we offer quick access, applicable to the analog text at hand, to an integrated digital/analog environment, only requiring common equipment such as the camera in a phone and a preferred web-enabled device. Simply photograph or upload an existing picture of a document, display it in Textension on a web-browser and start interacting. By making paper documents interactive on mobile devices Textension allows for a smooth transition between our history and our present by allowing users a quick way to digitize documents while working on-site in places like libraries. Our system produces in-line visualizations and interactive elements directly on the newly built digital document, allowing for work to continue while having an augmented digital document at the ready.

While we present a prototype in this paper, we see our main contribution as a discussion and amalgamation of what is possible when bringing together analog and digital affordances as it relates to text. We see Textension as a way to leverage past studies and computational approaches to natural language to quickly create digital documents from analog texts using readily accessible mobile technology.

Word Space is any portion of a document that has the printed word on it. The OCR engine we used, Tesseract, can identify and create bounding boxes around both printed and hand-written words, lines, and paragraphs. Depending on the specific implementation, word space could be considered any or all of these. In an analog book the actual printed type is an element that can be interacted with cognitively, but as we move into digital representations of that text it allows us to alter and query the text in interesting ways. The possibilities here are great, with one of the motivations being simply on-demand OCR. But when the text is digitized it opens up possibilities for text analysis, computational linguistics, and machine learning based on language. Marshall et al. reference both highlighting and annotation as a way that this space is interacted with on paper documents [Marshall 1997].

Line space is any space that exists between existing printed lines but remains inside the bounding box of the entire block of printed text on the page. Digitizing documents using our framework allows for on-demand opening and closing of these spaces. The manipulation of line space can create room for additional elements, such as ink annotations, inserted figures, and data visualizations that relate to the text. We synthesize the background of the document to avoid jarring the reader, and once this is done effectively any amount of space can be added to the document. Previous studies have found that this space is most often used for annotation, specifically in-line notes and for connectors such as arrows between words.

Margin space is any area outside of the bounding box of the text but still within the bounds of the original document. This is the space commonly used for free-form note-taking. For example, when studying, editing, or conducting a close reading of a document [Mehta 2017]. In addition to free-form note-taking, this space can be used for inserting new elements related to the text automatically, such as grammar trees, maps, or simply any placement of third-party or generated images.

Occlusion space is a layer that covers the document. Additions in the occlusion space can obscure the text, so semi-transparent and impermanent elements are appropriate to maintain document legibility. In an analog setting, the occlusion space is often used with physical additions such as sticky notes. This space can be accessed in multiple ways, but the underlying function of the space tends to be information that is needed in the moment, but not on a continual basis. We demonstrate how to interact with this space by using tool tips that show the definitions of words on demand. Most of the techniques we demonstrate in this paper could be taken from one of the other spaces and placed into occlusion space, the question of permanence or impermanence will be a decision that rests with the designer.

Canvas space is a concept that we could loosely attribute to the desk that a printed book is placed on or the space outside a page of text taped on a whiteboard. When we move to a digital representation of the book, the canvas space could be infinitely expandable allowing for the insertion of larger interactive elements. We have chosen to demonstrate how linguistic analysis could lead to automatic insertion of images, figures, and tables within the canvas space. It is important to note that external images could also be inserted in this space as Cheema et al. propose in AnnotateVis [Cheema 2016].

After creating an interactive, expandable document from the captured image, augmentations can be added to provide supportive features as required for the specific task and context. For example, a learner may require word definitions, while a literary scholar may be interested in the contemporary use of the words in the document. Augmentations can take the form of inserted glyphs, images, overlays, and annotations in the document spaces, or they may replace or change the words in the document. Augmentations can be temporary or permanent, as appropriate for their purpose and the document space in which they appear. The insertion and placement of augmentations and the provision of interactivity on the document and its augmentations is provided by the layout engine (see Figure 5).

The images after upload are broken into individual word and space objects that are then recompiled in order onto an HTML canvas to reproduce the original image with the added flexibility of moving, inserting, and changing elements. Augmentations are placed on the canvas as a layer on top of the image objects. Textension has been developed to support the creation of new augmentations, which can draw on custom data processing, local datasets, or public APIs and data. Textension was built using flask, a python server back-end; bootstrap, for UI elements; jinja, a template engine for python; and jquery, for data handling.

What we present in this section are a series of concrete implementations of document augmentations that demonstrate a subset of the possibilities of the Textension framework. The selected examples highlight the possible breadth available when considering the five document spaces and the possibilities for interaction with those spaces. We explore insertion augmentations, as well as temporary and permanent overlays. The availability of the plain text allows easy integration of natural language processing, and the fact that the digital document is built in pieces allows for easy insertion of space to accommodate for the adding of new features. In this way, we envision Textension as both a sandbox for designing interactive elements for digital documents and a way to use both digital and analog affordances simultaneously when working with texts.

The tension that exists between our analog pasts and our digital present can be addressed using our framework. Our prototype, Textension, leverages the power of OCR and digitally manipulates the five document spaces in near real-time. The system we present is an implementation of previous studies brought together in a way that can be extended easily for domain-specific analysis tasks. The web-based platform allows for easy integration with mobile technology and makes it possible to use Textension in a variety of locations and scenarios. We have demonstrated a breadth of possible use cases and have chosen widgets that demonstrate the utility of each of the document spaces.