This article, by two of the technical leads on Project Endings, represents the culmination of all we have learned over the last few years, during which we have rescued over a dozen projects from death by software obsolescence and reconstituted them as entirely static, standalone websites with minimal dependencies. We now know a great deal, mostly from our own mistakes, about how not to build robust, long-lasting digital resources, and we have developed a set of principles, practices, and software tools which we believe provide solid defences against digital extinction. Below, we describe the institutional context within which Project Endings was born, and lay out methodically the guiding principles by which we now develop digital editions and other web resources.

What is now the Humanities Computing and Media Centre (HCMC) at the University of Victoria was once simply the Language Centre, a unit whose role was to maintain language laboratories and computer workstations and to write software applications used for language teaching courses. We had both Macintosh and Windows workstations, and would typically write custom software for either platform but not both, since writing cross-platform software applications was quite difficult in the days before the World Wide Web (WWW) and Java. When the Web first appeared, we seized on it enthusiastically as a practical way to provide language-teaching resources which could be delivered not only on both kinds of workstation in our labs but also to other locations. When JavaScript emerged around 1996, we were blessed with the capability to create truly interactive programmed teaching exercises, and by the turn of the millenium we had built language-teaching exercise collections for many languages, as well as desktop authoring tools to make the creation of such pages easy for non-coders (Arneil and Holmes 1998, 1999, and 2001).

The materials we created during this period still exist, and they still work perfectly. Examples can be seen in the Indonesian course materials created in 1999 (https://web.uvic.ca/hrd/indonesian/) and the endearingly-dated “Web Language” online presentation [Holmes 1997], originally created in 1997 and last updated in 1999. The latter includes a feedback form which is still regularly used, so it still clearly has some relevance despite its appearance. The HTML and JavaScript is still fully functional, although non-standards-based cutting-edge technologies of the time, such as RealAudio, no longer work of course.

In the early 2000s, the Language Centre was transformed into the Humanities Computing and Media Centre, and given a wider mandate to support what was then commonly termed Humanities Computing, now Digital Humanities. Better-funded projects came along, and more ambitious coding was undertaken, supporting not only language-teaching resources but also digital edition projects, historical datasets, mapping initiatives, and many other research and teaching activities. In particular, we developed some expertise in TEI and the XML-related languages, and began to create large online document collections such as the Colonial Despatches project (https://bcgenesis.uvic.ca/) and Le Mariage sous l’Ancien Régime (https://mariage.uvic.ca/). In place of static (albeit interactive) web pages, we started to build more sophisticated web applications based on PHP/MySQL database stacks and the eXist XML database.[1] Now blessed with our own server infrastructure thanks to the ground-breaking Canadian TaPOR (Text-Analysis Portal) initiative, we were able to run Tomcat, Cocoon, Postgresql, and a host of other backends to support the growing variety of DH projects under way at our institution.

These server-side systems enabled us (along with the rest of the DH community) to build resources which were more coherent and easier to maintain; information about a single entity, individual, or place could be encoded or stored in a single record, but retrieved and displayed in a multitude of contexts wherever it was needed. Documents could be transcribed and encoded once, but decorated and rendered in many different ways based on the needs of the user or the application context. Sophisticated searching, grouping, and presentation options were easy to implement. Beginning in 2003 we generated a large number of these projects; our servers were well-populated and their CPUs were busy running XQuery, XSLT, PHP, SQL, and even ASP code.

As Lou Burnard (2016) remarks, “nothing in digital form is ever really finished.” DH projects may languish untended and unloved for periods of time, but their progenitors typically intend to continue working on them periodically or when funding allows. Setting up such projects implies a commitment to maintaining them, and as our project portfolio burgeoned, so did our maintenance burden. As we moved into the 2010s and onwards, a clear pattern began to emerge: projects created in the 1990s continued to work as intended with virtually no maintenance at all, but the projects we had built on sophisticated backend systems such as MySQL and eXist required frequent and time-consuming maintenance work. As an example, the Robert Graves Diary (https://graves.uvic.ca/), one of our earliest digital edition projects, was originally written based on eXist version 0.9 in 2003–2004; it had to be steadily revised throughout 2005–2007, and then completely rewritten for eXist 1.4 in 2012, because the original Tomcat/Cocoon/eXist 0.9 stack became obsolete. Then in 2014 it had to be updated again. This was a project on which the core academic work had been completed in the first phase of the project, but due to infrastructure churn it regularly returned to haunt us. Over the years, we amassed a maintenance burden which threatened to overwhelm our ability to support new projects coming through our doors.

This problem was the impetus for Project Endings. The Web had solved for us the challenge to “write once, run anywhere”;[2] the issue we were now addressing was how to “write once, run forever.” Project Endings brought together a team consisting of academic digital humanists, programmers, and librarians, along with a selection of active, complex digital edition projects, to determine how we could best plan, execute, and archive our projects in such a way that they would live on indefinitely without further maintenance.

In the early days of the Endings project, we began to examine in detail how our older digital editions and sites had progressively broken down. Some problems were obvious: obsolete versions of server-side applications and services (MapServer, MySQL, PHP) had been retired or replaced, and old code would not run on new versions, even if new versions existed. Other failures were more subtle. We investigated, for example, how successfully our applications had been archived by The Internet Archive (https://archive.org/) and reproduced on the Wayback Machine (https://archive.org/web/). It was immediately obvious that sites built on backend databases, where “pages” were in fact queries to the database, were very problematic. The project Le Mariage sous l’Ancien Régime, which linked to its anthology documents using URLs that queried the server (e.g. http://mariage.uvic.ca/xhtml.xq?id=le_bourgeoise_desprit), was scarcely archived at all, presumably because the archive crawler was unable to follow or parse the query-based links successfully, so for the most part, the archived versions of the site consist mainly of notices that “The Wayback Machine has not archived that URL.”

Mariage also had a rather gratuitous feature whereby the end user could choose between three available colour schemes, and because that functionality relied entirely on JavaScript, the required CSS files had been retrieved and stored as though they were HTML pages, and no styling appeared on the site pages at all. Site search engines failed completely, as did any annotation popups linked from texts which relied on retrieving data by querying the backend database, and JavaScript-based interactivity was generally broken. In fact, the majority of our large-scale, high-profile digital edition projects were scarcely functional at all in the Wayback Machine, and most of their content had not even been retrieved and stored.

The Internet Archive had never been one of our primary backup strategies, but it was disappointing to discover how unsuccessful it had been in its attempts to crawl and reproduce our sites. Although some of these failures could perhaps be laid at the door of the Internet Archive crawler, Heritrix, which had some limitations, most of them were clearly our fault, and resulted from our crude approach to URL construction. One site, the Robert Graves Diary, was represented by only a handful of static pages, since the majority of its content was designed to be accessed through explicit searches from the home page, with the search parameters encoded in URL query strings; the crawler would have had no way to know what pages existed on the site other than the few information and background pages linked directly from the home page. On the live versions of these sites, any pages accessed by a user were typically constructed on the fly by XQuery and XSLT, so they did not exist in any form which could be backed up even locally by any normal method.

Digging deeper into methods for backing up and replicating our projects within our own institution revealed other problems. Some sites and applications included hard-coded links to local resources using absolute URLs rather than relative ones, so if the site were replicated in a different location, it would still be dependent on, and prone to send the user over to, the original site.

Another problem faced by projects backed by online databases was a lack of consistency in versioning of content. Typically, multiple project members had rights to upload new material directly into the live database, thus “publishing” it, but there was no attempt to ensure that the current state of the live site was — in Endings parlance — “coherent, consistent, and complete” (see [Holmes & Takeda 2019b]). Someone editing a primary source text might upload a new chapter to the site, forgetting that it contained links to newly-added people in the personography or locations in the placeography which also needed to be uploaded, so those links would fail. Every “page” might have a different publication date, and the site overall had no version or edition information, so anyone citing it would be forced to fall back on the rather unfortunate “last accessed” date to anchor their citation, knowing that the specific version of the document they were citing might change tomorrow with no warning and no way to retrieve the previous version. This was a problem of project organization rather than a technical issue, and it arose largely out of the working methods emerging in the early 2000s. In the excitement provoked by the new affordances and publication possibilities inherent in digital projects, there was a tendency to publish anything and everything as quickly as possible. A measured and thoughtful approach to “rolling publication” is of course possible, but, as we discuss in section 5.5, our work on Endings has convinced us that for most projects, a more traditional edition-based methodology is better.

Finally, we were forced to acknowledge that few of our projects had adequate documentation, making remediation and reconstruction doubly difficult.

We had, it seemed, spent a dozen years enthusiastically generating fragile, unmanageable, unmaintainable, unarchivable contraptions, and we were going to pay the price for our thoughtlessness.

This is not an unusual situation. In fact, our research suggests that it is the norm for DH projects.[3] Many DH web applications are built rapidly, using off-the-shelf frameworks and libraries which date quickly and are ultimately guaranteed to fail, and most projects either rebuild their applications in repeated cycles of funding, or see them languish and ultimately disappear. Our situation was perhaps more pointed and critical because we are a base-funded unit with a long lifetime, committed to maintaining our projects even after their initial funding is exhausted. Units like ours are still relatively rare; in most cases, there is little or no institutional help and no expectation of any. But having caused the problem, we now had to fix it, not just by rebuilding multiple large projects, but by doing so in such a way that we would not have to do the same work again in another ten years.

The Endings Project was set up as a formal collaboration between the HCMC programmers grappling with the issue of project obsolescence, four Principal Investigators leading some of our largest and longest-running active projects, and a group from the University Library that specialized in digital preservation and collections. We began with the aim of researching, formalizing, and testing a range of strategies for rewriting four projects — The Map of Early Modern London, Le Mariage sous L’Ancien Régime, the Nxaʔamxcín Dictionary Database, and The Robert Graves Diary — with longevity in mind. The project, formally titled Endings: Concluding, Archiving, and Preserving Digital Projects for Long-Term Usability, was funded by the Social Sciences and Humanities Research Council of Canada (SSHRC) for four years starting in April 2016. Since then, we have added many more projects, some new and some dating back many years, to the project portfolio, and there are now 14 projects under the Endings umbrella. Each individual project has provided its own unique set of challenges and resulted in fresh ideas and strategies as we have learned to build static sites from them.

The Endings project has produced a range of outputs, including a survey with follow-up interviews, software, many articles and presentations, many static-site digital editions,[4] and the 2021 symposium from which this special issue was born. But perhaps the most important outcome of our work has been the set of principles that codifies our approach to site-building.

Readers of a certain age may remember the 1990s craze for the Tamagotchi, an electronic device that was designed to mimic a pet animal by requiring regular attention — feeding, education, play — from its owner in order to keep it healthy.[5] Our digital edition projects had become rather like Tamagotchis in requiring constant tending simply to keep them functional. Slightly older readers, however, may remember a similar, pre-digital fad from the 1980s. The Pet Rock was exactly what it purported to be: a piece of rock, which required nothing whatsoever from its owner. “A rock would not need to be fed, walked, bathed, or groomed, and it would not die, become sick, or be disobedient” (Wikipedia). Our Endings mission was to devise methods for turning our Tamagotchis into Pet Rocks.

The Endings Principles present a set of simple guidelines for this process, and are intended for two audiences: researchers who have Tamagotchi projects that are already too needy, and who are looking for an exit strategy; and (more importantly) researchers in the early stages of planning and setting up new projects, who are in a position to build a sustainable, low-maintenance digital edition from the outset.

This article lays out and discusses version 2.2 of the Endings Principles. This version does not differ substantially from the initial draft, first published on the Endings website in 2018, but the major sections have been re-ordered. Those sections cover the five principal aspects of a digital edition project which we consider to be crucial in designing for longevity: Data, Documentation, Processing, Products, and Release Management.

With these Principles in place, we were able to convert the dormant projects, such as The Robert Graves Diary, and the on-going projects, like The Map of Early Modern London, from unwieldy eXist applications into a large bundle of static HTML, CSS, and JavaScript. Our intention was that these static incarnations look and function identically to the previous live application and, as these sites were steadily released and tested, we were happy to find that this process was not only feasible, but significantly improved the projects. Our staticization of Graves, for instance, surfaced various encoding errors that had gone unnoticed for many years, which we could now easily diagnose and fix. And in moving from a dynamic application to the static site, we had the opportunity to re-write the codebase from scratch, removing the layers of obsolete code that had sedimented within the codebase over its long history of updates, maintenance, and modernization of the CSS and JavaScript. Projects in active development benefitted similarly; these re-writes provided a good opportunity to scrutinize the project, lopping off irrelevant or deprecated code. Thanks to our Jenkins Continuous Integration server, following any change to the source data or the processing, we no longer needed to click anxiously around the site to confirm that we hadn’t broken the anything: any encoder could simply commit, wait a few minutes, and then, so long as the Jenkins server didn’t send its “BUILD FAILED” message, proof their changes in the context of the site.

But these sites all lacked a significant and crucial part of their functionality: search. Most digital editions require some form of text-based searching and, in many cases, the ability to search across a large and disparate document collection and aggregate documents based on multiple metrics is the project’s raison d’être. Search was, in other words, non-negotiable — we could not sacrifice the robust search mechanisms on the existing site. We could quietly ignore search for the time being in some projects, like The Map Of Early Modern London, where search was peripheral to the main functionality, but The Robert Graves Diary forced us to come up with a real solution: the homepage for the project was essentially one large, faceted search page, which we were able to replicate perfectly, but since search itself was handled entirely by eXist, the static version was rendered completely inert.

We had always assumed that the kinds of search our projects required — complex faceting and filters, wildcards, and exact phrase queries split across multiple, collection specific search pages — was necessarily the stuff of servers and could not be made static. While some client-side search solutions were available, they were ill-equipped to handle the thousands of documents, millions of words, and complex metadata structures that comprise a standard digital edition.[18] Outsourcing indexing to an external service, like Elasticsearch, was out of the question as it would violate the core Endings principles.

Our intermediate solution was to stick with what we knew: package the static sites into a XAR bundle and spin up a simple eXist instance whose only job would be to index, query, and retrieve the search results from the static HTML pages. Of course, this worked perfectly, and we released the first version of a number of our static sites following this approach. But it was a disappointing compromise. We had gone through the trouble of re-writing these sites in their entirety to remove the fragility inherent in their dependency on eXist, and had celebrated our success in building completely static, serverless websites, and now here we were, returning to eXist, hat in hand.

If we were to return to eXist, we reckoned, then we should at least try to improve the situation; in the spirit of LOCKSS and following our principle for “Graceful Failure” (5.4.7), we decided to take a decidedly maximalist approach. Using Graves as our case-study, we supplemented eXist with three distinct approaches, which all had their own limitations with respect to functionality, archiving, and sustainability, but could provide something of a safety net for the project. We discuss these approach in greater detail in [Holmes & Takeda 2018], but, in brief, those three approaches were: 1) a Google search widget, which would not offer anything beyond what Google already provides, but at least gave users a shortcut for a site-specific search; 2) an experimental interface for querying a Solr index provided by the UVic Library (the eventual archival home for all of these projects), which could provide many of the faceting and filtering structures required by the project; and 3) a “standalone search,” a Javascript-only client-side text search interface that would stem user input and match it against an pre-built inverted index generated during the build process. In this process, a list of distinct words was created for each document; the words were then run through the Python implementation of the Porter Stemmer, and then documents were grouped by stem, so that JavaScript could return a simple list of the documents in which that stem appeared. This solution, as we noted then, worked well for small document collections, but lacked the necessary features — keyword-in-context, search faceting, and filtering — and thus should be considered the “ultimate fallback when all else fails”  [Holmes & Takeda 2018, 59].

Over the last three years, we have continued to improve and expand what we then called “standalone search” into “staticSearch”: a fully-featured and sustainable search engine for static websites that has no server side dependencies. Now implemented by nearly a dozen projects, staticSearch has not only allowed us to replicate server-side search functionality, but has significantly expanded what these searches can do. So to revise our answer to the question from our 2018 paper, “Why do I need four search engines?”: You don’t.

The technical principles that we have outlined so far demonstrate how we have created resources that can be archived, or that, at least, lack the technical barriers that have made digital preservation difficult (or impossible) in the past. However, there are still a number of issues and challenges that the Endings project has faced in preparing to archive these projects within institutional repositories and, in particular, in preserving the form and function of these projects such that they can continue to be useful as scholarly resources.

While librarians and archivists have long recognized the importance of secure archival storage, digital archiving initiatives have largely focused on data storage and management (see [Molloy 2011] or [InterPARES 2013] for examples). Back in 2002, for instance, Gelaw et al. noted that “It is now common knowledge that digital information is fragile in ways that differ from traditional technologies, such as paper or microfilm. The fact that information is increasingly stored in digital form,[sic] has led to an accelerated search for effective methods of managing electronic information resources.” However, the very title of that article, “A Metadata Approach to Preservation of Digital Resources,” underscores that, in many cases, archiving is primarily a matter of metadata; as they note, the “creation of preservation metadata for electronic files and digital collections are [sic] among the most important steps.” This approach treats understanding digital resources as discrete blobs to be stored in large repositories with their accessibility and findability contingent on the provision of effective descriptive metadata. As Goddard and Seeman write, “in order to ensure longevity, the library will make multiple copies of our digital collections, aggregate them into Archival Information Packages (AIPs), and distribute these archival copies in offsite locations”  [Goddard & Seeman 2020, 40].

For end-users not able to access archiving services from libraries or similar institutions, there are also archiving services available on the Web, such as Zenodo (offered by CERN, https://zenodo.org/), Preprints.org, or HAL (https://hal.archives-ouvertes.fr/). All of these, however, follow a similar model: a single binary object such as an article is uploaded and associated with metadata. Datasets may also be attached, but these are viewed as objects to download. This kind of archiving, while it may be a useful backup strategy (and we can never have enough of those), will not make a digital edition project (i.e. a complex website) available in the way a user would want to access it: as a functioning website. The Text Encoding Initiative, for example, archives each version of its standards package, consisting of schemas, exemplars, and other documentation, on Zenodo, but the end user can only download a zipped package of materials; they cannot browse and read the TEI Guidelines itself, as they can on the TEI’s own website. As we have suggested above, if a digital edition is to remain relevant, to continue to be read, cited, and hold its place in scholarly dialogue, it must be fully functional, available, and usable; after all, as Morreale reminds us, “Humanities scholars have been taught that what they produce should be, first and foremost, consumed and understood by other scholars. Once this is accomplished, they want those who have received their work to reuse or build upon it, so that their contributions become part of a larger conversation, one that may stretch out for years, decades, or even centuries”  [Morreale 2019, 4]. However, a shelved DVD copy with metadata in a catalogue or a zip archive in a distributed repository does not facilitate the kinds of experience that are at the heart of a digital project; only a functional website does. Sacchi (2015), based on a now-inaccessible technical report from the National Archives of Australia, offers a useful distinction between “stable artefacts” — resources preserved as blobs — and “performance”  [Sacchi 2015, 51ff]. “Performance” here means the direct, live access to digital objects, which, like any live performance, is a unique experience bound to a specific occasion and is contingent on the availability of hardware and software components.[20] Understanding digital archives as not just “stable artefacts” but also as “performance,” as Sacchi writes, “explicitly addresses the need for something concrete that needs to [be] experienced by a researcher in order [to] access the essence of a record”  [Sacchi 2015, 53]. In other words, while it is essential that digital resources are archived as “stable artefacts,” preserving the “essence” of a digital edition requires maintaining its functionality and allowing users to experience it as “performance.”

Librarians and archivists are now beginning to focus on this important distinction. The University of Victoria Library is currently developing a “Donation policy for legacy websites,” which outlines the conditions under which an individual or a project group may transfer an entire website to the library for long-term storage, with the expectation that it will actually remain accessible as a website on the Internet for public access. Offering such a service is not simple, though. A library presented with a Tamagotchi-style web application, with dependencies on back-end databases and server-side scripting, will (and should, quite rightly) reject the “donation” as it would, as we have argued, amount to the passing on of technical debt and the obligation for incessant maintenance. But the static site model we have proposed and implemented is not subject to these considerations. Assuming any functional webserver is available, a static site can be served for the foreseeable future with no maintenance burden whatsoever; only bandwidth and disk storage are required, just as is the case for binary-blob digital assets in the traditional archive model. With a static site approach, any given edition of a digital collection could be available in multiple “locations” (i.e. at multiple URLs), hosted by different institutions, providing resilient and reliable access to the resource even in the face of occasional failures or closures at specific institutions.

Such a scenario presents its own challenges, though. If a site was originally offered at a custom domain (www.ourfabulousproject.net), should the archiving institution agree to take on the ongoing cost of preserving that domain? If there are multiple incarnations of a site, how is it to be cited, and how is a reader to know that they are viewing the “same” site as that cited in a publication, albeit perhaps on a different URL? This is one reason that the edition management and release model proposed in section 5.5 is so crucial. Each coherent, consistent, and complete edition of a digital project may be archived and served from many locations, so it must be reliably and obviously consistent and show clear dating and versioning information, just like an edition of a printed book.

When Project Endings began, we really had no idea how practical it would be to build interactive digital editions without back-end services, but our initial work was very encouraging, and in fact it proved relatively straightforward to rewrite even very large and complex sites such as The Colonial Despatches (over 7,000 TEI-encoded documents) and The Map of Early Modern London (over 12,000 pages) into simple, fast, responsive, and easily searchable static sites. Other sites from our growing collection have taught us to handle a variety of different types of data such as video (Francotoile) and image collections (MyNDIR), while the largest dataset, the Landscapes of Injustice Archive, presents a searchable collection of over 150,000 pages representing diverse data including maps, oral histories, land title records, street directories (in Japanese and English), correspondence, and legal casefiles.

At this point, we can say with confidence that the approach we have taken is practical and effective. You can build digital editions this way, and it really doesn’t take much more work than it would to roll out a Tamagotchi of frameworks, libraries, software, and infrastructure. Once it’s done, it takes no maintenance at all, and you’re free to move on to the next project or focus on the next edition.

We’re doing it. You should do it too. You’ll regret it if you don’t.

This is a list of all the projects which have been constructed or reconstructed in accordance with the Endings principles outlined in this article.

We divide digital projects into five primary components: