The JSON input format of the collation tool CollateX allows for extra properties to be added to words [CollateX]. A number of projects make use of the JSON input format to “pass along” additional information about relevant markup elements through the collation pipeline. In a pre-processing step, the text is tokenized and transformed into JSON word tokens. Editors make a selection of elements that they wish to attach to the JSON token as a t property (“t” for token). The collation is executed on the value of the token’s n property (“n” for normalized), but the t property with the tag(s) is included in the JSON alignment table output and can be processed in the table’s visualization. This approach is used, among others, by the aforementioned BDMP; the online critical edition of the Primary Chronicles, or Povest’ vremennyx let, created by David J. Birnbaum; and the work on the Declamations of Calpurnius Flaccus done by Elisa Nury [Birnbaum 2015] [Nury 2018, 7.1]. The advantages of this approach are, first, that a witness with in-text variation can be collated as one text and, secondly, that the method approximates the goal of having nonlinear revisions marked as such in both in- and output. The main disadvantage is that the collation algorithm still treats the input witness as linear text: the witnesses are collated as such and any information about revision campaigns is ignored during the collation process.

Another approach is to transcribe a manuscript with in-text revisions as two or more separate plain text versions and then collate these temporary versions against each other. The advantage of this method is that it succeeds in aligning the two parts of a substitution vertically instead of horizontally (which the previous approach is forced to do), but the drawback is that the deletions and additions are spread over two or more (sub)versions. What is more, the revisions on a manuscript do not always lend themselves to a clean separation into different layers.

The approach is presented by Desmond Schmidt’s Multi-Version Document system, as integrated into Ecdosis, “A CMS for scholarly editors.”[10] In a 2019 article, Schmidt describes in detail how this layered approach also allows for the inclusion of in-text variation [Schmidt 2019, §60]. At the transcription phase, a manuscript text is divided into “levels” by the scholarly editor. All changes to the “baseline text” of a document (level 1) are grouped as being on level 2. Subsequent changes to level 2 are grouped as level 3. For each of these levels a separate text is created, a “layer:” “We do not claim that a layer is a text the author ever completed; it is not a version as such,” explains Schmidt, “It is simply a compact way to capture local changes.”[11] Each of these artificial layers is fed into Ecdosis as plain text, with a few additional features encoded in stand-off markup. The Ecdosis approach is implemented in the Charles Harpur Critical Archive. For example, the text of Harpur’s poem “To the Comet” is divided into three layers. The document identifier “english/harpur/poems/h595f/layer-2” points to the second layer of the poem’s text. Layer 2 is followed by “h595f/layer-final,” in which corrections on the third level are carried out on the text of layer 2.[12] The scholarly editors of the Charles Harpur Critical Archive see no issue in this “mechanical” grouping of in-text variation:

Assignment to levels is mechanical and describes a purely local succession of changes, which can almost always be determined. Authors leave many clues as to the succession of variants: position (above the line, then below, in the margin, on another page), the carrying over of changed text between levels, sense and crossing-out. Where the local temporal succession can’t be established this is usually because the corrections themselves can’t be read.  ((Ibid.))

The versions and layers in plain text are merged into one Multi-Version Document. It stores the text shared by each version and layer only once. Note that the comparison algorithm works at the character level, not at word level. In the table visualization, therefore, words may be cut up (see figure 1).

Evidently, XML is a widely used standard for (online) publishing and data-exchange, and there is a general need for comparing and versioning XML documents. Accordingly, there exists a wide variety of diff algorithms or deltas to find the differences between two XML documents, each with its own strengths and focus [Peters 2005]. Because most XML documents are data-centric and contain record-based data, deltas typically consider XML documents as hierarchically structured trees and compare them as such. Considering our type of material – text-centric XML documents in which the left-to-right order of the siblings is significant – we specifically focused on tools that can compare ordered XML trees.

We found a number of useful tools and algorithms, like the commercial tools Microsoft’s XML Diff and Patch or the XML compare tool of DeltaXML, that can compare both ordered and unordered XML trees.[13] Such tools can come in handy during the making of a digital edition, for instance when scholarly editors wish to compare and align TEI-XML transcriptions of the same text that were made by two different scholars of the same project. Furthermore, there exist XML editors with a built-in function to compare XML documents. Oxygen XML Editor, for example, allows its users to compare two XML documents on the level of lines, words, or characters, or to ignore attributes or namespaces.[14]

There is, however, an important difference between current XML comparison tools and algorithms, and how we designed HyperCollate. Existing tools compare XML documents and focus on the parent nodes, whereas HyperCollate interprets XML documents in order to better compare the versions of a text. It does not ignore the structure of the XML tree, but – in line with the research objectives of scholarly editors – it focuses on comparing text and is only concerned with the TEI-XML tags that determine the flow of a text with revisions. Given this fundamental difference in premise, an extensive analysis of the quality and use of diff algorithms and tools is beyond the scope of this article.[15]

Consider the example in figure 7, transcribed first with the TEI <del>/<add> method, and then with the TEI <app>/<rdg> method.

Instant revisions or currente calamo changes are revisions made within the initial writing of a sentence, with the change occurring on the same line, made in one and the same writing session. They pose a particular challenge for transcription, because they cannot be encoded as a substitution: the demarcation of the added text is problematic. Would all of the text following an instant deletion be an addition? The TEI Guidelines propose to add the attribute @instant with the value “true”’ to the <del> element, to distinguish between a regular correction.

In the following example, Beckett wrote “threw up his” and then crossed the three words out to continue the sentence with “gave such a jerk at Russell's arm that that poor little man was nearly pulled off his feet.”

In this case it not make sense to tag “gave such a jerk” as an addition, not would it make sense to tag the rest of the sentence as such.

Hence we argue that an instant revision does not constitute a case of nonlinear text. In contrast to regular in-text corrections and substitutions, there is no other way of reading the text: the instant revision is part of the same writing campaign as the text surrounding it.

Deletions followed by additions that are (semantically) related can be grouped by means of a <subst> (substitution) element, or as two readings in an <app> element. This can be illustrated with a simple example from Beckett (see figure 12).

In the case of the substitution of a group of words, there will usually be more than one way to align the witnesses. The differences in alignment stem from a difference in focus: does the scholarly editor want to give priority to the unit of the substitution, or should the alignment of matching words receive priority? The traditional approach in collation of starting from exact string matches and grouping the variants in columns between the matches has the advantage of accentuating the similarities between witnesses, but the unit of the substitution can become obscured as it is spread over multiple columns. Holding the substitution together in the alignment, on the other hand, keeps the focus on the nonlinear nature of manuscripts as witnesses. It marks the spot where “something happens” on the manuscript and the totality of a textual operation is presented as one block. The drawback is a potential loss of information if the occurrence of a matching word across witnesses is not indicated in the collation output.

Consider the examples below.

Given a collation with two slightly different witnesses (B: “Murphy stayed his hand.”, C: “Murphy stayed the arm.”), different scholarly editors might arrive at different alignment tables, but they will largely fall into one of two categories: alignment on matching tokens or on the unit of the substitution. Currently, HyperCollate aligns the witnesses on the unit of substitution and will produce the following alignment table:

In the future, HyperCollate aims to offer users two options in its output alignment: to give dominance to the matching of tokens, or to preserve the unit of the substitution as much as possible. Users will be then able to indicate their preference by way of a parameter in the collation command. The collation hypergraph and the two possible alignment options are discussed further in section 3.3.4.

The collation hypergraph can be visualized, among other formats, as an alignment table (see figure 17).

In figure 16 the “[+]” and “[-]” in the “A” version correspond to the <add> and <del> elements in the input XML. In the case of the <app>/<rdg> method, the value of the @varSeq attribute is included in the output, as shown in figure 18.

Collated against a fictional second witness B, the instant revision produces the following ASCII alignment table (figure 19). Note that the visualization options of the ASCII alignment table are limited, which is why the instant deletion is visualized as a regular deletion and the text that follows, “gave such a,” is placed in the same cell and directly above the instant deletion.

The HTML alignment table is more expressive and produces a more accurate visualization of the collation result (see figure 20).

The collation hypergraph visualization (figure 21), finally, demonstrates once more that the instant deletion does not produce an alternative path through the text of witness A. The graph also shows that even though the text tokens “&” and “and” as well as “jerk” and “pull” are aligned in the alignment table visualization, they are not considered a match by the collation algorithm.

Figure 22 shows the ASCII alignment tables for a collation of Beckett’s substitution of “Alice” with “Cathleen” against another version stating “Cathleen came.” When the substitution is encoded with the <del>/<add> method, the word in the <del> element is preceded by “[-]”, and the word in the <add> element with “[+]”. In the case of a transcription that uses the <app>/<rdg> method, HyperCollate reproduces the values of the @varSeq attributes in the <rdg> elements.

The collation hypergraph (see figure 23), from which the ASCII alignment tables are derived, holds the most contextual information: each node contains a full XPath expression. In a case where two witnesses match on a word, but are not on the same level of the XML tree, two different XPath expressions are listed in the node, bringing both the similarities and the differences to the user’s attention.

The instance of nonlinear text is visualized by two separate edges and nodes labeled with the same siglum, “A”. In order to make it easier to follow the path of a certain revision campaign, the edges contain not only information about the witness sigil, but also whether the text is part of a deleted or added branch. This information is visualized on the edges with [-] and [+] signs for deletions and additions respectively. Furthermore, information that is shared by more than one witness is given a thicker edge and node border (see e.g., Jänicke et al. [2015]). In this example, the words “Cathleen” and “came” are considered matches between A and B and thus share text nodes with a thicker border. Finally, the information about the location of the tag in the XML tree is added to the node as an XPath expression.

In the case of a collation of the Beckett example shown in figure15 with two other witnesses (B: “Murphy stayed his hand.”, C: “Murphy stayed the arm.”), the collation hypergraph identifies two variant groups across the witnesses (figure 24).

HyperCollate can express this graph in two possible alignment tables.