Close reading is an umbrella term for an assortment of reading strategies characterized by devout and detailed attention to the meaning and composition of art works. The approach was made famous by the New Critics, a group of Anglo-American literary scholars including Cleanth Brooks, William K. Wimsatt, and Monroe C. Beardsley. Inspired by I.A. Richards (author of Practical Criticism, 1929), Matthew Arnold, and T.S. Eliot, these scholars experienced their heyday of academic fame in the forties and fifties of the last century. Going against contemporary practices that, in their view, overvalued historical context and biographical information, the New Critics suggested that literary scholars should investigate the text itself. They wrote extensively on certain contemporary fallacies of literary analysis, for instance, letting your own emotions factor into the interpretation (the “affective fallacy”  [Wimsatt and Beardsley 1949]) or writing about authorial intentions (the “intentional fallacy”  [Wimsatt and Beardsley 1946]). Another practice they attacked was the paraphrasing of the contents or message of a work (the “heresy of paraphrase”  [Brooks 1947]).[1] Instead, this school propagated the careful examination of evidence offered by the text itself: images, symbols, and metaphors as part of a larger structure that gives the text its unity and meaning. Of particular interest to the close reader were devices that create ambiguities, paradoxes, irony, and other forms of tension within the text. Moving from close to distant reading, the level of analysis shifts from details within single texts to categories of many texts.

Distant reading is the practice of aggregating and processing information about, or content in, large bodies of texts without the necessity of a human reader to read these texts [Drucker 2013]. Distant reading corresponds to quantifying, computational reading methods and was introduced by Franco Moretti with the intention to identify the bigger picture in large collections of textual data that close reading cannot uncover. “Reading” is outsourced to a computer: it is in fact a form of data mining that allows information in (e.g., subjects, places, actors) or about (e.g., author, title, date, number of pages) the text to be processed and analyzed. The latter are called metadata: data about the data. Natural language processing can analyze the contents of “practically unreadably” large corpora of texts, while with data mining we can expose patterns or summarize on a scale that is beyond human capacity.

In his book Distant Reading (2013) Franco Moretti introduces the term polemically in explicit opposition to close reading, which, to his mind, fails to uncover the true scope of literature. Moretti is founder of the Stanford Literary Lab that seeks to confront literary “problems” by scientific means – computational modeling, hypothesis-testing, automatic text processing, algorithmic criticism, and quantitative analysis. The Lab’s first pamphlet suggested that literary genres “possess distinctive features at every possible scale of analysis” and that there are formal aspects of literature that people, unaided, cannot detect [Allison et al. 2011, 8]. The second pamphlet used network theory to re-envision plots [Allison et al. 2011]. Since then, Jockers (2013) has further developed distant reading in what he has called “macroanalysis”, a new approach to the literary reading and study designed for exploring digital texts in large quantities. Using computational tools to retrieve keywords, key phrases, and linguistic patterns across thousands of digital texts in databases allows researchers to attain quantifiable evidence on how literary trends have evolved over time and geographically. It can also determine what social, cultural, and historical connections exist between individual authors, texts, and genres [Jockers 2013]. In this article, we seek to combine such a macro-scaled method with close reading and manual annotation, and apply it to a non-literary dataset.

The text corpus used in the research described in this article comes from Reddit, a social content aggregation website and the self-styled “front page of the internet”. Functioning as “a bulletin of user-submitted text, links, photos, and videos ”  [Duggan and Smith 2013, 2], it is a message board wherein users submit content and discuss this content in different communities. The website is further referred to as a “social voting site”  [Gilbert 2013] as users (often referred to as redditors) vote submitted content up or down, sending the submissions with most upvotes to the Front page, i.e., the home page of reddit.com. Content on Reddit is organized in communities, so called subreddits. The nearly 900,000 communities of Reddit are organized around different topics like Technology, WorldNews, Music, Gaming, or PoliticalDiscussion. A single subreddit can be reached via, for example, reddit.com/r/PoliticalDiscussion. The single posts in one community are referred to as submissions. A submission usually contains a link, embedded images, gifs, or videos and may also contain a text written by the posting redditor. It is of special interest for textual analysis that other redditors can comment on a submission and reply to other comments. The comments are organized in a thread, a tree-like structure that allows for following the discussion chronologically and with regard to content. From any comment (every branch of the tree) further comments can emerge that yet again may receive comments (as more branches sprouting from the prior branch). Furthermore, redditors can make use of basic text formatting functions, such as quoting text of prior comments or highlighting. The text of the original submission and the comments on this submission are the primary source of information of this research.

The thread that formed the basis of our dataset was submitted on January 19th of 2017, and posed the following question: Should the Democrats nominate a celebrity in 2020? What would be the pros and cons?[2] Furthermore, the users added certain sub-questions, like: Would celebrity power help or hurt a presidential nominee in the next election? and If this plan actually goes forward, who would be the best choice?. Our dataset consists of 449 (461 including deleted comments) responses to these questions. Since the thread is still open, comments have been added after we collected and investigated our dataset. These new comments are not included in our research.

While investigations using Reddit data often aim at uncovering trends across single discussions and even across discussion forums (subreddits) (cf. [Zhang 2017]), the highest level of distant reading analysis in this research is on the level of a single discussion thread. This restricts the dataset to a size that can still be analysed by means of close reading and also constrains the comments to relate, even if only peripherally, to the one question or topic that initiated the discussion thread.

A common phenomenon in discussions in general is that the sub-questions emerge and that the focus of the topics shifts to new content. This illustrates that a technique is needed that can zoom in from the whole-thread level to prevalent topics that group the single posts into content categories. These single posts, on the other hand, can not only be members of topical (content) groups, but also point towards groups related to discourse function in Reddit discussions [Zhang 2017].

The hierarchical structure of Reddit discussions, ranging from single comments to a whole thread, matches our goal to contrast low-level close reading with high-level distant reading. For a scholarly reading of a discussion forum, neither isolated comments nor a high level view on the discussion thread as a whole are sufficient. Scholars need to know about topical groupings that provide a frame of reference for single comments. While grouping of comments can be achieved by manual labeling, a distant reading approach is promising as a method that is faster, that in principle scales up to larger discussion threads, and that may remove some human biases during the annotation process.

Yet, the role of irony, emotion, and humor in this thread warrants a close reading approach. The pervasiveness of irony and ironic detachment in contemporary (online) culture has been described by Ian Bogost as the “escape from having to choose between earnestness and disdain,”  [Bogost 2016, 59]. Poe’s Law, an adage of Internet culture, states that online, it’s impossible to know who’s joking and who’s being serious. In The Ambivalent Internet (2017), Whitney Phillips and Ryan Milner show how digital communications, e.g., through GIFs, memes, and videos, are operationalized to fundamentally destabilize the worldviews of others. It is almost impossible to determine when an ironic posture is adopted. Humor, irony, and role playing are central to the behaviors in online environments and communities like those on reddit. A manual annotation is to be expected to detect more of this humor and irony than a topic model. Therefore, we have chosen to call the level of human annotation “close reading”, which traditionally attends to tone and style as well as content of the message.

While the initial question of the Reddit thread regards viewpoints on the pros and cons of a future celebrity president and the names of potential Democrat candidates, the thread soon developed into a more complex conversation in which different “new” questions were discussed as well. In order to grasp these associative developments inside the discussion and the mechanics of a forum like Reddit, we choose to employ a hypothesis-free form of close reading, which means we first started looking for patterns in the material of the discussion in a rather open-ended way, without explicitly framing our horizon of expectation, or what we were expecting to find, beforehand. We chose this approach since our aim was first and foremost to comparatively analyze methodologies, and only secondarily, to find an answer to the question posed in the thread, regarding celebrities in politics.

This process took place in a bottom-up fashion: two human annotators analyzed the posts in the thread, and chose a word or phrase to summarize each post. During this process, they developed a collection of labels that were assigned to the posts. Based on this reading, fifteen classes of posts were identified and color-coded based on the classes. Ten of them turned out to be related to three different underlying questions that we formulated based on the classes. We describe these at more length under “report of the close reading analysis”. After manually annotating the 449 posts separately, we compared the lists of the outcome and slightly modified the categories to accommodate both our findings. Upon comparison, we discovered there was a high degree of congruence between the categories assigned to the posts in both annotations.

Parallel to the close reading based, manual annotations, we approach the Reddit thread from a distant reading perspective. Specifically, we adopt a data-driven ideal of distant reading, i.e. we want to start analysing without any human insight into the textual dataset. This results in two requirements on a distant reading method: First, the approach needs to be unsupervised, i.e. not relying on any prior labeling of the data. Second, the fact that Reddit threads are open-ended in the topics they comprise, the number of topics in the shape of clusters resulting from the distant reading algorithm needs to be variable. At least, there needs to be a possibility to model a range of clusters in a computationally feasible manner. These two requirements can be understood as equivalent to the hypothesis-free aspects (of the close reading approach) in our distant reading method.

One technique that fulfills both requirements is Latent Dirichlet Allocation (LDA) topic modeling [Blei et al. 2003]. This method allows for the automatic grouping of text documents according to latent content categories. These topics underlying a text corpus are modelled in a completely unsupervised manner, meaning that the algorithm does not know which texts belong together (for example, according to some human labeling) beforehand. This aspect of unsupervised LDA thus serves a data-driven ideal of distant reading. For each topic, LDA creates a different language model, as the underlying assumption is that different topics require different words and constructions. In our research, we expect that LDA is suitable as texts with different topics are expected to use a different “language”.

Importantly, the number of topics resulting from LDA is a parameter that is set manually by the user. While determining an adequate number of topics for a dataset is often a problematic challenge for which no definite solutions are agreed on, the variability of the resulting number of cluster matches the second requirement to our desired distant reading technique. Because it is unknown how many topics are to be expected from a discussion thread, a pass through a range of numbers of LDA topics is necessary. Investigating a whole range of numbers of topics may additionally reveal several different levels of topical granularity that can be captured using LDA.

LDA has already been used for text analysis in the area of Digital Humanities. For example, Emmery and van Zaanen (2015) investigated the application of LDA to identify changes in the number of comments on news articles on the topic of online security before and after the Snowden revelation in 2013.

Lastly, we choose LDA as counterpart to close reading, since both approaches focus on content of the texts (posts within the Reddit thread). We strengthen this perspective by removing stop words (which are mostly function words) from the textual data, before modeling topics with LDA.

As mentioned above, deciding on the optimal number of topics to be modeled is problematic. In the case of a Reddit thread, new topics unfold as the discussion proceeds, which means that the number of topics depends on the size of the Reddit thread. It is therefore difficult to make an informed decision about the number of topics for an LDA model, based on manual inspection of comments alone. To resolve this problem, we propose a possibility to determine an adequate number of topics for a set of already present manual annotations.

During the annotation process, each post (the original thread submission question or a comment on it) in the Reddit thread is treated as a separate document (Figure 1, left). For each document, the LDA topic with the highest probability is selected. This is certainly a limiting decision, but we find that the per-document probability distributions usually strongly favor one single topic with a very high probability, while the probabilities for the other topics are close to zero.

After analyzing the documents, they are thus represented by two labels, one from two distinct sets of annotations each: a close reading annotation and a (single) LDA topic. Based on this information, we now aim to identify LDA topics and manual annotations that express similar concepts. Going through the list of documents, co-occurrences of the two annotations are counted in a matrix (Figure 1, right, LDA topics in columns, manual annotation classes in rows).

Given the co-occurrence matrix, the aim is to find, for each LDA topic, the manual annotation class that is best represented by the topic. In order to do so, the matrix is passed columnwise, and the manual annotation class in the row with the highest count is selected as corresponding best to the LDA class of that column. At this point we introduce an optional step of normalisation, applied on the co-occurrence matrix (Figure 2). Collecting absolute counts in the matrix may give an unfair advantage to annotation tags with high class support. During normalisation, the counts are divided by the class support of the manual annotation class of this row, resulting in fractions instead of absolute numbers. The steps described in the rest of this article always make use of both the absolute counts and the normalised counts matrix separately.

For each manual/LDA pair of classes with highest co-occurrence, we generate a mapping from LDA classes to manual annotation classes (Figure 3). The mapping expresses classes of annotations that, from now on, we regard as corresponding to each other. Note that, as in the example of Figure 3, not necessarily all manual annotation classes are covered by the mapping.

Using the mapping, the LDA classes are translated into manual annotation classes (Figure 4). These can be regarded as the computer’s “guess” of the manual class that was assigned by a human. In the original list of documents, each document is now represented by two annotations that are drawn from one common pool of possible annotations. In machine learning terms, the manual annotations are treated as gold standard (or true labels) and the mapped LDA classes are regarded as predictions. To measure the extent to which the predictions and the gold standard overlap, we calculate the accuracy.

For a single topic model, we have now obtained a measure of overlap between close reading based, manual annotations and distant reading LDA topics. In order to find the optimal topic model, several topic models with varying numbers of topics can be computed and the overlap accuracy can be compared. For the present study, we produced topic models with a number of topics ranging from 1 topic to N topics, with an N equal to the number of documents in the collection. We expect a topic model with as many topics as are documents in the corpus to be of low expressive power. Similarly, a topic model with only one LDA class cannot adequately express several manual annotation classes.

For the Reddit thread on the question Should the Democrats nominate a celebrity in 2020? What would be the pros and cons?, overlap accuracy is calculated for LDA models with 1 to 461 topics. The topic model with the best fit to the manual annotations is defined as the topic model corresponding to the highest accuracy value.

Note that increasing the number of LDA topics leads to a higher likelihood of high accuracy. Imagine the situation in which each document receives a unique LDA class. This allows for a perfect accuracy, but the predictive power is extremely low as no document is comparable according to the LDA classes. To resolve this issue, a second possibility to determine the optimal number of topics is also investigated. Here, we reverse the mapping step and transform manual classes to LDA classes. The steps described above stay the same, except the co-occurrence matrix is passed row-wise instead of column-wise[3].

As seen in the results, both the forward mapping and the reverse mapping result in a perfect solution: Mapping LDA classes to manual classes leads to increasingly high accuracy with an increasing number of topics. Mapping manual classes to LDA classes, in turn, leads to complete overlap for a single LDA class, because all manual classes are correctly mapped to that single LDA class. The maximum accuracy values in the two scenarios are misleading because neither having a single LDA topic nor having hundreds is useful for the purpose of distant reading. In our approach we exploit this phenomenon by aiming for the right balance between the two perfect solutions. We do so by calculating the absolute difference between forward and reverse accuracy for each number of topics. Subsequently, we select the number of topics with the lowest absolute difference in the accuracies from the forward and reverse mapping.

Based on the close reading strategy described above, we identify fifteen classes of posts (Table 1). Ten of them are related to three different underlying questions that we formulate based on the classes: What characteristics should a president have in order to be a good leader? (Red, class 1-3), Which parties could influence the likeability of a potential president? (Green, class 4-5), In which political climate and context is the question discussed? (Blue, class 6-10). The other five are classified as “actions” that are divided over two groups, hyperlinks to other sources (Yellow, class 11-12) and (self-)referential comments/responses (Purple, class 13-15).

By assigning colors to the classes that relate to the same question or action, we can distinguish the variety of discussions inside one subreddit and see how these discussions develop throughout the thread (Figure 5). From left to right, the first 130 comments mostly discuss the political climate (Blue) (e.g., “Democrats didn’t show up. Republican turnout fairly steady.” (post 63)). These comments seem to create a context for the discussion as a whole. Thereafter, the characteristics of an ideal president are discussed (Red) (e.g., “I say charisma goes a long way, part of being president is to inspire people. Dems should go young and the candidate should be well spoken (Obama) and inspiring. Leave the dynasties, get someone younger, inspire people to come out and vote for them.” (post 120)). Near the end, the influence of media and celebrity culture are set against the former discussions (Green) (e.g., “Do you think he would’ve won the popular vote if not for that video” (post 17)). The “actions” of hyperlinking (Yellow), joking (e.g., “We’ll get him a box. Worked for Napoleon.” (post 239)) and (emotional) responding (e.g., “I’ll concede that that’s an excellent point.” (post 57)) (Purple) occur equally frequent throughout the whole subreddit. The variety of questions discussed in the subreddit does not seem to occur randomly, but in clusters: We can see a certain development in how the colors follow each other up.

Following application of the LDA topic modeling technique and the process to assign LDA topics to manual topics, we measure the overlap between manual and computational annotation. Accuracy measures are obtained for a varying number of topics in the LDA model. Figure 6 plots the overlap accuracy over the number of topics. It can be observed that the accuracy keeps increasing with higher numbers of topics in the LDA models. The reason for this is, that with more LDA topics, more and more topics get assigned to only one document. During the mapping, the single occurrence of an LDA class is necessarily converted to the correct manual annotation class. This creates the aforementioned perfect solution to the mapping problem. However, the model with the highest overlap accuracy has low generalising power. Such a detailed topic model moves away from the desired distant reading perspective, which would group the comments into a limited set of categories.

In order to resolve the problem of overspecification, we look at a combination of the forward and reverse accuracy. As outlined above, we balance the two LDA to manual assignment and manual to LDA assignment scores and inspect where the two curves intersect. The intersection is defined as the number of topics with the lowest absolute difference in accuracy. Figures 7 and 8 depicts forward and reverse accuracies over number of topics based on absolute and normalised co-occurrence counts.

To evaluate how well the LDA model overlaps with the manual annotations it is of fundamental importance to compare the performance against a baseline. As baseline we use a random assignment of the manual annotations to the documents. We explicitly decide not to make use of another common baseline, the majority class baseline. The reason for this is that we aim to cover all classes instead of just a single one and random assignment better reflects this goal. The random assignment labels are then compared to the actual manual assignment. This process is bootstrapped and averaged over 1000 trials. The accuracy measures derived from the random assignment and the LDA topic models are presented in Table 2. Regardless of whether the co-occurrence matrix is normalised for class support the LDA classes lead to a higher overlap with the manual annotations than the random assignment. The overlap between the LDA classes and the manual annotations is thus above chance level. Comparing the non-normalised model to the normalised model it can be observed that normalisation increases recall by covering relatively many manual classes to the mapping even when a small number of topics is modeled.

For the Reddit thread case study, the described process results in a topic model with 6 topics that has the highest overlap with the gold standard annotations. However, the measures based on absolute counts are still strongly biased towards the majority class. Since we intend to cover all classes as well as possible, the accuracy based on absolute counts is too optimistic for our purpose. The corresponding model using normalised counts may better generalise to all classes. According to the model normalised for class support, 23 topics best represent the manual annotation.