In our project we distinguished emotions from related categories such as mood, trait, cognitive state, physical state, edonic state, attitude and sensation (cf. WordNet-Affect - [Strapparava and Valitutti 2004]). We decided to focus on emotions proper, although the adjacent category of “moods” turned out to be hardly distinguishable from emotions in some cases. Additionally, we adopted the definition of emotions by Klaus Scherer who defines an emotion as a strong feeling deriving from one’s circumstances or relationships with others involving cognitive appraisal, bodily symptoms, (a readiness for) action, motoric expression (for instance in face or voice) and subjective awareness [Scherer 2005, 697, 703–5]. Besides emotions we are also interested in the simultaneous occurrence of bodily reactions. In total, we developed five annotation categories (Table 1).

In the annotation guidelines we further defined the five categories and provided lists of examples (see HEEM-Github for more detailed information). Starting from these definitions and after a preliminary exploration of a subset of corpus texts, we composed a list of 37 emotions relevant for the interpretation of early modern theatre texts (Table 2).

Words were annotated and interpreted in context, which means that a polysemous word like the Dutch noun “hoop” which has two meanings (heap and hope) will only be identified if it is used in its second meaning. Likewise, body related concepts, i.e. body parts, bodily processes and body sensations, are only identified if they are involved in an emotion expressed in the text. In the case of bodily processes, for instance, the reaction must be triggered by an emotion and not by any other phenomenon. For example, trembling is annotated in a sentence such as she trembles with anger, but is not annotated in: she trembles from the cold.

Each annotated concept is classified into one of the 37 classes of the HEEM emotion classification. Consider, for example, the following sentence and its annotations: Ex. (1) “Ik kreeg van haar daar zulk gekyf; / Dat noch het hart beefd in me lyf” (“She scolded at me so terribly that my heart is still trembling in my body”). The word “gekyf” (scolding) can be identified as an emotional action and linked to the emotion class Anger. “Hart” (heart) is annotated as a body part and, in this case, linked to the emotion class Fear. Likewise, “beefd in me lyf” (tremble in my body / breast) is identified as Bodily Process and also linked to the emotion class Fear. In the annotations, no distinction was made between the references to body parts or bodily processes in a literal and a metaphorical sense. The reason for this decision was that such a distinction is often quite difficult to make in early modern texts, where expressions which we now consider to be metaphorical often had a quite material basis in humoral theories of the passions.

The annotation was performed by a group of experienced readers of early modern Dutch (theatre) texts. An inter-annotator study was carried out to check the consistency and reliability of the annotations. Five annotators annotated independently 2 documents with a total of approx. 1100 sentences. Agreement is calculated following the metric agr as proposed by [Wiebe et al. 2005] and [Read and Carroll 2010], who calculate agreement on similar data. This metric calculates pair-wise agreement by first measuring precision of annotator A’s annotations on B’s annotations, and then measuring precision of annotator B’s annotations on annotator A’s annotations. Agreement between 2 annotators is the mean of the 2 scores. We only report overall agreement (Table 3 and 4), which is the mean of the pair-wise scores.

One of the problems of these kinds of annotations is that different annotators are likely to choose different unit lengths while marking up what is essentially the same. If we consider again example (1), we cannot know whether an annotator identifies only the word “beefd” (trembles) or whether he considers the whole sentence (“het hart beefd in me lyf”) as a Bodily Process. We took a lenient approach and considered an overlap of one word as matching.

We first measured agreement on the identification of the different concept types in text. Table 3 shows that the concept types Emotion, Body Part and to a lesser extent Sensation are reliably identifiable, with overall scores of 73, 73 and 61%, respectively. Bodily Processes and Emotional Actions, however, have low agreement scores, which do not allow for definite conclusions. On the basis of these results we decided to drop the categories Emotional Action and Bodily Process for our model development. The lack of agreement in these two categories is caused by the fact that some annotators tended to include more expressions than others. Plays are literary texts that artistically play with words and that are larded with metaphors. In the following sentence body part and emotion are easy to identify: “Wat nevel van verdriet bezwalkte uw blinckende ogen” (What haze of sorrow drapes your gleaming eyes). There is sorrow connected to the eyes, but what exactly happens to the eyes? And should that be taken literally? We agreed to take metaphors and expressions literally even though it could be a literary construction and thus that a phrase like this should be tagged as the description of a bodily process. Nonetheless this type of “bodily processes” could be easily overlooked.

We also measured agreement on the classifications of emotions into one of the specific HEEM emotion categories. On this task the annotators achieved an agreement of 85%. The numbers show that the identification of emotions (0.73) is harder than their classification into subcategories once the expressions are identified (0.85).

While we were developing HEEM, we also started experimenting with off-the-shelf sentiment mining tools. We chose Linguistic Inquiry and Word Count (LIWC) as the most promising sentiment and embodiment mining technique for further exploration. LIWC is a text analysis software program, which provides a simple but effective way of measuring tendency and strength of emotions (and other psychological properties) in text [Pennebaker et al. 2001] [Pennebaker et al. 2015]. The LIWC dictionary was translated into Dutch and tested by Dutch social scientists [Zijlstra et al. 2004].[2]

The LIWC software processes each word of a text by searching for matches with a categorized dictionary, including around 4,500 words and word stems. As the software is developed by and for psychologists, it provides lists of words on wellbeing, affect, cognition, biological processes, social practices, and some specific emotions, e.g. anger, anxiety, sadness [Tausczik and Pennebaker 2010] [Kahn et al. 2007].[3] The categories Posemo and Negemo cover a broad range of terms that express positive and negative appreciations and sentiments including adjectives and adverbs like nice or edgy, as well as states or situations that could allude to moods or indicate positive or negative appraisal such as alone or gloomy. See Table 4 for an example of some of the calculations of LIWC on an early modern Dutch theatre text: Achilles by Balthazar Huydecoper (1719). We will analyze the results of this table later in this paper. But first, we need to discuss the reliability of LIWC for researching early modern texts.

LIWC has been previously applied on historical data sets, without an evaluation of the performance of the software [Borowiecki 2013] [Leemans and Johannes 2013]. As LIWC is designed for modern texts, it underperforms in historical texts. To diminish this problem, we extended the Dutch LIWC dictionary with historical data by linking it to GiGaNT, a lexicon service provided by the INL (INL Lexicon Service). GiGaNT offers spelling and word variants of Dutch language, from the 15th to the 20th century. We used a subset covering the period from 1600-1800. The “historical Dutch LIWC” (HD-LIWC) was created by adding spelling variants for all words to the original Dutch LIWC (cf. the spelling normalization procedure applied in [Zwaan et al. 2015]). The original Dutch LIWC dictionary contains 6,512 terms, while the HD-LIWC contains 8,757 terms. Table 5 shows that most historical terms map unambiguously to a single modern variant (96%). For historical terms that mapped to multiple modern variants, a single variant was selected randomly.

Applying the “historical Dutch LIWC” (HD-LIWC) to our corpus resulted in higher coverage as more words were labeled with one or more LIWC categories. Table 4 shows percentages of words found for the positive emotions (Posemo), Negative emotions (Negemo), and Body categories using the original Dutch LIWC and HD-LIWC in one selected play: Achilles by Balthasar Huydecoper (1719). However, the higher percentages achieved by HD-LIWC must be interpreted with care, because our method of developing HD-LIWC introduced noise into the list. We identified three types of noise that were introduced. First, for historical words that map to multiple modern alternatives, one alternative was selected at random. For example, the historical term “vreed” is a spelling variant of both “vrede” (peace) and “wreed” (cruel). In the historical LIWC, “vreed” only maps to “vrede”, the meaning of “vreed” as “wreed” is missed. The second source of noise is caused by the way in which the LexiconService returns spelling variants for verbs. Regardless of the tense of the verb entered, the LexiconService returns spelling variants in all tenses. This means that LIWC categories Present and Past are unreliable and should not be used. The third source of noise is introduced by inconsistencies in GiGaNT/the LexiconService. For example, the term “gehoond” (mocked) is retrieved as a spelling variant of “gezond” (healthy) and counted in the LIWC category Body, where it obviously does not belong.

As an experiment, we also compiled another emotion wordlist for Dutch based on the English resource WordNetAffect [Strapparava and Valitutti 2004]. WordNetAffect is built as an extra layer on the Princeton WordNet [Fellbaum 1998] semi-automatically labeling each set of synonyms with a so-called affect type (such as mood, trait, emotion, behavior, etc.). Part of these synsets is additionally labeled with a set of fine-grained emotion classes such as anger, fear, disgust, etc. We transferred these labels from the Princeton WordNet to the Dutch WordNet [Vossen et al. 2008] using the equivalent links between the two WordNets. We then selected the words labeled with the affect type “emotion” and expanded the set with historic variants adopting the same strategy as followed for the historical version of LIWC. For the tests presented in this paper we did not take into consideration the fine-grained emotion classification.

The result in numbers are presented in Table 6. As this resource is automatically generated it includes also incorrect labels or omissions mainly due to erroneous or missing equivalent links or even to the fact that the original English resource has been built also in a partly automatic way.

The tables 9, 10 and 11 present correctly found, and missed emotion words, and words that HD-LIWC counts, but HEEM ignores. These lists provide insight into the way HEEM works in comparison to D-LIWC and HD-LIWC. Table 9 shows the most frequent words correctly found by HD-LIWC. The words in bold such as “vreezen” (fear), “vreugd” (joy) and “spyt” (reget) are historical variants of emotion words that are correctly found by HD-LIWC, and that are missed by D-LIWC. Table 10 shows the most frequent words that are still missed by HD-LIWC. Many words such as “gramschap” (anger), “toorn” (anger) “min” (love), “minnen” (to love), and “mededogen” (compassion) are archaic emotion words that are no longer in everyday use or even obsolete. These words are not included in the HD-LIWC as they are not variants of contemporaneous words with an affective meaning. It also shows that some historical form variants are still lacking in HD-LIWC (e.g. “jalouzij” (envy), “lijefd” (love), “liefd” (love), “haaten” (hate) etc.) although their modern counterparts (i.e. “jaloezie”, “liefde” and “haten”) are present in D-LIWC.

Table 11 constains the words that are found by HD-LIWC, but are ignored by HEEM. It includes words such as “goed” (good), “ryk” (rich), “held” (hero) which are appraisals and judgments rather than emotions in the stricter sense. In HEEM only descriptions of emotions as defined by Scherer (see paragraph IV) are annotated while LIWC includes all kinds of sentiment words including emotions, attitudes, appraisals and beliefs as well as concepts that are strongly associated with positive or negative feelings such as “hemel” (heaven) or “straf” (punishment).

A second set of highly frequent errors comes from incorrectly found form variants. The verb form “moet” (must) and the adverb “eer” (before) are wrongly considered as historic form variants of the noun “moed” (courage) and the noun “eer” (honor), respectively. It seems worth trying to solve these confusions by filtering on part-of-speech, but this is currently not an option as well-performing NLP tools are not yet available for historical Dutch.

Finally, LIWC does not take into account polysemy or context. LIWC overestimates the sentiment value of texts since it classifies ambiguous words like “alleen” (alone) and “vry” (free) as negemo and posemo, and lists “social” pronouns such as “we” as positive sentiments. The polysemous noun “hoop” has an affective meaning (hope) and a non-affective meaning (heap). The theater texts include both usages as in “van alle hoop ontbloot” (deprived from all hope) and “een hoop geboeyde slaven” (a lot of chained slaves). In HEEM the former usage is annotated as an emotion and the latter is not, whereas LIWC is not able to distinguish between them. Likewise, we can see that a word like “lust” (desire) appears both on the list of correctly found words (cf. Table 9) and on that of the incorrectly found words (cf. Table 11). It will therefore be difficult to achieve high scores with a word counting method like LIWC on an early modern literary corpus.

The LIWC category “Physical” will return body parts such as hands, eyes, womb or ears, and bodily processes or reactions such as trembling, bleeding or blushing, but also general physical states such as awake, sleepy, wounded, or illnesses (anorexia, angina, AIDS), and even loosely related body words such as alcohol, bathroom, bathrobe, cook, kitchen and dishes. HD-LIWC even broadens this category including words like “schoon”, which can mean clean and beautiful, but in early modern Dutch is in many cases used for however (“(of)schoon”). Also, the indicated problem that HD-LIWC introduces noise in spelling variants in verb tenses, is visible in the returns HD-LIWC gives on the historical corpus. One of the words with the highest recall rates in the category physical is was. This past tense of the verb to be, is indicated as physical since LexiconService has was listed as a verbal conjugation of “washing”. The broad range of the LIWC category Physical, and the noise created by HD-LIWC therefore renders this category as not very indicative for research into physical states or reactions to emotions.

The LIWC category Body is more precise and concise (400 words) than Physical (648 words). But again, noise is introduced by HD-LIWC. “zich” (his) is indicated as a body term, as a spelling variant of sight (“zicht”), just as “vry” (free), as a conjugation of “vrijen” (making love), “tien” as a variant of “teen” (toe), and “arme” (poor) as spelling variant of “arm” (arm). Altogether 25% of the indicated body terms is incorrect. Although this is a high percentage, it is quite easy to correct, since the same noise words keep coming back and their percentage is quite high (of the 25% of noise 75% is caused by the word “zich”).[4] For future use HD-LIWC could be cleared of these easily traceable mistakes. Apart from these apparent over-interpretations, HD-LIWC indicated many words as body terms that are ambiguous. “Vryen” can mean to make love, but in most instances it is an inflection of “vrij” (free). As long as LIWC only counts single words, without taking context in consideration, this will remain a problem.