Digital Humanities Questions & Answers

1. 

   mallios
   Member

   I'm looking for a tool or set of tools that allow, as simply and easily as possible, for the following kinds of word frequency tests to be performed on a relatively large set of textual data: (i) ranked quantification of most frequently used words in a textual archive; (ii) plotting such things over time (i.e., how word frequencies rise and fall over time, as the dates of the individual textual items in the archive progress); and most importantly (iii) collocation tests identifying and ranking most frequently used words within, say, 25 or 50 words, of a target word specified within the textual archive (e.g., what's the most frequently used word with 25 words of "Tolstoy"?)

   Thanks much for any help anyone can provide.

   • Tweet this question

   Posted 3 years ago Permalink
2. 

   Josh
   Member

   For a web-based solution, I believe Voyant can do most of this (though I'm not sure about collocation). The two best places to look for tools online are Bamboo DiRT and, especially for text analysis, TAPoR.

   Posted 3 years ago Permalink
3. 

   devinhiggins
   Member

   Based on my own very recent experience, I would also recommend the Natural Language Toolkit (NLTK) for Python. NLTK would make tasks (i) and (iii) quite simple, and I don't think task (ii) would be too much of a challenge either. If you have any familiarity with Python, NLTK would be an easy solution to implement, and the documentation in the form of this free e-book is great. You could also use NLTK to run MALLET topic modeling if so desired.

   Posted 3 years ago Permalink
4. 

   mallios
   Member

   Replying to @Josh Honn's post:

   Thanks very much, Josh.

   Posted 3 years ago Permalink
5. 

   mallios
   Member

   And thanks too, Devin.

   Posted 3 years ago Permalink
6. 

   Stéfan Sinclair
   Admin

   There's a collocates tool in Voyant Tools, but it's not included in the default "skin" (pre-defined arrangement of tools – users can create their own skins). For that part, you might want to use the collocates skin, as with this example from the Shakespeare corpus.

   Posted 3 years ago Permalink
7. 

   jlmcdonald@gmail.com
   Member

   Stéfan, I'd be interested in finding out more about what's underneath Voyant's collocates tool. The documentation seems to imply it's just showing term frequency in proximity to keyword (i.e. sort of aggregating the KWIC data), but I wonder if tool is subsequently doing more than that to get at the corpus linguist's definition of a collocate. That is, is it reflecting at all the likelihood that the co-location is occurring at a rate (statistically) significantly greater than chance?

   Posted 3 years ago Permalink
8. 

   Stéfan Sinclair
   Admin

   @Jarom: in lleu of proper documentation (ahem), let me just say that the default ordering of collocates in Voyant is by its relative frequency "ratio", which is essentially the difference between the term's relative frequency within the contexts (text segments in proximity to the keyword) and its relative frequency in the entire document. That value is available in the "ratio" column of the collocates table, as in this example.

   Posted 3 years ago Permalink

RSS feed for this topic

Reply

You must log in to post.