Digital Humanities Questions & Answers » Tag: text analysis

Digital Humanities Questions & Answers » Tag: text analysis - Recent Posts http://digitalhumanities.org/answers/tags/text-analysis Digital Humanities Questions & Answers » Tag: text analysis - Recent Posts en-US Tue, 21 May 2013 17:45:22 +0000 http://bbpress.org/?v=1.0.2 <![CDATA[Search]]> q http://digitalhumanities.org/answers/search.php johnlaudun on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1932 Wed, 13 Mar 2013 22:43:40 +0000 johnlaudun 1932@http://digitalhumanities.org/answers/ Replying to @Michael Widner's <a href="http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1872">post</a>: I didn't know about GenSim. What a great library for Python, and the site has some really nice explanatory material on it, too. Thanks for the link. Josh Honn on "How does one prepare and use data for network analysis with Gephi?" http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1885 Fri, 15 Feb 2013 03:02:35 +0000 Josh Honn 1885@http://digitalhumanities.org/answers/ John: Thanks for the links! As for more on what I want to do, I'll try my best to summarize it (though the idea is still in early formation/I don't necessarily know everything yet I want to do, look for, etc.). Also, the corpus, as it stands, is only 4 novels at around 800 pages, but this will increase as this author's novels are translated into English (and, in the future, I'd also like to add works from similar authors to the corpus). My ideas: 1. Extract names, places, and titles of works from the text 2. Perform some kind of frequency rankings (within works and across corpus) 3. Visualize connections: e.g. authors & their works mentioned in relation to each other, which authors are most mentioned in relation to each other, groupings of real and imaginary authors (something important and often ambiguous in these texts), and, eventually, locate overlaps in the network of this network with similar networks created by other authors. In other words, these novels, especially taken as a whole, embody a fairly vast historical (and sometimes fictional) network of literary figures and their works and I'd like to extract these from the corpus in order to (a) better access and analyze them, while at the same time exposing this labyrinthine network that might get lost (in its overwhelming totality) within each narrative and even more easily across multiple "distinct" works. Other questions: can narrative arrive just from, say, a network of authors? What makes metafiction more than that + annotation? etc. Again, these are just my initial ideas, and this is an intellectual side project but one that has implications in my daily work as a librarian working in the digital humanities; often times we need a project of our own to really acquire and retain substantive new skills. For more insight into Vila-Matas' metafictions, <a href="http://bit.ly/XHkPQ1">here's a good essay</a>. johnlaudun on "How does one prepare and use data for network analysis with Gephi?" http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1884 Fri, 15 Feb 2013 01:34:39 +0000 johnlaudun 1884@http://digitalhumanities.org/answers/ Replying to @Josh Honn's <a href="http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1883">post</a>: Depending on what you are doing, and depending on how much data we are talking about, you may not need to go the TEI route. If you are on a Mac, I've written about setting up NLTK here: <a href="http://johnlaudun.org/20121230-macports-the-key-to-python-happiness/" rel="nofollow">http://johnlaudun.org/20121230-macports-the-key-to-python-happiness/</a>. The TL;DR version is here: <a href="http://johnlaudun.org/20121230-macports-for-nltk/" rel="nofollow">http://johnlaudun.org/20121230-macports-for-nltk/</a>. Write back with more info and I'm sure more people will kick in with help. Josh Honn on "How does one prepare and use data for network analysis with Gephi?" http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1883 Thu, 14 Feb 2013 22:53:08 +0000 Josh Honn 1883@http://digitalhumanities.org/answers/ Oh great, another Python book to read! Just kidding. Thanks, Korey (and Justin)! NLTK looks like the way to go. Just glancing over the extracting information chapter, and from others who have suggested TEI, it's clear that I need to, in some way, go from unstructured to structured text before I do anything else. And now back to my irregularly scheduled Python reading/learning. Korey Jackson on "How does one prepare and use data for network analysis with Gephi?" http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1881 Thu, 14 Feb 2013 21:10:17 +0000 Korey Jackson 1881@http://digitalhumanities.org/answers/ Hey Josh, Just talking to a friend of mine (Justin Joque at the spatial and numeric data library here at Michigan). One thing he mentioned was that Cytoscape might be a little more user- and spreadsheet-friendly...though one strength of Gephi is its ability to animate networks over time, if that's something you're looking for. More generally, he thought it sounded like you were needing to do some named entity extraction, and recommended the Python-based <a href="http://nltk.org">nltk.org</a> for that. They also host a great resource--Natural Language Processing with Python--at <a href="http://nltk.org/book/">nltk.org/book</a>. Hope that helps! Josh Honn on "How does one prepare and use data for network analysis with Gephi?" http://digitalhumanities.org/answers/topic/how-does-one-prepare-and-use-data-for-network-analysis-with-gephi#post-1880 Thu, 14 Feb 2013 16:09:22 +0000 Josh Honn 1880@http://digitalhumanities.org/answers/ There are plenty of tutorials on the web that are useful for learning Gephi, but I've encountered a much steeper learning curve for the steps prior—such as extracting and preparing data from literary texts to be used with Gephi. In my example, I am interested in performing a network analysis of the social networks (both real and imaginary) in the works of Enrique Vila-Matas (I'd eventually like to expand this corpus to other authors). Acquiring the digital text and working with Gephi I can do (or the latter learn), but it's the very important intermediary steps of preparing the data (extraction of names and places to getting those in a form—database? spreadsheet?—analyzable by Gephi) that I need help with. Any good reading, tutorials, or other resources out there? Other recommendations? I'm working on a Mac and have a limited knowledge of Python, so any Mac-friendly software would be a big help. Thanks! Michael Widner on "Stopword list for Old English" http://digitalhumanities.org/answers/topic/stopword-list-for-old-english#post-1874 Fri, 01 Feb 2013 21:14:50 +0000 Michael Widner 1874@http://digitalhumanities.org/answers/ Thanks for the response. That fits what I was expecting given the variability of the language. scottkleinman on "Stopword list for Old English" http://digitalhumanities.org/answers/topic/stopword-list-for-old-english#post-1873 Fri, 01 Feb 2013 20:36:23 +0000 scottkleinman 1873@http://digitalhumanities.org/answers/ I've tried to generate Old English stop word lists for various experiments from time to time, but never to my satisfaction. Spelling variation is a problem, though not as bad as Middle English. But there is also a tremendous amount of homography--so much that it is almost better to generate a list of homographs and tag forms with separate meanings in the source text(s) before doing anything else. After that, you can just generate a list of the most frequent forms in the corpus. You'll probably then have to tailor this list by deleting forms like "wæs" or "cwom" if they're not relevant to your experiment. Michael Widner on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1872 Fri, 01 Feb 2013 19:11:14 +0000 Michael Widner 1872@http://digitalhumanities.org/answers/ If you're planning on working in Python anyway, you might want to look at the gensim library: <a href="http://radimrehurek.com/gensim/" rel="nofollow">http://radimrehurek.com/gensim/</a> It would let you perform some topic modeling in the code, so wouldn't require the stop-gap between word frequencies and mallet. Michael Widner on "Stopword list for Old English" http://digitalhumanities.org/answers/topic/stopword-list-for-old-english#post-1870 Fri, 01 Feb 2013 19:03:28 +0000 Michael Widner 1870@http://digitalhumanities.org/answers/ I haven't been able to turn up an existing stop word list for Old English. Does anyone know of one? Thanks! Ben Marwick on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1842 Mon, 31 Dec 2012 22:35:36 +0000 Ben Marwick 1842@http://digitalhumanities.org/answers/ Replying to @<a href='http://digitalhumanities.org/answers/profile/cforster'>cforster</a>'s <a href="http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1767">post</a>: Here are a few lines of R that I'm working with for topic modelling. These lines should: 1. read in the JSTOR CSV wordcount files to R 2. convert them from a table of words and their counts to a 'bag of words' 3. for each CSV file, create a txt file of the 'bag of words' ready for MALLET A sample of 1000 articles from 'American Antiquity' takes about 7 sec to read in the CSV files and about 6 sec to write the 'bag-of-words' txt files <div class="bb_syntax"><table><tr><td class="line_numbers"><pre>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 </pre></td><td class="code"><pre class="r" style="font-family:monospace;"># set working directory, ie. location of JSTOR DfR CSV # files on the computer setwd("C:\\some directory with JSTOR DfR CSV files")   # create a list of all the CSV files myFiles <- list.files(pattern="*.csv|CSV")   # read in all the CSV files to an R data object myData <- lapply(myFiles, read.csv)   # assign file names to each dataframe in the list names(myData) <- myFiles   # Here's the step where we turn the JSTOR DfR 'wordcount' into # the 'bag of words' that's typically needed for topic modelling # The R process is 'untable-ing' each CSV file into a # list of data frames, one data frame per file myUntabledData <- sapply(1:length(myData), function(x) {rep(myData[[x]]$WORDCOUNTS, times = myData[[x]]$WEIGHT)})   # And here's the step where we create individual txt files # for each data frame (formerly a CSV file) that should be suitable for # input into MALLET. names(myUntabledData) <- myFiles sapply(myFiles, function (x) write.table(myUntabledData[x], file=paste(x, "txt", sep="."), quote = FALSE, row.names = FALSE, eol = " " ))   # Look in the working directory to find the txt files</pre></td></tr></table></div> ` I have a few more snippets that use the citations CSV for filtering and attaching biblio data to the R data and topic modelling using R (both packages) and MALLET. Some of these are here: <a href="https://gist.github.com/benmarwick" rel="nofollow">https://gist.github.com/benmarwick</a> Andrew Goldstone on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1788 Mon, 12 Nov 2012 21:14:44 +0000 Andrew Goldstone 1788@http://digitalhumanities.org/answers/ Replying to @<a href='http://digitalhumanities.org/answers/profile/cforster'>cforster</a>'s <a href="http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1773">post</a>: Looked again at this, am embarrased by that snippet. Here's a whole script for making bags of words from jstor's wordcount csv files: <a href="https://github.com/agoldst/dfr-analysis/blob/master/count2txt" rel="nofollow">https://github.com/agoldst/dfr-analysis/blob/master/count2txt</a> . It is still in perl, so there. I couldn't get <code>mallet train-topics</code> to work on an instance file produced from <code>mallet import-svmlight</code>, but I didn't try very hard. cforster on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1773 Mon, 29 Oct 2012 13:48:31 +0000 cforster 1773@http://digitalhumanities.org/answers/ Replying to @Andrew Goldstone's <a href="http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1771">post</a>: Thanks very much; I'll likely cobble together a similar script in Python (Perl? What is this, the nineties? I kid... I kid...). Very much appreciate it. And thanks for the "fla" filter tip. I may revise my query with that in mind. I'm a little suspicious of the data because I ran a quick word count for one file against the data they provided and it didn't seem to match up. I'll look into that at greater length and then get in touch with the DfR folks should my suspicion be borne out. cforster on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1772 Mon, 29 Oct 2012 13:45:36 +0000 cforster 1772@http://digitalhumanities.org/answers/ Replying to @<a href='http://digitalhumanities.org/answers/profile/tedunderwood'>tedunderwood</a>'s <a href="http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1768">post</a>: Excellent; thanks very much. Andrew Goldstone on "Topic Modeling (MALLET) with JSTOR Data For Research" http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1771 Mon, 29 Oct 2012 13:25:53 +0000 Andrew Goldstone 1771@http://digitalhumanities.org/answers/ Replying to @<a href='http://digitalhumanities.org/answers/profile/cforster'>cforster</a>'s <a href="http://digitalhumanities.org/answers/topic/topic-modeling-mallet-with-jstor-data-for-research#post-1767">post</a>: Glad you're trying out DFR too. Lots of treasures buried in there, I feel sure. As Ted says, when I applied mallet to the data from jstor, I just reconstituted a bag of words from each word-count file in csv format, basically with this perl: <div class="bb_syntax"><table><tr><td class="line_numbers"><pre>1 2 3 4 5 6 7 8 9 </pre></td><td class="code"><pre class="perl" style="font-family:monospace;">my $header = <INFILE>; die unless $header =~ /^WORDCOUNTS,WEIGHT/;   while(<INFILE>) { my ($word,$count) = split ','; if($word) { print OUTFILE "$word " for (1..$count); } }</pre></td></tr></table></div> and then passed the resulting files to mallet import-dir. It was quite cheap in time and space for the set of about 10^4 articles we were working on. I don't think the mallet command-line tool can take a file of word counts. The MALLET java library must operate on word counts in the end, and I think if you interface with MALLET through java you can feed it the counts directly: cf. <a href="http://mallet.cs.umass.edu/import-devel.php">MALLET: Data Import for Java Developers</a>. I went with the fast-and-dumb method because my java is too weak to figure this out in short order and I wanted instant gratification. If your data looks funky, write the dfr support e-mail address; they were quite helpful to Ted and me. As Ted says it's a beta, but you are supposed to get wordcounts for the full articles. Remember that lots of jstor items are not articles but reviews, front and back matter, etc. You can filter by item type "fla" (full-length article) to get only articles (put <code>ty:fla</code> in your search field). edit: Or possibly you could convert the csv wordcounts to "SVMLight-style" feature:value pairs? Didn't try this, but see the bottom of: <a href="http://mallet.cs.umass.edu/import.php">http://mallet.cs.umass.edu/import.php</a>. edit: One last note; see below.