Voyant Tools is a free, open-source software with several built-in tools that can be used for text mining. Text mining essentially reduces the total amount of visible text and draws attention to only significant aspects or trends, which you would not have noticed by reading the text (Sinclair and Rockwell, 2016). I recently spent some time exploring the algorithms in Voyant Tools to see what I could learn about a historical corpus. The dataset was the WPA Slave Narratives, a collection of interviews with former slaves collected as part of the Federal Writers’ Project for the Works Progress Administration between 1936 and 1938 (See Musher, 2014).
I first explored the tools one at a time. The files I uploaded were organized by state, therefore, I could not only analyze the text of all the files together as one corpus but also by comparing individual files to examine trends between states. I added a stop list of words to exclude from my analysis in order to reduce the noise of the most used words. The Cirrus tool creates a word cloud by highlighting the highest frequency words in the text. First, I looked at the top words for the entire corpus using Cirrus, and I thought I had a pretty good sense of what themes were emerging. Second, I compared interviews between two individual states—Indiana and South Carolina. This was very instructive because I was able to see how each of these states’ interviews were different from the larger corpus and from each other. The most frequent words were not the same. I have a lot of theories about what this data means, so I jotted down some questions and observations to revisit later.
Indiana (top) vs. South Carolina (bottom)
In the same window when I clicked on Terms, a list of the most common words ranked according to count, as well as a trend line showing which states had the most instances of each word. Another algorithm, Links, created a visualization that connected words together, indicating a relationship between words the frequently appear together. This provided an opportunity to consider those relationships, generating more research questions.
The other default windows in Voyant Tools include Reader, Trends, Summary, and Context; I will just give a quick summary of what can be learned about a corpus from those tools. The Reader allowed me to look at a word in context. This was very helpful in investigating which words were significant. Sometimes a word appeared often, but upon further contextual investigation, I discovered it was being used in a section heading (e.g., the word folklore was used in headings), or it was a name (Samples was a name rather than the common noun I was expecting). The Trends algorithm had a few different trend visualization options. I used a line graph to compare word usage between states. The Summary tool provided statistical information like document lengths and average words. It also listed distinctive words for each state. These were words that stood out as different than what was being used across the larger corpus. I could see that the usage of the word whippers was used far more often in Indiana than in any other state, which generated more questions to investigate about why that might have been the case. The Contexts tool allowed me to see all the instances of a word briefly in context of the words directly to the left and right in the text.
The word whippers used by state
As I delved deeper into my exploration of Voyant Tools, I learned that using the tools together could reveal more than using them individually. They were designed to work together. While exploring the most distinctive words in the summary section of the tool, I began using the multiple tools in tandem. Once I clicked on a word, I was immediately able to see the context in another window. Then I could click on the word in the Context window and see it in in the Reader window, where I could evaluate the usage and distribution of the word. I could then click on the word in the Reader and see a graph come up in the trends window, comparing how often the word was used in each state’s interviews. In this way it was possible to toggle between windows to explore the use of words more deeply. A few of the potential research avenues that could be explored using Voyant Tools include how word usage changes over time, how usage varies between different documents or datasets, how much of a corpus is dedicated to a particular topic, or what texts are connected thematically.
The tools provide visual representation of the data, facilitating analyses of trends that may or may not be significant and clues to what they could mean; however, the tools themselves do not create meaning (see Sinclair and Rockwell, 2016.) Text mining requires an incredible amount of analysis, and the best use of text mining tools is as a companion to traditional methods of study. Only through an understanding of human behavior, motives, and context do digital patterns have any real meaning. Voyant Tools reveals connections that aren’t apparent from reading the texts and unearths layers of information about historical records for further inquiry. While historians are still trying to determine the best use of text mining and the best integration into historical study, the potential for understanding change over time and for modeling silences in the archive is undeniable (see Guldi, 2024).
References
Guldi, Jo. “The Revolution in Text Mining for Historical Analysis Is Here.” The American Historical Review 129, no. 2 (2024): 519–43.
Jockers, Matthew, and Ted Underwood. “Text-Mining the Humanities.” In The New Companion to Digital Humanities, edited by Susan Schreibman, Ray Siemens, and John Unsworth. Chichester: Wiley Blackwell, 2016.
Sharon Ann Musher, “The Other Slave Narratives: The Works Progress Administration Interviews,” in John Ernest, ed, The Oxford Handbook of the African American Slave Narrative (New York: Oxford University Press, 2014).
Sinclair, Stefan, and Geoffrey Rockwell. “Text Analysis and Visualization: Making Meaning Count.” In The New Companion to Digital Humanities, edited by Susan Schreibman, Ray Siemens, and John Unsworth, 330-349. Chichester: Wiley Blackwell, 2016.