Voyant Tools is a free, open-source software with several built-in tools that can be used for text mining. Text mining essentially reduces the total amount of visible text and draws attention to only significant aspects or trends, which you would not have noticed by reading the text (Sinclair and Rockwell, 2016). I recently spent some time exploring the algorithms in Voyant Tools to see what I could learn about a historical corpus. The dataset was the WPA Slave Narratives, a collection of interviews with former slaves collected as part of the Federal Writers’ Project for the Works Progress Administration between 1936 and 1938 (See Musher, 2014).

I first explored the tools one at a time. The files I uploaded were organized by state, therefore, I could not only analyze the text of all the files together as one corpus but also by comparing individual files to examine trends between states. I added a stop list of words to exclude from my analysis in order to reduce the noise of the most used words. The Cirrus tool creates a word cloud by highlighting the highest frequency words in the text. First, I looked at the top words for the entire corpus using Cirrus, and I thought I had a pretty good sense of what themes were emerging. Second, I compared interviews between two individual states—Indiana and South Carolina. This was very instructive because I was able to see how each of these states’ interviews were different from the larger corpus and from each other. The most frequent words were not the same. I have a lot of theories about what this data means, so I jotted down some questions and observations to revisit later.

Indiana (top) vs. South Carolina (bottom)

In the same window when I clicked on Terms, a list of the most common words ranked according to count, as well as a trend line showing which states had the most instances of each word. Another algorithm, Links, created a visualization that connected words together, indicating a relationship between words the frequently appear together. This provided an opportunity to consider those relationships, generating more research questions.

The other default windows in Voyant Tools include Reader, Trends, Summary, and Context; I will just give a quick summary of what can be learned about a corpus from those tools. The Reader allowed me to look at a word in context. This was very helpful in investigating which words were significant. Sometimes a word appeared often, but upon further contextual investigation, I discovered it was being used in a section heading (e.g., the word folklore was used in headings), or it was a name (Samples was a name rather than the common noun I was expecting). The Trends algorithm had a few different trend visualization options. I used a line graph to compare word usage between states. The Summary tool provided statistical information like document lengths and average words. It also listed distinctive words for each state. These were words that stood out as different than what was being used across the larger corpus. I could see that the usage of the word whippers was used far more often in Indiana than in any other state, which generated more questions to investigate about why that might have been the case. The Contexts tool allowed me to see all the instances of a word briefly in context of the words directly to the left and right in the text.

The word whippers used by state

As I delved deeper into my exploration of Voyant Tools, I learned that using the tools together could reveal more than using them individually. They were designed to work together. While exploring the most distinctive words in the summary section of the tool, I began using the multiple tools in tandem. Once I clicked on a word, I was immediately able to see the context in another window. Then I could click on the word in the Context window and see it in in the Reader window, where I could evaluate the usage and distribution of the word. I could then click on the word in the Reader and see a graph come up in the trends window, comparing how often the word was used in each state’s interviews. In this way it was possible to toggle between windows to explore the use of words more deeply. A few of the potential research avenues that could be explored using Voyant Tools include how word usage changes over time, how usage varies between different documents or datasets, how much of a corpus is dedicated to a particular topic, or what texts are connected thematically.

The tools provide visual representation of the data, facilitating analyses of trends that may or may not be significant and clues to what they could mean; however, the tools themselves do not create meaning (see Sinclair and Rockwell, 2016.) Text mining requires an incredible amount of analysis, and the best use of text mining tools is as a companion to traditional methods of study. Only through an understanding of human behavior, motives, and context do digital patterns have any real meaning. Voyant Tools reveals connections that aren’t apparent from reading the texts and unearths layers of information about historical records for further inquiry. While historians are still trying to determine the best use of text mining and the best integration into historical study, the potential for understanding change over time and for modeling silences in the archive is undeniable (see Guldi, 2024).

References

Guldi, Jo. “The Revolution in Text Mining for Historical Analysis Is Here.” The American Historical Review 129, no. 2 (2024): 519–43.

Jockers, Matthew, and Ted Underwood. “Text-Mining the Humanities.” In The New Companion to Digital Humanities, edited by Susan Schreibman, Ray Siemens, and John Unsworth. Chichester: Wiley Blackwell, 2016.

Sharon Ann Musher, “The Other Slave Narratives: The Works Progress Administration Interviews,” in John Ernest, ed, The Oxford Handbook of the African American Slave Narrative (New York: Oxford University Press, 2014).

Sinclair, Stefan, and Geoffrey Rockwell. “Text Analysis and Visualization: Making Meaning Count.” In The New Companion to Digital Humanities, edited by Susan Schreibman, Ray Siemens, and John Unsworth, 330-349. Chichester: Wiley Blackwell, 2016.

Humanists do not use data the same ways as other disciplines. We do not simply pull datasets and find that they can easily be interpreted in useful ways. Most of the time data must be mined from available sources and modeled in a way that is helps to answer a research question. Any data could be mined, but without a careful, methodical data modeling, it would be difficult to evaluate the gaps in the data as well as the nuances that historians and other humanists are trained to address and interpretation through their expertise. The upside to the work that goes into these projects, though, is that mining and structuring data allow us to study sources in new ways.

Creating data for humanities projects requires a lot of interpretation of sources. Because the formatting of historical sources, for instance, is not standard, it requires interpretation. In evaluating interviews conducted and collected of former slaves in the 1930s as part of a project done by the Works Progress Administration, I noticed how varied the length, format, and content was between the interviews. There is no reliable way to automate the aggregation of meaningful data across these interviews as a group. Instead, I followed a checklist suggested by Johanna Drucker to create a data model, which includes things like identifying information that can be described unambiguously, standardizing formats, and checking for assumptions and bias before utilizing the data (2021).

Working with the data is an important part of the research project in digital humanities. Transparency about the process of data handling is a crucial part of the overall project. Why was the scope pared down to what it was? What data was ultimately eliminated and why. What decisions were made about standardization of the data and what does that mean for the outcome? One project that I evaluated this week was the Early African American Film Project, which involved collecting data about early race films. One of the things I appreciated about their project was the transparency about some of the decisions the researchers had to make and the inclusion of data they ultimately cut. They did this so that other researchers could evaluate their decisions because the aggregation of data and the data modeling is part of the research process that others need to understand in order to evaluate the work (see Hoekstra and Koolen, 2019).

When humanities source data is put in a structured format, new methods of study are possible. Another project I evaluated this week was Tudor Networks of Power. Researchers used computers to an assisted cleaning of data, to link data, and to create visualizations of the information cleaned from the datasets, which illuminated patterns in the ways letters were used in a political network in a way that could never be understand by a researcher simply reading all the letters over time and trying to see the patterns on their own. The data was carefully mapped and cleaned, and the researchers were very detailed in their descriptions of their processes.

We have moved way beyond digitizing sources and thinking that is the extent of the usefulness of technological applications for the humanities. There is so much more that can be evaluated by mining and modeling historical data that does not appear in a surface level reading alone.

 

References

Drucker, Johanna. “Data Modeling and Use.” In The Digital Humanities Coursebook: An Introduction to Digital Methods for Research and Scholarship, 1st ed. Taylor and Francis, 2021.

Hoekstra, Rick, and Marijn Koolen. “Data scopes for digital history research.” Historical Methods: A Journal of Quantitative and Interdisciplinary History 52, 2 (2019): 79-94.

With the expansion of digital humanities work, the line between historian or DH practitioner and archivist has blurred and it has become vital for us to learn something about archival principles to create more responsible digital projects that are also discoverable and helpful for other researchers. Metadata is one aspect of archival study that bleeds into the work of digital humanities. Metadata is often described as data about data (Carbajal and Caswell 2021). It includes, among other things, the title, description, dimensions, and format of our items that allow context for our resources.

Metadata is useful for DH practitioners for several reasons. I’ll focus just on just three. First, I would argue that we can have a better grasp of the resources we are describing, analyzing, creating, or preserving if we are considering all their properties. According to Carbajal and Caswell, “The interpretation of records requires context. If users do not know who created a record, or where or why, it becomes virtually meaningless to preserve that record over time . . .  [Metadata] allows us to keep in mind the context of when, where, and from what perspective that information was collected” (2021). Metadata can help explain the significance of an item, so, the quality and depth of our study could be influenced by a greater understanding of our sources. The design of a digital project could also be informed by and structured according to information gleaned from the metadata.

Second, metadata helps to create and maintain structure to large amounts of data. Keeping the data organized lends to a more thought-out project. Also, we aren’t likely to forget what we learn from the metadata over the weeks, months, or years, that we are working on a project because the data is attached to the items or sources we are working with. Context and order in the data structure help keep a project from becoming unmanageable or unfocused.

Third, I would argue that metadata helps the researchers or users of our projects to understand our sources and find what they need. This is accomplished in several ways: items are contextualized for those who visit a digital project, items are searchable through carefully linked subject associations, tagging, and quality titles descriptions, and, if mapped correctly, resources are discoverable through internet searching. We live in a digital world where consumers are used to being able to navigate quickly through interconnected information and recommendations based on their preferences, and careful metadata practice allows for increased visibility and searchability for the best user experience (see Vogelsang, 2024).  Standardized metadata practices and information architecture also provide a shared language of understanding between cultural heritage professionals, making your work mappable and understandable for others.

I recommend a couple of tools for using and keeping track of metadata for digital humanities professionals. Tropy is a free, open-source desktop software tool that helps to keep digital documents and photos organized. It allows historians and DH practitioners to keep dates, archival data, and rights information organized and attached to the sources, rather than having to keep separate files. It also features tagging and grouping capabilities, which makes it possible to filter materials by different criteria depending on the workflow. For a digital humanities project, sources could be grouped in Tropy by section of a project, for instance, keeping research clean and organized. Another benefit of Tropy is the ability to seamlessly transfer metadata to Omeka via a CSV file. I’ve started adding my sources in Tropy at the time I collect them, and it has made for a smoother process between research and writing or between research and digital project.

One of the reasons that Omeka is such a great platform for digital humanities projects (in addition to being free and open source) is the built-in metadata fields to follow recognized industry standards (such as Dublin Core) for describing resources. In addition to the built-in fields, it’s possible to create your own custom vocabulary if your sources would benefit from additional descriptive categories. I find this incorporated structure beneficial in building quality metadata that will help others find my work and make good use of it.

 

Resources

Carbajal, Itza, and Michelle Caswell. “Critical Digital Archives: A Review from Archival Studies.” American Historical Review 126, 3 (2021): 1102-1120.

Vogelsang, Henning von. “Categories Or Tagging? Differences In Taxonomy And IA.” Bootcamp, November 22, 2024.

Digitization is an important topic for professionals in the digital humanities community. DH work includes engagement with digitized historical records, artifacts, and images as well as corpora and other machine processible datasets created from historical sources (Terras 2012).  One of the benefits of understanding the digitization process and environment is a better grasp of the capabilities and drawback of digital representations. While online collections are of great benefit for research, teaching, and engagement, it is important to understand that working with digital proxies of documents and material culture can be fundamentally different than working with concrete, physical items. It can change how we perceive the items and what we are able to do with them.

When an item is digitized, some things are captured and some are not. Even with the most advanced technology, digital representations cannot communicate to the senses what cannot be captured (e.g., I cannot currently digitize the smell of an old book). In my experience photographing and scanning items, paper documents are by far easier to work with. Two-dimensional scans are far less expensive and time consuming than more advanced technologies (Terras 2022). Text is obviously captured by the scan, and sometimes folds or textures of paper can be seen as well. If a document is colored, it is difficult to distinguish whether the color on the screen is a true representation of the physical version. Other than the obvious smell and sound associated with paper documents, contextual clues about the care of the document and position within the collection are difficult to include in the capture and can only be described in associated metadata.

Some items are not easily captured or require more advanced 3D imaging techniques for a more accurate representation. Photos or two-dimensional scans of three-dimensional objects are insufficient in many ways: all sides of an object are not visible, textures look flat, and a lack of context for understanding size and scale. Technologies such as photogrammetry can create a 3D rendering based off many overlapping photos taken from different angles that are stitched together (Drucker 2021). I used a free app called Kiri Engine, which is available for iPhone and Android. I discovered that it took some practice and that lighting and being patient about taking enough overlapping photos was crucial for good results. I found that these 3D scans gave a better representation of objects and artifacts dimensions, angles, and textures and allowed the object to be rotated and inspected from all angles. Some things that were still hard to capture were the weight of an item and size to some extent. Smell and sound and feel, of course, are also not captured with this technology. The processing was slow and took several minutes for each item, so that’s one consideration when applying the use of the technology to a large project.

Engaging with documents online is different from pulling physical papers out of an archival box. When I smell and touch a box of old papers, different questions are raised in my mind beyond the text I see on the pages. I think about where the documents were kept, how they were preserved, and I think about provenance. Those questions don’t surface as naturally with digitized documents. However, one of the benefits of digitizing text documents is the ability to use technologies like OCR (optical character recognition) to make text easier to read, process, search, index, and create data sets. The technology is not perfect; bad lighting, colored paper, or unusual formatting could affect the output. I found that my phone can quickly transform any text from a photo into plain text that I can use elsewhere. In my testing of this technology, I found that my sample worked well, but I would not trust it to do large sets of text for a project without quality assurance checks.

The constraints of a digital screen determine how we can interact with an item. Even a photo or scan of a document or photo could look slightly different on two different screens depending on color settings, so accuracy of representation is not even entirely controllable. I can see an item, but I can’t touch it. Maybe the weight might be something worthy of analysis on an archaeological artifact, but I wouldn’t even think to address that in my study because I didn’t experience that characteristic of the item in my digital interaction.

Digital tools on the other hand afford new opportunities for study and preservation. For instance, Joanna Drucker discusses several use cases in which technologies allow for study or preservation of items without unearthing them or displaying fragile items without exposing them to human touch. Digitization can prevent fragile documents from being handled repeatedly. Digitization also allows for the display of multiple interpretations of restoration for an artifact without altering the appearance of the actual item (Drucker 2021).

In addition to understanding what digitization can and cannot do and how it affects DH projects, it’s a good idea to be aware that there are digitization standards as well as cost and sustainability concerns for institutions regarding digitization projects. Understanding the digitization process helps DH professionals to contribute to the conversation on what to digitize in the future that will do the greatest good.

References

Drucker, Joanna. “Photogrammetry.” In The Digital Humanities Coursebook: An Introduction to Digital Methods for Research and Scholarship, 1st ed. Taylor and Francis, 2021, 162-168.

Terras, Melissa. “Digitisation and Digital Resources in the Humanities,” In Digital Humanities in Practice, edited by Claire Warwick, Melissa Terras, and Julianne Nyhan. Facet Publishing, 2012.

Terras, Melissa. “Digital humanities and Digitised Cultural Heritage.” In James O’Sullivan (ed.), The Bloomsbury Handbook to the Digital Humanities. London: Bloomsbury Academic, 2022, 255-266.

Digital Humanities projects are simplest when working with sources already digitized. Research strategies for finding sources have changed over last two decades with a rising number of digitized sources and academic databases available to universities.

One benefit to academic databases of historical collections is that I can access archival collections from across the country. I have access to so many resources, but navigating the search can be daunting.

I did some searching in the databases available through my university and found that I had to browse through the databases first and figure out which ones might apply to my research. Second, I had to come up with a list of search terms that I was going to use in my searches. I was searching for material on Mormon settlement in the West, so I searched for Mormon pioneers, Mormon settlement, Mormons Utah, Mormons West, LDS settlement, Mormon migration, etc. One thing to consider while searching is that the same term might not apply throughout history. There may have been a different way a certain group of people was characterized or nicknamed in the past. It’s important to keep those in mind when searching. Once I found a document of interest, I paid attention to the other keywords associated with it in the metadata. I found exploring this to be a very helpful practice. Some of the databases also allowed browsing by topics as a finding tool. I found this to be helpful as well.

Although I was pretty sure I knew most of the Mormon collections and the repositories that held them, doing searches in academic databases opened my eyes to collections of interest held in New Jersey, Illinois, and California that I wouldn’t have known about otherwise. Once I was able to identify all the applicable collections, then I could evaluate their usefulness. Some of my research questions have a lot of applicable digital sources, others have very little. Now I have a better grasp on what collections exist and I can make a decision about how to use the sources available for my project.

Once I decided what collections were applicable to me, I had to remember what I learned previously about copyright and fair use. Some of the documents I found are protected by copyright still.

As I start brainstorming about all the projects I can do in the future, I will consider what digital records exist first before I finalize my project idea. It’s amazing to see what have been digitized in the last 25 years and made publicly available, although a lot of materials are still protected by copyright. I enjoyed perusing the databases and learning about what different repositories hold. While going to an archive has its benefits as well, having a wealth of digitized primary sources opens up many new avenues of research not possible in the past.

Imagine you found a collection of historic photographs online that pertain to your research, and you want to use two or three for a digital exhibit. How can you determine if you have permission to use them? I’ve put together a short guide to help answer that question for those photographs and any other source you might find online.

I think it’s important to remember the definition of rights or copyright as you consider using someone else’s content. According to Franky Abbot from the Digital Public Library of America “’Rights’ refer to reproduction or publication restrictions on an item—its copyright status according to US law.” The creator of a work, therefore, can determine how their work is copied and shared for a set number of years, after which it becomes available to the public (Abbott 2015).

One interesting thing is that copyright doesn’t just apply to published materials. The key is that it exists to protect creative work (because you can’t copyright facts), but even a letter or a draft of a story you’ve been keeping tucked away falls under the umbrella of copyright. (see Dombrowski and Sag 2020).

Where to Find Rights Information

If you are lucky and you are perusing a library or archive catalog when you find something of interest, the metadata for an item will usually have a rights and access statement about how the item can be used. Sometimes you have to poke around to find it. For this map at the New York Public Library, there is a slider menu on the right side of the image, where you’ll find the rights information. Occasionally for smaller collections rights information is listed in one place for the entire site. The Willis Bell Photographic Archive has a small link in the footer of the website with copyright information. If no rights information is listed, you can usually direct your question via email to the institution that put the item online.

Whether the rights are explicitly stated or not, it’s a good idea to understand how to determine a copyright status yourself by following these steps:

PART I – Determine the Copyright Status

Step 1. Establish whether the work is in the public domain

If the work is in the public domain, you’re free to use it without restriction or attribution but determining whether something is in the public domain will require knowing some things about the content and its creator. If you are looking on the website of a library or archive, much of that information should be in the metadata.

There are five ways content typically comes into the public domain. I’m borrowing from Franky Abbott for this list.

1. Copyright has expired. For published work in 2026 that includes items published before 1930 (95+ years since creation). For unpublished works, the rule is 70 years after the death date of the creator or 120 years from the date the work was created. That’s right, in many cases unpublished works have greater protection under copyright law!

Side note: published means something slightly different in copyright law than you might think. It is determined by the date a work is first distributed to the public. Franky Abbot’s presentation, she gives an example of a menu and states that it is published according to copyright law because it is distributed to the public.

2. Copyright owner published work without copyright notice (only applicable for items published between 1923 and 1977). The copyright owner had to put a label on the work stating that it was copyrighted, including a date and the name of the copyright owner (something like Copyrighted 2026 Kristin Andrews). If it wasn’t included at that time, it wasn’t copyrighted and is considered in the public domain.

3. Copyright owner failed to renew copyright status. This is only for items published between 1923 and 1963 that DID have a copyright notice added to the work BUT the creator didn’t file renewal paperwork. This requires research, but it is possible to see if something that looks copyrighted during this time is now in the public domain.

4. Copyright owner dedicates the work to the public domain and waives rights by using the Creative Commons Zero dedication (You would need to see this designation on the materials you are evaluating. Read on to hear more about Creative Commons and where to look for this).

5. Copyright does not protect the type of work. Documents created by the Federal government and its employees (as part of their work) are public domain. This also applies to some states.

Step 2. If the work is in the public domain, skip to Part II (Terms of Use). If the work is not in the public domain (copyright applies), look to see if there is a license for use.

Many copyright holders are willing to share their creative works under specific terms. These are known as Creative Commons licenses. You may find that a creator is allowing you to use their photo or video, for instance, if you give them credit or don’t alter their work. There are six different types of Creative Commons licenses. Rights information on many websites will include standardized licensing information (usually in the metadata for an item but more information can often be found in a footer link or in the About section for a collection). For more information on the types of licenses, visit the Creative Commons website.

Also, remember that attribution is best practice, even when not required.

Step 3. If there is no license for use, consider what options you might still have for using copyrighted material.

Consider the following options for using content:

1. Wait until the term of copyright is over.
2. Contact the creator (if known) and try to get permission.
3. Determine if Fair Use might apply to your use of the item.

Fair Use

Fair use is the gray area. If you want to use a copyrighted source in a limited way and you haven’t been able to get permission, you can think about whether your use of the work would fall in fair use territory. There are four main principles to consider as to whether you can get away with using a source. (See Abbott 2015 and Dombrowski and Sag 2020).

Non-profit or transformative vs. commercial or duplication – Simply republishing someone else’s work or using it to sell something is not going to hold up well. Using a portion for a research project would be OK according to fair use.

Factual vs. creative – Facts can’t be copyrighted and can be used more freely than creative works.

Small amount vs. complete work – Fair use says a small portion of the work could be permissible while reproducing a complete work would be outright copyright infringement.

Doesn’t hurt market vs. potential to hurt market – Will the use of the work hurt sales for the creator?

If you weight these four factors and determine that you have a good case for 3 out of 4, you are probably safely in the fair use zone. There is always a certain amount of risk in publishing something that is copyrighted. If the author or creator doesn’t like that you’ve used their work, they may ask you to remove their content, so it’s a good idea to ahead of time whether you are operating in safe territory or infringing on their rights.

Now before you get too excited about the fact that you determined that your source is available to use, please consider the terms of use for the website on which you found it.

Part II. Terms of Use

At this point you have determined if the content you want to use is free to use under public domain, available to use under the terms of a license, or you’ve sought alternative solutions to resolving the copyright problem. Let’s say that you have determined that you can use it for your purposes according to copyright law. The next step is to check the Terms of Use for the website on which it is published.

Probably every possible website where you could find a source has terms and conditions associated with the use of the site. Unfortunately, that means that the company or organization running the site can control how you use the sources found on their site, even when they are clear of copyright concerns. Judy G. Russell of The Legal Genealogist explains that this contract law and that it is enforceable because you can own things that you don’t hold copyright for. This is an annoying extra layer, but it’s very important to pay attention. The terms of use for a website are usually found linked at the footer on a website and can be full of legal jargon, so make sure you read carefully.

That’s it! You made it through. It seems like a lot to consider, but you don’t want to spend months or years working on a project that you can’t ultimately publish because of copyright concerns. It’s much easier to make the assessment when you are evaluating source options upfront.

 

References

Abbott, Franky. “Understanding Copyright.” Public Library Partnership Program, Digital Public Library of America, 2015

Dombrowski, Quinn and Matthew Sag. “DSC #7: The DSC and Mean Copyright Law.” The Data-Sitters Club, September 22, 2020. https://datasittersclub.github.io/site/dsc7.html

Russell, Judy G. “A terms of use intro.” Links to an external site.The Legal Genealogist (blog), April 27, 2012.