Skip to content

Loretta C. Duckworth Scholars Studio

⠀

Menu
  • Scholars Studio Blog
    • Digital Methods
      • coding
      • critical making
      • data visualization
      • digital pedagogy
      • immersive technology (AR/VR)
      • mapping
      • textual analysis
      • web scraping
    • Disciplinary Fields
      • Anthropology
      • Archaeology
      • Architecture
      • Art History
      • Business
      • Computer Science
      • Critical Digital Studies
      • Cultural Studies
      • Dance
      • Economics
      • Education
      • Environmental Studies
      • Film Studies
      • Gaming Studies
      • Geography
      • History
      • Information Science
      • Linguistics
      • Literary Studies
      • Marketing
      • Media and Communication Studies
      • Music Studies
      • Political Science
      • Psychology
      • Public Health
      • Sculpture
      • Sociology
      • Urban Studies
      • Visual Art
    • Cultural Analytics Practicum Blogposts
  • Current Staff
  • Newsletter
  • About
    • Games Group 
Menu

Counting Words, Predicting Grades: A Computational Study of Student Writing

Posted on December 16, 2022October 18, 2023 by Megan Kane

By Megan Kane

Introduction

Should computational tools be used to assess student writing? Their use is certainly becoming popular, particularly in high-stakes tests like the GRE, TOEFL, and College Board placement exams. However, many writing instructors remain skeptical of automated scoring systems, and the use of computational scoring systems is also opposed by the Conference on College Composition and Communication (CCCC), the largest organization for college writing professionals.

As a writing instructor myself, I agree that computational tools can’t replicate human reading practices that are critical to good writing assessment. Still, as a digital humanities researcher I see the merits of computational text analysis to study significant language patterns within large text corpora. 

In this blog post, I suggest this analysis can be leveraged to assist (rather than replace) current writing program assessment practices. My Fall 2022 Cultural Analytics Practicum project explored one possible application for these tools by asking: Can computational tools detect language patterns in student writing that correlate to high grades? 

In the rest of this post, I’ll walk through the Python code pipeline I developed to explore my research question. My code is also available in full on this Github repository.

Data Context and Preparation

For this study, I used a collection of around 150 student papers written in Temple University’s First-Year Writing Program, and the grades given to these papers by the course instructor. The papers I collected were written portfolio-style, so each submission actually contained four papers: a reflection, a rhetorical analysis, a topic exploration, and an argumentative essay. 

Importantly, each portfolio was assigned a singular grade, so students were assessed based on how they conducted a variety of writing tasks. My study aimed to computationally identify and analyze language patterns correlated with students’ success in two of these tasks: 1) use of rhetorical terminology to analyze a source and 2) identification and engagement with arguments used in secondary sources. 

To prepare the data for analysis, I converted the original versions of all papers (.docx files) to .txt format so they would be readable by Python. I uploaded the documents and scores to Google Colaboratory, a browser-based tool for running Python code, and organized them in a Pandas data frame (similar to a spreadsheet or table). Before conducting analysis, I used the Natural Language Toolkit to remove all instructor and student names from the essays and conduct some basic cleaning (lowercasing, punctuation removal and stop-word removal). I also segmented each text into paragraphs, since my pipeline aims to identify the immediate contexts within which students were using language to conduct the two tasks of interest. 

Linear Regression of Rhetorical Terms Across Papers 

To determine what types of language computational tools could identify that indicated the successful performance of rhetorical analysis, I turned to the course’s syllabus and assignment guidelines. Three classical rhetorical terms featured prominently and were clearly expected to appear in student work: “pathos” (the appeal to emotion), “ethos” (appeal to credibility) and “logos” (appeal to logic). As such, I conducted a computational keyword analysis of these terms to see how frequently they appeared in each portfolio. 

Data Frame with Essay Paragraphs and Rhetorical Analysis Term Counts
Figure 1: Output of Data Frame with Essay Paragraphs and Rhetorical Analysis Term Counts

Regression analysis revealed a weak positive correlation between the use of rhetorical terms and high scores (R-squared=0.025; 2.5 percent of the variance in essay scores can be explained the model). In essence, the rhetorical term usage is not a strong predictor of essay score. As the regression plot shows, rhetorical terms were used to varying degrees across all grade ranges:

Figure 2: Linear Regression Plot for Rhetorical Analysis Terminology by Score

Such analysis indicates that computational identification of rhetorical analysis terminology cannot strongly predict instructors’ scores. In short, simply counting words–even ones instructors prize in the classroom–cannot significantly explain why instructors are making certain scoring decisions. This also tells students not to expect high scores simply because they use these terms frequently in their writing. 

Along these lines, it is critical to note that these student portfolios are graded based on a constellation of factors, rather than their performance on a single task. Let’s look at another type of language that might better correlate to high scores: citation practices.

Linear Regression of Citation Practices Across Papers 

To capture how students used language to engage with secondary sources in their work, I used two forms of computational analysis: string-detection of parenthetical citations and the tool DocuScope‘s metric for ‘citation’. First, knowing that students in this class used MLA citation style, which requires parenthetical-style in-text citations wherever sources are referenced, I ran code to search for parenthetical expressions used in the portfolios. I then removed parenthetical expressions without features indicating citation, like capital letters and/or numbers.

Figure 3: Linear Regression Plot for Parenthetical Citations by Score

To complement this analysis, I used the DocuScope computational rhetorical analysis platform to identify language related to citation. DocuScope is a dictionary-based tool which “tags” texts based on a number of categories its authors deem as rhetorically meaningful. In this case, I was interested in the language DocuScope labeled as “Citation,” or language described as “statements that attribute words, thoughts, and positions to previous others.” Examples include “According to”, “based on,” and “as cited in”. This analysis was conducted outside of Python using the DocuScope desktop app, but the resulting file with citation language counts was uploaded to my script for analysis. 

Figure 4: Linear Regression Plot for DocuScope Citations by Score

Both parenthetical citations (R-squared=0.057) and DocuScope terminology (R-Squared=0.062) had weak positive correlations to essay grades. However, both were stronger than rhetorical analysis terminology, signaling that there may be a significant correlation between citation practices and essay grades that could be further isolated with future study. For students, this indicates that it’s important to be aware of the number of citations used in a paper, while also attending to other factors (such as, perhaps, their quality) which are not quantified here.

Conclusions and Next Steps

So, can computational tools detect language patterns linked to score? My project reveals that the answer is complicated. Neither rhetorical or citation-based language was proven to strongly correlate to high essay grades. However, even low R-squared values may be significant when studying human behavior (like grading), and the weak correlations yielded interesting insights about the relationship between score and language use for instructors and students. The data set used in this pilot study was also limited in size and spread (e.g., there were far more A- and B-range essays than C- and D-range grades). Research using a larger, more representative dataset could yield more accurate results.

In future study, my plans include include: 1) broadening the types of terms I’m counting as related to analysis or citation, 2) calculating the correlation between grades and students’ usage of both rhetorical and citation practices, using ANOVA or PCA, 3) investigating language features related to other valued writing tasks, like argumentation, and 4) more comprehensively studying the contexts in which students are utilizing rhetorical terms, citations, and other language practices. Ultimately, this project encourages my continued exploration of how computational tools can help us develop more robust ways of analyzing and assessing student work. 

Recent Posts

  • The Untold History of Fletcher Street’s Stables April 21, 2025
  • Building an Immersive Archive of the Greek Orthodox Churches in Istanbul April 15, 2025
  • Tracing Influence in Genealogies of Communication Theory April 14, 2025

Tags

3D modeling 3D printing arduino augmented reality banned books coding corpus building critical making Cultural Heritage data cleaning data visualization Digital Preservation digital reconstruction digital scholarship film editing game design games gephi human subject research linked open data machine learning makerspace makerspace residency mapping network analysis oculus rift omeka OpenRefine Photogrammetry Python QGIS R SketchUp stylometry text analysis text mining textual analysis top news twitter video analysis virtual reality visual analysis voyant web scraping webscraping

Recent Posts

  • The Untold History of Fletcher Street’s Stables April 21, 2025
  • Building an Immersive Archive of the Greek Orthodox Churches in Istanbul April 15, 2025
  • Tracing Influence in Genealogies of Communication Theory April 14, 2025

Archives

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Archives

Blog Tags

3D modeling (11) 3D printing (14) arduino (8) augmented reality (5) banned books (3) coding (12) corpus building (4) critical making (7) Cultural Heritage (11) data cleaning (4) data visualization (11) Digital Preservation (3) digital reconstruction (9) digital scholarship (12) film editing (3) game design (3) games (6) gephi (3) human subject research (3) linked open data (4) machine learning (6) makerspace (8) makerspace residency (4) mapping (30) network analysis (17) oculus rift (8) omeka (3) OpenRefine (4) Photogrammetry (5) Python (8) QGIS (10) R (9) SketchUp (4) stylometry (8) text analysis (10) text mining (4) textual analysis (32) top news (102) twitter (5) video analysis (4) virtual reality (17) visual analysis (5) voyant (4) web scraping (16) webscraping (3)

Recent Posts

  • The Untold History of Fletcher Street’s Stables April 21, 2025
  • Building an Immersive Archive of the Greek Orthodox Churches in Istanbul April 15, 2025
  • Tracing Influence in Genealogies of Communication Theory April 14, 2025
  • From Theory to Practice: Weaving in Response to the Grid in the Global Context March 26, 2025
  • Visiting a Land of Twilight February 24, 2025

Archives

©2025 Loretta C. Duckworth Scholars Studio | Design: Newspaperly WordPress Theme
Menu
  • Scholars Studio Blog
    • Digital Methods
      • coding
      • critical making
      • data visualization
      • digital pedagogy
      • immersive technology (AR/VR)
      • mapping
      • textual analysis
      • web scraping
    • Disciplinary Fields
      • Anthropology
      • Archaeology
      • Architecture
      • Art History
      • Business
      • Computer Science
      • Critical Digital Studies
      • Cultural Studies
      • Dance
      • Economics
      • Education
      • Environmental Studies
      • Film Studies
      • Gaming Studies
      • Geography
      • History
      • Information Science
      • Linguistics
      • Literary Studies
      • Marketing
      • Media and Communication Studies
      • Music Studies
      • Political Science
      • Psychology
      • Public Health
      • Sculpture
      • Sociology
      • Urban Studies
      • Visual Art
    • Cultural Analytics Practicum Blogposts
  • Current Staff
  • Newsletter
  • About
    • Games Group