{"id":162,"date":"2014-08-12T12:01:37","date_gmt":"2014-08-12T16:01:37","guid":{"rendered":"https:\/\/sites.temple.edu\/tudsc\/?p=162"},"modified":"2019-08-26T11:31:13","modified_gmt":"2019-08-26T15:31:13","slug":"text-scrubbing-hacks-cleaning-your-ocred-text","status":"publish","type":"post","link":"https:\/\/sites.temple.edu\/tudsc\/2014\/08\/12\/text-scrubbing-hacks-cleaning-your-ocred-text\/","title":{"rendered":"Text Scrubbing Hacks: Cleaning Your OCRed Text"},"content":{"rendered":"

By Beth Seltzer<\/p>\n

<\/p>\n

I believe that as digital methods grow in prominence, we\u2019ll develop a huge collection of beautiful, correctly spelled, carefully curated texts for scholars to use for textual analysis.<\/p>\n

The problem (as I point out in my earlier post<\/a>) is that we\u2019re not quite there yet. A lot of books are only available as messy OCR that\u2019s never gotten any type of spell-checking, and other texts you have to OCR yourself from a PDF.<\/p>\n

Fortunately, there are a few things you can do to clean up your texts without programming.<\/p>\n

I used a program called ABBYY FineReader Pro to OCR my collection of Victorian detective novels that are only available as PDFs. I also tried out Adobe Acrobat Pro in an admittedly not particularly controlled experiment. I found they were both relatively accurate for OCR, but ABBYY FineReader was faster and more intuitive to use, which makes sense because it is just designed for OCR. I also tested out the Pro vs. the free versions of ABBYY with the help of one of our librarians, and discovered that the Pro version is significantly better than the free version.<\/p>\n

One issue with OCRed texts is that old books often repeat the book title and\/or chapter title on each page, and the computer will read these as text like any other text. And if your file repeats the title “Murder or Manslaughter?” two hundred times, it will significantly skew your textual analysis results! ABBYY solves this problem by automatically detecting page features such as headers and footers.<\/p>\n

\"Screenshot<\/a>
Abbyy analyzes the page layout automatically.<\/figcaption><\/figure>\n

Then, you can change your settings so that headers and footers aren’t included when you create a text version of the file:<\/p>\n

\"Screenshot<\/a>
Make sure “headers and footers” isn’t checked.<\/figcaption><\/figure>\n

You can save your file in many different formats. I chose to save as plain-text, the format most often used for textual analysis. It\u2019s still got plenty of OCR errors, but it looks a little nicer than the plain-text I got from the Internet Archive, and it only takes 10-15 minutes to OCR a book (depending on the size of your PDF).<\/p>\n

But once everything\u2019s OCRed, how do you spell-check it? You could check each text file individually\u2014but even with a relatively small corpus of 200 novels, that would take weeks of work. You can speed up the process with a free program called Notepad++.<\/p>\n

\"Screenshot<\/a>
Notepad++ opens all your files at once in separate tabs.<\/figcaption><\/figure>\n

Not particularly fancy-looking, but this program will let you open all of your text files at once and do a find-and-replace on your whole corpus with just one step\u2014a huge time-saver!<\/p>\n

My sense is that we\u2019re only a few years, at most, from a fancy spellchecker that will clean up your OCR automatically. In the meantime, here are some of the top corrections I needed in my 19th<\/sup>-century corpus:<\/p>\n

    \n
  • Remove all page numbers*: 55,776 corrections<\/li>\n
  • Recombine all hyphenated words beginning with com-, con-, in-, re-, and un-*: 8,002 corrections<\/li>\n
  • Recombine all hyphenated words ending with ment, tion, and ing*: 5,612 corrections<\/li>\n
  • Fix some common misspellings:<\/li>\n
  • delete \u25a0 (960)<\/li>\n
  • bis -> his (422)<\/li>\n
  • delete “digitized by” (1331)<\/li>\n
  • liis -> his (250)<\/li>\n
  • tbe -> the (337)<\/li>\n
  • tiie -> the (698)<\/li>\n<\/ul>\n

    And so on!<\/p>\n

    *I did these more complex searches through something called “regular expressions.” Basically, instead of searching for and deleting every number individually: 1, 2, 3, 4\u2026 301, 302, etc., you can tell the computer \u201cdelete all the numbers\u201d by entering a short code you can look up on the internet. (In this case, the code is [0-9]+ ) Pretty convenient!<\/p>\n","protected":false},"excerpt":{"rendered":"

    By Beth Seltzer<\/p>\n","protected":false},"author":1420,"featured_media":163,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[2,288],"tags":[334,333,37],"class_list":["post-162","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-grad-students","category-literary-studies","tag-abbyy","tag-digitization","tag-textual-analysis"],"_links":{"self":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/162","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/users\/1420"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/comments?post=162"}],"version-history":[{"count":0,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/162\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/media\/163"}],"wp:attachment":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/media?parent=162"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/categories?post=162"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/tags?post=162"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}