Written by\u00a0 Liz Rodrigues<\/p>\n
<\/p>\n
After I had assembled a list of US immigrant autobiography and checked to see which were available in full text and plain text files, the next step was to get those files. This list of about fifty books was probably short enough to do this manually–by looking up each title and copying & pasting the file into notepad–but to prepare for future projects, I wanted to learn how to do it programmatically. <\/span><\/p>\n My first step was to do some googling and find out what kinds of access Project Gutenberg, the Internet Archive, and HathiTrust provided to their text files. The first two both offer an API–basically allowing you to write a script that will query their server for particular kinds of data and then download it. HathiTrust also offers an API, but it takes more sophisticated programming to get it. So, I decided to focus on Gutenberg and IA, which offered the majority of my texts anyway. Before grabbing anything, it\u2019s a good idea to check the terms of service. Both sites allow moderate downloading for personal use but suggest that if you want the entire site you should set up your own mirror.<\/span><\/p>\n It took a little bit of tweaking to do what I specifically wanted to do which was get the full text, put it into a folder where I could quickly keep track of it, and not have a bunch of extra copies on my computer. Here is what I came up with–stripped down versions of more ambitious scripts adapted to my specific need.<\/span><\/p>\n For getting texts from Internet Archive, I used <\/span>Jacob Johnson\u2019s Internet Archive package for Python<\/span><\/a>. It is well-documented and does many things beyond grabbing full text, but since that\u2019s what I was focused on that\u2019s what I started with. It ran smoothly last fall and just today, using this very basic code <\/span>(sorry I can\u2019t make it prettier, our installation of wordpress doesn\u2019t currently support markdown)<\/span>:<\/span><\/p>\n import internetarchive<\/span><\/p>\n from internetarchive import get_item<\/span><\/p>\n import shutil, os<\/span><\/p>\n <\/p>\n corpus = [“text_identifier_1\u201d, \u201ctext_identifier_2\u201d ]<\/span><\/p>\n #the text identifier is a number found in the url for the text<\/span><\/p>\n <\/p>\n for text in corpus:<\/span><\/p>\n \u00a0 \u00a0 item = get_item(text)<\/span><\/p>\n #identifies the files available for the text you want<\/span><\/p>\n \u00a0 \u00a0 t = text + “_djvu.txt”<\/span><\/p>\n #specifies that you want the plain text file specifically<\/span><\/p>\n \u00a0 \u00a0 f = item.get_file(t)<\/span><\/p>\n #gets that plain text file<\/span><\/p>\n \u00a0 \u00a0 f.download(t)<\/span><\/p>\n #downloads the plain text file<\/span><\/p>\n new_name = text[0:(len(text))]<\/span><\/p>\n \u00a0 \u00a0 shutil.copyfile(t, new_name)<\/span><\/p>\n #renames the file with a sleek version of the text id<\/span><\/p>\n \u00a0 \u00a0 shutil.move(new_name, ‘path_to_where_you_want_the_file\/’)<\/span><\/p>\n #moves the file to a place where you know where it is<\/span><\/p>\n \u00a0 \u00a0 os.remove(t)<\/span><\/p>\n