The extension can get things done. However, I didn’t have enough control over\u00a0what I want in my data set. Here are some of the troubles:<\/p>\n
(1) I don’t have much control on what the data structure looks like.<\/p>\n
(2) If some information is not displayed on the web, the visual-cue-based selectors of Web Scraper might not work. For example, I want to use ‘post ids’ to label each post, but the web page does not display the ids, which means that it is difficult to “click to select” with Web Scraper.<\/p>\n
(3) If a tag has a child, and there are texts both under and outside the child, it is difficult to “click to select” only those texts outside the child with Web Scraper. Unfortunately, this is how the post contents are structured in html on the forum. This is crucial because I need to separate each post’s original content from its quoted content (if there is any).<\/p>\n
(4) I can’t find a way to identify myself to the web server of the forum. This is important considering that identifying oneself is one of the scraping courtesies.<\/p>\n
To deal with these issues, I turn to python. Inspired by Andrea Siotto’s (2016) post on scraping Wikipedia<\/a><\/span>, I thought it was a good time to start learning python and to do some scrapings as exercise. I managed to write a scraper in python to get the data I wanted. It is doable in a few weeks once you have basic understanding of the python language, the data structures in python, the Beautiful Soup python library, and basics of regular expression. The rest is to google for troubleshooting.<\/p>\n I’m sharing my codes below and it might not be the most elegant way to do the work. If you have any ideas to improve the scraper, feel free to leave a comment. Also, I hope some parts of the codes would be useful if you are scraping similar websites, or come across similar troubles.<\/p>\n Due to the space limit, I am not going into details of what each line of code does. I will explain the general logic of my scraping based on the discussion of nested dictionary in python, and how I dealt with the specific troubles I mentioned above.<\/p>\n General logic:<\/strong><\/p>\n (1) Store the scraped data in python as a nested dictionary.<\/p>\n A basic dictionary in python is composed of multiple key-value pairs. Values of keys can simply be texts and numbers. The values can also be dictionaries. In this later case, inner dictionaries are nested under an outer dictionary. A simple two-layer nested dictionary is shown above. “test” is the outer dictionary, where ‘A’ and ‘B’ are the outer keys, ‘a’, ‘b’, ‘c’, ‘d’, and ‘e’ are the inner keys, and the numbers (as strings) are the values of inner keys. To make it more interpretable, ‘A’ and ‘B’ can be labels of two observations, ‘a’ to ‘e’ can be five attributes of observations, and the numbers can be measures of the attributes. If we write the dictionary into a csv file, it looks like this:<\/p>\n Note that we can reproduce the same csv file with the following lists:<\/p>\n Why is a single nested dictionary preferred to a group of lists? Two reasons: (I) Lists are mutable in python. A list can be easily changed without knowing how changes are made. If, for example, the items in lists are somehow reordered, the resulting csv will present the data differently from the ways measures are originally recorded. If it is a nested dictionary, it is safer because the key-value pair is fixed after being recorded; (II) It is more efficient to search through a nested dictionary than a group of lists.\u00a0Frances Zlotnick (2014) provides an excellent explanation<\/span> on this matter.<\/p>\n In my case, the unit of analysis is post. Each post has a unique id number and 17 other attributes I’m interested in. Therefore, I assign the ‘post ids’ to be the outer keys and the 18 attributes as 18 inner keys.<\/p>\n (2) A big trunk of the codes below (from line 24 to line 125) is doing one thing: creating a list for each attribute containing information that will be used later as values of inner keys in the nested dictionary. This task includes locating the appropriate html tags, looping through the tags and retrieving information, and append each piece of retrieved information to the corresponding lists.<\/p>\n For example, lines 32 to 38 do the following: 1. create an empty list called “authorlist;” 2. locate all the<\/p>\n tags of the class “popupmenu memberaction” (because post authors can be found under these tags by examining the web’s source code), which returns a list containing these tags; 3. loop through this list: for each<\/p>\n tag, find the tags of the class “username online popupctrl” and the class “username offline popupctrl.” We need two classes of <\/a> because the forum makes a distinction between post authors who are online and offline at the moment; 4. extract the author name and append it to “authorlist.” After looping, each author name becomes an item in “authorlist.”<\/a><\/p>\n (3) The next trunk of code (lines 132 to 157) loops through all the lists together, assigns post ids as outer keys, assigns attribute names as inner keys, and pairs inner keys with the values from the 18 lists just created.<\/p>\n (4) The nested dictionary is now created but it’s only for one page at the post level (there are 10 posts maximum on each page under a thread). Some threads have hundreds of posts. Also, at the thread level, there are also multiple pages (20 threads maximum on each page). In total, there are more than 1,200 threads as of 10\/24\/2016.<\/p>\n Therefore, the scraper needs to perform the following three tasks to deal with pagination:<\/p>\n A. Make a thread-to-post connection. After the scraper collects thread urls at the thread-level page, it will “click and open” each of those urls and parse it (see lines 11 to 22);<\/p>\n B. Loop through all pages at the post level under each thread until no “next page” button can be found (see the “while” loop statement at line 20, and lines 161-167);<\/p>\n C. When it finishes scraping at one page at the thread level, loop through following\u00a0pages at the thread level until no “next page” button can be found (see the “while” loop statement at line 10, and lines 169-175).<\/p>\n (5) Writing the nested dictionary to csv (lines 177-213). The output looks like this:<\/p>\ntest=<\/span>{'A'<\/span>:{'a'<\/span>:'5'<\/span>,'b'<\/span>:'4'<\/span>,'c'<\/span>:'5'<\/span>,'d'<\/span>:'7'<\/span>,'e'<\/span>:'8'<\/span>},\r\n 'B'<\/span>:{'a'<\/span>:'3'<\/span>,'b'<\/span>:'8'<\/span>,'c'<\/span>:'8'<\/span>,'d'<\/span>:'10'<\/span>,'e'<\/span>:'1'<\/span>}}\r\n<\/pre>\n<\/div>\n![\"nested](\"https:\/\/sites.temple.edu\/tudsc\/files\/2016\/10\/Screen-Shot-2016-10-17-at-1.45.18-PM.png\")
<\/a><\/p>\ncases=<\/span>['A'<\/span>,'B'<\/span>]\r\na=<\/span>['5'<\/span>,'3'<\/span>]\r\nb=<\/span>['4'<\/span>,'8'<\/span>]\r\nc=<\/span>['5'<\/span>,'8'<\/span>]\r\nd=<\/span>['7'<\/span>,'10'<\/span>]\r\ne=<\/span>['8'<\/span>,'1'<\/span>]\r\n<\/pre>\n<\/div>\n
![\"screen-shot-2016-10-17-at-6-14-31-pm\"](\"https:\/\/sites.temple.edu\/tudsc\/files\/2016\/10\/Screen-Shot-2016-10-17-at-6.14.31-PM.png\")
<\/a><\/p>\n