By mining the same textual data, I’ve done some preliminary work on Principal Component Analysis<\/a> (PCA). Section II explains why I chose PCA.<\/p>\n If you need some R code for doing PCA, feel free to go to Section III.<\/p>\n The interpretation of unsupervised leaning’s results is necessarily data-, algorithm-, and model-dependent (Grimmer & Stewart, 2013<\/a>). The burden of validation is now on the analyst. The key question is: How do we interpret and evaluate the results in a systematic way?<\/p>\n For example, if we look at a dendrogram (a tree diagram for taxonomy), we have to decide where we want to cut the branches. That is, how many clusters do we want to see from the data? If based on the dendrogram only, the decision is often pretty arbitrary. If we use k-means clustering<\/a>, then we must specify a number of clusters in advance. Is there a systematic (automated) way to decide the number of clusters based on data-generating assumptions?<\/p>\n Yes. For example,\u00a0Fraley and Raftery (1998)<\/a> proposed to use the Gaussian mixture model (GMM) to compare models with different parameters (including cluster numbers) by looking at the Bayesian Information Criterion (BIC<\/a>). The greater the BIC of a model, the stronger the evidence for the model. The Gaussian mixture model is straightforward enough to understand: A data point is assumed to first come from the j<\/em>th cluster from a total of K<\/em> clusters (according to a set of probabilities, each for a cluster), and a value for that data point is then drawn from the multivariate normal distribution<\/a> for the j<\/em>th cluster.<\/p>\n All sounds good. BUT, we are working on textual data, which means that the assumption of multivariate normal distribution is highly questionable. In a typical document-term matrix, there tend to be many zeros. For a cluster, a term’s frequency distribution tends to be skewed to the right. Here is a great post on why we should avoid GMM when the distribution assumption is not met<\/a>. In other words, GMM’s don’t seem ideal for textual data, which tend to have unpredictable distribution patterns.<\/p>\n LDA (Latent Dirichlet Allocation) Topic Modeling (Blei & Lafferty, 2009<\/a>) seems to have more plausible assumptions about how words are generated (but it still requires setting parameters ahead of time). As argued in Grimmer and Stewart’s (2013) critique, the way we use language is so complex that automated textual analysis may be inherently premised on more or less incorrect\u00a0<\/i>data-generating models. Therefore, I decided to forget about the model-based unsupervised learning methods for a moment, and used a method that suits my textual data better.<\/p>\n PCA is firstly a dimensionality reduction technique that does not rely on any explicit distribution assumptions (see Section 18.1.4 in this link<\/a>). Everything PCA does is reducing the number of features\/terms\/variables,\u00a0 while minimizing information loss [check out this article for a non-technical explanation<\/a>]. This would be extremely helpful in my case because I have a sparse matrix with over 10,000 columns (words). Also, after dimensionality reduction, I can still do a clustering analysis based on the reduced data.<\/p>\n 1. Preprocessing:<\/strong><\/p>\n This part has been addressed in my previous posts [I just used the standard ‘tm’ package in R]. Assume that we now have a document-term matrix called “dtm.”<\/p>\n 2. Scaling\/normalization:<\/strong><\/p>\n The PCA results rely heavily on the scales of features. Here, word frequency is just discrete count data. Therefore, scale is not a problem. Still, we need to take into account the fact that some document’s length is longer than others. I normalized my document-term matrix by dividing the frequency of a term in a document with the Euclidean norm of that document’s vector:<\/p>\n <\/p>\n 3. The ‘irlba’<\/a> package in R:<\/strong><\/p>\n Because we are dealing with a high-dimensional sparse matrix, to make the calculation more efficient, we use ‘irlba.’<\/p>\n 4. How many principal components?<\/strong><\/p>\n In general, the maximum number of principal components we can specify is the smaller value of (the row length \u2014 1) and (the column length \u2014 1). I have 17 documents and more than 10,000 words, so the maximum number of components is 16.<\/p>\n We can first run a PCA with 16 components and check the cumulative explained variance by component graph: Here is the graph showing the proportion of explained variance by component: 5. What are the important words in a component?<\/strong><\/p>\n The code in this section was adapted from Silge’s (2018) wonderful blog post<\/a>. The additional required R packages are: ‘tidyverse,’ ‘broom,’ and ‘scales.’ Now, we would like to know which words have the greatest absolute loadings on the first component, PC1. On PC1, if we look at the words with large negative factor loadings and the ones with large positive loadings, we may interpret that this component is about a contrast between different topical aspects of the 2016 Presidential Election. On one hand, with key words like “hillari,” “trump,” “win,” and “poll,” people are focusing on the “horse race” aspect of the election; On the other, with words like “parti,” “republican,” “white,” “christian,” “black,” people are talking about the candidates’ and\/or voters’ demographics.<\/p>\n What about PC2?<\/p>\n We can do one graph for each component (out of 10), but I’m not showing it here.<\/p>\n 6. A 3-D projection<\/strong><\/p>\n By Luling Huang<\/p>\n","protected":false},"author":3163,"featured_media":6097,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[296,2,287,290],"tags":[64,375,372,69,329,6],"class_list":["post-6076","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-critical-digital-studies","category-grad-students","category-media-studies","category-political-science","tag-coding","tag-machine-learning","tag-principal-component-analysis","tag-r","tag-text-analysis","tag-top-news"],"_links":{"self":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/6076","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/users\/3163"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/comments?post=6076"}],"version-history":[{"count":1,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/6076\/revisions"}],"predecessor-version":[{"id":9197,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/6076\/revisions\/9197"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/media\/6097"}],"wp:attachment":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/media?parent=6076"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/categories?post=6076"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/tags?post=6076"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}II. Why Principal Component Analysis?<\/h2>\n
III. PCA in R<\/h2>\n
dtm_norm <- t(apply(dtm, 1, function(x) x\/sqrt(sum(x^2))))<\/span>\n<\/pre>\n<\/div>\n
\n<\/p>\np_dtm_norm_16 <- prcomp_irlba(dtm_norm, n=16)<\/span>\npercent_variation_16 <- p_dtm_norm_16$sdev^2 \/ sum(p_dtm_norm_16$sdev^2) ## sdev is the standard deviation of each component<\/span>\nplot(cumsum(percent_variation_16), xlab = \"Principal Component\",<\/span>\n ylab = \"Cumulative Proportion of Variance Explained\",<\/span>\n type = \"b\")<\/span>\n<\/pre>\n<\/div>\n![\"\"](\"https:\/\/sites.temple.edu\/tudsc\/files\/2019\/02\/CumProp_ExplainedVar_PCA.png\")
<\/p>\n![\"\"](\"https:\/\/sites.temple.edu\/tudsc\/files\/2019\/02\/Prop_ExplainedVar_PCA.png\")
\nThe first graph above does not look like a line, which is good news. The second graph shows that the first 4 components explained more variance than the others. After the 10th component, there is not much difference in explained variance. Let’s just set the number of components as 10 for now.
\n<\/p>\np_dtm_norm_10 <- prcomp_irlba(dtm_norm, n=10)<\/span><\/pre>\n<\/div>\n
\n<\/p>\ntidied_pca_10 <- bind_cols(Tag = colnames(dtm_norm),<\/span>\n tidy(p_dtm_norm_10$rotation)) %>%<\/span>\n gather(PC, Contribution, PC1:PC10)<\/span>\n\ntidied_pca_10 %>%<\/span>\n filter(PC == \"PC1\") %>%<\/span>\n top_n(40, abs(Contribution)) %>%<\/span>\n mutate(Tag = reorder(Tag, Contribution)) %>%<\/span>\n ggplot(aes(Tag, Contribution, fill = Tag)) +<\/span>\n geom_col(show.legend = FALSE, alpha = 0.8) +<\/span>\n theme(axis.text.x = element_text(angle = 90, hjust = 1, vjust = 0.5), <\/span>\n axis.ticks.x = element_blank()) + <\/span>\n labs(x = \"Terms\",<\/span>\n y = \"Relative importance in principle component\")<\/span>\n<\/pre>\n<\/div>\n![\"\"](\"https:\/\/sites.temple.edu\/tudsc\/files\/2019\/02\/words_pc1_10PC-1.png\")
<\/p>\n![\"\"](\"https:\/\/sites.temple.edu\/tudsc\/files\/2019\/02\/words_pc2_10PC-1.png\")
\nPC2 seems to be a contrast regarding the topics\/persons between the two major U.S. political parties.<\/p>\n![\"A](\"https:\/\/plot.ly\/~manuyee\/3.png?share_key=K3qZIVQCf5OotIUFA2ri7K\")
<\/a>
\nhttps:\/\/plot.ly\/embed.js<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"