{"id":8072,"date":"2021-12-15T16:34:46","date_gmt":"2021-12-15T20:34:46","guid":{"rendered":"https:\/\/sites.temple.edu\/tudsc\/?p=8072"},"modified":"2021-12-17T12:57:09","modified_gmt":"2021-12-17T16:57:09","slug":"querying-wikidata","status":"publish","type":"post","link":"https:\/\/sites.temple.edu\/tudsc\/2021\/12\/15\/querying-wikidata\/","title":{"rendered":"Querying Wikidata with SPARQL"},"content":{"rendered":"

By jay winkler<\/p>\n

For many of us, adding \u201cwiki\u201d to the end of a Google search is like a cheat code for getting a quick answer. I know the structure of a Wikipedia article, and if I want an answer to a simple question, like \u201cwhere was E.B. Lewis born\u201d, just finding it on Wikipedia is the fastest way to get it.<\/span><\/p>\n

But what about more complex questions? What if I wanted to know, say, who are all of the Black artists who were born in Philadelphia? For that, I can turn to <\/span>Wikidata<\/span><\/a>. Wikidata is an attempt to convert all of the world\u2019s information into a structured, queryable dataset. There are Wikidata items for a huge range of topics and concepts. People, places, corporations, foods, animals, basketball teams, emotions, works of art, and just about anything else can have a Wikidata item.<\/span><\/p>\n

In the second half of 2021 I worked with the Temple University Library as part of the <\/span>LEADING Fellowship<\/span><\/a>. Through a collaboration with the Philadelphia Museum of Art, our project mentors Synatra Smith, Alex Wermer-Colan, and Holly Tomren focused our project on updating and improving Wikidata\u2019s collection of Black artists<\/a>, and especially those from the Philadelphia area. I was also paired with a second LEADING fellow on this project Rebecca Y. Bayeck, CLIR postdoctoral Fellow in Data Curation at the Schomburg Center for Research in Black Culture. We developed a complicated query that enabled us to quickly collect a great deal of information about local artists. In this tutorial, I’ll discuss the building blocks of that query.<\/span><\/p>\n

Wikidata Basics<\/b><\/h3>\n

Where Wikipedia has articles, Wikidata has \u201citems.\u201d Like I said above, an item can be anything that a Wikidata editor thinks people might want queryable structured data for. For each Wikidata item, statements can be added. These statements add properties of the item, such as \u201coccupation\u201d, \u201cplace of birth\u201d, \u201crace or ethnicity\u201d, or other potential descriptors of the item. There are also properties for external identifiers, such as the items <\/span>Virtual International Authority File<\/span><\/a>\u00a0 catalog number. These properties are represented by triples: the item (1) has a property (2) with a given value (3).<\/span><\/p>\n

Each Wikidata item is assigned a QID, which is a unique identifier that starts with a Q and ends with a number. The numbers are assigned sequentially, and have varying numbers of digits. Items then have a label. If you look at <\/span>E.B. Lewis\u2019s Wikidata item<\/span><\/a>, the QID is Q5321733, and the human-readable English label is \u201cE.B. Lewis\u201d.\u00a0<\/span><\/p>\n

Using the QID as the primary identifier is useful for two main reasons. First, it allows for Wikidata items with duplicate labels (such as a band and their self-titled album or two identically named people) to be easily distinguishable, without the sort of parenthetical disambiguation common on Wikipedia. It also makes translation easier; as the whole Wikidata item does not need to be translated, just the label, while property labels can be updated individually.<\/span><\/p>\n

Properties work similarly: each property has a property ID (PID) that can be used in queries. For many properties, the value also is a Wikidata item. For example, E.B. Lewis (Q5321733) has \u201coccupation (P106)\u201d property with the value \u201cartist (Q483501)\u201d. For other properties, the value can be a date, geographic coordinate, URL, or even an image. This is not an exhaustive list of potential property values. While anyone can create an item about anything, property creation does require a formal proposal and discussion with the Wikidata editor community. My colleague, Dr. Rebecca Y. Bayeck, has released a blog post in tandem with this one that goes deeper into editing Wikidata<\/a>.<\/span><\/p>\n

Basic Queries<\/b><\/h3>\n

Wikidata queries are built with SPARQL, a query language based on SQL. SPARQL makes it possible to query databases in the Research Data Framework (RDF) format, which are stored in triples. Queries can be typed directly into the <\/span>Wikidata Query Service<\/span><\/a> (WDQS), and there is also a SPARQL endpoint to collect data through Python.<\/span><\/p>\n

The basic syntax for a SPARQL query is simple:<\/span><\/p>\n

SELECT ?item ?itemLabel\u00a0<\/span>\r\nWHERE {<\/span>\r\n?item wdt:p# wd:q#<\/span>\r\nSERVICE wikibase:label { bd:serviceParam wikibase:language \"[AUTO_LANGUAGE],en\". }<\/span>\r\n}<\/span>\r\n\r\n<\/pre>\n
To break this down: the SELECT statement tells the WDQS what variables we want to retrieve. In this case, we\u2019ll retrieve the item, and because Wikidata will automatically return the QID for each item, we\u2019ll also ask it to return the item\u2019s label so our results are human readable.<\/span><\/p>\n
The WHERE statement is where we put our conditions for what we want wikidata to return. In general, we\u2019ll name a variable, and then say what properties that variable is required to have to be added to our list. If we want Wikidata to generate a list of all artists, the statement would look like this:<\/span><\/p>\n
?item wdt:P106 wd:Q483501<\/span><\/pre>\n
\u201cwdt\u201d indicates a property, with P106 being the \u201coccupation\u201d property, and \u201cwd\u201d indicates we want the value of that property to be a specific Wikidata item, with Q484501 being the item for \u201cartist\u201d. So, we\u2019re telling wikidata that to add an item to our list of items, it needs to have a statement with the occupation property, and that occupation\u2019s value needs to be artist.<\/span><\/p>\n
This query<\/span><\/a> returns far too many results and may time out, so we\u2019ll have to narrow it down a bit. Let\u2019s look for artists <\/span>from Philadelphia<\/span><\/i>.\u00a0<\/span><\/p>\n
SELECT ?item ?itemLabel\u00a0<\/span>\r\nWHERE {<\/span>\r\n\u00a0\u00a0\u00a0\u00a0?item wdt:P106 wd:Q483501;<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0wdt:P19 wd:Q1345<\/span>\r\nSERVICE wikibase:label { bd:serviceParam wikibase:language \"[AUTO_LANGUAGE],en\". }<\/span>\r\n}<\/span><\/pre>\n
The SERVICE statement tells WDQS that when we ask for labels, we want the English ones. No one needs to memorize this line of code, as WDQS\u2019s tools make it very easy to add it automatically.<\/span><\/p>\n
Retrieving More Variables<\/b><\/h3>\n
Let\u2019s say we want to retrieve a few more variables. We don\u2019t just want Black artists from Philadelphia, we want to know where those artists went to college and their gender. We do that by adding more variables to the SELECT statement, and then to the query itself. You can see our new query, and the rest of the full queries in this blog, <\/span>directly on the Wikidata Query Service<\/span><\/a>.<\/span><\/p>\n
Our main focus here is the changes to the SELECT statement, and the corresponding changes to the query. We\u2019ve changed our main variable from \u201c?item\u201d to \u201c?artist\u201d, just to make things a little easier on ourselves. We\u2019ve also added two new variables that will become columns in our results: <\/span>genderLabel<\/span><\/i> and <\/span>educationLabel<\/span><\/i>.<\/span><\/p>\n
In the query, we\u2019ve added two lines that tell WDQS what to put in the new columns: for each artist, check for a P21 (which is the sex or gender property), and put the result in a column we\u2019re calling <\/span>gender<\/span><\/i>. Do the same thing for P69, \u201ceducated at\u201d, and put it in a column called <\/span>education<\/span><\/i>.<\/span><\/p>\n
It is also important that in your SELECT statement you add \u201c<\/span>Label<\/span><\/i>\u201d to the end of your variables. So, if in the query you have the \u201c<\/span>Gender\u201d <\/span><\/i>variable, in the SELECT statement you want the variable to be \u201c<\/span>GenderLabel\u201d<\/span><\/i>. This way, your data will include the human-readable labels, and not the QIDs.\u00a0<\/span><\/p>\n
Let\u2019s make one important change <\/span>to our query<\/span><\/a>.<\/span><\/p>\n
I\u2019ve now wrapped the requests for gender and education in the OPTIONAL tag and brackets. In the previous query, if an artist did not have either a P21 or a P69, they would simply be excluded from the dataset. By wrapping them in the optional tag, any artist that fits the parameters for the artist variable gets a row, and <\/span>if<\/span><\/i> it has a statement for gender or education, that gets included as well.<\/span><\/p>\n
Concatenation<\/b><\/h3>\n
In the query above, both gender and education can potentially carry multiple statements. Many people were educated at multiple institutions, so both of those institutions are included in the results. Wikidata places these each in their own row, meaning several artists have multiple rows. We can fix this through concatenation.<\/span><\/p>\n
Concatenation commands go before the WHERE statement. Each concatenation command replaces that variable in the SELECT statement:<\/span><\/p>\n
SELECT ?artist ?artistLabel<\/span>\r\n\u00a0\u00a0(group_concat(DISTINCT(?genderLabel);separator=\", \") as ?genders)<\/span>\r\n\u00a0\u00a0(group_concat(DISTINCT(?educationLabel);separator=\", \") as ?educations)<\/span><\/pre>\n
Here I\u2019ve removed the genderLabel and educationLabel variables and replaced them with two concatenation statements. I\u2019ve asked WDQS to concatenate, I\u2019ve asked it to only provide me distinct values so as to remove duplicates (this may occur when different sources are cited, for example), and I\u2019ve asked it to use a comma followed by a space as a separator. I\u2019ve also given the concatenated column its own name by pluralizing the main variable name.<\/span><\/p>\n
Most of the WHERE statement is the same, but I do need to make a change to the SERVICE statement. This is due to a bug in WDQS that should be fixed eventually, but is consistent enough that most documentation will tell you to call in all of the labels manually:<\/span><\/p>\n
SERVICE wikibase:label { bd:serviceParam wikibase:language \"[AUTO_LANGUAGE],en\".\u00a0<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0?artist rdfs:label ?artistLabel .<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0?gender rdfs:label ?genderLabel .<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0?education rdfs:label ?educationLabel .<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}<\/span><\/pre>\n
Finally, at the very end of my query, after the final curly bracket, I need to add a GROUP BY statement. You need to include every variable that is NOT being concatenated in your GROUP BY statement. If you are concatenating 2 out of your 14 variables, there needs to be 12 variables in your GROUP BY statement. Here, it looks like this:<\/span><\/p>\n
GROUP BY ?artist ?artistLabel<\/span><\/pre>\n
Put it all together, <\/span>and you get the full query<\/span><\/a>, which is better looked at within the linked Wikidata Query service.<\/span><\/p>\n
I\u2019ve also added an ORDER BY statement so the data displays in alphabetical order.<\/span><\/p>\n
Larger Statements<\/b><\/h3>\n
Now that we have the basics down, the queries are nearly endlessly extensible. To expand our Philadelphia Black artists dataset, we added a <\/span>great number of variables<\/span><\/a>.<\/span><\/p>\n
In this query we\u2019re able to gather artist names, birthplaces, residences, sexual orientation, employers, and some geographic coordinates using the same basic formula we used in the rest of our queries. I have cheated here: in the very first part of the WHERE statement we restrict results to a preselected list of artists. This prevents the system from timing out, but still gathers the dataset we\u2019re looking for.<\/span><\/p>\n
You can use these queries as the building blocks for your own queries. While the query just above might be complicated, try going back to the <\/span>basic concatenated data,<\/span><\/a> and replacing the QID for artists with a different occupation you\u2019re interested in. Replace the city with one closer to you, and add some of the options in this final query to gather more data. There are many other SPARQL tricks, but these basics will hopefully help you understand how you can start from scratch on your own.\u00a0<\/span><\/p>\n
Next Steps<\/b><\/h3>\n
Developing the queries above required a great deal of trial and error, not only in the code, but in conceptualizing what exactly it was we wanted represented in our dataset. I would run the query, and find that while the code worked it lacked some piece of information that might be useful in drawing connections between these artists. In a forthcoming blog post, me and my colleague Rebecca will discuss how we used these results, to create visualizations using Python.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
By jay winkler<\/p>\n","protected":false},"author":24646,"featured_media":8089,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[289,286,405,86],"tags":[259,321,434,6],"class_list":["post-8072","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-art-history","category-cultural-studies","category-digital-humanities","category-services","tag-cultural-heritage","tag-digital-scholarship","tag-linked-open-data","tag-top-news"],"_links":{"self":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/8072","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/users\/24646"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/comments?post=8072"}],"version-history":[{"count":0,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/posts\/8072\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/media\/8089"}],"wp:attachment":[{"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/media?parent=8072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/categories?post=8072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sites.temple.edu\/tudsc\/wp-json\/wp\/v2\/tags?post=8072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
If we make our full query look like this, we return a <\/span>much more manageable set of results<\/span><\/a>. The semicolon indicates we\u2019re still adding parameters for the item variable, and the second line indicates the parameters we\u2019ve added: the item must have a P19 place of birth statement, which must be Q1345, Philadelphia. We can add more parameters to narrow what artists we want to return, and the basic <\/span>list of Black artists from Philadelphia<\/span><\/a> was the building block of our work in the LEADING fellowship.<\/span><\/p>\n