{"id":322,"date":"2020-07-30T14:41:00","date_gmt":"2020-07-30T18:41:00","guid":{"rendered":"https:\/\/sites.temple.edu\/klugman\/?p=322"},"modified":"2020-07-30T16:29:41","modified_gmt":"2020-07-30T20:29:41","slug":"oh-pisa","status":"publish","type":"post","link":"https:\/\/sites.temple.edu\/klugman\/2020\/07\/30\/oh-pisa\/","title":{"rendered":"Oh PISA"},"content":{"rendered":"
\"\"<\/a>
Trying not to go for something obvious here<\/figcaption><\/figure>\n

In Damned Lies and Statistics<\/em>, Joel Best argues that consumers of statistics need to especially scrutinize international comparisons because there are so many opportunities to mix up apples and oranges (I have discussed<\/a> this with regard to the conceptual definitions used to quantify police-related deaths in different countries).\u00a0 \u00a0One of Best’s examples was international comparisons of test scores; he pointed out that sampling strategies used vary across countries and often countries’ performance levels could be chalked up to the broadness of their sampling strategy.\u00a0 In particular, countries with comprehensive secondary school systems (like the United States, where all, or nearly all, adolescents are exposed to an academic-focused curriculum) would sample from the entire population of schools, while countries with\u00a0 “streaming” systems (like in Germany), where some adolescents go to academic high schools while others go to more vocationally-oriented schools, would sample from the academic high schools only.\u00a0 This would stack the deck against countries like the United States.<\/p>\n

Damned Lies and Statistics<\/em> came out in 2001, and the international testing comparisons Best talked about have been supplanted by the Programme for International Student Achievement (PISA), run by the OECD.\u00a0 When I teach Best in my statistics class, I show students the general sampling strategy<\/a> of PISA:<\/p>\n

The desired base PISA target population in each country consisted of 15-year-old students attending educational institutions located within the country. This meant that countries were to include (i) 15-year-olds enrolled full-time in educational institutions, (ii) 15-year-olds enrolled in educational institutions who attended on only a part-time basis, (iii) students in vocational training types of programmes, or any other related type of educational programmes…<\/em><\/p>\n

Sure, there were some problems<\/a> with China, but what you are going to do?\u00a0 Surely PISA must be good for comparing democratic countries, right?<\/p>\n

Well, no.\u00a0 A team of UCL educational researchers headed by Jake Anders have analyzed<\/a> the 2015 Canadian sample for PISA and their analysis raises questions about the quality of comparisons involving Canada, which does very well on the PISA in terms of high average scores.<\/p>\n

Their article is nice for walking the reader through the sampling strategy of PISA countries.<\/p>\n