Rooduijn and Pauwels (2011) compared a classical content analysis of party manifestos, in which coders judged the joint presence of people-centrism and anti-elitism in each paragraph, with a computerised dictionary method that counted only anti-elitism words, and left four issues open: the conceptual adequacy of anti-elitism-only measurement in other settings, the cross-language validity of translated and locally supplemented dictionaries, unexplained high populism scores for mainstream parties (type case: the Italian post-communist social democrats in the early 1990s), and a proposed hybrid man-machine procedure that was never empirically tested. Subsequent work produced component-complete dictionaries for single languages, supervised machine-learning scorers, expert surveys, a global holistic-grading database, and large-language-model graders, yet the field's own validation studies conclude that measuring populism validly over time and between countries remains an open challenge. The problem is to construct and validate a scalable text-based measure of populism that is conceptually complete (both components with joint-presence aggregation), demonstrably equivalent across languages and political contexts, validated against human coding across text types, and accountable for anomalous mainstream-party scores.

Background

Populism is studied in comparative politics mainly through its ideational definition, articulated most influentially by Mudde (2004): populism is a 'thin-centred' ideology, i.e. a partial belief system that attaches to fuller host ideologies rather than standing alone, which holds that society is ultimately separated into two homogeneous and antagonistic camps — 'the pure people' and 'the corrupt elite' — and that politics should express the general will of the people. Two textual components operationalise this definition for content analysis: people-centrism (references centring 'the people', such as citizens, society, or the nation, interpreted in context) and anti-elitism (generalised criticism of political elites, as distinct from criticism of a specific rival party or named politician). Under the ideational definition, a text unit counts as populist only when both components are present together.

Because populism is a property of discourse rather than of policy platforms, it is measured by analysing political texts, chiefly election manifestos and speeches. Three families of text-based measurement dominated when Rooduijn and Pauwels (2011) wrote. Classical content analysis has trained human coders read texts unit by unit (the paragraph being a typical unit of measurement, the whole manifesto the unit of analysis) and apply a codebook; quality is assessed with inter-coder reliability coefficients, most commonly Krippendorff's alpha, a chance-corrected measure of agreement among coders. Holistic grading, developed by Hawkins, instead has coders read an entire speech and assign one integrated 0-to-2 populism score. Dictionary-based computerised content analysis counts the share of words matching a theoretically selected word list (Rooduijn and Pauwels used the open-source Yoshikoder software), with the dictionary translated across countries and supplemented with context-specific terms; its internal reliability can be probed by split-half testing corrected with the Spearman-Brown prophecy formula, a standard correction that estimates the reliability of the full word list from the correlation between its random halves.

Rooduijn and Pauwels (2011) coded manifestos of parties in the United Kingdom, the Netherlands, Germany, and Italy with both approaches. Their dictionary deliberately counted only anti-elitism words: candidate words for people-centrism such as 'we' and 'our' are frequently ambiguous between a party referring to itself and a reference to the people, and in their data people-centrism and anti-elitism correlated only weakly at the paragraph level (r = 0.04), although almost every anti-elitist paragraph also contained a reference to the people, which is why anti-elitism alone tracked the full measure closely in this dataset (Pearson r = 0.80 between the two party-level scores). National inter-coder reliabilities for the classical method ranged from 0.66 to 0.92 (Krippendorff's alpha), while the adjusted dictionary reliabilities ranged from 0.37 (Germany) to 0.70 (United Kingdom). Based on the comparison they proposed a three-step 'man-machine' procedure — a maximally inclusive dictionary, a computer scan marking candidate passages, then human coders judging the marked passages with a codebook — which combined machine speed with contextual human judgment but was not tested.

Four limitations of this measurement programme, stated in the source itself, define the problem. First, conceptual completeness: measuring only anti-elitism omits people-centrism, and the empirical harmlessness of this omission was established only for the study's own four-country manifesto dataset. Second, cross-language validity: the dictionary had to be translated and supplemented with context-specific vocabulary (for instance the Dutch word 'regenten', a historically loaded term for ruling elites used only by Dutch populists), and the equivalence of such instruments across languages was assumed rather than tested. Third, anomaly accountability: some mainstream parties scored unexpectedly high, most strikingly the Italian post-communist social democrats (labelled 'DS' in the source, whose 1992 manifesto was included) in the early 1990s, and the source explicitly calls for further research on the reasons. Fourth, scalability: the untested hybrid procedure was meant to make comparative measurement across countries, elections, and text types (written manifestos and spoken discourse) feasible at medium scale.

The later landscape transformed each ingredient. Supervised machine-learning classifiers (models trained on human-coded texts) and, most recently, transformer-based language models (neural network language models pre-trained on large text corpora) and large-language-model (LLM) graders (general-purpose text-generation models prompted to score texts) now automate scoring; expert surveys rate parties' populism with large panels of country experts; and the Global Populism Database applies holistic grading by native speakers to leaders' speeches across dozens of languages. Yet validation work has repeatedly found that scalable instruments fail exactly the criteria above — conceptual completeness, cross-language equivalence, and validity against human coding — so the composite measurement objective stated in 2011 survives as an open methodological problem for the field.

Problem Statement

Construct and validate a text-based measurement of populism in party texts (manifestos and comparable party communication, with spoken and written text types) that simultaneously satisfies the four linked criteria left open by Rooduijn and Pauwels (2011):

  1. Conceptual completeness and correct aggregation. The instrument must capture both people-centrism and anti-elitism, and its aggregation rule must respect the ideational definition's joint-presence requirement — a textual unit or actor counts as populist only to the extent that both components co-occur — rather than scoring components additively or measuring anti-elitism alone. The instrument must state its unit of analysis (e.g. the manifesto or speech) and unit of measurement (e.g. the paragraph), and its output must permit graded (not merely binary) comparison of actors.

  2. Cross-language and cross-context validity. The instrument must be shown to measure the same concept in at least two languages and national political contexts, with equivalence demonstrated empirically across the language-specific implementations, not assumed from literal translation of a word list; context-specific vocabulary (such as the Dutch 'regenten' type) must be handled by an explicit, documented procedure.

  3. Validation against human coding across text types. The instrument's scores must be validated against human content-analytic or holistic-grading benchmarks, reporting inter-coder or inter-rater reliability for the human benchmark, concurrent validity against it on held-out data, and coverage of both written and spoken political text; the reliability and validity evidence must meet the customary statistical standards of political text analysis.

  4. Anomaly accountability. The instrument must confront the anomalous high populism scores of mainstream parties observed in the source, with the Italian post-communist social democrats ('DS') of the early 1990s as the type case: applying the instrument to such cases, the study must either substantiate the high scores as genuine populist discourse (with textual and historical evidence) or identify the specific measurement artifacts responsible, thereby establishing whether the anomaly is real populism or an instrument failure.

These four criteria are components of one scientific objective — a valid, reliable, and scalable comparative measure of populism in party texts.

References

  1. Rooduijn, Matthijs & Pauwels, Teun (2011). 'Measuring Populism: Comparing Two Methods of Content Analysis'. West European Politics 34(6): 1272-1283. DOI: 10.1080/01402382.2011.616665. https://doi.org/10.1080/01402382.2011.616665
  2. Mudde, Cas (2004). 'The Populist Zeitgeist'. Government and Opposition 39(4): 541-563. DOI: 10.1111/j.1477-7053.2004.00135.x. https://doi.org/10.1111/j.1477-7053.2004.00135.x
  3. Pauwels, Teun (2011). 'Measuring Populism: A Quantitative Text Analysis of Party Literature in Belgium'. Journal of Elections, Public Opinion & Parties 21(1): 97-119. DOI: 10.1080/17457289.2011.539483. https://doi.org/10.1080/17457289.2011.539483
  4. Rooduijn, Matthijs (2014). 'The Nucleus of Populism: In Search of the Lowest Common Denominator'. Government and Opposition 49(4): 573-599. DOI: 10.1017/gov.2013.30. https://doi.org/10.1017/gov.2013.30
  5. Norris, Pippa (2020). 'Measuring populism worldwide'. Party Politics 26(6): 697-717. DOI: 10.1177/1354068820927686. https://doi.org/10.1177/1354068820927686
  6. Meijers, Maurits J. & Zaslove, Andrej (2021). 'Measuring Populism in Political Parties: Appraisal of a New Approach'. Comparative Political Studies 54(2): 372-407. DOI: 10.1177/0010414020938081. https://doi.org/10.1177/0010414020938081
  7. Gründl, Johann (2022). 'Populist ideas on social media: A dictionary-based measurement of populist communication'. New Media & Society 24(6): 1481-1499. DOI: 10.1177/1461444820976970. https://doi.org/10.1177/1461444820976970
  8. Di Cocco, Jessica & Monechi, Bernardo (2022). 'How Populist are Parties? Measuring Degrees of Populism in Party Manifestos Using Supervised Machine Learning'. Political Analysis 30(3): 311-327. DOI: 10.1017/pan.2021.29. https://doi.org/10.1017/pan.2021.29
  9. Jankowski, Michael & Huber, Robert A. (2023). 'When Correlation Is Not Enough: Validating Populism Scores from Supervised Machine-Learning Models'. Political Analysis 31(4): 591-605. DOI: 10.1017/pan.2022.32. https://doi.org/10.1017/pan.2022.32
  10. Bernhard, Laurent & Chueri, Juliana (2024). 'It's Conditional: Improving the Measurement of Populism'. Populism 7(2): 226-242. DOI: 10.1163/25888072-bja10067. https://doi.org/10.1163/25888072-bja10067
  11. Celico, Andrea; Rode, Martin & Rodriguez-Carreño, Ignacio (2024). 'Will the real populists please stand up? A machine learning index of party populism'. European Journal of Political Economy 82: 102529. DOI: 10.1016/j.ejpoleco.2024.102529. https://doi.org/10.1016/j.ejpoleco.2024.102529
  12. Tóth, Tamás; Goyanes, Manuel & Demeter, Márton (2024). 'Extend the context! Measuring explicit and implicit populism on three different textual levels'. Communications 49(2): 222-242. DOI: 10.1515/commun-2022-0009. https://doi.org/10.1515/commun-2022-0009
  13. López Robles, Adalberto de Jesús (2024). 'Populismo de izquierda y democracia en América Latina: un estudio comparativo'. Revista Temas Sociológicos 34: 159-193. DOI: 10.29344/07196458.34.3800. https://doi.org/10.29344/07196458.34.3800
  14. Tao, Yuzhou; Zhan, Zhiqin; Zhou, Han; Kang, Jingshi & Sun, Shaojing (2025). 'Measuring Chinese online populist discourse: an automated semantic text analysis method'. Chinese Journal of Communication 18(2): 121-141. DOI: 10.1080/17544750.2024.2420622. https://doi.org/10.1080/17544750.2024.2420622
  15. Erhard, Lukas; Hanke, Sara; Remer, Uwe; Falenska, Agnieszka & Heiberger, Raphael Heiko (2025). 'PopBERT. Detecting Populism and Its Host Ideologies in the German Bundestag'. Political Analysis 33(1): 1-17. DOI: 10.1017/pan.2024.12. https://doi.org/10.1017/pan.2024.12
  16. Makhortykh, Mykola; de León, Ernesto; Christner, Clara; Sydorova, Maryna; Urman, Aleksandra; Adam, Silke; Maier, Michaela & Gil-Lopez, Teresa (2025). 'Is a single model enough? The systematic comparison of computational approaches for detecting populist radical right content'. Quality & Quantity 59(S2): 1163-1207. DOI: 10.1007/s11135-024-02034-1. https://doi.org/10.1007/s11135-024-02034-1
  17. Jung, Yujin J.; Tamaki, Eduardo Ryô; Chatterley, Julia; Mitchell, Grant; Dzebo, Semir; Sandoval, Cristóbal; Littvay, Levente & Hawkins, Kirk A. (2025). 'Populism Meets AI: Advancing Populism Research with LLMs'. arXiv: 2510.07458. https://arxiv.org/abs/2510.07458
  18. Hawkins, Kirk A.; Aguilar, Rosario; Castanho Silva, Bruno; Jenne, Erin K.; Kocijan, Bojana & Rovira Kaltwasser, Cristóbal (2019). 'Global Populism Database'. Harvard Dataverse (version 1, 2019; version 2, 2022; version 3, 2025). DOI: 10.7910/DVN/LFTQEZ. https://doi.org/10.7910/DVN/LFTQEZ