Anfossi, Ciolfi, Costa, Parisi and Benedetto (2016) proposed a method for scoring research publications at the scale of a national assessment exercise: each publication is located by its citation percentile and its journal-metric percentile within a reference field, and merit classes are defined by parallel linear thresholds in this unit square, calibrated so that class fractions match a prescribed distribution per reference field and year. This weighted combination was the bibliometric algorithm deployed in Italy's VQR 2011-2014. Two construction problems for the percentile inputs remain unresolved: the first is flagged explicitly in the paper's own footnote, and the published literature contains no criterion for either. First, the cumulative distribution function (CDF) that converts a journal's impact metric into a percentile can be built by counting each journal once or by weighting journals by their number of published items; the source flags the journal-count ordering as "not the only strategy" and defers the comparison to a passage in its closed-access full text, but no criterion has been established in the published literature for which construction is appropriate. Second, the reference fields are predefined Web of Science Subject Categories, which handle heterogeneous and interdisciplinary or multi-topic journals poorly; more homogeneous reference sets derived from semantic clustering or citation-network communities have been developed for pure citation indicators but never integrated into this calibrated hybrid scheme, and their feasibility at national-exercise scale is undemonstrated. The open problem is to determine, with explicit criteria and reproducible evidence, the appropriate journal-metric CDF construction and whether cluster-based reference sets can replace Subject Categories within such a scheme.

Background

National research assessment exercises — such as the UK Research Excellence Framework and Italy's Valutazione della Qualità della Ricerca (VQR), run by Italy's National Agency for the Evaluation of Universities and Research Institutes (ANVUR) — score samples of research publications and feed the results into performance-based university funding. Because a full peer review of every output is costly, such exercises have used bibliometric indicators for journal articles: each publication is located by two coordinates. The first is a citation percentile: the rank of the publication's citation count among all world publications of the same year in a given reference field. The second is a journal-metric percentile: the rank of its journal's impact factor (the journal's average number of citations per recent paper, such as the two-year journal Impact Factor) or a similar journal-level metric among the journals of the same reference field. In Italy the reference fields were the predefined Web of Science (WoS) Subject Categories (SCs), a journal-level classification maintained by the citation database; a journal, and hence every paper in it, inherits one or a few SCs regardless of its actual citation audience.

The 2004-2010 exercise — the first of the VQR series, and Italy's second national assessment after the 2001-2003 VTR (Ancaiani et al., 2015) — scored each article by classifying both coordinates into four percentile classes and mapping the resulting 4x4 block to a merit class through a panel-defined matrix, with discordant cases (the two indicators more than two classes apart) routed to informed peer review (peer review in which referees are shown the bibliometric indicators). Anfossi, Ciolfi, Costa, Parisi and Benedetto (2016) — the ANVUR group behind the design — showed the limitations of this block scheme (corner effects, non-robustness to small threshold shifts, and the impossibility of exactly calibrating class sizes per field and year) and proposed a linear-threshold method: the unit square spanned by the citation percentile CIT and the journal-metric percentile IF is partitioned into merit classes by three parallel straight lines of the form CIT=A⋅IF+Bn\mathrm{CIT} = A \cdot \mathrm{IF} + B_n (n=1,2,3n = 1, 2, 3). The slope A fixes the relative weight of citations versus journal prestige and is chosen by the disciplinary panels; the intercepts BnB_n are then determined so that the fraction of publications in each class matches a prescribed probability distribution (in their illustration, 20% Excellent, 20% Good, 10% Acceptable, 50% Limited) within each Subject Category and publication year. This weighted combination of the two percentiles became the bibliometric algorithm actually deployed in the following exercise, VQR 2011-2014.

Two construction choices in the percentile inputs remain open; the first is acknowledged in the paper itself. First, the cumulative distribution function (CDF, i.e., the reference distribution used to convert a value into a percentile) underlying the journal-metric percentile can be built in two ways: by ordering the journals of the SC and year and counting each journal as one unit (journal-count CDF), or by weighting each journal by the number of items (papers) it published (item-count CDF). The paper's own footnote flags the journal-count construction as "not the only strategy to build the cumulative distribution function for the IF variable, as we will discuss later in the paper"; that deferred discussion and the paper's figures lie in the closed-access full text, and no published work states a criterion for the choice. The difference is structural: with journal-count weighting the reference population is the population of journals, so a paper in a very large high-volume journal is compared against hundreds of small journals counted equally; with item-count weighting the reference population is effectively the population of papers, matching the citation-percentile axis but compressing differences between journals of similar size. Second, the reference sets themselves are the predefined Subject Categories, which are known to be internally heterogeneous and to treat interdisciplinary or multi-topic journals poorly. More homogeneous reference sets could be derived from semantic clustering or from citation-network communities of publications; the open question is whether such clustering-based reference sets can be constructed and used at the scale of a national formal assessment exercise — a feasibility question that no published work has settled.

A substantial adjacent literature exists on field normalization, the construction of reference sets against which citation impact is measured. Publication-level classifications built by clustering direct-citation networks were developed by Waltman and van Eck (2012) and used for field-normalized citation indicators by Ruiz-Castillo and Waltman (2015); Perianes-Rodriguez and Ruiz-Castillo (2018) compared the effect of different classification systems on the evaluation of the universities of the Leiden Ranking (an annual bibliometric ranking of world universities); Leydesdorff and Bornmann (2016) critically examined the operationalization of "fields" as WoS subject categories; and Bornmann (2020) proposed citing-side normalization, which weights each citation according to the citing side's characteristics and avoids any explicit field classification of the cited paper. This literature, however, concerns pure citation indicators: it has not been integrated with the calibrated hybrid citation-plus-journal scheme of Anfossi et al., and it does not address the journal-count versus item-count CDF question at all.

Problem Statement

For the calibrated weighted-combination evaluation scheme of Anfossi et al. (2016) — in which a publication is scored by its citation percentile and its journal-metric percentile within a reference set, with merit-class thresholds calibrated to a prescribed class distribution per reference set and publication year — the construction of the percentile inputs poses two open problems. The first is explicitly flagged and left unresolved in the source paper; the second concerns the scheme's retention of predefined Subject Categories as reference sets, which no subsequent work has replaced or adjudicated within the scheme.

(1) Journal-metric percentile CDF construction. The cumulative distribution function that converts a journal's impact metric into a percentile within a Subject Category and year can be built by counting each journal as one unit, or by weighting each journal by the number of items it published. Determine which construction is appropriate for this scheme: state the criteria explicitly and justify them, then show analytically or empirically which construction they support and with what consequences for the resulting classifications, calibrations, and cross-field comparability. The criteria are for the solver to choose and justify. Because the adopted criteria need not agree, determine the conditions under which each construction is preferable.

(2) Reference-set construction. Determine whether more homogeneous reference sets — derived from semantic clustering or citation-network communities of publications rather than predefined Subject Categories — can replace Subject Categories as the reference sets of the scheme. The question has three parts: (i) whether such reference sets can replace Subject Categories in the scheme in general; (ii) how they compare for the interdisciplinary and multi-topic journals that Subject Categories handle poorly; and (iii) whether constructing them is feasible at the scale of a national formal assessment exercise. Establishing the feasibility of (iii) requires an explicit account of the construction's data requirements, computational cost, and coverage at the scale of the exercise.

The two questions are coupled: both fix the reference distributions from which the two percentiles are computed, so together they determine what 'percentile' means in the scheme and whether its calibrated class distributions are comparable across fields.

References

  1. Anfossi A., Ciolfi A., Costa F., Parisi G., Benedetto S. (2016). Large-scale assessment of research outputs through a weighted combination of bibliometric indicators. Scientometrics 107(2):671-683. DOI: 10.1007/s11192-016-1882-9. https://doi.org/10.1007/s11192-016-1882-9 (source paper)
  2. Ancaiani A., Anfossi A. F., Barbara A., Benedetto S., Blasi B., Carletti V., Cicero T., Ciolfi A., Costa F., Colizza G., Costantini M., di Cristina F., Ferrara A., Lacatena R. M., Malgarini M., Mazzotta I., Nappi C. A., Romagnosi S., Sileoni S. (2015). Evaluating scientific research in Italy: The 2004-10 research evaluation exercise. Research Evaluation 24(3):242-255. DOI: 10.1093/reseval/rvv008. https://doi.org/10.1093/reseval/rvv008
  3. Abramo G., D'Angelo C. A. (2015). The VQR, Italy's second national research assessment: Methodological failures and ranking distortions. Journal of the Association for Information Science and Technology 66(11):2202-2214. DOI: 10.1002/asi.23323. https://doi.org/10.1002/asi.23323
  4. Abramo G., D'Angelo C. A. (2016). Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014). Scientometrics 109(3):2053-2065. DOI: 10.1007/s11192-016-2153-5. https://doi.org/10.1007/s11192-016-2153-5
  5. Abramo G., D'Angelo C. A. (2017). On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011-2014. Journal of Informetrics 11(3):783-787. DOI: 10.1016/j.joi.2017.06.003. https://doi.org/10.1016/j.joi.2017.06.003
  6. Baccini A., De Nicolao G. (2016). Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise. Scientometrics 108(3):1651-1671. DOI: 10.1007/s11192-016-1929-y. https://doi.org/10.1007/s11192-016-1929-y
  7. Benedetto S., Checchi D., Graziosi A., Malgarini M. (2017). Comments on the correspondence 'On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011-2014', J. Informetr., 11 (2017), 783-787. Journal of Informetrics 11(3):838-840. DOI: 10.1016/j.joi.2017.07.002. https://doi.org/10.1016/j.joi.2017.07.002
  8. Bornmann L. (2020). How can citation impact in bibliometrics be normalized? A new approach combining citing-side normalization and citation percentiles. Quantitative Science Studies 1(4):1553-1569. DOI: 10.1162/qss_a_00089. https://doi.org/10.1162/qss_a_00089
  9. Cappelletti-Montano B., Columbu S., Montaldo S., Musio M. (2022). Interpreting the outcomes of research assessments: A geometrical approach. Journal of Informetrics 16(1):101254. DOI: 10.1016/j.joi.2022.101254. https://doi.org/10.1016/j.joi.2022.101254 (preprint: arXiv:2104.06495)
  10. Carrabs F., Cerulli M., Sorgente C. (2026). On the Complexity of the Paper Selection Problem for the Italian Research Quality Evaluation (VQR). Networks, advance online publication, article net.70068. DOI: 10.1002/net.70068. https://doi.org/10.1002/net.70068
  11. Checchi D., Ciolfi A., De Fraja G., Mazzotta I., Verzillo S. (2021). Have you Read This? An Empirical Comparison of the British REF Peer Review and the Italian VQR Bibliometric Algorithm. Economica 88(352):1107-1129. DOI: 10.1111/ecca.12373. https://doi.org/10.1111/ecca.12373
  12. Cicero T., Malgarini M. (2020). On the use of journal classification in social sciences and humanities: evidence from an Italian database. Scientometrics 125(2):1689-1708. DOI: 10.1007/s11192-020-03581-8. https://doi.org/10.1007/s11192-020-03581-8
  13. D'Angelo C. A. (2025). The 2015-2019 VQR, Italy's third national research assessment: an intended informed peer review or a DIY bibliometric exercise? Research Evaluation 34, article rvaf045 (published online 29 October 2025). DOI: 10.1093/reseval/rvaf045. https://doi.org/10.1093/reseval/rvaf045
  14. Franceschini F., Maisano D. (2017). Critical remarks on the Italian research assessment exercise VQR 2011-2014. Journal of Informetrics 11(2):337-357. DOI: 10.1016/j.joi.2017.02.005. https://doi.org/10.1016/j.joi.2017.02.005
  15. Leydesdorff L., Bornmann L. (2016). The operationalization of 'fields' as WoS subject categories (WCs) in evaluative bibliometrics: The cases of 'library and information science' and 'science & technology studies'. Journal of the Association for Information Science and Technology 67(3):707-714. DOI: 10.1002/asi.23408. https://doi.org/10.1002/asi.23408
  16. Perianes-Rodriguez A., Ruiz-Castillo J. (2018). The impact of classification systems in the evaluation of the research performance of the Leiden Ranking universities. Journal of the Association for Information Science and Technology 69(8):1046-1053. DOI: 10.1002/asi.24017. https://doi.org/10.1002/asi.24017
  17. Ruiz-Castillo J., Waltman L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal of Informetrics 9(1):102-117. DOI: 10.1016/j.joi.2014.11.010. https://doi.org/10.1016/j.joi.2014.11.010
  18. Traag V. A., Malgarini M., Sarlo S. (2025). Metrics and peer review agreement at the institutional level. Quantitative Science Studies 6:328-350. DOI: 10.1162/qss_a_00352. https://doi.org/10.1162/qss_a_00352 (preprint: arXiv:2006.14830)
  19. Waltman L., Traag V. A. (2021). Use of the journal impact factor for assessing individual articles: Statistically flawed or not? F1000Research 9:366 (version 2). DOI: 10.12688/f1000research.23418.2. https://doi.org/10.12688/f1000research.23418.2
  20. Waltman L., van Eck N. J. (2012). A new methodology for constructing a publication-level classification system of science. Journal of the American Society for Information Science and Technology 63(12):2378-2392. DOI: 10.1002/asi.22748. https://doi.org/10.1002/asi.22748