Identifying the composition of performance-measure distortions and the dynamic design of measurement systems

Courty and Marschke (2008) proposed a general test for distortions in performance measures: within a linear multitask principal-agent model, the regression slope of organizational value on a performance outcome, Cov(p,V)/Var(p), falls after a measure is activated (given incentive weight) if and only if the measure is distorted; a fall in the covariance Cov(p,V) itself specifically indicates value-reducing gaming. Applying the test to US job-training data, they found both slope and covariance declines. The test's own assumptions and framework leave two linked theoretical problems open: (1) the activation-induced moment changes only partially identify the composition of distortions - gaming ($w_{i,\alpha}>0$, with value-reducing gaming the sub-case $w_{i,\alpha}>0$ together with $\xi_{i,\alpha}>0$) versus random measure-specific marginal productivity - and the diagnostic's validity rests on strong conditions (the researcher observes a sufficiently fine project partition, activation is exogenous to agents' actions, agents are myopic, and measurement error is classical); (2) the dynamic measurement-design problem - a principal who does not know the distortion technology should trade off current incentive provision against the information that activation reveals - is not formalized or solved. Later literature applied the test (economic-value-added and residual-income measures, research assessments, child-support enforcement, payment-by-results) and sketched dynamic frameworks, but resolved neither core. The open problem is to build the identification theory of distortion composition under realistic observability and noise conditions, and to characterize optimal dynamic measurement choice.

Background

In the multitask principal-agent framework (an agency relationship in which an agent performs several tasks and is compensated on measures that only imperfectly reflect them; Holmstrom and Milgrom 1991; Baker 1992), a performance measure is called distorted when rewarding it induces actions that raise the measure without raising, or while reducing, the organization's true objective; strategic responses of this kind are known as gaming (also dysfunctional or unintended responses). Demonstrating gaming empirically is difficult because such behavior is typically hidden from the researcher. Building on their case studies of the US Job Training Partnership Act (JTPA), a federal job-training program in which grant awards to local training centers depended on reported performance outcomes, Courty and Marschke (2008) developed a general and easily implemented test.

Their model has projects (or tasks) of type alpha, indexed by a partition that the researcher observes; the agent allocates common productive effort $e_{0,\alpha}$, measure-specific productive efforts $e_{i,\alpha}$, and gaming efforts $g_{i,\alpha}$ for measures $i=1,2$. The performance outcome and the organizational value are

$p_{i,\alpha}=v_{0,\alpha}e_{0,\alpha}+(v_{i,\alpha}+\eta_{i,\alpha})e_{i,\alpha}+w_{i,\alpha}g_{i,\alpha},\qquad V_{\alpha}=v_{0,\alpha}e_{0,\alpha}+\sum_i v_{i,\alpha}e_{i,\alpha}-\sum_i \xi_{i,\alpha}g_{i,\alpha},$

where $v_{i,\alpha}$ are common marginal productivities, $\eta_{i,\alpha}$ is a measure-specific random marginal-productivity component (with mean and third moment zero), $w_{i,\alpha}\ge 0$ scales how strongly gaming raises the measure, and $\xi_{i,\alpha}\ge 0$ scales how strongly gaming destroys value. The agent is rewarded on aggregate measures with weights $(\beta_1,\beta_2)$ and has quadratic effort costs. The key observable is the slope of a regression of value on the performance outcome, $b=\mathrm{Cov}(p,V)/\mathrm{Var}(p)$ (the empirical analog is the coefficient from regressing estimated program impacts on reported outcomes across subgroups). Within the model, $b$ falls after a measure's relative incentive weight rises if and only if the measure is distorted, meaning $\eta_{i,\alpha}\neq 0$ or $w_{i,\alpha}>0$ for some project types. These two conditions exhaust the model's distortion taxonomy: reallocation of effort across measures and tasks is the multitasking margin internal to the productive channel rather than a separate distortion channel, and selection enters only through the adoption of the measures themselves (a validity condition discussed below). Moreover, the covariance $\mathrm{Cov}(p,V)$ falls only if value-reducing gaming ($w_{i,\alpha}>0$ together with $\xi_{i,\alpha}>0$) is present, whereas distortions operating only through $\eta$ do not by themselves lower the covariance. Activation means introducing a measure into the incentive system, i.e. raising its weight from zero to positive. Using subgroup-level experimental impact estimates from the National JTPA Study as $V_{\alpha}$ and reported subgroup outcome levels as $p_{\alpha}$, the authors found that the slope fell after activation of the follow-up measures and that the covariance fell significantly for two measure-impact pairs, which they read as evidence of value-reducing gaming; they also showed that pre-activation correlations are an unreliable guide for selecting measures, since rankings by association can reverse after activation.

The test's validity conditions are explicit. The researcher must observe a project partition at least as fine as the one agents condition on (agents must not hold substantially richer private information); the change in incentive weights must be exogenous to agents' actions (in the JTPA case, staggered adoption across US states between 1986 and 1990, determined by federal and state authorities); agents must be myopic, not manipulating current actions to influence the principal's future measure choices; and the estimates of $p_{\alpha}$ and $V_{\alpha}$ must not be too contaminated by measurement error and sampling variability (the impact estimates use a correction, due to Bloom 1984, for sample members who drop out of the experimental follow-up). The dynamic dimension is assumed away rather than analyzed: the paper assumes that the introduction of a measure is exogenously given, interpreted as the principal not knowing ex ante which measure is likely to perform well; it ignores agents' anticipation of future measure changes because each of the more than 600 independent agents in the application has very little influence on the principal's decision; and a footnote concedes that in a single-agent model the equilibrium dynamic of gaming investment and the decision to change the measure would be more complex. The dynamic measurement-design problem - how a principal should choose measures over time as activation reveals distortion margins - is neither formalized nor solved in the paper. The open question thus has two linked components: what activation data can and cannot identify about the composition of distortions under realistic observation and noise, and how a principal should choose measures over time in light of that identification mapping.

Problem Statement

In the linear multitask principal-agent environment of Courty and Marschke (2008), or a stated generalization of it, two linked questions are open.

(1) Identification of the composition of distortions. Given the observable moment changes induced by activation of a performance measure (changes in the mean and variance of the measure, in the covariance $\mathrm{Cov}(p,V)$ between measure and organizational value, and in the slope $\mathrm{Cov}(p,V)/\mathrm{Var}(p)$), characterize the identified set for the two source-defined distortion channels: gaming ($w_{i,\alpha}>0$, with value-reducing gaming the sub-case $w_{i,\alpha}>0$ together with $\xi_{i,\alpha}>0$) and random measure-specific marginal productivity ($\eta_{i,\alpha}\neq 0$). Beyond the known special case in which the covariance itself falls while overall incentive weights do not decrease (which identifies value-reducing gaming), establish which features of the composition are and are not identified from these moments; determine what additional moment conditions, panel structure, or design variation would restore identification; and establish the validity or failure of the diagnostic under realistic observation conditions: (a) the researcher observes only a coarser project partition than the agent conditions on, (b) adoption or activation of the measure is endogenous to agents' actions or to selection, (c) agents are forward-looking and internalize their influence on the principal's future measure choices, and (d) $p_{\alpha}$ and $V_{\alpha}$ are estimated with measurement error and sampling noise (as with experimentally estimated impacts), including robust inference for the slope and covariance tests under such noise.

(2) Dynamic measurement design. Formulate the problem of a principal who does not know the distortion technology $(\eta,w,\xi)$ ex ante, chooses measure weights and whether to activate candidate measures over time, and trades off current incentive provision against the information about distortion margins revealed by the induced outcomes; characterize the optimal policy or prove structural properties of it (including impossibility results where they hold), at least in a canonical version of the environment.

The two components share one scientific objective: turning the one-shot detection test into a complete theory of measurement choice under distortion, in which what activation data identify about distortion composition feeds back into optimal sequencing of measurement decisions. Both components are open problems left undeveloped by the Courty-Marschke framework, and no finite-size, single-application, or method-class restriction is part of the question. Component (1) concerns the characterization of the identified set for the two channels, the determination of what additional moment conditions, panel structure, or design variation would restore identification, and the establishment of the validity or failure of the diagnostic under all four observation conditions (a)-(d); the characterization may establish either identification or non-identification. Component (2) concerns the characterization of the optimal policy or of its structural properties in a canonical version of the environment.

References

1. Pascal Courty and Gerald Marschke. A General Test for Distortions in Performance Measures. Review of Economics and Statistics 90(3):428-441, 2008. DOI: 10.1162/rest.90.3.428. https://doi.org/10.1162/rest.90.3.428
2. Pascal Courty and Gerald Marschke. A General Test of Gaming. EUI Working Paper ECO No. 2004/33, European University Institute, 2004. https://hdl.handle.net/1814/2705 (earlier full-text version of the published article)
3. Bengt Holmstrom and Paul Milgrom. Multitask Principal-Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. Journal of Law, Economics, and Organization 7(special issue):24-52, 1991. DOI: 10.1093/jleo/7.special_issue.24. https://doi.org/10.1093/jleo/7.special_issue.24
4. George P. Baker. Incentive Contracts and Performance Measurement. Journal of Political Economy 100(3):598-614, 1992. DOI: 10.1086/261831. https://doi.org/10.1086/261831
5. Howard S. Bloom. Accounting for No-Shows in Experimental Evaluation Designs. Evaluation Review 8(2):225-246, 1984. DOI: 10.1177/0193841X8400800205. https://doi.org/10.1177/0193841X8400800205
6. Pascal Courty and Gerald Marschke. An Empirical Investigation of Gaming Responses to Explicit Performance Incentives. Journal of Labor Economics 22(1):23-56, 2004. DOI: 10.1086/380402. https://doi.org/10.1086/380402
7. Pascal Courty and Gerald Marschke. Dynamics of Performance-Measurement Systems. Oxford Review of Economic Policy 19(2):268-284, 2003. DOI: 10.1093/oxrep/19.2.268. https://doi.org/10.1093/oxrep/19.2.268
8. Carolyn J. Heinrich and Gerald Marschke. Incentives and their dynamics in public sector performance management systems. Journal of Policy Analysis and Management 29(1):183-208, 2010. DOI: 10.1002/pam.20484. https://doi.org/10.1002/pam.20484
9. Randolph Sloof and Mirjam van Praag. Testing for Distortions in Performance Measures: An Application to Residual Income-Based Measures like Economic Value Added. Journal of Economics and Management Strategy 24(1):74-91, 2015. DOI: 10.1111/jems.12080. https://doi.org/10.1111/jems.12080
10. Michael J. Gibbs, Kenneth A. Merchant, Wim A. Van der Stede, and Mark E. Vargus. Performance Measure Properties and Incentive System Design. Industrial Relations 48(2):237-264, 2009. DOI: 10.1111/j.1468-232X.2009.00556.x. https://doi.org/10.1111/j.1468-232X.2009.00556.x
11. Sofronis Clerides, Panos Pashardes, and Alexandros Polycarpou. Peer Review vs Metric-based Assessment: Testing for Bias in the RAE Ratings of UK Economics Departments. Economica 78(311):565-583, 2011. DOI: 10.1111/j.1468-0335.2009.00837.x. https://doi.org/10.1111/j.1468-0335.2009.00837.x
12. Paul Clist. Payment by Results in Development Aid: All That Glitters Is Not Gold. World Bank Research Observer 31(2):290-319, 2016. DOI: 10.1093/wbro/lkw005. https://doi.org/10.1093/wbro/lkw005
13. Edwin Gerrish. The Effect of the Child Support Performance and Incentive Act of 1998 on Rewarded and Unrewarded Performance Goals. SSRN working paper, 2014. DOI: 10.2139/ssrn.2476033. https://doi.org/10.2139/ssrn.2476033
14. Ed Gerrish. The Impact of Performance Management on Performance in Public Organizations: A Meta-Analysis. Public Administration Review 76(1):48-66, 2016. DOI: 10.1111/puar.12433. https://doi.org/10.1111/puar.12433
15. Alex Frankel and Navin Kartik. Improving Information from Manipulable Data. Journal of the European Economic Association 20(1):79-115, 2022. DOI: 10.1093/jeea/jvab017. https://doi.org/10.1093/jeea/jvab017 (print issue February 2022; first published online June 2021)