Empirically testing the group size and task difficulty predictions of the crowdsourcing collective problem-solving model

The evolutionary knowledge-dynamics model of crowdsourcing by Guazzini, Vilone, Donati, Nardi and Levnajić (2015) predicts, from simulation alone, that group size and task difficulty jointly shape the balance between collaboration and free-riding in collective problem solving: collaboration falls as groups grow for tasks of non-extreme difficulty, free-riding depresses the payoff of very large groups, harder tasks restore cooperation, and an optimal group size exists at which collaboration propensity is independent of difficulty and individual and collective performance are maximized together. The authors identified testing the potentials of crowdsourcing through real experiments — controlled social experiments and uncontrolled online settings — as the field's most important direction, noting that no standard experimental framework for systematically varying group size and task difficulty in real human populations exists. No prior real-human study jointly varies group size and task difficulty while measuring the collaboration/free-riding balance together with both per-capita payoff and group-level accumulated knowledge, so this validation has not yet been performed. The problem is to design and conduct real-human experiments that jointly manipulate group size and task difficulty, measure the emergent collaboration/free-riding balance together with individual and collective performance, adjudicate the model's central predictions in controlled and/or online crowdsourcing settings, and, where predictions fail, examine whether the deviations are explained by the idealizations the source flags (fixed collaboration probabilities, no within-lifetime learning, selection by individual fitness rather than collective capacity) or by experimental factors.

Background

Crowdsourcing is the solicitation of ideas, information or work from a large pool of independent participants, typically via the Internet, with the aim of finding good solutions to a given challenge. Because contributions usually add to a shared pool of knowledge from which everyone benefits, crowdsourced problem solving has the structure of a social dilemma: an individual who withholds effort — a free-rider — can still exploit the knowledge accumulated by collaborating others, so individual and collective interests need not coincide. Whether, and how, the size of the participating group and the difficulty of the task jointly determine this balance between collaboration and free-riding is an empirical question about human behavior, and it is the subject of this problem.

Guazzini, Vilone, Donati, Nardi and Levnajić (2015) studied this question with an evolutionary simulation model built to mimic crowdsourcing as collective problem solving. A fixed population of $N=100$ players is divided into equal-size groups of $S$ players (in the simulations, $S$ takes the seven values $S \in \{1, 2, 5, 10, 25, 50, 100\}$). Each player $i$ carries a fixed collaboration probability $p_i$, drawn uniformly from $[0,1]$: in every iteration the player joins the group's collaborating subgroup (a 'collectivist') with probability $p_i$, and otherwise works alone (an 'individualist'). The group faces a task of simplicity $R$, which acts as the probability that a collaborating player solves it in one attempt, so task difficulty is $1-R$. Collectivists solve the task with probability $R$; individualists solve it with probability $R^{G_i}$, where $G_i$ is a random integer between 1 and 10 — since $R \leq 1$, a larger $G_i$ means a smaller chance of solving, so an individualist faces a harder task but keeps the larger gain $G_i$ when successful, whereas knowledge produced by a successful collectivist is shared with the whole group. Two observables track knowledge: a group's capacity, the number of iterations in which at least one collectivist solves the task, modeling accumulated collective knowledge; and each player's fitness, the individual payoff, including gains received from others' shared solutions. Iterations are grouped into rounds of 1000; after each round a random 20% of the players whose fitness is below average is replaced by new players with freshly drawn collaboration probabilities — an evolutionary selection acting on the group's distribution of $p_i$, run for 2000 rounds per game.

The simulations yield a coherent phenomenology. For tasks of non-extreme difficulty, small groups evolve substantially higher collaboration than large groups, and very large groups end with lower average fitness because individualism (free-riding on the collaborating majority) is stimulated in them. As tasks become harder, large groups discourage individualism and cooperation returns toward parity. Most notably, at an intermediate optimal group size (about 50 in the model) the collaboration propensity becomes independent of task difficulty, and groups of this size simultaneously reach the highest average fitness and the best capacity — individual and collective optima coincide, contrary to typical game-theoretic expectations — with a moderate amount of free-riding remaining compatible with the accumulation of collective knowledge; that free-riding may even be beneficial is an extra-model conjecture the authors raise about the real world, speculating that individually gained unique knowledge might eventually be shared with the community and prove advantageous for society in the long run.

All of these results are computational. The authors emphasize in their Discussion that testing the potentials of crowdsourcing via real experiments — social experiments with controlled rules and known participants, as well as uncontrolled online experiments where participation is spontaneous — is the field's most important open direction, note that a standard framework for such experiments has yet to be established, and flag the model's idealizations: the collaboration probabilities $p_i$ never adapt to outcomes, players do not learn or mutate strategies within their lifetime, and selection favors individual fitness rather than collective capacity. They present more realistic variants that relax these idealizations — adaptive collaboration probabilities, within-lifetime learning, or selection tuned to collective capacity — as an open question for the model family rather than a settled extension. Precedents for experiments on group cooperation exist — web-based networked public-goods experiments (Suri and Watts 2011), collaborative learning in networks of 16-person groups (Mason and Watts 2012), coordination experiments on human subject networks (Kearns, Suri and Montfort 2006), and the demonstration that online labor markets such as Amazon Mechanical Turk can host behavioral experiments (Rand 2012) — and group performance has separately been shown to carry a collective intelligence factor, a statistical factor summarizing a group's general problem-solving capability (Woolley et al. 2010). None of these works, however — nor the human-group studies that manipulate overlapping constructs, the laboratory cooperation experiment varying group size and problem complexity (Guazzini, Stefanelli, Imbimbo, Vilone, Bagnoli and Levnajić 2019) and the empirical study of collective intelligence in real and online groups (Imbimbo, Stefanelli and Guazzini 2020) — jointly manipulates the model's construct pair — group size and task difficulty — with the collaboration/free-riding and individual-versus-collective performance readouts needed to adjudicate its predictions.

Problem Statement

Determine, by real experiments with human groups, whether the group-size × task-difficulty phenomenology predicted by the knowledge-dynamics model of crowdsourcing (Guazzini et al. 2015) holds for actual human populations, and refine the model where it fails.

Concretely: design and conduct experiments — in a controlled social-experimental setting with known participants and/or in an uncontrolled online crowdsourcing environment with spontaneous engagement — that (i) operationalize the model's constructs for real tasks: group size $S$, task difficulty (the analogue of $1-R$), the choice between contributing to shared collective knowledge and working individually (collaboration versus free-riding), an individual payoff analogue of fitness (per-capita payoff within a group, matching the model's mean players' fitness), and a collective knowledge analogue of capacity (group-level accumulated knowledge, matching the model's per-group capacity); (ii) systematically vary group size and task difficulty over multiple levels of each, including group sizes well beyond small teams, so that their joint effect is identifiable; and (iii) measure the emergent balance between collaboration and free-riding together with both individual and collective performance as functions of group size and difficulty.

The model's central predictions, to be tested against real-human evidence, are: (a) collaboration propensity decreases with group size for tasks of non-extreme difficulty, with very large groups suffering reduced average payoff from free-riding individualism; (b) increasing task difficulty restores cooperation in large groups toward parity with individualism; (c) there exists an optimal group size at which collaboration propensity is roughly independent of task difficulty and at which individual and collective performance — per-capita payoff and group-level accumulated knowledge, respectively, compared across group sizes at a fixed total participant pool as in the model's fixed-$N$ comparison — are simultaneously maximized, with the same group size achieving both optima across the difficulty levels studied; (d) a moderate level of free-riding is compatible with a group's accumulated collective knowledge — and, as a conjecture the source raises about the real world rather than a simulation result, that individually gained unique knowledge might eventually be shared with the community and prove beneficial to society. For each of (a)–(d), the verdict in each experimental setting used is supported, refuted, or untested according to whether the available real-human evidence bears on it; no dataset or participant pool accompanies the problem, so where the available evidence leaves a prediction untested, the question extends to what new data would confirm or refute it. No data are to be invented or imported to fill a gap.

Where predictions fail, the question is whether the deviations are explained by the idealizations the source flags — the fixed collaboration probabilities $p_i$ that do not adapt to iteration outcomes, the absence of within-lifetime learning or strategy mutation, or the selection of players by individual fitness rather than collective capacity — or by experimental factors such as self-selection, attrition, incentive effects, task misunderstanding, or non-transfer between settings. The experimental design specifies the task paradigm, the operationalization of difficulty, and measures of collaboration, free-riding, individual payoff, and collective knowledge, given the absence of a standard experimental framework noted by the source.

References

1. Guazzini, A., Vilone, D., Donati, C., Nardi, A. & Levnajić, Z. (2015). Modeling crowdsourcing as collective problem solving. Scientific Reports 5, 16557. DOI: 10.1038/srep16557. https://doi.org/10.1038/srep16557 (preprint: arXiv:1506.09155)
2. Guazzini, A., Vilone, D., Donati, C., Nardi, A. & Levnajić, Z. (2016). Correction: Corrigendum: Modeling crowdsourcing as collective problem solving. Scientific Reports 6, 33105. DOI: 10.1038/srep33105. https://doi.org/10.1038/srep33105
3. Guazzini, A., Stefanelli, F., Imbimbo, E., Vilone, D., Bagnoli, F. & Levnajić, Z. (2019). Humans best judge how much to cooperate when facing hard problems in large groups. Scientific Reports 9, 5497. DOI: 10.1038/s41598-019-41773-2. https://doi.org/10.1038/s41598-019-41773-2 (preprint: arXiv:1808.07670, titled 'Humans make best use of social heuristics when confronting hard problems in large groups')
4. Guazzini, A., Duradoni, M., Lazzeri, A. & Gronchi, G. (2018). Simulating the cost of cooperation: A recipe for collaborative problem-solving. Future Internet 10(6), 55. DOI: 10.3390/fi10060055. https://doi.org/10.3390/fi10060055
5. Carletti, T., Guarino, A., Guazzini, A. & Stefanelli, F. (2020). Problem solving: When groups perform better than teammates. Journal of Artificial Societies and Social Simulation 23(3), 4. DOI: 10.18564/jasss.4292. https://doi.org/10.18564/jasss.4292
6. Imbimbo, E., Stefanelli, F. & Guazzini, A. (2020). Adolescent's collective intelligence: Empirical evidence in real and online classmates groups. Future Internet 12(5), 81. DOI: 10.3390/fi12050081. https://doi.org/10.3390/fi12050081
7. Song, M., Gong, X., Du, J., Lu, T. & Jiao, R. J. (2024). Population dynamics modeling of crowdsourcing as an evolutionary Cooperation-Competition game for fulfillment capacity balancing and optimization of smart manufacturing services. Computers & Industrial Engineering 197, 110572. DOI: 10.1016/j.cie.2024.110572. https://doi.org/10.1016/j.cie.2024.110572
8. Almaatouq, A., Alsobay, M., Yin, M. & Watts, D. J. (2021). Task complexity moderates group synergy. Proceedings of the National Academy of Sciences 118(36), e2101062118. DOI: 10.1073/pnas.2101062118. https://doi.org/10.1073/pnas.2101062118
9. Suri, S. & Watts, D. J. (2011). Cooperation and contagion in web-based, networked public goods experiments. PLoS ONE 6(3), e16836. DOI: 10.1371/journal.pone.0016836. https://doi.org/10.1371/journal.pone.0016836
10. Mason, W. & Watts, D. J. (2012). Collaborative learning in networks. Proceedings of the National Academy of Sciences 109(3), 764-769. DOI: 10.1073/pnas.1110069108. https://doi.org/10.1073/pnas.1110069108
11. Kearns, M., Suri, S. & Montfort, N. (2006). An experimental study of the coloring problem on human subject networks. Science 313, 824-827. DOI: 10.1126/science.1127207. https://doi.org/10.1126/science.1127207
12. Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N. & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science 330, 686-688. DOI: 10.1126/science.1193147. https://doi.org/10.1126/science.1193147
13. Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology 299, 172-179. DOI: 10.1016/j.jtbi.2011.03.004. https://doi.org/10.1016/j.jtbi.2011.03.004