Ruffle&Riley (Schmucker, Xia, Azaria, and Mitchell, 2024) is an LLM-based conversational tutoring system in which a human learner teaches an LLM student agent, with dialogues orchestrated from tutoring scripts automatically induced from lesson text. In two between-subject online studies (N = 200), users rated the system as engaging and helpful and spent more time on it than on a reading activity, yet showed no significant short-term learning gains over reading. This crystallized an open question for the field: under what instructional designs, assessment formats, learner populations, and usage conditions will LLM-based conversational tutoring produce reliable, objectively measured learning gains rather than only engaging experiences? Later evidence is heterogeneous rather than settling: supplementary conversational AI tutoring raised math scores in Ghanaian schools; an unguarded generative-AI tutor harmed subsequent unassisted performance in high-school mathematics while guardrails mitigated the harm; large-scale randomized trials found that LLM-generated feedback in an online middle-school mathematics platform improved immediate next-problem performance, especially for low-knowledge learners; a pedagogically fine-tuned model supervised by expert tutors matched human tutors in UK classrooms; and perception-gain disconnects have been replicated. No controlled study yet maps instructional-design factors of unsupervised LLM conversational tutoring itself (feedback precision, dialogue structure, dosage, assessment format, learner population) to objective gains against active controls, so the question remains open.

Background

Intelligent tutoring systems (ITSs) are software systems that deliver individualized instruction, and conversational tutoring systems (CTSs) are the variant that conducts learning through natural-language dialogue; dialogue-based tutoring is recognized for promoting cognitive engagement and improving learning outcomes, especially on reasoning tasks, but historically required prohibitively expensive manual authoring of tutoring content. Large language models (LLMs), text generators such as the GPT-4 model behind ChatGPT, promised to remove this obstacle. Ruffle&Riley (Schmucker et al., 2024) is an early and carefully evaluated instance: an instructor supplies a lesson text, the system automatically induces an editable tutoring script structured as topics (review questions) together with related expectations (anticipated correct responses), building on the Expectation Misconception Tailoring design framework in which the tutor targets feedback to specific expectations and misconceptions; unlike traditional EMT-based systems, however, these scripts do not attempt to anticipate misconceptions ahead of time, instead relying on GPT-4 to detect and respond to misconceptions in the learner's responses during the active teaching process. A learning-by-teaching workflow is then orchestrated by two LLM agents acting as a student (whom the human learner teaches) and a professor (who moderates), following the ITS-typical inner and outer loop structure (the inner loop supports the learner within one step or question, the outer loop selects and sequences problems). The system was evaluated in two between-subject online studies with 200 adults recruited on the Prolific crowdsourcing platform, covering biology lessons and comparing against simpler question-answering chatbots and a reading activity. Learning was measured objectively with pre-tests and post-tests, separately from subjective engagement and understanding ratings. The headline result is a disconnect: Ruffle&Riley users reported high engagement and perceived helpfulness and spent more time on the activity, but short-term learning gains did not differ significantly from the much shorter reading activity. Conversation analysis further showed the system sometimes omits scripted expectations, repeats requests, and accepts partial or imprecise learner explanations too readily, pointing to feedback precision as one candidate design lever. The authors list further unresolved conditions: evaluation only with adult online participants (not K-12 or college students), only a single learning session (so adaptation and repeated exposure are unknown), the need for certification of safe and trustworthy behavior before use with younger learners, and recurring API costs raising equity concerns. The broader context makes the question pointed: one-to-one tutoring is widely considered the gold standard of instruction (as restated in the LearnLM/Eedi randomized trial), and generative AI is promoted as a route to a personal tutor for every learner, yet controlled evidence is heterogeneous. A WhatsApp-based conversational AI math tutor used as a supplement to regular instruction for eight months in Ghanaian schools raised math growth (effect size about 0.37); a high-school randomized experiment (Bastani et al., PNAS 2025) found an ordinary ChatGPT-style tutor boosted assisted performance but harmed later unassisted performance, with pedagogical guardrails (hints rather than answers) largely mitigating the harm; a pedagogically fine-tuned model whose drafts were supervised by expert human tutors performed at least as well as human tutors alone in UK secondary schools; and AI assistance to human tutors improved K-12 mastery rates, especially for weaker tutors. Most recently, two large-scale randomized trials in an online middle-school mathematics platform (Worden et al., 2026, 21,478 students combined) found that LLM-generated feedback on commonly occurring wrong answers made students 16% more likely to correct their current answer and 7% more likely to succeed on the subsequent problem, with benefits concentrated in low-knowledge students — evidence that generative-AI feedback can support learning in a practice-platform context, though the outcomes are immediate performance rather than delayed unassisted learning, and the setting is not conversational tutoring. Meanwhile benchmark work (TutorBench) shows frontier LLMs still score below 56% on expert-rubric measures of core tutoring skills, and the LearnLM position paper observes that the field predominantly evaluates such systems with pedagogical rubrics and preference judgments rather than learning outcomes. None of these studies manipulates the instructional design of an unsupervised conversational tutoring system against an active control and measures objective learning gains, which is precisely the gap Ruffle&Riley's null result exposed.

Problem Statement

Under what instructional designs, assessment formats, learner populations, and usage conditions does an LLM-based conversational tutoring system produce reliable, objectively measured learning gains, rather than only engaging learning experiences? The paradigm instance is the Ruffle&Riley class of systems, in which a human learner teaches an LLM student agent through dialogues orchestrated from automatically induced tutoring scripts (topics and expectations) without a human in the loop; in the original evaluation such systems produced high engagement but no significant short-term learning gains over reading the same lesson text. Candidate conditions identified in the source's own limitations and future-work discussion include the precision of targeted feedback on imprecise learner explanations (instead of premature acceptance), the structure of the dialogue and tutoring script, the number and spacing of sessions (single-session versus repeated exposure), the format of the learning assessment, and the learner population (online adults versus K-12 or college students, with safe-behavior certification as a precondition for younger learners); recurring API costs are raised separately as an equity and accessibility concern for deployment conditions. Evidence bearing on the question is controlled and comparative, against active control conditions and with objective pre/post learning measures, as in the source's own evaluation. Self-reported engagement, perceived helpfulness, or assisted-performance gains while the tool is available do not by themselves answer the question, since the source's null result shows these can dissociate from learning; the guardrail evidence from high-school mathematics (harm to subsequent unassisted performance without safeguards) shows unassisted post-tests are the decisive outcome. Each condition stands for a distinct claim about the direction of an effect on objectively measured learning gains in a particular setting, established by controlled comparative studies against an active control with objective pre/post learning measures. These conditions are independent sub-questions of one mapping: a rigorous characterization of any single one of them is a scientific result in its own right, and none of the others is a precondition for it.

References

  1. Robin Schmucker, Meng Xia, Amos Azaria, Tom Mitchell (2024). Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring System. arXiv:2404.17460. https://arxiv.org/abs/2404.17460
  2. Robin Schmucker, Meng Xia, Amos Azaria, Tom Mitchell (2024). Ruffle&Riley: From Lesson Text to Conversational Tutoring. Proceedings of the Eleventh ACM Conference on Learning @ Scale. DOI: 10.1145/3657604.3664719. https://doi.org/10.1145/3657604.3664719
  3. Robin Schmucker, Meng Xia, Amos Azaria, Tom Mitchell (2023). Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systems. arXiv:2310.01420. https://arxiv.org/abs/2310.01420
  4. Owen Henkel, Hannah Horne-Robinson, Nessie Kozhakhmetova, Amanda Lee (2024). Effective and Scalable Math Support: Evidence on the Impact of an AI-Tutor on Math Achievement in Ghana. arXiv:2402.09809. https://arxiv.org/abs/2402.09809
  5. Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Ozge Kabakci, Rei Mariman (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences 122(26). DOI: 10.1073/pnas.2422633122. https://doi.org/10.1073/pnas.2422633122
  6. Eamon Worden, Luca Dang, Wen Chiang Ivan Lim, Sarah Miller, Jiayi Zhang, Aaron Haim, Adam C. Sales, Ashish Gurung, Neil T. Heffernan (2026). A Large Scale Randomized Control Trial Showing LLM Generated Feedback Helps Low-Knowledge Middle School Math Students with Short-Term Learning. Proceedings of the Thirteenth ACM Conference on Learning @ Scale. DOI: 10.1145/3774398.3811594. https://doi.org/10.1145/3774398.3811594
  7. LearnLM Team, Eedi, Albert Wang, Aliya Rysbek, Andrea Huber, Anjali Nambiar, Anna Kenolty, Ben Caulfield, Beth Lilley-Draper, Bibi Groot, Brian Veprek, Chelsea Burdett, Claire Willis, Craig Barton, Digory Smith, George Mu, Harriet Walters, Irina Jurenka, Iris Hulls, James Stalley-Moores, Jonathan Caton, Julia Wilkowski, Kaiz Alarakyia, Kevin R. McKee, Liam McCafferty, Lucy Dalton, Markus Kunesch, Pauline Malubay, Rachel Kidson, Rich Wells, Sam Wheeler, Sara Wiltberger, Shakir Mohamed, Simon Woodhead, Vasco Brazao (2025). AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms. arXiv:2512.23633. https://arxiv.org/abs/2512.23633
  8. Rose Wang, Ana Ribeiro, Carly Robinson, Susanna Loeb, Dora Demszky (2024). Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise. Research Square preprint. DOI: 10.21203/rs.3.rs-5363154/v1. https://doi.org/10.21203/rs.3.rs-5363154/v1
  9. Lu Fang, Ge Tang, Lu Zhang (2025). Helpful or Harmful? Comparative Study of Perceived and Actual Effectiveness of LLM-Driven Tutors in Game-Based CFL Learning. Education Sciences 15(11):1502. DOI: 10.3390/educsci15111502. https://doi.org/10.3390/educsci15111502
  10. Rakshith S Srinivasa, Zora Che, Chen Bo Calvin Zhang, Diego Mares, Ernesto Hernandez, Jayeon Park, Dean Lee, Guillermo Mangialardi, Charmaine Ng, Ed-Yeremai Hernandez Cardona, Anisha Gunjal, Yunzhong He, Bing Liu, Chen Xing (2025). TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models. arXiv:2510.02663. https://arxiv.org/abs/2510.02663
  11. Irina Jurenka, Markus Kunesch, Kevin R. McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, Ankit Anand, Miruna Pislar, Stephanie Chan, Lisa Wang, Jennifer She, Parsa Mahmoudieh, Aliya Rysbek, Wei-Jen Ko, Andrea Huber, Brett Wiltshire, Gal Elidan, Roni Rabin, Jasmin Rubinovitz, Amit Pitaru, Mac McAllister, Julia Wilkowski, David Choi, Roee Engelberg, Lidan Hackmon, Adva Levin, Rachel Griffin, Michael Sears, Filip Bar, Mia Mesar, Mana Jabbour, Arslan Chaudhry, James Cohan, Sridhar Thiagarajan, Nir Levine, Ben Brown, Dilan Gorur, Svetlana Grant, Rachel Hashimshoni, Laura Weidinger, Jieru Hu, Dawn Chen, Kuba Dolecki, Canfer Akbulut, Maxwell Bileschi, Laura Culp, Wen-Xin Dong, Nahema Marchal, Kelsie Van Deman, Hema Bajaj Misra, Michael Duah, Moran Ambar, Avi Caciularu, Sandra Lefdal, Chris Summerfield, James An, Pierre-Alexandre Kamienny, Abhinit Mohdi, Theofilos Strinopoulous, Annie Hale, Wayne Anderson, Luis C. Cobo, Niv Efron, Muktha Ananda, Shakir Mohamed, Maureen Heymans, Zoubin Ghahramani, Yossi Matias, Ben Gomes, Lila Ibrahim (2024). Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach. arXiv:2407.12687. https://arxiv.org/abs/2407.12687
  12. Scott Crossley, Wesley Morris, Joon Suh Choi, Langdon Holmes (2025). Exploratory Assessment of Learning in an Intelligent Text Framework: iTELL RCT. Proceedings of the Twelfth ACM Conference on Learning @ Scale. DOI: 10.1145/3698205.3729548. https://doi.org/10.1145/3698205.3729548