Recent Reading About LLMs

I’m co-teaching a lesson for the Carpentries next week about the impact of LLMs on teaching. Here are a few things I’ve been reading to prepare:

Barba2026

Lorena A. Barba and Laura Stegner: “The Conversational Exam: A Scalable Assessment Design for the AI Era”. https://arxiv.org/abs/2601.10691, 2026. Conversational exam (live coding + explanation in small groups) restores assessment validity against generative AI cheating; 58 students examined in 2 days; combines authentic practice with inherent validity.

Bielaczyc1995

Katerine Bielaczyc, Peter L. Pirolli, and Ann L. Brown: “Training in Self-Explanation and Self-Regulation Strategies: Investigating the Effects of Knowledge Acquisition Activities on Problem Solving.” Cognition and Instruction. 13(6), 1995. https://doi.org/10.1207/s1532690xci1302_3. Training study (24 novice programmers) showing self-explanation and self-regulation strategy training causally improves programming task performance; instructional group showed significantly greater strategy use and performance gains.

Bridgeford2025

Eric W. Bridgeford, Iain Campbell, Zijao Chen, et al.: Ten Simple Rules for AI-Assisted Coding in Science. https://arxiv.org/abs/2510.22254, 2025. 10 practical rules for AI-assisted coding in scientific computing; addresses problem preparation, context management, testing/validation, and code quality; emphasizes human agency and domain expertise for reproducible research.

Butler2024

Jenna Butler, Jina Suh, Sankeerti Haniyur, and Constance Hadley: “Dear Diary: A Randomized Controlled Trial of Generative AI Coding Tools in the Workplace.” https://doi.org/10.48550/arxiv.2410.18334, 2024. Mixed-methods study (survey + RCT + 3-week diary) on generative AI coding tools at a large multinational; sustained use increases perceived usefulness and enjoyment; trustworthiness perceptions unchanged; 84% report positive daily work changes; unexpected uses include web search replacement.

Deslauriers2019

Louis Deslauriers, Logan S. McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin: “Measuring Actual Learning Versus Feeling of Learning in Response to Being Actively Engaged in the Classroom.” Proc. National Academy of Sciences, 116, Sept. 2019. https://doi.org/10.1073/pnas.1821936116. RCT shows active learning produces more learning but lower perceived learning; increased cognitive effort is misread as poorer learning; early instructor intervention corrects this misperception.

FinnieAnsley2022

James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather: “The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming.” Proc. 24th Australasian Computing Education Conference, https://doi.org/10.1145/3511861.3511863, 2022. OpenAI Codex outscores most students on intro programming exams; handles Rainfall problem variants well; generates diverse solutions for identical prompts; raises challenges and opportunities for CS education.

Jiao2026

Yuling Jiao and Qiuli Wang: “Large language models for formative feedback in writing instruction: a systematic review of classroom interventions, feedback quality, and student outcomes”. Frontiers in Education, 11, 2026, https://doi.org/10.3389/feduc.2026.1834085. Studies in which teachers discussed AI-generated feedback, helped students interpret it, or combined it with their own comments generally reported better learning outcomes than studies where students worked independently with AI.

Leinonen2023a

Juho Leinonen, Paul Denny, Stephen MacNeil, et al.: “Comparing Code Explanations Created by Students and Large Language Models.” Proc. 2023 Conference on Innovation and Technology in Computer Science Education, https://doi.org/10.1145/3587102.3588785, 2023. LLM-generated code explanations are rated significantly more accurate and understandable than student-generated ones in a 1000-student course; scalable on-demand explanations can scaffold introductory programming learning.

Leinonen2023b

Juho Leinonen, Arto Hellas, Sami Sarsa, et al.: “Using Large Language Models to Enhance Programming Error Messages.” Proc. 54th ACM Technical Symposium on Computer Science Education, https://doi.org/10.1145/3545945.3569770, 2023. LLMs enhance Python error messages with plain-language explanations and fix suggestions; sometimes surpass original messages in interpretability and actionability for novice programmers.

Ma2024

Qianou Ma, Hua Shen, Kenneth Koedinger, and Sherry Tongshuang Wu: “How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging.” Lecture Notes in Computer Science, https://doi.org/10.1007/978-3-031-64302-6_19, 2024. HypoCompass trains students to debug LLM code by having them hypothesize error causes while LLMs handle code completion; improves debugging performance 12% over pre-test with fourfold efficiency vs. human tutors.

Ma2025

Qianou Ma, Weirui Peng, Chenyang Yang, Hua Shen, Ken Koedinger, and Tongshuang Wu: “What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use.” ACM Transactions on Computer-Human Interaction, 32(4), https://doi.org/10.1145/3731756, 2025. Randomized experiment with 30 novices finds Requirement-Oriented Prompt Engineering (ROPE) training achieves 20% gains vs. 1% for conventional prompt engineering training.

OBrien2026

Gabrielle O’Brien, Alexis Parker, Nasir Eisty, and Jeffrey Carver: “A survey of generative AI adoption and perceived productivity among scientists who program.” 2026, https://doi.org/10.48550/arXiv.2512.19644. Survey of 868 scientists who program as part of their work, reporting that 80% use GenAI tools in their programming, with 77.5% of those using general purposing tools like ChatGPT over specialised coding tools.

Peng2023

Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer: “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.” 2023, https://doi.org/10.48550/arXiv.2302.06590. Randomized controlled experiment claiming that GitHub Copilot users completed a JavaScript coding task 55.8% faster than the control group.

Richards2026

Jonan Richards, Bruno Alves de Oliveira, Iury Oliveira, Igor Wiese, and Mairieli Wessel: “No Two Developers Think Alike: How Problem-Solving Styles and Experience Shape Needs in Conversational Interaction with Copilot”. 2026, https://arxiv.org/abs/2606.19216. Characterizes 5 distinct interaction modes and 10 underlying needs in developers’ interactions with AI tools.

Sadowski and Zimmerman 2019

Caitlin Sadowski and Thomas Zimmermann (eds.): Rethinking Productivity in Software Engineering. Apress, 2019, 9781484242216. Edited volume collecting research and practitioner perspectives on how to understand, define, and measure software developer productivity.

Stray2026

Viktoria Stray, Elias Goldmann Brandtzæg, Viggo Wivestad, Astri Barbala, and Nils Brede Moe: “Developer Productivity with and Without GitHub Copilot: A Longitudinal Mixed-Methods Case Study.” Proceedings of the 59th Hawaii International Conference on System Sciences, https://doi.org/10.24251/hicss.2026.880. 2026. Mixed-methods study of 703 NAV IT repositories finds Copilot users were more active even before adoption and shows no statistically significant changes in commit-based metrics after adopting the tool.