Another Modest Proposal
RSECon26 is happening in Sheffield in September. After spending two days talking to people about LLMs and research software engineering, I think the conference will be a golden opportunity to find out how people are currently using LLMs in their work.
-
Invite people to take part in a 90-minute session (which, once they’re set up, will give you about an hour of useful time).
-
Give them two small problems to solve: one that requires writing code from scratch, one in which they need to debug existing code. (I’d use data ingestion and cleanup problems, because I think they’ll be most familiar to most people.)
-
Have them record their screens while they work. They can use whatever tools they want: in particular, they can use or not use LLMs when and as they normally would.
Analyzing those recordings will give us a baseline reference against which to to compare future runs of this experiment. It will also help us identify places where some people are more or less advanced than others, which will give us some idea of what to teach and a reason for people to pay attention. Most importantly, I think it will give people in the RSE community something to look forward to and a reason to believe that people they trust are trying to figure out what should happen next.