Three Research Questions About AI
Sebastian Baltes recently asked, “In your view, what don’t we know about software development with agentic AI tools (but should)? To me, ‘know’ means proper empirical evidence. Not ‘some random dude wrote a blog post about it’.” Here are my answers:
-
What heuristics developers use to evaluate claims about AI, i.e., how do they decide which claims are believable and which aren’t. (Note that I’m not asking which claims they believe, but rather how they decide.)
-
How much of the discussion in Sadowski & Zimmermann (eds) Rethinking Productivity in Software Engineering (2019) has proven to be relevant/useful since the AI tidal wave hit? I.e., were researchers thinking about the right things seven years ago? (I hope that will give us an idea of how much of what’s being said now is likely to prove useful in hindsight a few years in the future.)
-
Which of the tools requested in What Developers Actually Want From AI would we be able to evaluate if they magically appeared tomorrow? I.e., for which ones would we be able to say, “Here’s what impact it actually has on the development process” with some degree of reliability?