How to Not Be Wrong About AI

This tutorial is a short introduction to research methods you can use to figure out whether AI-based programming tools are helping your team or not. Even if you don't do such studies yourself, we hope that understanding how they work will help you decide which claims about them to trust. Help is welcome, but please note that all contributors must abide by our Code of Conduct.

Lessons

1. Introduction
2. Defining Productivity
3. How to Be Wrong
4. Qualitative Methods: Interviews and Surveys
5. Quantitative Methods: Controlled Experiments
6. Observational Studies and Natural Experiments
7. Reading Studies Critically
8. Running Studies in Your Organization
9. Starting Cheap
10. Conclusion: What to Do Next

Appendices

1. License
2. Code of Conduct
3. Contributing
4. Bibliography
5. Glossary
6. Further Reading
7. Statistical Reference

Slides

1. How To Not Be Wrong About AI

start where you are · use what you have · help who you can