Qualitative Methods: Interviews and Surveys

Qualitative methods are how you find out what is actually going on when numbers alone cannot tell you. For example, in 2024, a team led by Jenny Liang surveyed 410 professional developers across industries about their experience with AI programming assistants [Liang2024]. They did not just ask "is this useful?" They asked about specific scenarios, frustrations, workarounds, and the kinds of tasks where developers had learned to distrust the tools. The result was a detailed map of where AI assistance actually helps and where it gets in the way—knowledge that would have been invisible to a study that only measured task completion times.

A smaller but deeper example: Johnson et al. interviewed 20 developers to ask why static analysis tools are so rarely used despite being widely available [Johnson2013]. Every participant believed the tools are beneficial, but false positives and poor warning presentation were the main barriers. Where Liang provides breadth (410 developers across industries), Johnson provides depth (20 developers with detailed follow-up), which illustrates the scale-versus-richness trade-off between surveys and interviews.

When Qualitative Methods Are the Right Choice

• When you do not yet know what to measure
  • Qualitative work often precedes quantitative work by identifying the right variables
• When the phenomenon involves meaning, interpretation, or context that a number cannot capture
• When your sample is small or access is limited
  • Surveys of 10 people are not statistically powerful, but interviews with 10 people can be rich
• When you want to understand why a pattern exists, not just that it does
• When stakeholders need to be heard rather than counted

Designing Interviews

• Three types of interviews
  • Structured interviews use a fixed script
  • Semi-structured interviews have a guide but allow follow-up
  • Unstructured interviews follow the participant's lead
  • Semi-structured is the most common in software engineering research: consistent enough to compare across participants, flexible enough to surface surprises
• Open questions invite narrative: "Tell me about a time when the AI suggestion was wrong in a way that surprised you"
• Closed questions invite classification: "On a scale of 1 to 5, how often do you accept AI suggestions without reading them?"
• Probing follows up on what a participant says: "You mentioned you don't trust it for security-related code—can you say more about that?"
• Avoid leading questions: "Don't you find it faster?" assumes the answer
• Avoid hypothetical questions: "Would you use X if Y?" People are poor predictors of their own behavior

Designing Surveys

• Surveys trade depth for breadth: you can reach many more people but get shallower data from each
• Likert scales ("Strongly agree" to "Strongly disagree") are common but have known problems:
  • People cluster toward the middle or toward the extremes
  • The same label means different things to different people
  • Use at least 5 points: 7 points gives more resolution
• Response bias: people answer differently depending on question order, framing, and what they think the researcher wants to hear
• Sampling strategy determines who you can generalize to
  • Convenience samples (whoever responds to a social media post) are common and usually biased toward engaged, technical users
  • Stratified samples deliberately include proportional representation of subgroups
• Pilot your survey with 3-5 people and revise before distributing
• E.g., Bettenburg et al. surveyed Apache, Eclipse, and Mozilla developers and users to find out what makes a good bug report [Bettenburg2008]
  • A concrete question, a targeted audience, and actionable results

Thematic Analysis

• Thematic analysis is the standard approach for analyzing qualitative data
• Open coding: read through the data and tag segments with descriptive labels (e.g., "trust in AI suggestions," "time saved on boilerplate")
  • Use gerund coding: frame codes as actions rather than nouns, so "avoiding AI for security tasks" rather than "AI distrust"
  • This preserves the temporal and causal structure of what participants are actually doing [Thornberg2014]
• A theme is not a bucket for related quotes:
  • It should express a claim you could write as a sentence, such as "developers trust AI output for syntax but not for logic" [Braun2019]
  • Strong themes name tensions as well as patterns: "trusting the output while suspicious of the process" is more informative than "attitudes toward AI"
• Axial coding: group codes into higher-level themes and examine relationships between them
  • Ask which themes co-occur across participants, and why
  • Look for conditions, contexts, and consequences: what makes a theme appear, in what setting, and with what effect on the participant
  • If a code keeps appearing alongside very different themes, it may be doing double duty: split it and check whether each half is coherent on its own
  • Separate participants' hypotheses and actions from their emotions and evaluations
    • "I think AI will replace junior devs" is a claim
    • "I hate how deskilled it makes me feel" is an affect
• Saturation: stop collecting data when new interviews or responses stop introducing new codes
  • A practical definition of "enough data"
• Code with a second researcher and measure intercoder reliability to reduce individual bias
  • We'll discuss Cohen's kappa later
• Keep an audit trail: document coding decisions so others can evaluate them

Experience Sampling

• Experience sampling prompts participants at random or scheduled intervals during their workday to report their current task, perceived state, or productivity
• It captures in-the-moment data that retrospective surveys miss
  • A developer's sense of how productive a day was often differs from what they reported each hour during that day
• A field study using hourly self-reports found that developers categorized into six types (focused, social, lone, balanced, leading, goal-oriented) and that productivity patterns varied substantially by time of day and individual [Meyer2017]
• When measuring subjective states, use validated instruments rather than writing your own questions
  • Scales like SPANE (Scale of Positive and Negative Experience) have known psychometric properties [Diener2010]
  • A five-item Likert question drafted for a single study does not

Triangulation

• Triangulation means combining multiple data sources or methods to increase confidence in a finding
• If interviews, survey, and observations all point to the same conclusion, that conclusion is more credible
• Types of triangulation: data (multiple sources), investigator (multiple coders), method (qualitative + quantitative), theory (multiple interpretive frameworks)

Grounded Theory

• Grounded theory is a qualitative methodology in which theory develops inductively from data through iterative coding, rather than testing a predetermined hypothesis
• A researcher records observations or interviews, codes them, identifies patterns, collects more data to challenge those patterns, and continues until saturation
• Grounded theory is expensive and slow but produces theory that is directly grounded in the messiness of real practice
• Examples:
  • Sedano et al. spent 2.5 years embedded at Pivotal and produced a nine-category taxonomy of software development waste [Sedano2017]
  • Zieris and Prechelt recorded 60+ pair programming sessions at a dozen companies and identified the knowledge gap constellations that student experiments had controlled away [Zieris2014]
  • Wessel et al. interviewed 21 open-source practitioners about their experience with bots in pull requests and developed a theory of how bot output becomes noise that overwhelms and distracts developers [Wessel2021]

Ethical Considerations

• Even internal research on colleagues requires ethical care
  • Which is often ignored
• Informed consent: participants must know what data is being collected and how it will be used
• Privacy: interview and survey data can reveal individual performance, dissatisfaction, and private information
• Power dynamics: interviewing colleagues or subordinates creates pressure to participate and to answer in ways that please the researcher
• Harm: findings about team practices can be used for performance evaluation, even if that was not the study's intent
• Involve legal, HR, or ethics reviewers before collecting data from individuals in your organization

Misconceptions

Leading questions don't bias interview results if you're aware of them.

Even when you know a question is leading, the participant may still be pushed toward the expected answer. "Most developers find AI helpful — would you agree?" presupposes the answer. Rewrite as an open question: "What has your experience been with AI coding tools?"

Surveying your own team or users is sufficient for internal research.

Convenience sampling gives you data about a specific group that may not represent the population you want to generalize to. If only people who already agree with you respond, the results will confirm your assumptions.

You can stop analyzing data once you have enough to support your hypothesis.

Confirmation bias pushes researchers to stop analysis when the data supports their expectations. Rigorous qualitative work continues until saturation.

A study of 20 developers at one company can describe "what developers think."

Overgeneralization turns findings from a specific sample into claims about a broader population. "Developers find AI tools frustrating" from a study of 20 junior developers at one company is overgeneralized. The claim should specify who, where, and under what conditions.

A low response rate doesn't matter if you have enough responses.

If only 10% of people surveyed respond, the 90% who did not may have very different views. Non-response bias can be larger than sampling error, and a high absolute number of responses does not correct for it.

Qualitative research is just asking people what they think.

Rigorous qualitative work involves systematic data collection, structured analysis, documentation of decisions, and assessment of intercoder reliability. Asking a few colleagues over lunch is not a study.

More interviews always produce better qualitative results.

Depth matters more than volume. A study with twenty rich, well-analyzed interviews reaching saturation is more informative than one with a hundred superficial ones that never probe below the surface.

A high response count makes a survey representative.

Representativeness depends on who responds relative to who you want to generalize to, not on the raw number of responses. A million responses from a self-selected online audience is still a biased convenience sample.

Thematic analysis is subjective and therefore unreliable.

The subjectivity of interpretation is a known feature, not a flaw. Qualitative researchers manage it through audit trails, multiple coders, and transparent documentation of how codes and themes were derived.

Any descriptive label counts as a valid code.

A code named "AI" or "trust" is a noun bucket, not an analysis. Good codes capture what a participant is doing: "switching off AI suggestions after a bad experience" tells you something; "negative AI attitude" does not. The same principle applies to themes: a theme that cannot be expressed as a claim is a filing category, not a finding.

Check Understanding

Exercises