Software Engineering's Greatest Hits

<div align="center" style="padding-top: 2rem;">
  <h1>Software Engineering's Greatest Hits</h1>
  <p><strong>what we actually know about software development<br/>and why we believe it's true</strong></p>
  <p><img src="../img/codebender-logo.svg" alt="Third Bit logo" width="20%" /></p>
  <h4>Greg Wilson</h4>
  <p><small><a href="http://third-bit.com/talks/greatest-hits/">http://third-bit.com/talks/greatest-hits/</a></small></p>
  <p><img src="../img/cc-by.svg" width="20%" alt="CC-BY" /></p>
</div>
---
<div class="split-title">
  <h2>In the Beginning</h2>
</div>
  <div align="center">
    <img src="ships.jpg" alt="Sailing ships"/>
  </div>

- Seven Years' War (actually lasted nine years)
  - British lost 1512 sailors to enemy action…
  - …and almost 100,000 to scurvy
</div>
---
name: james-lind
<div class="split-title">
  <h2>Irony</h2>
</div>
<div class="flex-row">
  <div class="flex-col" width="30%">
    <img src="james-lind.jpg" alt="James Lind"/>
  </div>
  <div class="flex-col">
    <ul>
      <li>James Lind (1716–94)</li>
      <li>1747: first controlled medical experiment</li>
    </ul>
    <table>
      <tr>
	<td>cider</td>
	<td>sea water</td>
      </tr>
      <tr>
	<td>vitriol</td>
	<td>vinegar</td>
      </tr>
      <tr>
	<td>barley water</td>
	<td>oranges</td>
      </tr>
    </table>
  </div>
</div>

---

- Those given oranges survived
- Admiralty ignored the result until 1794
---
name: doctors
<div class="split-title">
  <h2>It Took a While</h2>
</div>

- 1950: Hill & Doll publish a case-control study comparing smokers with non-smokers
- Found that smoking causes lung cancer…
---
template: doctors
- …and that many people would rather fail than change
---
name: introduction
<div class="split-title">
  <h2>Is Software Engineering?</h2>
</div>

- "The use of the scientific method to design novel things"
---

---

<div class="split-title">
  <h2>Chickens and Eggs</h2>
</div>

- We know a lot about software and how it's built
- But students aren't taught methods or results in school
- And we're all too busy to learn them later

---

<div class="split-title">
  <h2>We Can Do Better</h2>
</div>

- Steady growth of empirical studies over 30 years
- Fueled in part by data from open source projects
- Many studies are small, and not all are well done
- But we can refute some myths and reveal new insights

---

<div class="split-title">
  <h2>Are Some Languages Better Than Others?</h2>
</div>

<a href="#Stefik2013">Stefik</a>: "An Empirical Investigation into Programming Language Syntax"

- Measured how easily novices could read:
  - Perl
  - Quorum: the language their team is building
  - Randomo: syntax "designed" by rolling D&D dice
---

template: stefik-1
- Perl is as hard for novices to learn as a language with a randomly-designed syntax
---

<div class="split-title">
  <h2>Are Some Languages Better Than Others?</h2>
</div>
- Second study
  - More subjects
  - Multiple assessment strategies
- Languages in the C family are as hard for novices to learn to read as a randomly-designed language
- Ruby and Python are significantly easier
- Quorum is easier still

---

<div class="split-title">
  <h2>Nobody Knows Anything</h2>
</div>

<a href="#Altadmri2016">Altadmri2016</a>: "37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data"

- Ask educators for learners' most common mistakes
- Compare their answers to empirical data
- Weak consensus among educators
- Weak correlation with observations
- Educator experience had only weak impact on results

---

<div class="split-title">
  <h2>Nobody Knows Anything</h2>
</div>

Most common actual errors are:

- Mis-matched parentheses (*not* confusing `=` with `==`)
- Invoking methods with the wrong arguments
- Reaching end of non-`void` method without `return`
---

The three that take the most time to fix are:

- Confusing  short-circuit logical and bitwise operators
- Using `==` instead of `.equals` to compare strings
- Ignoring the return value from a non-void method

---

<div class="split-title">
  <h2>Let's Talk About Test-Driven Development</h2>
</div>

<a href="#Shull2010">Shull2010</a>: "How Effective is Test-Driven Development?"

> …evidence from controlled experiments suggests an
> improvement in productivity when TDD is used.
> However…pilot studies provide mixed evidence, some in
> favor of and others against TDD. In the industrial
> studies…evidence suggests that TDD yields worse
> productivity. Even when considering only the more rigorous
> studies…the evidence is equally split for and against a
> positive effect.

---

<div class="split-title">
  <h2>Let's Talk About Test-Driven Development</h2>
</div>

<a href="#Fucci2016">Fucci2016</a>: "An External Replication on the Effects of Test-driven Development Using a Multi-site Blind Analysis Approach"

- Replication study, 39 professionals, real projects
- No significant difference between test-first and test-last development
---

template: fucci-1
name: fucci-2
- "The claimed benefits of TDD may…rather [be] due to the fact that [it] encourages fine-grained steady steps that improve focus and flow."
---

template: fucci-2
> I practice TDD…and it works great. We don't need to prove that it works anymore.
---

<div class="split-title">
  <h2>Code Metrics</h2>
</div>

- Lots of code metrics have been proposed
  - Halstead's measures
  - Cyclomatic complexity
  - Many object-oriented measures
---
template: metrics-1
- But nothing works better than counting lines of code
  - <a href="#ElEmam2001">ElEmam2001</a>: "Confounding Effects of Class Size on the Validity of Object-Oriented Metrics"
  - <a href="#Herraiz2010">Herraiz2010</a>: "Beyond Lines of Code: Do We Need More Complexity Metrics?"
---

<div class="split-title">
  <h2>Check Your Error Handling</h2>
</div>

<a href="#Yuan2014">Yuan2014</a>: "Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems"

- 198 randomly selected, user-reported failures on Cassandra, Hadoop MapReduce, etc.
- Majority of catastrophic failures could easily have been prevented by performing simple testing on error handling code
---

<div class="split-title">
  <h2>Check Your Error Handling</h2>
</div>

<a href="#Nakshatri2016">Nakshatri2016</a>: "Analysis of Exception Handling Patterns in Java Projects: An Empirical Study"

- Most common `catch` block logs the error rather than trying to recover from it
- Next most common does nothing at all (20% of cases)
  - Or converts the checked exception into an unchecked one so it can be ignored.
- Most programmers simply catch `Exception` (78%) or `Throwable` (84%) rather than their subclasses

---

<div class="split-title">
  <h2>Open Source</h2>
</div>

<a href="#Steinmacher2015">Steinmacher2015</a>: "Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects"

- Identify 58 potential barriers
- What matters most?
  1. How easy it is to get set up to make a contribution
  1. How easy it is to find a task to start with
- Other work has also identified "how warmly first contribution was received"

---

<div class="split-title">
  <h2>There Is No "Geek Gene"</h2>
</div>

<a href="#Patitsas2016">Patitsas2016</a>: "Computer Science Grades Are Not Bimodal"

- The "geek gene" is computing's most enduring and damaging myth
- In fact, only 5.8% of course grade distributions at a large university were actually multi-modal
- But CS faculty are more likely to see distributions as bimodal if they think they're from a CS class
- Even *more* likely if they believe some students are innately predisposed to do well in CS

---

<div class="split-title">
  <h2>Productivity</h2>
</div>

- <a href="#Prechelt2000">Prechelt2000</a> had programmers implement the same thing in the language of their choice
- How long did it take?

---

<div class="split-title">
  <h2>Productivity</h2>
</div>

- Shortest and longest times were 0.6 and 63 hours
  - Ratio is 150X (!)
- But people used 7 different languages
- If we only look at Java, times are 3.8 vs. 63 hours (17X)
- But comparing best and worst exaggerates differences
- 90th percentile vs. 50th gives 3.7X

---

<div class="split-title">
  <h2>Productivity</h2>
</div>

<a href="#Begel2014">Begel2014</a>: "Analyze This! 145 Questions for Data Scientists in Software Engineering"

- Ask a bunch of developers to propose questions, then ask another bunch to rate them
- Developers want to know how people actually use their products
- They *don't* want assessments of individual performance
  - Because they're afraid such measures will be weaponized

---

## Sidebar

- My colleagues and I did a similar study of CS teachers and CS education researchers
- No overlap between the top 5 questions selected by each group :-(

---