7 Pedagogical Content Knowledge

Every teacher needs three things:

content knowledge: such as how to program;
general pedagogical knowledge: such as an understanding of the psychology of learning; and
pedagogical content knowledge: (PCK), which is the domain-specific knowledge of how to teach a particular concept to a particular audience. In computing, PCK includes things like what examples to use when teaching how parameters are passed to a function or what misconceptions about nesting HTML tags are most common.

We can add technical knowledge to this mix (Koehler et al. 2013), but that doesn’t change the key point: it isn’t enough to know the subject and how to teach—you have to know how to teach that particular subject (Mayer 2004). This chapter therefore summarizes some results from computing education research that will add to your store of PCK.

As with all research, some caution is required when interpreting results:

Theories change as more data becomes available.: Computing education research (CER) is a young discipline: the American Society for Engineering Education was founded in 1893 and the National Council of Teachers of Mathematics in 1920, but the Computer Science Teachers Association didn’t exist until 2005. While a steady stream of new insights are reported at conferences like SIGCSE, ITiCSE, and ICER, we simply don’t know as much about learning to program as we do about learning to read, play a sport, or do basic arithmetic.
Most of these studies’ subjects are WEIRD:: they are from Western, Education, Industrialized, Rich, and Democratic societies (Henrich et al. 2010). What’s more, they are also mostly young and in institutional classrooms, since that’s the population most researchers have easiest access to. We know much less about adults, members of marginalized groups, learners in free-range settings, or end-user programmers, even though there are far more of them.

If this was an academic treatise, I would therefore preface most claims with qualifiers like, “Some research may seem to indicate that…” But since actual teachers in actual classrooms have to make decisions regardless of whether research has clear answers yet or not, this chapter presents actionable best guesses rather than nuanced perhapses.

Jargon

Like any specialty, CER has jargon. CS1 refers to an introductory semester-long course in which learners meet variables, loops, and functions for the first time, while CS2 refers to a second course that covers basic data structures like stacks and queues, and CS0 refers to an introductory course for people without any prior experience who aren’t intending to continue with computing right away. Full definitions for these terms can be found in the ACM Curriculum Guidelines.

7.1 What Are We Teaching Them Now?

Very little is known about what coding bootcamps and other free-range initiatives teach, in part because many are reluctant to share their curriculum. We know more about what is taught by institutions (Luxton-Reilly et al. 2017):

Topic	%	Topic	%
Programming Process	87%	Data Types	23%
Abstract Programming Thinking	63%	Input/Output	17%
Data Structures	40%	Libraries	15%
Object-Oriented Concepts	36%	Variables & Assignment	14%
Control Structures	33%	Recursion	10%
Operations & Functions	26%	Pointers & Memory Management	5%

High-level topic labels like these can hide a multitude of sins. A more tangible result comes from (Rich et al. 2017), which reviewed a hundred articles to find learning trajectories for computing classes in elementary and middle schools. Their results for sequencing, repetition, and conditionals are essentially collective concept maps that combine and rationalize the implicit and explicit thinking of many different educators (Figure 7.1).

Figure 7.1: Learning trajectory for conditions (from (Rich et al. 2017))

7.2 How Much Are They Learning?

There can be a world of difference between what teachers teach and how much learners learn. To explore the latter, we must use other measures or do direct studies. Taking the former approach, roughly two-thirds of post-secondary students pass their first computing course, with some variations depending on class size and so on, but with no significant differences over time or based on language Watson and Li (2014).

How does prior experience affect these results? To find out, (Wilcox and Lionelle 2018) compared the performance and confidence of novices with and without prior programming experience in CS1 and CS2 (see below). They found that novices with prior experience outscored novices without by 10% in CS1, but those differences disappeared by the end of CS2. They also found that women with prior exposure outperformed their male peers in all areas, but were consistently less confident in their abilities (Section 10.4).

As for direct studies of how much novices learn, (McCracken et al. 2001) presented a multi-site international study that was later replicated by (Utting et al. 2013). According to the first study, “…the disappointing results suggest that many students do not know how to program at the conclusion of their introductory courses.” More specifically, “For a combined sample of 216 students from four universities, the average score was 22.89 out of 110 points on the general evaluation criteria developed for this study.” This result may say as much about teachers’ expectations as it does about student ability, but either way, our first recommendation is to measure and track results in ways that can be compared over time so that you can tell if your lessons are becoming more or less effective.

7.3 What Misconceptions Do Novices Have?

Chapter 2 explained why clearing up novices’ misconceptions is just as important as teaching them strategies for solving problems. The biggest misconception novices have—sometimes called the “superbug” in coding—is the belief that computers understand what people mean in the way that another human being would (Pea 1986). Our second recommendation is therefore to teach novices that computers don’t understand programs, i.e. that calling a variable “cost” doesn’t guarantee that its value is actually a cost.

(Sorva 2018) presents over forty other misconceptions that teachers can also try to clear up, many of which are also discussed in (Qian and Lehman 2017)’s survey. One is the belief that variables in programs work the same way they do in spreadsheets, i.e. that after executing:

grade = 65
total = grade + 10
grade = 80
print(total)

the value of total will be 90 rather than 75 (Kohn 2017). This is an example of the way in which novices construct a plausible-but-wrong mental model by making analogies; other misconceptions include:

A variable holds the history of the values it has been assigned, i.e. it remembers what its value used to be.
Two objects with the same value for a name or id attribute are guaranteed to be the same object.
Functions are executed as they are defined, or are executed in the order in which they are defined.
A while loop’s condition is constantly evaluated, and the loop stops as soon as it becomes false. Conversely, the conditions in if statements are also constantly evaluated, and their statements are executed as soon as the condition becomes true regardless of where the flow of control is at the time.
Assignment moves values, i.e. after a = b, the variable b is empty.

7.4 What Mistakes Do Novices Make?

The mistakes novices make can tell us what to prioritize in our teaching, but it turns out that most teachers don’t know how common different kinds of mistakes actually are. The largest study of this is (Brown and Altadmri 2017), which found that mismatched quotes and parentheses are the most common type of errors in novice Java programs, but also the easiest to fix, while some mistakes (like putting the condition of an if in {} instead of ()) are most often made only once. Unsurprisingly, mistakes that produce compiler errors are fixed much faster than ones that don’t. Some mistakes, however, are made many times, like invoking methods with the wrong arguments (e.g. passing a string instead of an integer).

Not Right vs. Not Done

One difficulty in research like this is distinguishing mistakes from work in progress. For example, an empty if statement or a method that is defined but not yet used may be a sign of incomplete code rather than an error.

(Brown and Altadmri 2017) also compared the mistakes novices actually make with what their teachers thought they made. They found that, “…educators formed only a weak consensus about which mistakes are most frequent, that their rankings bore only a moderate correspondence to the students in the…data, and that educators’ experience had no effect on this level of agreement.” For example, mistaking = (assignment) and == (equality) wasn’t nearly as common as most teachers believed.

Not Just for Code

(Park et al. 2015) collected data from an online HTML editor during an introductory web development course and found that nearly all learners made syntax errors that remained unresolved weeks into the course. 20% of these errors related to the relatively complex rules that dictate when it is valid for HTML elements to be nested in one another, but 35% related to the simpler tag syntax determining how HTML elements are nested. The tendency of many teachers to say, “But the rules are simple,” is a good example of expert blind spot discussed in Chapter 3…

7.5 How Do Novices Program?

Soloway (1986) pioneered the exploration of novice and expert programming strategies. The key finding is that experts know both “what” and “how,” i.e. they understand what to put into programs and they have a set of program patterns or plans to guide their construction. Novices lack both, but most teachers focus solely on the former, even though bugs are often caused by not having a strategy for solving the problem rather than to lack of knowledge about the language. Recent work has shown the effectiveness of teaching four distinct skills in a specific order (Xie et al. 2019):

	semantics of code	templates related to goals
reading}	1. read code and predict behavior	3. recognize templates and their uses
writing}	2. write correct syntax	4. use templates to meet goals

Our next recommendations are therefore to have learners read code, then modify it, then write it, and to introduce common patterns explicitly and have learners practice using them. (Muller et al. 2007) is just one of many studies proving the benefits of teaching common patterns explicitly, and decomposing problems into patterns creates natural opportunities for creating and labeling subgoals Margulieux et al. (2016).

7.6 How Do Novices Debug?

A decade ago, (McCauley et al. 2008) wrote, “It is surprising how little page space is devoted to bugs and debugging in most introductory programming textbooks.” Little has changed since: there are hundreds of books on compilers and operating systems, but only a handful about debugging, and I have never seen an undergraduate course devoted to the subject.

Lister et al. (2009) found that many novices struggled to predict the output of short pieces of code and to select the correct completion of the code from a set of possibilities when told what it was supposed to do. More recently, (Harrington and Cheng 2018) found that the gap between being able to trace code and being able to write it has largely closed by CS2, but that novices who still have a gap (in either direction) are likely to do poorly.

Our fifth recommendation is therefore to explicitly teach novices how to debug. Murphy et al. (2008) found that good debuggers were good programmers, but not all good programmers were good at debugging. Those who were used a symbolic debugger to step through their programs, traced execution by hand, wrote tests, and re-read the spec frequently, which are all teachable habits. However, tracing execution step by step was sometimes used ineffectively: for example, a novice might put the same print statement in both parts of an if-else. Novices would also comment out lines that were actually correct as they tried to isolate a problem; teachers can make both of these mistakes deliberately, point them out, and correct them to help novices get past them.

Teaching novices how to debug can also help make classes easier to manage. (Alqadi and Maletic 2017) found that learners with more experience solved debugging problems significantly faster, but times varied widely: 4–10 minutes was a typical range for individual exercises, which means that some learners need 2–3 times longer than others to get through the same exercises. Teaching the slower learners what the faster ones are doing will help make the group’s overall progress more uniform.

Debugging depends on being able to read code, which multiple studies have shown is the single most effective way to find bugs Bacchelli and Bird (2013). The code quality rubric developed in Stegeman et al. (2016) is a good checklist of things to look for, though it is best presented in chunks rather than all at once.

Having learners read code and summarize its behavior is a good exercise (Section 5.1), but often takes too long to be practical in class. Having them predict a program’s output just before it is run, on the other hand, helps reinforce learning (Section 9.11) and also gives them a natural moment to ask “what if” questions. Teachers or learners can also trace changes to variables as they go along, which (Cunningham et al. 2017) found was effective (Section 12.2).

7.7 What About Testing?

Novice programmers seem just as reluctant to test software as professionals. There’s no doubt that doing it is valuable—(Carter and Hundhausen 2017) found that high-performing novices spent a lot of time testing, while low performers spent much more time working on code with errors—and many teachers require learners to write tests for assignments. But how well do they do this? One answer comes from (Brian et al. 2015), which scored learners’ programs by how many teacher-provided test cases those programs passed, and conversely scores test cases written by learners according to how many deliberately-seeded bugs they caught. They found that novices’ tests often have low coverage (i.e. they don’t test most of the code) and that they often test many things at once, which makes it hard to pinpoint the causes of errors.

Another answer comes from (Edwards and Shams 2014), which looked at all of the bugs in all novices’ code submissions combined and identified those detected by the novices’ test suite. They found that novices’ tests only detected an average of 13.6% of the faults present in the entire program population. What’s more, 90% of the novices’ tests were very similar, which indicates that novices mostly write tests to confirm that code is doing what it’s supposed to rather than to find cases where it isn’t.

One approach to teaching better testing practices is to define a programming problem by providing a set of tests to be passed rather than through a written description (Section 12.1). Before doing this, though, take a moment to look at how many tests you’ve written for your own code recently, and then decide whether you’re teaching what you believe people should do, or what they (and you) actually do.

7.8 Do Languages Matter?

The short answer is “yes”: novices learn to program faster and learn more using blocks-based tools like Scratch (Figure 7.2) (Weintrop and Wilensky 2017). One reason is that blocks-based systems reduce cognitive load by eliminating the possibility of syntax errors. Another is that block interfaces encourage exploration in a way that text does not: like all good tools, Scratch can be learned accidentally (Maloney et al. 2010).

But what happens after blocks? (Chen et al. 2018) found that learners whose first programming language was graphical had higher grades in introductory programming courses than learners whose first language was textual when the languages were introduced in or before early adolescent years. Our sixth recommendation is therefore to start children and teens with blocks-based interfaces before moving to text-based systems. The age qualification is there because Scratch deliberately looks like it’s meant for younger users, and it can still be hard to convince adults to take it seriously.

Scratch has probably been studied more than any other programming tool. One example is (Aivaloglou and Hermans 2016), which analyzed over 250,000 Scratch projects and found (among other things) that about 28% of projects have some blocks that are never called or triggered. As in the earlier aside about incomplete versus incorrect Java programs, the authors hypothesize that users may be using these blocks as a scratchpad to keep track of bits of code they don’t (yet) want to throw away. Another example is Mladenović et al. (2017), which studied novices learning about loops in Scratch, Logo, and Python. They found that misconceptions about loops are minimized when using a block-based language rather than a text-based language. What’s more, as tasks become more complex (such as using nested loops) the differences become larger.

Harder Than Necessary

The creators of programming languages make those languages harder to learn by not doing basic usability testing. For example, (Stefik and Siebert 2013) found that, “…the three most common words for looping in computer science, for, while, and foreach, were rated as the three most unintuitive choices by non-programmers.” Their work shows that C-style syntax (as used in Java and Perl) is just as hard for novices to learn as a randomly designed syntax, but that the syntax of languages such as Python and Ruby is significantly easier to learn, and the syntax of a language whose features are tested before being added to the language is easier still. (Stefik et al. 2017) is a useful brief summary of what we actually know about designing programming languages and why we believe it’s true, while (Guzdial 2016) lays out five principles that programming languages for learners should follow.

Object-Oriented Programming

Objects and classes are power tools for experienced programmers, and many educators advocate an objects first approach to teaching programming (though they sometimes disagree on exactly what that means (Bennedsen and Schulte 2007)). (Sorva and Seppälä 2014) describes and motivates this approach, and (Kölling 2015) describes three generations of tools designed to support novice programming in object-oriented environments.

Introducing objects early has a few challenges. (Miller and Settle 2016) found that most novices using Python struggled to understand self (which refers to the current object): they omitted it in method definitions, failed to use it when referencing object attributes, or both. (Ragonis and Shmallo 2017) found something similar in high school students, and also found that high school teachers often weren’t clear on the concept either. On balance, we recommend that teachers start with functions rather than objects, i.e. that learners not be taught how to define classes until they understand basic control structures and data types.

Type Declarations

Programmers have argued for decades about whether variables’ data types should have to be declared or not, usually based on their personal experience as professionals rather than on any kind of data. Fischer and Hanenberg (2015) found that requiring novices to declare variable types does add some complexity to programs, but it pays off fairly quickly by acting as documentation for a method’s use—in particular, by forestalling questions about what’s available and how to use it.

Variable Naming

(Kernighan and Pike 1999) wrote, “Programmers are often encouraged to use long variable names regardless of context. This is a mistake: clarity is often achieved through brevity.” Lots of programmers believe this, but (Hofmeister et al. 2017) found that using full words in variable names led to an average of 19% faster comprehension compared to letters and abbreviations. In contrast, (Beniamini et al. 2017) found that using single-letter variable names didn’t affect novices’ ability to modify code. This may be because their programs are shorter than professionals’ or because some single-letter variable names have implicit types and meanings. For example, most programmers assume that i, j, and n are integers and that s is a string, while x, y, and z are either floating-point numbers or integers more or less equally.

How important is this? (Binkley et al. 2012) reported that reading and understanding code is fundamentally different from reading prose: “…the more formal structure and syntax of source code allows programmers to assimilate and comprehend parts of the code quite rapidly independent of style. In particular…beacons and program plans play a large role in comprehension.” It also found that experienced developers are relatively unaffected by identifier style, so our recommendation is just to use consistent style in all examples. Since most languages have style guides (e.g. PEP 8 for Python) and tools to check that code follows these guidelines, our full recommendation is to use tools to ensure that all code examples adhere to a consistent style.

7.9 Do Better Error Messages Help?

Incomprehensible error messages are a major source of frustration for novices (and for experienced programmers as well). Several researchers have therefore explored whether better error messages would help alleviate this. For example, (Becker et al. 2016) rewrote some of the Java compiler’s messages so that instead of:

C:\stj\Hello.java:2: error: cannot find symbol
        public static void main(string[ ] args)
^
1 error
Process terminated ... there were problems.

learners would see:

Looks like a problem on line number 2.
If "string" refers to a datatype, capitalize the 's'!

Sure enough, novices given these messages made fewer repeated errors and fewer errors overall.

(Barik et al. 2017) went further and used eye tracking to show that despite the grumblings of compiler writers, people really do read error messages—in fact, they spend 13–25% of their time doing this. However, reading error messages turns out to be as difficult as reading source code, and how difficult it is to read the error messages strongly predicts task performance. Teachers should therefore show learners how to read and interpret error messages. (Marceau et al. 2011) has a rubric for responses to error messages that can be useful in grading such exercises.

Does Visualization Help?

Visualizing program execution is a perennially popular idea, and tools like the Online Python Tutor (Guo 2013) and Loupe (which shows how JavaScript’s event loop works) are useful teaching aids. However, people learn more from constructing visualizations than they do from viewing visualizations constructed by others Cetin and Andrews-Larson (2016), so does visualization actually help learning?

To answer this, (Cunningham et al. 2017) replicated an earlier study of the kinds of sketching learners do when tracing code execution. They found that not sketching at all correlates with lower success, while tracing changes to variables’ values by writing new values near their names as they change was the most effective strategy.

One possible confounding effect they checked was time: since sketchers take significantly more time to solve problems, do they do better just because they think for longer? The answer is no: there was no correlation between the time taken and the score achieved. Our recommendation is therefore to teach learners to trace variables’ values when debugging.

Flowcharts

One often-overlooked finding about visualization is that people understand flowcharts better than pseudocode if both are equally well structured (Scanlan 1989). Earlier work showing that pseudocode outperformed flowcharts used structured pseudocode and tangled flowcharts; when the playing field was leveled, novices did better with the graphical representation.

7.10 What Else Can We Do to Help?

(Vihavainen et al. 2014) examined the average improvement in pass rates of various kinds of intervention in programming classes. They point out that there are many reasons to take their findings with a grain of salt: the pre-change teaching practices are rarely stated clearly, the quality of change is not judged, and only 8.3% of studies reported negative findings, so either there is positive reporting bias or the way we’re teaching right now is the worst possible and anything would be an improvement. And like many other studies discussed in this chapter, they were only looking at university classes, so their findings may not generalize to other groups.

With those caveats in mind, they found ten things teachers can do to improve outcomes (Figure 7.3):

Collaboration: Activities that encourage learner collaboration either in classrooms or labs.
Content Change: Parts of the teaching material were changed or updated.
Contextualization: Course content and activities were aligned towards a specific context such as games or media.
CS0: Creation of a preliminary course to be taken before the introductory programming course; could be organized only for some (e.g. at-risk) learners.
Game Theme: A game-themed component was introduced to the course.
Grading Scheme: A change in the grading scheme, such as increasing the weight of programming activities while reducing that of the exam.
Group Work: Activities with increased group work commitment such as team-based learning and cooperative learning.
Media Computation: Activities explicitly declaring the use of media computation (Chapter 10).
Peer Support: Support by peers in form of pairs, groups, hired peer mentors or tutors.
Other Support: An umbrella term for all support activities, e.g. increased teacher hours, additional support channels, etc.

Figure 7.3: Effectiveness of interventions

This list highlights the importance of cooperative learning. (Beck and Chizhik 2013) looked at this specifically over three academic years in courses taught by two different teachers and found significant benefits overall and for many subgroups. The cooperators had higher grades and left fewer questions blank on the final exam, which indicates greater self-efficacy and willingness to try to debug things.

Computing Without Coding

Writing code isn’t the only way to teach people how to program: having novices work on computational creativity exercises improves grades at several levels (Shell et al. 2017). A typical exercise is to describe an everyday object (such as a paper clip or toothbrush) in terms of its inputs, outputs, and functions. This kind of teaching is sometimes called unplugged; the CS Unplugged site has lessons and exercises for doing this.

7.11 Where Next?

For those who want to go deeper, (Fincher and Robins 2019) is a comprehensive summary of CER, (Ihantola et al. 2016) summarizes the methods that studies use most often. I hope that some day we will have catalogs like (Ojose 2015) and more teacher training materials like Sentance et al. (2018) to help us all do better.

Most of the research reported in this chapter was publicly funded but is locked away behind paywalls: at a guess, I broke the law 250 times to download papers from sites like Sci-Hub while writing this book. I hope the day is coming when no one will need to do that; if you are a researcher, please hasten that day by publishing your work in open access venues.

7.12 Exercises

7.12.1 Your Learners’ Misunderstandings (small groups/15 min)

Working in small groups, re-read Section 7.3 and make a list of misconceptions you think your learners have. How specific are they? How would you check how accurate your list is?

7.12.2 Checking for Common Errors (individual/20 min)

These common errors are taken from a longer list in (Sirkiä and Sorva 2012):

Inverted assignment: The learner assigns the value of the left-hand variable to the right-hand variable rather than the other way around.
Wrong branch: The learner thinks the code in the body of an if is run even if the condition is false.
Executing function instead of defining it: The learner believes that a function is executed as it is defined.

Write one exercise for each to check that learners aren’t making that mistake.

7.12.3 Mangled Code (pairs/15 min)

(Cheng and Harrington 2017) describes exercises in which learners reconstruct code that has been mangled by removing comments, deleting or replacing lines of code, moving lines, and so on. Performance on these correlates strongly with performance on assessments in which learners write code, but these questions require less work to mark. Take the solution to a programming exercise you’ve created in the past, mangle it in two different ways, swap with a partner, and see how long it takes each of you to answer the other’s question correctly.

7.12.4 The Rainfall Problem (pairs/10 min)

(Soloway 1986) introduced the Rainfall Problem, which has been used in many subsequent studies of programming Seppälä et al. (2015). Write a program that repeatedly reads in positive integers until it reads the integer 99999. After seeing 99999, the program prints the average of the numbers seen.

Solve the Rainfall Problem in the programming language of your choice.
Compare your solution with that of your partner. What does yours do that theirs doesn’t and vice versa?

7.12.5 Roles of Variables (pairs/15 min)

Sajaniemi et al. (2006) presented a set of single-variable patterns that I have found very useful in teaching beginners:

Fixed value: A data item that does not get a new proper value after its initialization.
Stepper: A data item stepping through a systematic, predictable succession of values.
Walker: A data item traversing in a data structure.
Most-recent holder: A data item holding the latest value encountered while going through a succession of values.
Most-wanted holder: A data item holding the best or most appropriate value encountered so far.
Gatherer: A data item accumulating the effect of individual values.
Follower: A data item that always gets its new value from the old value of some other data item.
One-way flag: A two-valued data item that cannot get its initial value once the value has been changed.
Temporary: A data item holding some value for a very short time only.
Organizer: A data structure storing elements that can be rearranged.
Container: A data structure storing elements that can be added and removed.

Choose a 5–15 line program and classify its variables using these categories. Compare your classifications with those of a partner. Where you disagreed, did you understand each other’s view?

7.12.6 What Are You Teaching? (individual/10 min)

Compare the topics you teach to the list developed in (Luxton-Reilly et al. 2017) (Section 7.1). Which topics do you cover? Which don’t you cover? What extra topics do you cover that aren’t in their list?

7.12.7 Beneficial Activities (individual/10 min)

Look at the list of interventions developed by (Vihavainen et al. 2014) (Section 7.10). Which of these things do you already do in your classes? Which ones could you easily add? Which ones are irrelevant?

7.12.8 Misconceptions and Challenges (small groups/15 min)

The Professional Development for CS Principles Teaching site includes a detailed list of learners’ misconceptions and exercises. Working in small groups, choose one section (such as data structures or functions) and go through their list. Which of these misconceptions do you remember having when you were a learner? Which do you still have? Which have you seen in your learners?

7.12.9 What Do You Care Most About? (whole class/15 min)

(Denny et al. 2019) asked people to propose and rate various CER questions, and found that there was no overlap between those that researchers cared most about and those that non-researchers cared most about. The researchers’ favorites were:

What fundamental programming concepts are the most challenging for students?
What teaching strategies are most effective when dealing with a wide range of prior experience in introductory programming classes?
What affects students’ ability to generalize from simple programming examples?
What teaching practices are most effective for teaching computing to children?
What kinds of problems do students in programming classes find most engaging?
What are the most effective ways to teach programming to various groups?
What are the most effective ways to scale computing education to reach the general student population?

while the most important questions for non-researchers were:

How and when is it best to give students feedback on their code to improve learning?
What kinds of programming exercises are most effective when teaching students Computer Science?
What are the relative merits of project-based learning, lecturing, and active learning for students learning computing?
What is the most effective way to provide feedback to students in programming classes?
What do people find most difficult when breaking problems down into smaller tasks while programming?
What are the key concepts that students need to understand in introductory computing classes?
What are the most effective ways to develop computing competency among students in non-computing disciplines?
What is the best order in which to teach basic computing concepts and skills?

Have each person in the class independently give one point to each of the eight questions from the combined lists that they care most about, then calculate an average score for each question. Which ones are most popular in your class? In which group are the most popular questions?

Aivaloglou, Efthimia, and Felienne Hermans. 2016. “How Kids Code and How We Know.” 2016 International Computing Education Research Conference (ICER’16). https://doi.org/10.1145/2960310.2960325. Presents an analysis of 250,000 Scratch projects.

Alqadi, Basma S., and Jonathan I. Maletic. 2017. “An Empirical Study of Debugging Patterns Among Novice Programmers.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017761. Reports patterns in the debugging activities and success rates of novice programmers.

Bacchelli, Alberto, and Christian Bird. 2013. “Expectations, Outcomes, and Challenges of Modern Code Review.” 2013 International Conference on Software Engineering (ICSE’13), May. A summary of work on code review.

Barik, Titus, Justin Smith, Kevin Lubick, et al. 2017. “Do Developers Read Compiler Error Messages?” 2017 International Conference on Software Engineering (ICSE’17), May. https://doi.org/10.1109/icse.2017.59. Reports that developers do read error messages and doing so is as hard as reading source code: it takes 13-25% of total task time.

Basili, Victor R., and Richard W. Selby. 1987. “Comparing the Effectiveness of Software Testing Strategies.” IEEE Transactions on Software Engineering SE-13 (12): 1278–96. https://doi.org/10.1109/tse.1987.232881. An early and influential summary of the effectiveness of code review.

Beck, Leland, and Alexander Chizhik. 2013. “Cooperative Learning Instructional Methods for CS1: Design, Implementation, and Evaluation.” ACM Transactions on Computing Education 13 (3): 10:1–21. https://doi.org/10.1145/2492686. Reports that cooperative learning enhances learning outcomes and self-efficacy in CS1.

Becker, Brett A., Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. “Effective Compiler Error Message Enhancement for Novice Programming Students.” Computer Science Education 26 (2-3): 148–75. https://doi.org/10.1080/08993408.2016.1225464. Reports that improved error messages helped novices learn faster.

Beniamini, Gal, Sarah Gingichashvili, Alon Klein Orbach, and Dror G. Feitelson. 2017. “Meaningful Identifier Names: The Case of Single-Letter Variables.” 2017 International Conference on Program Comprehension (ICPC’17), May. https://doi.org/10.1109/icpc.2017.18. Reports that use of single-letter variable names doesn’t affect ability to modify code, and that some single-letter variable names have implicit types and meanings.

Bennedsen, Jens, and Michael E. Caspersen. 2007. “Failure Rates in Introductory Programming.” ACM SIGCSE Bulletin 39 (2): 32. https://doi.org/10.1145/1272848.1272879. Reports that 67% of students pass CS1, with variation from 5% to 100%.

Bennedsen, Jens, and Carsten Schulte. 2007. “What Does ‘Objects-First’ Mean?: An International Study of Teachers’ Perceptions of Objects-First.” 2007 Koli Calling Conference on Computing Education Research (Koli’07), 21–29. Teases out three meanings of “objects first” in computing education.

Binkley, Dave, Marcia Davis, Dawn Lawrie, Jonathan I. Maletic, Christopher Morrell, and Bonita Sharif. 2012. “The Impact of Identifier Style on Effort and Comprehension.” Empirical Software Engineering 18 (2): 219–76. https://doi.org/10.1007/s10664-012-9201-4. Reports that reading and understanding code is fundamentally different from reading prose, and that experienced developers are relatively unaffected by identifier style, but beginners benefit from the use of camel case (versus pothole case).

Brian, Samuel A., Richard N. Thomas, James M. Hogan, and Colin Fidge. 2015. “Planting Bugs: A System for Testing Students’ Unit Tests.” 2015 Conference on Innovation and Technology in Computer Science Education (ITiCSE’15). https://doi.org/10.1145/2729094.2742631. Describes a tool for assessing students’ programs and unit tests and finds that students often write weak tests and misunderstand the role of unit testing.

Brown, Neil C. C., and Amjad Altadmri. 2017. “Novice Java Programming Mistakes.” ACM Transactions on Computing Education 17 (2). https://doi.org/10.1145/2994154. Summarizes the authors’ analysis of novice programming mistakes.

Byckling, Pauli, Petri Gerdt, and Jorma Sajaniemi. 2005. “Roles of Variables in Object-Oriented Programming.” 2005 Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’05). https://doi.org/10.1145/1094855.1094972. Presents single-variable design patterns common in novice programs.

Carter, Adam Scott, and Christopher David Hundhausen. 2017. “Using Programming Process Data to Detect Differences in Students’ Patterns of Programming.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017785. Shows that students of different levels approach programming tasks differently, and that these differences can be detected automatically.

Cetin, Ibrahim, and Christine Andrews-Larson. 2016. “Learning Sorting Algorithms Through Visualization Construction.” Computer Science Education 26 (1): 27–43. https://doi.org/10.1080/08993408.2016.1160664. Reports that people learn more from constructing algorithm visualizations than they do from viewing visualizations constructed by others.

Chen, Chen, Paulina Haduong, Karen Brennan, Gerhard Sonnert, and Philip Sadler. 2018. “The Effects of First Programming Language on College Students’ Computing Attitude and Achievement: A Comparison of Graphical and Textual Languages.” Computer Science Education 29 (1): 23–48. https://doi.org/10.1080/08993408.2018.1547564. Finds that students whose first language was graphical had higher grades than students whose first language was textual when the languages were introduced in or before early adolescent years.

Cheng, Nick, and Brian Harrington. 2017. “The Code Mangler: Evaluating Coding Ability Without Writing Any Code.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017704. Reports that student performance on exercises in which they undo code mangling correlates strongly with performance on traditional assessments.

Cunningham, Kathryn, Sarah Blanchard, Barbara J. Ericson, and Mark Guzdial. 2017. “Using Tracing and Sketching to Solve Programming Problems.” 2017 Conference on International Computing Education Research (ICER’17). https://doi.org/10.1145/3105726.3106190. Found that writing new values near variables’ names as they change is the most effective tracing technique.

Denny, Paul, Brett A. Becker, Michelle Craig, Greg Wilson, and Piotr Banaszkiewicz. 2019. “Research This! Questions That Computing Educators Most Want Computing Education Researchers to Answer.” 2019 Conference on International Computing Education Research (ICER’19). Found little overlap between the questions that computing education researchers are most interested in and the questions practitioners want answered.

Edwards, Stephen H., and Zalia Shams. 2014. “Do Student Programmers All Tend to Write the Same Software Tests?” 2014 Conference on Innovation and Technology in Computer Science Education (ITiCSE’14). https://doi.org/10.1145/2591708.2591757. Reports that students wrote tests for the happy path rather than to detect hidden bugs.

Endrikat, Stefan, Stefan Hanenberg, Romain Robbes, and Andreas Stefik. 2014. “How Do API Documentation and Static Typing Affect API Usability?” 2014 International Conference on Software Engineering (ICSE’14). https://doi.org/10.1145/2568225.2568299. Shows that types do add complexity to programs, but it pays off fairly quickly by acting as documentation hints for a method’s use.

Fincher, Sally, and Anthony Robins, eds. 2019. The Cambridge Handbook of Computing Education Research. Cambridge University Press. ISBN 978-1108721899. A 900-page summary of what we know about computing education.

Fischer, Lars, and Stefan Hanenberg. 2015. “An Empirical Investigation of the Effects of Type Systems and Code Completion on API Usability Using TypeScript and JavaScript in MS Visual Studio.” 11th Symposium on Dynamic Languages (DLS’15). https://doi.org/10.1145/2816707.2816720. Found that static typing improved programmer efficiency independently of code completion.

Fisler, Kathi. 2014. “The Recurring Rainfall Problem.” 2014 International Computing Education Research Conference (ICER’14). https://doi.org/10.1145/2632320.2632346. Reports that students made fewer low-level errors when solving the Rainfall Problem in a functional language.

Fitzgerald, Sue, Gary Lewandowski, Renée McCauley, et al. 2008. “Debugging: Finding, Fixing and Flailing, a Multi-Institutional Study of Novice Debuggers.” Computer Science Education 18 (2): 93–116. https://doi.org/10.1080/08993400802114508. Reports that good undergraduate debuggers are good programmers but not necessarily vice versa, and that novices use tracing and testing rather than causal reasoning.

Grover, Shuchi, and Satabdi Basu. 2017. “Measuring Student Learning in Introductory Block-Based Programming.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017723. Reports that middle-school children using blocks-based programming find loops, variables, and Boolean operators difficult to understand.

Guo, Philip J. 2013. “Online Python Tutor.” 2013 Technical Symposium on Computer Science Education (SIGCSE’13). https://doi.org/10.1145/2445196.2445368. Describes the design and use of a web-based execution visualization tool.

Guzdial, Mark. 2015. Learner-Centered Design of Computing Education: Research on Computing for Everyone. Morgan & Claypool Publishers. ISBN 9781627053518. Argues that we must design computing education for everyone, not just people who think they are going to become professional programmers.

Guzdial, Mark. 2016. Five Principles for Programming Languages for Learners. Https://cacm.acm.org/blogs/blog-cacm/203554-five-principles-for-programming-languages-for-learners/fulltext. Explains how to choose a programming language for people who are new to programming.

Harrington, Brian, and Nick Cheng. 2018. “Tracing Vs. Writing Code: Beyond the Learning Hierarchy.” 2018 Technical Symposium on Computer Science Education (SIGCSE’18). https://doi.org/10.1145/3159450.3159530. Finds that the gap between being able to trace code and being able to write it has largely closed by CS2, and that students who still have a gap (in either direction) are likely to do poorly in the course.

Hazzan, Orit, Tami Lapidot, and Noa Ragonis. 2014. Guide to Teaching Computer Science: An Activity-Based Approach. Second. Springer. ISBN 9781447166290. A textbook for teaching computer science at the K-12 level with dozens of activities.

Henrich, Joseph, Steven J. Heine, and Ara Norenzayan. 2010. “The Weirdest People in the World?” Behavioral and Brain Sciences 33 (2-3): 61–83. https://doi.org/10.1017/s0140525x0999152x. Points out that the subjects of most published psychological studies are Western, educated, industrialized, rich, and democratic.

Hofmeister, Johannes, Janet Siegmund, and Daniel V. Holt. 2017. “Shorter Identifier Names Take Longer to Comprehend.” 2017 Conference on Software Analysis, Evolution and Reengineering (SANER’17), February. https://doi.org/10.1109/saner.2017.7884623. Reports that using words for variable names makes comprehension faster than using abbreviations or single-letter names for variables.

Ihantola, Petri, Kelly Rivers, Miguel Ángel Rubio, et al. 2016. “Educational Data Mining and Learning Analytics in Programming: Literature Review and Case Studies.” 2016 Conference on Innovation and Technology in Computer Science Education (ITiCSE’16). https://doi.org/10.1145/2858796.2858798. A survey of methods used in mining and analyzing programming data.

Kemerer, Chris F., and Mark C. Paulk. 2009. “The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data.” IEEE Transactions on Software Engineering 35 (4): 534–50. https://doi.org/10.1109/tse.2009.27. Uses individual data to explore the effectiveness of code review.

Kernighan, Brian W., and Rob Pike. 1999. The Practice of Programming. Addison-Wesley. ISBN 9788177582482. A programming style manual written by two of the creators of modern computing.

Koehler, Matthew J., Punya Mishra, and William Cain. 2013. “What Is Technological Pedagogical Content Knowledge (TPACK)?” Journal of Education 193 (3): 13–19. https://doi.org/10.1177/002205741319300303. Refines the discussion of PCK by adding technology, and sketches strategies for building understanding of how to use it.

Kohn, Tobias. 2017. “Variable Evaluation: An Exploration of Novice Programmers’ Understanding and Common Misconceptions.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017724. Reports that students often believe in delayed evaluation or that entire equations are stored in variables.

Kölling, Michael. 2015. “Lessons from the Design of Three Educational Programming Environments.” International Journal of People-Oriented Programming 4 (1): 5–32. https://doi.org/10.4018/ijpop.2015010102. Compares three generations of programming environments intended for novice use.

Kuittinen, Marja, and Jorma Sajaniemi. 2004. “Teaching Roles of Variables in Elementary Programming Courses.” ACM SIGCSE Bulletin 36 (3): 57. https://doi.org/10.1145/1026487.1008014. Presents a few patterns used in novice programming and the pedagogical value of teaching them.

Lister, Raymond, Colin Fidge, and Donna Teague. 2009. “Further Evidence of a Relationship Between Explaining, Tracing and Writing Skills in Introductory Programming.” ACM SIGCSE Bulletin 41 (3): 161. https://doi.org/10.1145/1595496.1562930. Replicates earlier studies showing that students who cannot trace code usually cannot explain code and that students who tend to perform reasonably well at code writing tasks have also usually acquired the ability to both trace code and explain code.

Lister, Raymond, Otto Seppälä, Beth Simon, et al. 2004. “A Multi-National Study of Reading and Tracing Skills in Novice Programmers.” 2004 Conference on Innovation and Technology in Computer Science Education (ITiCSE’04). https://doi.org/10.1145/1044550.1041673. Reports that students are weak at both predicting the outcome of executing a short piece of code and at selecting the correct completion for short pieces of code.

Luxton-Reilly, Andrew, Jacqueline Whalley, Brett A. Becker, et al. 2017. “Developing Assessments to Determine Mastery of Programming Fundamentals.” 2017 Conference on Innovation and Technology in Computer Science Education (ITiCSE’17). https://doi.org/10.1145/3174781.3174784. Synthesizes work from many previous works to determine what CS instructors are actually teaching, how those things depend on each other, and how they might be assessed.

Maloney, John, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. “The Scratch Programming Language and Environment.” ACM Transactions on Computing Education 10 (4): 1–15. https://doi.org/10.1145/1868358.1868363. Summarizes the design of the first generation of Scratch.

Marceau, Guillaume, Kathi Fisler, and Shriram Krishnamurthi. 2011. “Measuring the Effectiveness of Error Messages Designed for Novice Programmers.” 2011 Technical Symposium on Computer Science Education (SIGCSE’11). https://doi.org/10.1145/1953163.1953308. Looks at edit-level responses to error messages, and introduces a useful rubric for classifying user responses to errors.

Margulieux, Lauren E., Richard Catrambone, and Mark Guzdial. 2016. “Employing Subgoals in Computer Programming Education.” Computer Science Education 26 (1): 44–67. https://doi.org/10.1080/08993408.2016.1144429. Reports that labelled subgoals improve learning outcomes in introductory computing courses.

Margulieux, Lauren E., Mark Guzdial, and Richard Catrambone. 2012. “Subgoal-Labeled Instructional Material Improves Performance and Transfer in Learning to Develop Mobile Applications.” 2012 International Computing Education Research Conference (ICER’12), 71–78. https://doi.org/10.1145/2361276.2361291. Reports that labelled subgoals improve outcomes and transference when learning about mobile app development.

Mayer, Richard E. 2004. “Teaching of Subject Matter.” Annual Review of Psychology 55 (1): 715–44. https://doi.org/10.1146/annurev.psych.55.082602.133124. An overview of how and why teaching and learning are subject-specific.

McCauley, Renée, Sue Fitzgerald, Gary Lewandowski, et al. 2008. “Debugging: A Review of the Literature from an Educational Perspective.” Computer Science Education 18 (2): 67–92. https://doi.org/10.1080/08993400802114581. Summarizes research about why bugs occur, why types there are, how people debug, and whether we can teach debugging skills.

McCracken, Michael, Tadeusz Wilusz, Vicki Almstrum, et al. 2001. “A Multi-National, Multi-Institutional Study of Assessment of Programming Skills of First-Year CS Students.” 2001 Conference on Innovation and Technology in Computer Science Education (ITiCSE’01). https://doi.org/10.1145/572133.572137. Reports that most students still struggle to solve even basic programming problems at the end of their introductory course.

Miller, Craig S., and Amber Settle. 2016. “Some Trouble with Transparency: An Analysis of Student Errors with Object-Oriented Python.” 2016 International Computing Education Research Conference (ICER’16). https://doi.org/10.1145/2960310.2960327. Reports that students have difficulty with self in Python.

Mladenović, Monika, Ivica Boljat, and Žana Žanko. 2017. “Comparing Loops Misconceptions in Block-Based and Text-Based Programming Languages at the K-12 Level.” Education and Information Technologies, ahead of print, November. https://doi.org/10.1007/s10639-017-9673-3. Reports that K-12 students have fewer misconceptions about loops using Scratch than using Logo or Python, and fewer misconceptions about nested loops with Logo than with Python.

Muller, Orna, David Ginat, and Bruria Haberman. 2007. “Pattern-Oriented Instruction and Its Influence on Problem Decomposition and Solution Construction.” 2007 Technical Symposium on Computer Science Education (SIGCSE’07). https://doi.org/10.1145/1268784.1268830. Reports that explicitly teaching solution patterns improves learning outcomes.

Murphy, Laurie, Gary Lewandowski, Renée McCauley, Beth Simon, Lynda Thomas, and Carol Zander. 2008. “Debugging: The Good, the Bad, and the Quirky - a Qualitative Analysis of Novices’ Strategies.” ACM SIGCSE Bulletin 40 (1): 163. https://doi.org/10.1145/1352322.1352191. Reports that many CS1 students use good debugging strategies, but many others don’t, and students often don’t recognize when they are stuck.

Ojose, Bobby. 2015. Common Misconceptions in Mathematics: Strategies to Correct Them. UPA. ISBN 0761858857. A catalog of K-12 misconceptions in mathematics and what to do about them.

Park, Thomas H., Brian Dorn, and Andrea Forte. 2015. “An Analysis of HTML and CSS Syntax Errors in a Web Development Course.” ACM Transactions on Computing Education 15 (1): 1–21. https://doi.org/10.1145/2700514. Describes the errors students make in an introductory course on HTML and CSS.

Pea, Roy D. 1986. “Language-Independent Conceptual ‘Bugs’ in Novice Programming.” Journal of Educational Computing Research 2 (1): 25–36. https://doi.org/10.2190/689t-1r2a-x4w4-29j2. First named the "superbug" in coding: most newcomers think the computer understands what they want, in the same way that a human being would.

Qian, Yizhou, and James Lehman. 2017. “Students’ Misconceptions and Other Difficulties in Introductory Programming.” ACM Transactions on Computing Education 18 (1): 1–24. https://doi.org/10.1145/3077618. Summarizes research on student misconceptions about computing.

Ragonis, Noa, and Ronit Shmallo. 2017. “On the (Mis)understanding of the This Reference.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017715. Reports that most students do not understood when to use this, and that teachers are also often not clear on the subject.

Rich, Kathryn M., Carla Strickland, T. Andrew Binkowski, Cheryl Moran, and Diana Franklin. 2017. “K-8 Learning Trajectories Derived from Research Literature.” 2017 International Computing Education Research Conference (ICER’17). https://doi.org/10.1145/3105726.3106166. Presents learning trajectories for K-8 computing classes for Sequence, Repetition, and Conditions gleaned from the literature.

Sajaniemi, Jorma, Mordechai Ben-Ari, Pauli Byckling, Petri Gerdt, and Yevgeniya Kulikova. 2006. “Roles of Variables in Three Programming Paradigms.” Computer Science Education 16 (4): 261–79. https://doi.org/10.1080/08993400600874584. A detailed look at the authors’ work on roles of variables.

Scanlan, David A. 1989. “Structured Flowcharts Outperform Pseudocode: An Experimental Comparison.” IEEE Software 6 (5): 28–36. https://doi.org/10.1109/52.35587. Reports that students understand flowcharts better than pseudocode if both are equally well structured.

Sentance, Sue, Erik Barendsen, and Carsten Schulte, eds. 2018. Computer Science Education: Perspectives on Teaching and Learning in School. Bloomsbury Press. ISBN 135005710X. A collection of academic survey articles on teaching computing.

Seppälä, Otto, Petri Ihantola, Essi Isohanni, Juha Sorva, and Arto Vihavainen. 2015. “Do We Know How Difficult the Rainfall Problem Is?” 2015 Koli Calling Conference on Computing Education Research (Koli’15). https://doi.org/10.1145/2828959.2828963. A meta-study of the Rainfall Problem.

Shell, Duane F., Leen-Kiat Soh, Abraham E. Flanigan, Markeya S. Peteranetz, and Elizabeth Ingraham. 2017. “Improving Students’ Learning and Achievement in CS Classrooms Through Computational Creativity Exercises That Integrate Computational and Creative Thinking.” 2017 Technical Symposium on Computer Science Education (SIGCSE’17). https://doi.org/10.1145/3017680.3017718. Reports that having students work in small groups on computational creativity exercises improves learning outcomes.

Simon. 2013. “Soloway’s Rainfall Problem Has Become Harder.” 2013 Conference on Learning and Teaching in Computing and Engineering, March. https://doi.org/10.1109/latice.2013.44. Argues that the Rainfall Problem is harder for novices than it used to be because they’re not used to handling keyboard input, so direct comparison with past results may be unfair.

Sirkiä, Teemu, and Juha Sorva. 2012. “Exploring Programming Misconceptions: An Analysis of Student Mistakes in Visual Program Simulation Exercises.” 2012 Koli Calling Conference on Computing Education Research (Koli’12). https://doi.org/10.1145/2401796.2401799. Analyzes data from student use of an execution visualization tool and classifies common mistakes.

Soloway, Elliot. 1986. “Learning to Program = Learning to Construct Mechanisms and Explanations.” Communications of the ACM 29 (9): 850–58. https://doi.org/10.1145/6592.6594. Analyzes programming in terms of choosing appropriate goals and constructing plans to achieve them, and introduces the Rainfall Problem.

Soloway, Elliot, and Kate Ehrlich. 1984. “Empirical Studies of Programming Knowledge.” IEEE Transactions on Software Engineering SE-10 (5): 595–609. https://doi.org/10.1109/tse.1984.5010283. Proposes that experts have programming plans and rules of programming discourse.

Sorva, Juha. 2018. “Misconceptions and the Beginner Programmer.” In Computer Science Education: Perspectives on Teaching and Learning in School, edited by Sue Sentance, Erik Barendsen, and Carsten Schulte. Bloomsbury Press. ISBN 135005710X. Summarizes what we know about what novices misunderstand about computing.

Sorva, Juha, and Otto Seppälä. 2014. “Research-Based Design of the First Weeks of CS1.” 2014 Koli Calling Conference on Computing Education Research (Koli’14). https://doi.org/10.1145/2674683.2674690. Proposes three cognitively plausible frameworks for the design of a first CS course.

Stasko, John, John Domingue, Mark H. Brown, and Blaine A. Price, eds. 1998. Software Visualization: Programming as a Multimedia Experience. MIT Press. ISBN 0262193957. A survey of program and algorithm visualization techniques and results.

Stefik, Andreas, Patrick Daleiden, Diana Franklin, et al. 2017. Programming Languages and Learning. Https://quorumlanguage.com/evidence.html. Summarizes what we actually know about designing programming languages and why we believe it’s true.

Stefik, Andreas, and Susanna Siebert. 2013. “An Empirical Investigation into Programming Language Syntax.” ACM Transactions on Computing Education 13 (4): 1–40. https://doi.org/10.1145/2534973. Reports that curly-brace languages are as hard to learn as a language with randomly-designed syntax, but others are easier.

Stegeman, Martijn, Erik Barendsen, and Sjaak Smetsers. 2014. “Towards an Empirically Validated Model for Assessment of Code Quality.” 2014 Koli Calling Conference on Computing Education Research (Koli’14). https://doi.org/10.1145/2674683.2674702. Presents a code quality rubric for novice programming courses.

Stegeman, Martijn, Erik Barendsen, and Sjaak Smetsers. 2016. “Designing a Rubric for Feedback on Code Quality in Programming Courses.” 2016 Koli Calling Conference on Computing Education Research (Koli’16). https://doi.org/10.1145/2999541.2999555. Describes several iterations of a code quality rubric for novice programming courses.

Utting, Ian, Juha Sorva, Tadeusz Wilusz, et al. 2013. “A Fresh Look at Novice Programmers’ Performance and Their Teachers’ Expectations.” 2013 Conference on Innovation and Technology in Computer Science Education (ITiCSE’13). https://doi.org/10.1145/2543882.2543884. Replicates an earlier study showing how little students learn in their first programming course.

Vihavainen, Arto, Jonne Airaksinen, and Christopher Watson. 2014. “A Systematic Review of Approaches for Teaching Introductory Programming and Their Influence on Success.” 2014 International Computing Education Research Conference (ICER’14). https://doi.org/10.1145/2632320.2632349. Consolidates studies of CS1-level teaching changes and finds media computation the most effective, while introducing a game theme is the least effective.

Watson, Christopher, and Frederick W. B. Li. 2014. “Failure Rates in Introductory Programming Revisited.” 2014 Conference on Innovation and Technology in Computer Science Education (ITiCSE’14). https://doi.org/10.1145/2591708.2591749. A larger version of an earlier study that found an average of one third of students fail CS1.

Weintrop, David, and Uri Wilensky. 2017. “Comparing Block-Based and Text-Based Programming in High School Computer Science Classrooms.” ACM Transactions on Computing Education 18 (1): 1–25. https://doi.org/10.1145/3089799. Reports that students learn faster and better with blocks than with text.

Wilcox, Chris, and Albert Lionelle. 2018. “Quantifying the Benefits of Prior Programming Experience in an Introductory Computer Science Course.” 2018 Technical Symposium on Computer Science Education (SIGCSE’18). https://doi.org/10.1145/3159450.3159480. Reports that students with prior experience outscore students without in CS1, but there is no significant difference in performance by the end of CS2; also finds that female students with prior exposure outperform their male peers in all areas, but are consistently less confident in their abilities.

Xie, Benjamin, Dastyni Loksa, Greg L. Nelson, et al. 2019. “A Theory of Instruction for Introductory Programming Skills.” Computer Science Education 29 (2-3): 205–53. https://doi.org/10.1080/08993408.2019.1565235. Lays out a four-part theory for teaching novices based on reading vs. writing and code vs. templates.