Exaptation and the Future of Software Engineering
Back in the 1980s, we knew that software engineering was going to become mathematically rigorous like electrical and civil engineering. Where they used calculus to figure out an antenna's gain and whether a dam would stand up, we would use logic and discrete mathematics to prove that programs were correct.
That obviously hasn't happened, and isn't going to any time soon1. However, I think software engineering actually is going to be mathematized over the next few years. It's just not going to be the discrete math that we expected; instead, it's going to be statistics, and instead of proving programs correct, tomorrow's software engineers are going to analyze data from real projects to predict those projects' futures.
To explain why, I need to borrow an idea from evolutionary biology. According to Wikipedia, "Exaptation…describes a shift in the function of a trait during evolution. For example, a trait can evolve because it served one particular function, but subsequently it may come to serve another." Bird feathers are a classic example: they probably evolved to help dinosaurs stay warm and for display, and were only then adapted for flight.
Exaptation (also called "pre-adaptation") is occurring right now in thousands of software companies. Every day, programmers are gathering and analyzing data about how their web sites are used. Those statistical methods can also be used to predict when the next release will be ready to ship based on historical data about past releases. And thanks to open source and the Internet, software engineers now have the data they need to build useful models. Where it tooks pioneers like Barry Boehm months or years to collect data back in the 1970s, today's developers can download and analyze information from thousands of projects.
Mature methods now exist as well. The first conference on mining software repositories was held in 2004; by 2015, the number and rigor of submissions had increased significantly. New books like The Art and Science of Analyzing Sofware Data, Sharing Data and Models in Software Engineering, and the upcoming Perspectives on Data Science for Software Engineering show that this community can answer questions that matter.
As I argued last year, the missing piece is an undergraduate textbook to organize this and make it digestible to 20-year-olds who aren't ready to read primary research literature. Once that is in place, software engineering will finally become the mathematically rigorous discipline it has always aspired to be, just not in the way we imagined.
Despite decades of work, formal methods are still too hard for everyday use. Programmers can't learn enough in their first course to solve real problems, and once they do master techniques powerful enough to be useful, those techniques are too labor-intensive to be economical for all but the most exacting needs.