Why is software creation hard?

(Apparently, I’m not the sort of guy to address the smaller issues in life? ;))

The one thing that has been mystifying ever since (and possibly before) Fred Brooks wrote his famous The Mythical Man-Month book, is the question why creating software seems to be so much harder than it ought to be. After all, nowadays, upfront investments in the form of specialist equipment is negligible (a few hundred $’s per seat), anybody with at least a modicum of education can call him-/herself a programmer and essentially the only resource which is destructively consumed during the process is the caffeine poured into said programmers. And the best thing is of course that software’s so flexible -that’s why it’s called software, remember?

I have the nagging feeling that that’s exactly the problem, especially in interaction with people to have the word ‘manager’ in their job descriptions.

Getting rid of bad analogies

Part/indicative of the problem might be the type of analogies that are used in communication with said managerial types. The car factory-analogy has turned into cliché for at least a decade even though the analogy doesn’t quite hold. The premis of a car factory (or any mass production factory, typically using ye ole’ conveyor belt) is that it’s able to turn out a large number of incredibly homogeneous products efficiently and continuously.

This is in marked contrast to the actual software creation process where we typically turn out one product at a time, or a very small run of variations of essentially the same product (either a product line, or a small set of different versions or the same version across dev/test/acceptance/production). Any two or three projects/products tend to be heterogeneous enough to prohibit manufacturing them in the same software factory and the critical threshold for such factories to become economical is much larger than two or three.

When looking at certain, isolated aspects of projects, the situation is much less bleak: (well-chosen) aspects on themselves often exhibit enough homogeneity to make creating a factory for those aspects separately economical across a very limited number of projects and often enough within one project. It is the premis of Model-Driven Software Development that by capturing the variability of specific aspects using models written down in a DSL and building an execution engine for these models, we achieve a significant production gain. Similarly, using a framework might already go a long way in achieving a speedup.

In fact, the very, very few software factories that I know that actually work (in the sense that they’re being using to deliver working software to paying customers), do so by catering for a well-chosen category of aspects of a well-chosen category of software products. And even then, not all aspects of the project are realized by the software factory alone but are added manually.

So, let’s do away with the car factory-analogy and look for something more fitting. Personally, I like the analogy with building a large, fixed structure (a building, a bridge, etc.) for the following characteristics it shares with software creation: it takes a lot of specialist and specialistic knowledge and skills to get it done; there’s a lot of integration with existing infrastructure (characteristics of building site, surrounding structures, utility infrastructure like water, electricity, sewage, etc.); architectural/design decisions have very definite consequences on the overall process; once it’s done you, it’s very hard to change anything about it and the original builders move on to something else. Note that I’m not proposing that software creation should be like building a large structure, just that the shared characteristics might make it a more useful analogy than building cars.

A nice ‘feature’ of this analogy is the abundantly-clear importance of design: the structure is designed up-front after which the building is ‘simply’ a matter of executing that design according to the construction specifications (Dutch: “bestek”). As Jack C. Reeves reasons in his 1992 article, writing code is design and the analogy of the actual construction would ‘just’ be compilation. This also indicates the failure of the analogy: compilation is cheap, while construction isn’t.

Soft is the new hard

Back to the problem at hand: software seems pliable and malleable on the outside, but takes on properties of common concrete over time. This phenomenon is to some extent unavoidable: no matter how well-engineered and -factored your code base is, no matter how good you’re able to manage technical debt, software always ages to a certain degree because technologies, frameworks and languages used go out of style – just as with buildings. However, it often happens within a project that technical debt is accrued at such a rate that it quite soon stops all progress in its tracks. Martin Fowler calls this the Design Stamina Hypothesis.

Over the years, there have been numerous articles on how and why this occurs, but I’d like to sum the problem up as follows:

The powers-that-be fail to realize that many of their decisions may have near-irreversible consequences (based on the false premis that software is extremely malleable) and software developers (aka ‘the powerless’) do not make enough of a stand to prevent those consequences.

Point in case: the very few managers I have encountered which didn’t think that software creation was particularly hard, approached software not as something which is intrinsically malleable but rather as something that’s quite the opposite. The funny thing is that these are actually the people who consistently score at least two of ‘in time, within budget, with quality’ in an industry which often tends to score around 0.

What do you think: is software/code too malleable? And who’s to blame for abusing that property?