A Generic Application Language

This blog is a follow-up to a previous one with, as promised, some of my thoughts on a Generic Application Language (GAL). This is about what I personally would like to see in a modeling/programming language and its tooling, based on my experience actually building applications. As a consequence, the following will be extremely opinionated and in places downright rude. Deal with it or go away! 😉

I’ll start with pitching a few terms which I would have apply to the GAL. Some of these are purely oriented towards functionality, some are technical, some none of the previous. Here goes…

TL;DR: this explains what a language for application development and its IDE should look like

The tooling actually supports software development

Many IDEs are essentially not much more than glorified file tree navigators that open files in editors, with a lot of micro services tacked on top of it. This is great for configurability, but that also means that you actually have to configure it. It typically takes an inordinate amount of work to marry up your team’s/company’s workflow (version control, reviews, Agile, etc.) and the particular architecture of your code base to the IDE. So much even, that I’ve never seen it done to any degree of intimate satisfaction – wait, that sounds wrong, doesn’t it? Usually, this results in many manual actions and/or lengthy and cumbersome build cycles, hindering feedback/validation for the developer. (In all fairness, IDEs on the level of IDEA/IntelliJ provide reasonably smooth sailing these days.)

So, some features I want to see (roughly in that order):

1. Truly smooth integration (and configuration of, if/when required) with workflow, including Continuous Build and Deployment.
2. Live coding everywhere. Point to some code, give it some input, see what happens, debug it. No mucking about with making tests, providing all dependencies by hand  or running the entire application, setting a breakpoint and hoping for the best. Upgrade a live coding session to a test afterwards – unit or integrated, whatever you wanna call it; I don’t care.
3. Being able to reason about the code base within the tool itself. Run a query that’s language- and context-aware on your code base. Define code conventions and  architecture that way. Do impact analyses. Verify assumptions and invariants. Also: these things should be part of the code base itself, versioned and all.
4. Advanced and automatic Refactoring suggestions. If you have two sub classes with identical members, surely an IDE should be able to inform you of that and offer help? Recognise an Anti-Pattern as soon as it crops up. The possibilities are effectively endless, especially when you get to add your own wizard thingies.

Static typing

I know a lot of people experience pronounced localised vascular throbbing because of thinking dynamic languages are intrinsically way better than statically-typed ones, but I’m entirely willing to call those out as innophobic, pseudo-religiously insane bastards (m/f). Static typing helps. A lot. No amount of TDD can compensate for the lack of API discovery, content assist and avoiding stupid-but-hard-to-detect bugs – all things made possible by having a real type system. Studies show it, so get over it.

Sure, you need the following to be in place:

1. A healthy type system. This means reified generics. It does, however, not mean full recursiveness. Because that’s called Reflection-induced Hell in actual practice. Because someone will use it – probably yourself, thinking you’re smart. Same goes for meta programming: easy to abuse, hard to maintain by the rookie once the meta cowboy has moved on to fresher pastures, makes reasoning about your code base nigh impossible. ‘Nough said.
2. A good type inference engine. Reduce syntactic noise: no-one wants another Java. Scala and Xtend already show the way.
3. Polymorphism. Comes quite naturally for dynamic languages but is actually not so hard for statically-typed ones. (Xtend puts it on top of Java essentially effortlessly, e.g.)
4. Tags/tagged values/annotations. A lightweight, orthogonal way of tagging parts of your code and adorning them with extra info. Crucial to those LISP-like macros I’m going to be talking about later.

As long as you can keep the type system reasonable (Scala not being a good example), this is something that’s somewhat tedious and not trivial but also not frightfully difficult. Once you have these you can start thinking about using proof assist software such as Coq could help to reason about null safety, arguments validity, reachability and such. This ties in naturally with points 3 and 4 of the previous section.

Integration of front- and backend

Most applications today are Web applications. Typically, this means that front- and backend run on platforms that are technologically effectively disjoint: e.g. HTML5 vs. JEE. Equally typically, front- and backend are blissfully unaware of their respective details and any change to either requires patches up the other (1) by hand and (2) in several places.

To add insult to injury: if the architecture is set up with this in mind, the code on either end will look quite similar which begs the question why you can’t transpile from say, Java to JavaScript. (If it isn’t: well, good luck with the mess you got yourself in…) Also: tests are confined to their platforms, so these will not alert you to the fact that you’ve missed something.

Surely, in this day and age we can create a language that unifies both aspects using static typing. Google’s Web Toolkit got this, as does Safely-Typed JavaScript. Oh, this also means you could do tests that cross over that Great Divide.

Projectional editing

If you want to play around with a rectangular grid of characters, go play WordFeud™ (or Scrabble™). If you think free form text is the best thing since sliced bread, break out and dust off your grandfather’s type writer and bash out a novel.

In order to make my case, let’s consider changing exactly one character in the entire code base and assume that your entire application is not immediately breaking either because it doesn’t compile anymore or because it plainly blows up in your or someone else’s face when running it. Then, in decreasing order of likelihood:

1. You’re replacing whitespace with whitespace. This means you’re either a fascist or you’ve got a severe case of OCD. Go see either a criminal court or an episode of tBBT.
2. It’s inside a comment. Why is that comment there? And why isn’t it statically checked? In any case, probably endemic of a bigger problem.
3. You’re commenting out code. Planning to commit that, ‘guv?
4. It’s inside a text that’s purely for human consumption. Is it internationalised? Are you secretly writing a novel after all? If so, better go and search for that Underwood.
5. It’s inside a magic value. Hopefully you made that fact explicit. Otherwise, good luck with detecting and fixing the bugs that will inevitably result from that magic.
6. It’s inside dead code (provided the code even compiles). Why is it still there? Why didn’t your IDE alert you to that?
7. You’re just lucky. Well, for now at least. You’ll have some Heisenbugs later on, don’t worry.

So, why going to the trouble of give your developers the freedom to type in essentially random characters in random places, slapping them on the wrist through validation or trying to make up for the mess by means of syntax checking, highlighting, content assist, etc. after the fact? What more is an IDE than an application that visualises a code base and provides editability and other micro services with a defined behaviour on it? Is editor behaviour then not a matter of UX design? Even the command line guys are exploring that  direction.

(Another disadvantage of the textual format is that it leads to code layout wars – see also point 1 above. This applies less to Go and Python, and not at all to Brainf*ck, though.)

Oh, and you could even consider text to be a projection of the original code’s AST to some extent.

Once you’ve got projectional editing working as such, the following features are fairly easy to pull off (and although not impossible, extremely difficult to do for purely textual notations):

1. Multiple projections. The same AST can be visualised in different ways, with altering behavior of the micro services.
2. Great syntax and layout. Once you get started, there’s no end to how good you can make code look. Goodbye to mono-spaced fonts and CGA-era colour schemes.
3. Language embedding. Really difficult for the textual case – I know of only one tool that’s able to do that and it’s pretty much academic in every sense. Much, much easier for projectional editing. One especially important use case for this is LISP-like macros/templates. More on that below.
4. Context-aware editing. Got to put in a date here? Date picker! Same for colours, easy. This often obviates the need for language embedding.

Creating DSLs with macros/templates

I already alluded to this in a previous post, but this requires more explanation and examples. The general idea here is to be able to create code from code, in a way that allows you to gradually move from the level of abstraction that you want to the level that you (or the platform) need. (I guess you could also call this “desugaring”.) I’m going to talk about it in another post, because this post is long enough as it is and I think the idea deserves its own podium.

Functional programming

This is a bit of a no-brainer these days, luckily. Many GPLs are already firmly going that way, with 1st-class support for immutability, closures (I guess “lexically-scoped coroutines” to some), lazy evaluation, etc.. To me the most important take-aways from FP are (in that order!):

1. A good expression sub-language that also innately covers querying and comprehending of collections(/streams), is extremely powerful – essential, even.
2. Expressions are easier than statement (block)s. “Everything is an expression” is easy to achieve. Expressions can easily be upgraded to closures.
3. Pure functions are easy to reason about.
4. Pure functions are trivial to parallelise. In fact, it’s pretty much the only productive way to do actual concurrency/parallelism.

(Note that there’s a distinction between expressions and functions.) In fact, you can view an(y) application as a function of a stream of inputs to a stream of outputs/effects. This notion might not help matters when applying it to the application as a whole, but it certainly can when applied to individual layers of it. E.g., a UI is for the most part a function of application (its data) and client-specific state (which screen, etc.).

Things that are less important to me:

1. Monads. If you need them to make software development easier, you’re missing the point. That a lot of things are easier to reason about once you realise they’re monads, does not imply that you need to forcefully make everything a monad before they could be made to work. Use/apply them where it makes sense, don’t where it doesn’t. “But: side effects!”, you say. “Where?”, I say. It’s extremely useful to make side effects explicit. So much so that you should really consider creating a 1st-class language construct (with decent, non-leaky semantics) for that, instead of providing your developers with a cabinet full of Hello Kitty!™-themed bandage to plaster them over with.
2. While we’re at it, and pretty much for the same reasons: category theory. I know Erik Meijer does magical things with it, but face it: he’s pretty much the only guy that knows how to do that in real life. Again: use/apply it where useful, don’t where it doesn’t and certainly don’t make it a prerequisite for joining ye’olde Clubb.
3. Immutability as an opt-in concept. I see more value in explicitly specifying lifecycle and mutability only where it’s intrinsically relevant (e.g., in the data model). In expressions, immutability is the default and lifecycle irrelevant.
4. Whatever you can name as “super important, enabling concept that can only be found in Our Tool™”. (With the exception, of course, of anything intrinsic about the GAL ;)) I’ll be eternally puzzled by product descriptions along the lines of: “Our Tool™ allows business users to do X. That’s because we’ve built it according to vary-vague-and-academic-principle Y!” Huh? Interest in X and understanding of Y is mutually exclusive, so who should be enthused by this pitch, exactly? Exactly…

Reactiveness

Hey, there’s Erik Meijer again! 🙂 To me, reactiveness is several things at the same time:

• A philosophy.
• An extension of FP: everything is a stream of things and functions thereof.
• An implementation detail of FP: there’s an obvious mapping between pure functions on data to reactive streams of “delta events” on that data.

The latter two points especially pertain to modeling UIs – more on that in a later blog post (perhaps).

 

In the trenches: MDSD as guerilla warfare

However much I would’ve liked for things to be different, adoption of MDSD is still quite cumbersome, even though it has been around for quite some time in different guises and various names. Particularly, the MDA variant may have caused more damage than brought goodwill because of its rigidity and reliance on UML which, frankly and paradoxically, is both unusably large and largely semantics-free at the same time.

Therefore, it’s “somewhat” unlikely that you’ll be able to ride the silver bullet train and introduce MDSD top-down as a project- or even company-wide approach to doing software. This means you’ll have to dig in and resort to guerilla tactics, so be prepared to have a lot of patience as well as suffer frustration, boredom and even a number of casualties – doing all this in the knowledge that if/when you succeed, you will have furthered the MDSD Cause and made the world a ever-so-slightly better place.

Remarkably, opposition to MDSD has -at least in my experience- equally come from both non-techies as well as techies alike but with very differing concerns. From the managers’ viewpoint, you’re introducing yet another “technology”, or more accurately: methodology, so you’ll have to convince them of the gains (e.g., increase of productivity, longer-term reliability) and placate them regarding the risks (e.g., ramp-up costs, specialty knowledge). That to be expected, as well entirely reasonable.

Your colleague developers, on the other hand, are a tougher crowd to convince and any manager or team/project lead worth his or her salt is going to take a team’s gut feeling into account when deciding on proclaiming yay or nay on adopting MDSD in a given project. What struck me, at first, as extremely odd is the reluctance of the “average developer” to use MDSD. This was at a point in my career that I still had to find out that most developers were much more interested in prolonging the usefulness of their current skill set than in acquiring new skills, and much less seeing that same skill set become much less valuable because of something that’s not a buzz word (which MDSD isn’t). From a personal economic viewpoint this makes imminent sense as it makes for a steady and predictable longer-term income.

“Unfortunately,” that’s not the way I work: I have a fair bit of Obsessive Compulsive Disorder which leaves me quite incapable of not continuously improving a code base and encourages me to find ever better ways of succinctly expressing intent in it. I stand by Ken Thompson’s quote: “One of my most productive days was throwing away 1000 lines of code.” In hindsight, it is no surprise that I immediately gravitated towards MDSD when I came to know it.

But enough about me: what would the Average Developer™ think about MDSD? As already mentioned: it can easily be perceived as a threat because adoption usually means that less of total developer time is spent on using an existing skill set (never mind that that skill set comes with dull, tedious and repetitive tasks!) which begs the question what the remainder of the developer time is spent on. The answer can be: on something which requires a new skill set or it is not spent at all because the work can be done in less hours which amounts to another blow to one’s job security.

The other reason that our AD doesn’t like MDSD is that they are, well…average, meaning that it’s probably a bit of leap for him/her to get used to it and especially to acquire some of the required skill set to actively participate in it. Meta modeling comes natural to some of us, but these people tend to occupy the far right of the bell curve of the normal distribution. So, MDSD is also a threat to one’s occupational pride and confidence.

A useful approach to convincing the AD that MDSD might hold some sway and can make life easier, is by “doing it from the inside out”: instead of converting the entire project to an MDSD approach, convert a small part of it, focusing on a low-hanging fruit which comes with a lot of daily pain for the developers involved. A fruit that usually hangs within grabbing range is the data model, together with all its derived details in the various application layers: DB schema, ORM configuration, DTOs, views, etc. It’s convenient that it’s already called a data model, so it’s not a big leap to formalize/”modelize” its description sufficiently, e.g. using a DSL, and to write a generator to generate the tedious bits of code.

It’s important to pick your battles intelligently: don’t try and subjugate entire regions of the existing code base to an MDSD regime, even if you already could with the current state-of-the-art of the model. This is for two reasons: any conversion will have impact on the project and have some unforeseen consequences, but more importantly, you’re effectively throwing away someone’s code and replacing it with generated code – most developers are uneasy with that and if you do this too harshly you will not gain traction for MDSD.

An important ingredient of guerilla tactic in general is to appease the local population: “The people are the key base to be secured and defended rather than territory won or enemy bodies counted”. For MDSD it’s no different which means it’s crucial to keep responsibility in the same place. As Daniel Pink explains, autonomy, mastery and purpose are crucial to anyone’s motivation. So, make the original developer of the replaced code responsible for the MDSD solution as well: show him/her how to change the model, the generator and (even) the DSL and demonstrate how this makes life easier.

Once small parts of a project are model-driven, you can stimulate growth in two directions: (1) “conquer” more areas of the project, and (2) glue parts together. E.g., if at first you’re only generate the DB schema from the data model, then start generating the DTOs/POJOs as well, after which you naturally can generate the ORM configuration as well. If you’ve also got a model for the backend services, you could now start writing down contracts for those services in terms of the data model.

Again, you’ll have to keep track of how your fellow programmers are doing, where you can help them with MDSD (and where you can’t), whether you can “hide” the ramp-up costs enough to not be considered a cost center, and whether they are adopting the MDSD way as their own.

A (slightly) better switch statement in JavaScript

The switch statement in JavaScript suffers from the usual problems associated with C-style switch statements: fall through. This means that each case guard needs to be expressly closed with a break statement to avoid falling through to the first executable code after that – no matter which case that code belongs to. Fall through has been the source of very many bugs. Unfortunately, the static code analysis for JavaScript (JSLint, JSHint and Google’s Closure compiler) do not check for potential fall through (yet?).

Today I thought I could improve the switch statement slightly with the following code pattern:

var result = (function(it) {
switch(it) {
case 'x': return 1;
case 'y': return 2;
/* ... */
default: return 0;
}
})(my_it);

(Apologies for the lack of indentation: couldn’t get that to work…)

The advantage of using return statements is two-fold:

1. it exits the switch statement immediately,
2. it usually comes right after the case guard, making visual inspection and verification much easier than hunting for a break either in- or outside of a nice pair of curly braces.

This approach also has a definite functional programming flavor, as we’ve effectively turned the switch statement into an expression, since the switch statement is executed as part of a function invocation.

Postscript

Yes, I do write JavaScript from time to time. I usually don’t like the experience very much, mostly because of inadequate tool support and the lack of static typing (and the combination thereof: e.g. the JS plug-ins for Eclipse often have a hard time making sense of the code at all). But we do what we can to get by 😉

The ASmaP Principle (part 2)

To recount the gist of the previous blog: the ASmAP-principle strives to minimize the footprint of your project/application in terms of code base size, dependencies and configuration complexity. (In fact, it’s a particular re-statement of the KISS-principle.)

What’s the use?

The main reason to follow this principle is that projects/applications tend to grow fast to a point where it’s quite impossible to fit a good mental model of it into your brain in any reasonable amount of time. In turn, this inhibits you in becoming productive on those projects and making the right coding decisions. By constantly making an effort to reduce incidental complexity, you’ll make development on the project more effective and productive – not only in the long run, but also in the short run. You should see a decrease of time-to-market, bug count, maintenance costs after a short time, while hooking up extra devs to the project and effectively communicating with your customer becomes easier.

The ASmaP Principle is not about being as smart as possible in as few lines of code, it’s about obtaining/using/achieving the simplest possible solution, but not simpler – free after Albert Einstein. It also pertains to documentation: provide just enough text to clarify what your code is doing beyond what the code already clearly expresses on its own. This is also an incentive to write your code in a way which makes documentation largely unnecessary. I usually get away with some explanation about a class as a whole and some usage patterns.

Now, I’m not advocating premature optimization (of the performance of implemented functionality) here. I do think that it is the root of at least a lot of evil and any optimization effort should be based on profiling. Moreover, optimization is usually the source of a lot of incidental complexity since it doesn’t expand functionality as such only the performance of the implementation. As an example, consider caching: it requires adding a layer of indirection, thinking about invalidation and usually comes with a bunch of assumptions and runtime behaviors which are

Oh, and I certainly don’t advocate minification or obfuscation 🙂

Kill your dependencies

Dependencies on external libraries, frameworks, components, runtimes, environments and what-have-you -i.e., all this is within your direct sphere of influence but which you didn’t actually create yourself- are often the biggest source of incidental complexity. Each of these dependencies add their own dependencies but more importantly, they add to the information you need to carry around in your head: what’s the library’s API like, how to include the library correctly, how to instantiate the component, what are the peculiarities of this runtime, what do I need to configure in that environment to have it switch between dev/test/acc/prod?

Obviously, you can document much of this, but documentation has to be kept up-to-date and you’ll always need to remember a non-trivial amount of information to even be able to get things working at all, let alone productively.

Furthermore, dependencies tend to solve only half of a problem that technical rather than functional in nature – in other words: it’s falls squarely in the category of incidental complexity. My favorite example is Hibernate: it gives you (un-)marshalling of Java objects from/to a relational database, but at the price of having to write XML mapping files (well, in the pre-@annotation days, at least) which you need to keep in sync with both the Java classes and DB schema. Were I to generate the latter, I could just as well generate the necessary (un-)marshalling methods. (With annotations most of the meta data resides in the Java class, at least.)

To gauge whether you have too many dependencies in your Java project, I propose the following:

The Maven Rule: if your project uses Maven for dependency management, you have too many dependencies. Period.

While Maven undeniably makes the whole dependency management problem more bearable, it generally expands the same problem by making it just too friggin’ easy to drag in more dependencies without every really wondering whether that’s a good idea. Think about what you really need from dependency #37 that you put in your project. Some questions you could ask yourself:

1. Is the same functionality already implemented (more-or-less equivalently) somewhere else in your project (typically using another dependency-of-ill-repute)? – If so, Refactor and remove all but the most-used dependency. Sometimes, you’ll find that you’re better off writing your own implementation without using external dependencies to snugly fit your needs.
2. Does it solve your entire technical problem? If not, are better frameworks available?
3. Does it provide an complete paradigm/way-of-thinking/-working (such as the Google Guice or Apache Wicket frameworks) or does it “only” provide some utility functions (such as Google Guava)? In the latter case, see the first point.

Note that I’m not advocating NIH here, but mind that there’s also no point in shoehorning a problem in a particular shape so that it happens to fit some “reusable” code: the fit is often quite bad and the primary code not very aesthetically pleasing. That’s a general problem with reuse (I have): all code makes a lot of implicit assumptions and unless these assumptions are met by your problem (domain) on a grand scale, things are not going to fly. The most rewarding uses of reusable components are often those where the component either matches the problem domain perfectly or is completely functionally orthogonal.

Not only can you reduce the sheer number of artifacts on which you’re dependent, you can also make sure that the number of direct dependencies (i.e., part X makes direct use of dependency Y by explicit reference such as an import) is as small as possible. This is analogous to your project architect insisting that application layers only interact in very controlled ways (and preferably through DTOs). Often, you can hide a large number of dependencies in a specific sub system which then governs use of and acces to its dependencies.

New Year’s Resolution: the ASmAP Principle (part 1)

Happy New Year, everyone!

Seeing that we’ve safely progressed into 2012, it’s time for New Year’s Resolutions. As a suggestion for a NYR in software engineering, I’d like to suggest following the

ASmAP-principle, where ASmAP == As Small As Possible

Ever since I started to program on the MSX2, I’ve been wondering where all that extra computing power, memory and disk space that all my subsequent work horses have been endowed with, went to. Obviously, the amount of pixel estate has grown quite a bit (from a measly 0.1Mpixel in 4-bit color to over 1.2 in 32-bit color on the quite modest screen of my MBP 15″), but that doesn’t really explain why my Eclipse typically reports around 200MB of heap space in use: I sincerely doubt that the size of my source code (or even the sheer bytecode size of my compiled classes for a fairer comparison) has grown by a factor of  7000 from the roughly 24KB of memory that was available for MSX BASIC. It also doesn’t explain why whole slews of applications aren’t really more responsive these days than they were in “ye’ olden days”.

Of course, I know that running a JVM and OO languages on it is intrinsically more memory intensive than assembler or what are essentially token streams (the in-memory representation of an MSX BASIC program). But I simply refuse to accept that we need so many more bytes to write even the simplest of programs on our modern-day OSs, VMs and IDEs. I think a large part of this growth stems precisely from Moore’s Law in action: because we have a lot of memory and extra CPU cycles per second, we tend to use them – not per se for more functionality, nicer graphics and such, but also to make programming easier for ourselves without actually delivering more business value or quality.

I’ve seen my fair share of “programming by copy-paste” and the consequences of that: bloated code bases with a lot of (usually inconsistent) duplication, a build process that takes ages to finish and is brittle, a development environment that takes days to set up (possibly after sacrificing a few kittens in the process, just to get it to work at all) and an enormous amount of dependencies (“Maven makes this so easy to manage”) consisting of all kinds of stop-gap frameworks which never seem to solve a business problem and only seem to solve halve of a technical problem. The end result is a gargantuan and monolithic Thing™ that’s hard to fix, hard to alter and hard to hand off to anyone so you’re stuck with it until eternity. Sounds familiar? 🙂

I think it’s this tendency to copy-paste our way around development, without taking the time to properly Refactor your stuff and reflect on your solutions, which accounts for the enormous gap between a typical application’s footprint and the value it delivers. As a solution, I propose to optimize something that’s relatively easy to measure: the footprint of your code base, in terms of pure source size but also taking into account the amount of re-used “stuff” such as libraries/frameworks, the time needed to build your project and deploy the application and the number of dependencies. In other words: make your project As Small As Possible. (Note that size is a lot easier to quantify than, say, complexity or quality.)

The overruling motivation for doing this is something which hasn’t changed the last few millennia: the amount of information our brains can interpret, process and store (in a limited amount of time). It is this characteristic which is the prime limitation in any project and especially those involving more than one developer – i.e., the typical ones. The amount of information in a software project is a lot bigger than the sheer code size conveys because there are all kinds of connections between various parts of the code base which you need to understand in order to add or change anything in the code. It’s my impression that the complexity is roughly a function of the code size base multiplied by the number of layers/aspects in the architecture.

So, the larger a project’s footprint, the harder it is to understand: not only for your co-workers (who are presumably stupid gits which couldn’t program their way out of a tin can), but even for yourself. The less you understand about your project, the less effective and productive you’re going to be. In other words: you need to build up a mental model of the project (which is something I’ve blogged about earlier) and hope it will fit in your brain in the short time you have. Given that you and your team mates can’t grow extra brains, you’ll have to utilize the available mental processing power as efficiently as possible. Note that adding extra single brains to the project doesn’t provide a solution, because each individual brain will not be able to grok the entire project if you couldn’t do so yourself to begin with.

In the next installment, I’ll discuss some simple approaches to follow the ASmAP-principle.

RTFM!!

As a follow-up of sorts to a previous blog, I’m going to rant a bit now about the situation that there is decent documentation, but people refuse to read it.

In ye’ olden days, I have tried to study books like this as part of my Master’s studies in Mathematics, books which are really hard because they deal with very abstract topics and do so in a comprehensive but still quite compact manner so you have to ’embellish’ the reasoning by interpolating between the factual statements given in order for you, mere mortal, to be able to understand it completely.

And by ‘hard’ I mean ‘time consuming’ as much as ‘headache-inducing’, the juxtaposition of which precludes that “Googling for 5 hours” will get you anywhere: it may be time consuming (it certainly is) and might be ‘frustration-inducing’ but your brain is not engaged enough by that type of effort to actually attain a bit of knowledge about something from which you can come up with a solution to your problem. It also precludes finding solution for problems which nobody ever encountered before or which have been worded sufficiently different.

Going back to math books: I always told the (business/psychology/A.I.) students I had while being a student assistant teacher, that if they worked their way through one page of mathematics in less than a quarter of an hour, they could be 100% certain that they hadn’t understand one bit of it and would therefore fail their examinations. I’ve never been challenged about the truth of that statement, only the minimum amount of time varies per student and per book/course: usually it takes a lot more than 15 minutes to really grok the material. For the book I cited earlier, I could substitute ’15 minutes’ with ‘8 hours’. In the end, I’ve given up on that particular book. Luckily, that didn’t stop me from getting my MSc.

The reason that learning takes time is that the human brain is surprisingly good at a great number of things, but remembering factual statements rigorously and executing their semantics in a totally reliable manner isn’t one of them. I read once that “the human brain is very good at avoiding being eaten by a mountain lion and distinguishing the nourishing berries from the poisonous ones, but we aren’t that well equipped for thinking about 11-dimensional spaces”. We are exceedingly good at pattern recognition and are able to deal with incomplete or faulty ‘input’ very well -ftr ll, y cn prbbly rd ths sntnc jst fn- but logical thought takes a bit of a backseat, comparatively speaking.

To do ‘intellectual stuff’ like reasoning about parser implementations and such, we have to build up a sort of mental model which we can hold in our brains. This mental model holds all sort of knowledge: both factual knowledge, remembered verbatim, but also vague stuff which we could file under ‘intuition’, as well ‘executive knowledge’ which can be fed input and which helps you form thought processes. All this will hopefully enable you to solve problems; if not, you have to adapt and expand the mental model. The mental model is unique for each combination of a particular human brain and a particular domain and not readily expressed in any natural language, which also explains for a bit why teaching is such an art: you have to find a modus to tap into the brain and the relevant mental model of the people you’re teaching.

(Disclaimer to the previous paragraphs: I’m not a psychologist nor a neurologist, so what I’m saying here is essentially my mental model of how learning works.)

Building up the mental model takes time, and appreciably more time than just reading some text book: your brain has got to remember a lot of ‘stuff’, make connections between that stuff and validate all new inputs (the next sentence) against the mental model that’s being formed. And to add insult to injury: often enough it’s very difficult to say that your mental model has actually expanded at the end of the day, because once you really understand something, it tends to become trivial and the mental model feels ‘compressed’.

I realize that I have yet to start my rant for real…

In stark contrast to the former, this day and age apparently is one where all answers should be one Google search away. This is unrealistic for many reasons, the most important being that it would imply that all answers are already known, which would be a depressing thought in itself. However, the most demeaning aspect of this school of ‘thought’ (or lack thereof) is that it is implied that knowledge is unnecessary and that only facts matter. But facts in itself are essentially inert matter and you have to make it come alive by embedding in a body of knowledge: a.k.a. the mental model. Without that we would be the proverbial infinite amount of monkeys hammering away at typewriters to try and come up with a Shakespeare play.

Also annoying to the max is the idea that communities (such as the plethora of Eclipse fora) exist to do the work for you, rather than helping you do it by guiding you in acquiring more knowledge on the subject. It seems that home schoolwork is often done these days by posting the problems to some Q&A site and expecting, nay, demanding that someone else solve these for them. Instead of being glad that someone who has already gone to the trouble to attain the knowledge required for performing that task free up some of his/her time to try and help a complete stranger by giving hinters, tips and useful pointers, abuse seems to be in order…

For myself, I’ve decided already some time ago to only help those people who show at least some willingness to learn and put in the (right kind of) effort themselves. In fact, I tend to try and help everyone but stop helping as soon as my help isn’t taken seriously, no effort is being expended by the other party, I receive abuse or any combination thereof – there’s no point in feeding the troll, after all: he/she can go gnaw some rocks, as far as I’m concerned…

Addendum

Funny point in case: how to post on xda.

On documentation (or lack thereof)

There are probably enough books, articles, blogs and tweets on documentation to fill a library, so there’s absolutely no need that I vent my thoughts. Fsck it, I’m going to do so anyway, spurred on by a recent tweet “discussion” and a general lack of Javadoc on classes of one of my favorite frameworks. And don’t worry: I’ll make it personal and not based on any existing literature or body of knowledge whatsoever, so it’s completely fresh and unadulterated 😉

My initial grumbling tweet read “Hmmm, why do devs confuse beta/experimental with ‘no need to document’.” To which I got the reply from @markijbema: “Most documentation is still too far away from coding. I’m willing to document if it’s DRY and away from my normal workflow.” (Note that this post is not a rant about that tweet: the tweet just happened to trigger me to write up the beliefs I’ve had for some time now.)

Now, I completely agree with the DRY-part. After all, we’ve all seen code like this:

/**
 * Increments the given integer with 1.
 *
 * @return The given integer, incremented with 1.
 * @param i
 *            - the integer to be incremented.
 */

public int increment(int i) {
  return ++i; // increment i with 1 and return
}

Good for your LoC count, bad for everything else. This complies with Mark’s remark about ‘documentation being (still) too far away from coding’. However, good documentation is not too close to the code, as documentation should augment the code but let the code itself speak for itself as much as possible. To give one good reference, despite all my promises in the first paragraph: go read Kent Beck’s Implementation Patterns.

I completely disagree with Mark’s assertion that documenting shouldn’t be part of a normal workflow, though. There are a couple of reasons for this. First of all, if it isn’t part of a normal workflow, it’s not going to happen, period. Now, a lack of documentation could very well be more desirable than bad documentation, since it can’t put anyone on the wrong foot and doesn’t cost any effort to produce.

Alas, the real world often insists on some documentation which usually means it will be done poorly. By someone else than an original coder. That someone else being a non-coder or a very bad one since he/she isn’t allowed to touch the code anymore. And it’s done about 5 minutes before the last deadline in about as many minutes. (This goes some way in explaining why 70% of all software never makes it to Production.)

Secondly, the effort it takes to write good documentation is an inverse measure of the understandability or the engineering quality of the code: if it’s difficult to describe what some piece of code does, then I’ll bet that the code is difficult to understand, hence difficult to use, to maintain and to change. If it’s easy to write good documentation, then you’re done in no time at all and there’s no reason why you shouldn’t have taken the effort 🙂

How I do it (or at least: try to)

For myself, I tend to document my code on the ‘module’ level (which can mean anything from a singleJava package to a coherent set of Eclipse projects – whatever constitutes a sensible, functional/technical breakup of the code base), on the class level and one the feature level (meaning fields or methods). Also, I tend to document design decisions in the code, right at the point where I made them.

On the module level, I stick to high-level stuff (what am I trying to achieve, what are the dependencies in terms of infrastructure, libraries and development environment, what does the road map look like and how can someone influence it, how can one contribute), possibly combined with pointers to ‘hooks’ in the code base which act as starting points from which you can traverse the code base in an meaningful direction.

On the class level I tend to document aspects like the class’ responsibility and usage patterns –What is this class intended to achieve? How can this class be used by other classes to that end?- and nothing more (apart from design decisions and such). Note that the usage patterns tend to include dependencies.

I’ve often noticed that documenting the class’ responsibility in a high-level manner is often already useful for yourself, but it’s especially important for other people who try to understand your code base. As an example: I know quite a bit about Xtext, but its code base generally lacks JavaDoc which means I have to second-guess about the function of a class and how to use. This also means I’m only going to notice gems like o.e.x.util.OnChangeEvictingCache until I stumble across them and see someone else use it.

On the feature level, I try to be as succinct as possible, especially with private features, but I don’t restrict documentation to public features. After all, visibility should be about separation of concerns, coherence and loose coupling and doesn’t pertain to the code’s understandability as it transcends the boundaries of visibility. I think quite a bit about good method names, but also about parameter names, even in languages that only have positional instead of named parameters (like Java) since these do show up in the documentation and can be used to convey meaning. Or the other way around: badly chosen names can cause more confusion than documentation can cure.

Overall, I try to tie the documentation as much as possible to the actual code by use of JavaDoc or similar DSLs, since that gives me a kind of limited static type safety on the documentation. At any point I reserve the “how” for in-code comments (e.g., Java single or multi-line comments) and the “what” and “why” for the ‘official documentation’ (e.g., JavaDoc).

My favorite quote

It’s about time for a new blog, seeing that the last one was more than a month ago, and this time it’s going to be as non-technical as I can make it.

“It’s easier to ask for forgiveness than it is to get permission.” – attributed to Grace Hopper

has become one of my favorite quotes during my years working in the software industry, to the extent that it has become something of a personal mantra. I’ve used it a fair number of times in the face of managers or processes that only had short-term incentives or optimizing measurable-but-less-than-sensible metrics in mind, rather than long-term actual goals. In fact, the end result usually was forgiveness being extended rather than retaliation being dealt.

In each of the cases where I decided on not obtaining permission, I went with what I genuinely considered to be the most sensible approach to a real hard problem but which was met by the usual counter-“arguments” like “too much risk and not enough short-term Return-On-Investment” – these particularly crop up when trying to introduce model-driven development, obviously. One such situation I described in my first blog (which in hindsight was much too long).

My conviction to stand up to superiors (and sometimes peers as well) was fueled in part by my Frisian genes -which tend to provide a propensity for mule-headed stubbornness- but more so by the realization that people with MBAs are generally the people with the least suitable background (in terms of education and experience relevant to software engineering) to weigh the pros and cons lying on the table and come up with the Right Decision™.

It’s interesting to note that Grace Hopper was an officer in the US Navy. To me this proves that even in an organization as hierarchical as the military, it is sometimes possible or even required to do something that’s not actually approved or sanctioned by the powers that be in order to answer the call of duty and achieve the important goals. In her case it certainly did lead to forgiveness -at least in the sense that it outweighed the severity of the retaliation that did occur- and even recommendation: Hopper eventually rose up to the lofty rank of Rear Admiral.

Finally, it’s interesting to note that Hopper had a large hand in coming up with the first COBOL language implementation. I’m sure she must have thought at the time that COBOL was A Good Thing™. I’d wonder what she would think of our present IT world which apparently runs for a good 80% on good’ole COBOL, with millions of new lines of code of it being written every day (maybe even on weekend days!). Maybe we need some more boldness-without-permission to go and rid the world of it.

Why is software creation hard?

(Apparently, I’m not the sort of guy to address the smaller issues in life? ;))

The one thing that has been mystifying ever since (and possibly before) Fred Brooks wrote his famous The Mythical Man-Month book, is the question why creating software seems to be so much harder than it ought to be. After all, nowadays, upfront investments in the form of specialist equipment is negligible (a few hundred $’s per seat), anybody with at least a modicum of education can call him-/herself a programmer and essentially the only resource which is destructively consumed during the process is the caffeine poured into said programmers. And the best thing is of course that software’s so flexible -that’s why it’s called software, remember?

I have the nagging feeling that that’s exactly the problem, especially in interaction with people to have the word ‘manager’ in their job descriptions.

Getting rid of bad analogies

Part/indicative of the problem might be the type of analogies that are used in communication with said managerial types. The car factory-analogy has turned into cliché for at least a decade even though the analogy doesn’t quite hold. The premis of a car factory (or any mass production factory, typically using ye ole’ conveyor belt) is that it’s able to turn out a large number of incredibly homogeneous products efficiently and continuously.

This is in marked contrast to the actual software creation process where we typically turn out one product at a time, or a very small run of variations of essentially the same product (either a product line, or a small set of different versions or the same version across dev/test/acceptance/production). Any two or three projects/products tend to be heterogeneous enough to prohibit manufacturing them in the same software factory and the critical threshold for such factories to become economical is much larger than two or three.

When looking at certain, isolated aspects of projects, the situation is much less bleak: (well-chosen) aspects on themselves often exhibit enough homogeneity to make creating a factory for those aspects separately economical across a very limited number of projects and often enough within one project. It is the premis of Model-Driven Software Development that by capturing the variability of specific aspects using models written down in a DSL and building an execution engine for these models, we achieve a significant production gain. Similarly, using a framework might already go a long way in achieving a speedup.

In fact, the very, very few software factories that I know that actually work (in the sense that they’re being using to deliver working software to paying customers), do so by catering for a well-chosen category of aspects of a well-chosen category of software products. And even then, not all aspects of the project are realized by the software factory alone but are added manually.

So, let’s do away with the car factory-analogy and look for something more fitting. Personally, I like the analogy with building a large, fixed structure (a building, a bridge, etc.) for the following characteristics it shares with software creation: it takes a lot of specialist and specialistic knowledge and skills to get it done; there’s a lot of integration with existing infrastructure (characteristics of building site, surrounding structures, utility infrastructure like water, electricity, sewage, etc.); architectural/design decisions have very definite consequences on the overall process; once it’s done you, it’s very hard to change anything about it and the original builders move on to something else. Note that I’m not proposing that software creation should be like building a large structure, just that the shared characteristics might make it a more useful analogy than building cars.

A nice ‘feature’ of this analogy is the abundantly-clear importance of design: the structure is designed up-front after which the building is ‘simply’ a matter of executing that design according to the construction specifications (Dutch: “bestek”). As Jack C. Reeves reasons in his 1992 article, writing code is design and the analogy of the actual construction would ‘just’ be compilation. This also indicates the failure of the analogy: compilation is cheap, while construction isn’t.

Soft is the new hard

Back to the problem at hand: software seems pliable and malleable on the outside, but takes on properties of common concrete over time. This phenomenon is to some extent unavoidable: no matter how well-engineered and -factored your code base is, no matter how good you’re able to manage technical debt, software always ages to a certain degree because technologies, frameworks and languages used go out of style – just as with buildings. However, it often happens within a project that technical debt is accrued at such a rate that it quite soon stops all progress in its tracks. Martin Fowler calls this the Design Stamina Hypothesis.

Over the years, there have been numerous articles on how and why this occurs, but I’d like to sum the problem up as follows:

The powers-that-be fail to realize that many of their decisions may have near-irreversible consequences (based on the false premis that software is extremely malleable) and software developers (aka ‘the powerless’) do not make enough of a stand to prevent those consequences.

Point in case: the very few managers I have encountered which didn’t think that software creation was particularly hard, approached software not as something which is intrinsically malleable but rather as something that’s quite the opposite. The funny thing is that these are actually the people who consistently score at least two of ‘in time, within budget, with quality’ in an industry which often tends to score around 0.

What do you think: is software/code too malleable? And who’s to blame for abusing that property?

 

A common misconception about aesthetics in software

In case you hadn’t noticed: I’m a programmer at heart 😉

Like many or most of my fellow programmers, I’m endowed both with OCD (but only ever so slightly as not to make me dysfunctional…) and a sort of built-in sense of beauty, of “doing things the right way”, when it comes to programming. Chances are that if you’re reading this, you know perfectly well what I’m talking about. It’s that nagging feeling that Separation of Concerns isn’t really observed, that a necessary Refactoring has only been abandoned (temporarily -one can always dream, can’t he/she?) in the face of short-term managerial thinking and that things have not yet been “set right” -a situation that obviously needs correcting. Unfortunately, this type of beauty is hard to make quantifiable, which lends programming a bit of an “art edge” which probably doesn’t sit well with types more interested with things like burn rate, delivered features/solved bugs and such.

The common misconception with managerial types is that this sense of beauty is totally irrelevant or even harmless to the project. Well, it’s not…

It’s both an indication of the craftsmanship (or craftswomanship) of the developer talking about it, as well as an indication of an impending problem (or the presence of technical debt, as you might call it). When a developer says something like “We should have implemented this using the Strategy Pattern.” it’s not the equivalent of “We should have hung this picture frame on the wall using wire instead of a nail.” for interior decoration. It’s the equivalent of saying “We just knocked a hole in the supporting wall. It’ll hold for now, but the wall needs strengthening and will collapse once you knock another hole into it.”. Dismissing him/her means that you’re both throwing caution in the wind as well as that you’re effectively disrespecting your developer and he/she will remember when you ask him/her to work overtime on that particularly nasty bug that crops a few months later after someone knocked another hole in that wall.

(For those of you who think that you have a Process in place for this: in my experience, having a process typically means that people shut off their brains regarding anything governed by that process. Process should be about achieving the common goal, ensuring productivity and providing safety nets, not about placing blame.)

Not listening to your developer having this sense of beauty can and will hurt your project. The bigger issue is that, from a managerial viewpoint, there’s a focus on delivering functional features. Most of the time, a developer’s sense of aesthetics doesn’t concern functionality but rather the infrastructure and architecture of the code -the look-and-feel of the code, but not so much what the code’s actually doing for the business. So, spending time on improving that will have no or no discernible effect on the number of features delivered and will not provide any business value in the near future. Again, the analogy with building a house is useful: the family will want to live in the house as soon as possible, even while it’s still under construction. Given that, having furniture and drapes seems much more important for the end users (the occupants) than having a construction which allows expansion in a few years’ time, even though some walls are creaking under stress somewhat… But, there’s a time that that expansion will be required or that those walls are cracking or even collapsing. At that point, you’ll remember that worker’s warning.

There’s another side to my message: developers should voice their opinions more clearly and loudly and should stand up more to the powers-that-be. When a project manager asks me for an estimate and says “It’s too much. You should be able to do it in half the time, if you just employ these-and-these hacks”, my response simply is: “If you don’t like the estimate, please find some other chap to do your work and don’t call when this person’s work falls apart (unless you’re willing to take me up on the original estimate, of course).”. On one occasion, I’ve simply refused to provide an estimate for doing something the wrong/hacky way, while I was willing to put a number to the right way, which was much more risky but reduced technical debt considerably. In the end, I both proved to be right as well as earned some extra kudos for using my skills in this way for the benefit of the project.

By the way: there’s a very nice article by Jeff Ello about managing geeks.