In the trenches: MDSD as guerilla warfare

However much I would’ve liked for things to be different, adoption of MDSD is still quite cumbersome, even though it has been around for quite some time in different guises and various names. Particularly, the MDA variant may have caused more damage than brought goodwill because of its rigidity and reliance on UML which, frankly and paradoxically, is both unusably large and largely semantics-free at the same time.

Therefore, it’s “somewhat” unlikely that you’ll be able to ride the silver bullet train and introduce MDSD top-down as a project- or even company-wide approach to doing software. This means you’ll have to dig in and resort to guerilla tactics, so be prepared to have a lot of patience as well as suffer frustration, boredom and even a number of casualties – doing all this in the knowledge that if/when you succeed, you will have furthered the MDSD Cause and made the world a ever-so-slightly better place.

Remarkably, opposition to MDSD has -at least in my experience- equally come from both non-techies as well as techies alike but with very differing concerns. From the managers’ viewpoint, you’re introducing yet another “technology”, or more accurately: methodology, so you’ll have to convince them of the gains (e.g., increase of productivity, longer-term reliability) and placate them regarding the risks (e.g., ramp-up costs, specialty knowledge). That to be expected, as well entirely reasonable.

Your colleague developers, on the other hand, are a tougher crowd to convince and any manager or team/project lead worth his or her salt is going to take a team’s gut feeling into account when deciding on proclaiming yay or nay on adopting MDSD in a given project. What struck me, at first, as extremely odd is the reluctance of the “average developer” to use MDSD. This was at a point in my career that I still had to find out that most developers were much more interested in prolonging the usefulness of their current skill set than in acquiring new skills, and much less seeing that same skill set become much less valuable because of something that’s not a buzz word (which MDSD isn’t). From a personal economic viewpoint this makes imminent sense as it makes for a steady and predictable longer-term income.

“Unfortunately,” that’s not the way I work: I have a fair bit of Obsessive Compulsive Disorder which leaves me quite incapable of not continuously improving a code base and encourages me to find ever better ways of succinctly expressing intent in it. I stand by Ken Thompson’s quote: “One of my most productive days was throwing away 1000 lines of code.” In hindsight, it is no surprise that I immediately gravitated towards MDSD when I came to know it.

But enough about me: what would the Average Developer™ think about MDSD? As already mentioned: it can easily be perceived as a threat because adoption usually means that less of total developer time is spent on using an existing skill set (never mind that that skill set comes with dull, tedious and repetitive tasks!) which begs the question what the remainder of the developer time is spent on. The answer can be: on something which requires a new skill set or it is not spent at all because the work can be done in less hours which amounts to another blow to one’s job security.

The other reason that our AD doesn’t like MDSD is that they are, well…average, meaning that it’s probably a bit of leap for him/her to get used to it and especially to acquire some of the required skill set to actively participate in it. Meta modeling comes natural to some of us, but these people tend to occupy the far right of the bell curve of the normal distribution. So, MDSD is also a threat to one’s occupational pride and confidence.

A useful approach to convincing the AD that MDSD might hold some sway and can make life easier, is by “doing it from the inside out”: instead of converting the entire project to an MDSD approach, convert a small part of it, focusing on a low-hanging fruit which comes with a lot of daily pain for the developers involved. A fruit that usually hangs within grabbing range is the data model, together with all its derived details in the various application layers: DB schema, ORM configuration, DTOs, views, etc. It’s convenient that it’s already called a data model, so it’s not a big leap to formalize/”modelize” its description sufficiently, e.g. using a DSL, and to write a generator to generate the tedious bits of code.

It’s important to pick your battles intelligently: don’t try and subjugate entire regions of the existing code base to an MDSD regime, even if you already could with the current state-of-the-art of the model. This is for two reasons: any conversion will have impact on the project and have some unforeseen consequences, but more importantly, you’re effectively throwing away someone’s code and replacing it with generated code – most developers are uneasy with that and if you do this too harshly you will not gain traction for MDSD.

An important ingredient of guerilla tactic in general is to appease the local population: “The people are the key base to be secured and defended rather than territory won or enemy bodies counted”. For MDSD it’s no different which means it’s crucial to keep responsibility in the same place. As Daniel Pink explains, autonomy, mastery and purpose are crucial to anyone’s motivation. So, make the original developer of the replaced code responsible for the MDSD solution as well: show him/her how to change the model, the generator and (even) the DSL and demonstrate how this makes life easier.

Once small parts of a project are model-driven, you can stimulate growth in two directions: (1) “conquer” more areas of the project, and (2) glue parts together. E.g., if at first you’re only generate the DB schema from the data model, then start generating the DTOs/POJOs as well, after which you naturally can generate the ORM configuration as well. If you’ve also got a model for the backend services, you could now start writing down contracts for those services in terms of the data model.

Again, you’ll have to keep track of how your fellow programmers are doing, where you can help them with MDSD (and where you can’t), whether you can “hide” the ramp-up costs enough to not be considered a cost center, and whether they are adopting the MDSD way as their own.

When to solve stuff with a DSL

One of the great benefits of DSLs (textual or graphical alike), is that it allows you to separate essential complexity, i.e. complexity which is intrinsic to the problem you’re trying to solve, from incidental complexity, i.e. complexity which is caused by the approach chosen to solve that problem. This is certainly the case for external DSLs, since these often evolve from a ‘technical clean slate’. Whether you’re actually able to achieve that separation depends on a couple of things:

1. the measure in which you understand the problem space or domain;
2. the measure in which you understand the solution space;
3. your skill as a DSL/language designer.

But first and foremost, it depends on your ability to recognize a situation in which ‘death by incidental complexity’ is likely to occur or even already occurring which in turn depends on the quality of your communication with the project (team). During one of my recent projects, I found myself in the situation that I didn’t (allow myself to) recognize such a situation until it was almost too late and a lot of effort had been wasted.

A tale from the crypt

This particular project entailed building a custom middleware Web application, which we were luckily able to do rather successfully and efficiently using Model-Driven Software Development. The central use case for this application consisted of quite a lot of complicated screen, with one screen being extremely complex…and becoming more and more complex as time progressed, due to an liberal amount of change requests (scope creep, anyone?). The screen was complex for a number of reasons:

1. it was big: the number of possible input fields, buttons, etc. almost ran to the triple digits;
2. it was highly dynamic: depending on values of certain input fields, check boxes and such, other input fields were or were not visible and/or editable;
3. no roundtrips to the server were allowed for performance and usability reasons, which led to duplication of logic in both the server code (Java) as well as the client code (JavaScript);
4. the mapping from the object model to values on the screen and (especially) back, as well as roundtripping to other screens, was tedious;
5. testability was problematic and only partially automated.

To make matters worse: because it was one of the few screens which didn’t fit the modeling language used (UML2 + custom profiles), almost nothing of the screens functionality could be modeled so everything had to be hand-coded and hand-integrated with the rest of the application. In the process, the requirements for this screen were separated from the application model and documented in the usual amalgam of freeform Word and Excel documents.

So, over a period of about half a year a developer colleague of mine slaved away over this gargantuan screen for what must have been at least a full workload, guided and assisted much of the time by a business analyst colleague who was maintaining said Excel documents and who had previously come to like and rely on the rigor of modeling quite a lot. All this happened within eyeshot…which unfortunately didn’t actually prompt me to take some interest and see why this damned screen was taking a lot more time to complete than initially expected. To further implicate myself: I had implemented the initial version of the screen when it wasn’t very dynamic yet, and I didn’t really mind not being part of the actual developing effort. As karma usually has it: the developer colleague decided to continue his career with another company and ‘The Beast’ was handed back to me, together with a rather hefty bunch of change requests 😉 Despite being knowledgeable about our architecture, I couldn’t really make head or tails of the implementation which suffered from the essential and incidental complexities essentially being multiplied instead of added up -always a bad sign in (non-generated) code, as is an almost-exponential effort curve for implementing changes.

So there I was with a few hundred K’s of (not so homogeneous) Java and JSP code serving a Web page with a lot of JavaScript code, plus some change requests for which I had very little idea of implementing in said code base. The first thing I usually do in these circumstances, is to Refactor away. After about a week, this Refactoring hardly made a dint in reducing the total complexity and only increased the quality of the JavaScript code a bit. Time for plan B, which was to mentally throw the existing solution out of the window and think about how a DSL solution (which was plausible given that we already were an MDSD shop) would look.

The Solution

For a couple of days I feverishly but happily worked on implementing a textual DSL (using Xtext, obviously) to capture the screens’ requirements and generate a Web page mockup from it, as that would allow the business analyst to validate her DSL instance functionally on the push of a button (literally). In total, it took me about 1.5 week to implement the DSL, a half-working mockup generator (with the mockups already being quite close to an actual implementation inside the application’s architecture), a DSL instance capturing about 80% of the screen in all its dynamic and purely essential detail plus some additional stuff to be able to interact with the entity model (in UML). When I showed what I had to my colleagues, the business analyst said ‘Please move over, I have a screen to finish!’ which is by far the nicest response I’ve ever had to anything I’ve created. Unfortunately, the project was cancelled before we could progress from here, for reasons beyond our control, but not before having obtained a full GO from the project manager for replacing the existing implementation with a DSL-based one.

The most important thing I learned from this is that I should have taken an interest in what was going on a mere 1m from my desk which would have allowed me to recognize the potential of a DSL to help and get a grip on the complexity of the things at hand at an earlier stage which could have saved a lot of effort and head aches. Interesting is the fact that it took me only a few days to come up with a quite complex and rather mature DSL (having a sub-DSL for Boolean expressions, references of features of an entity model using path expressions, among various other not-so-trivial constructs) using Xtext even though it was my first DSL built with Xtext version 0.7.2 using not-so-trivial scoping -future posts will discuss and detail some of the features mentioned. This means that the ramp-up of a complete DSL+generator solution in these or similar circumstances can be measured in weeks rather than in months or longer. Also, progress was quite linear and number and complexity of features -no exponential curve or 20%-80%-rule in sight.

---

Addendum

Ron Kersic had already argued that ‘incidental complexity’ is a much better nomer than ‘accidental complexity’ since the primary semantics of incidental is ‘happening in connection with or resulting from something more important’ while that of accidental is ‘occurring by chance, unexpectedly, or unintentionally’. Although I agreed with him, ‘accidental complexity’ is a reasonably established concept…until Fred Brooks chose to forego on it and use ‘incidental complexity’ instead. So, I happily replaced all occurrences of ‘accidental’ with ‘incidental’ in the body of the blog. And thanks again to Ron for pointing this out 🙂