When to solve stuff with a DSL

One of the great benefits of DSLs (textual or graphical alike), is that it allows you to separate essential complexity, i.e. complexity which is intrinsic to the problem you’re trying to solve, from incidental complexity, i.e. complexity which is caused by the approach chosen to solve that problem. This is certainly the case for external DSLs, since these often evolve from a ‘technical clean slate’. Whether you’re actually able to achieve that separation depends on a couple of things:

1. the measure in which you understand the problem space or domain;
2. the measure in which you understand the solution space;
3. your skill as a DSL/language designer.

But first and foremost, it depends on your ability to recognize a situation in which ‘death by incidental complexity’ is likely to occur or even already occurring which in turn depends on the quality of your communication with the project (team). During one of my recent projects, I found myself in the situation that I didn’t (allow myself to) recognize such a situation until it was almost too late and a lot of effort had been wasted.

A tale from the crypt

This particular project entailed building a custom middleware Web application, which we were luckily able to do rather successfully and efficiently using Model-Driven Software Development. The central use case for this application consisted of quite a lot of complicated screen, with one screen being extremely complex…and becoming more and more complex as time progressed, due to an liberal amount of change requests (scope creep, anyone?). The screen was complex for a number of reasons:

1. it was big: the number of possible input fields, buttons, etc. almost ran to the triple digits;
2. it was highly dynamic: depending on values of certain input fields, check boxes and such, other input fields were or were not visible and/or editable;
3. no roundtrips to the server were allowed for performance and usability reasons, which led to duplication of logic in both the server code (Java) as well as the client code (JavaScript);
4. the mapping from the object model to values on the screen and (especially) back, as well as roundtripping to other screens, was tedious;
5. testability was problematic and only partially automated.

To make matters worse: because it was one of the few screens which didn’t fit the modeling language used (UML2 + custom profiles), almost nothing of the screens functionality could be modeled so everything had to be hand-coded and hand-integrated with the rest of the application. In the process, the requirements for this screen were separated from the application model and documented in the usual amalgam of freeform Word and Excel documents.

So, over a period of about half a year a developer colleague of mine slaved away over this gargantuan screen for what must have been at least a full workload, guided and assisted much of the time by a business analyst colleague who was maintaining said Excel documents and who had previously come to like and rely on the rigor of modeling quite a lot. All this happened within eyeshot…which unfortunately didn’t actually prompt me to take some interest and see why this damned screen was taking a lot more time to complete than initially expected. To further implicate myself: I had implemented the initial version of the screen when it wasn’t very dynamic yet, and I didn’t really mind not being part of the actual developing effort. As karma usually has it: the developer colleague decided to continue his career with another company and ‘The Beast’ was handed back to me, together with a rather hefty bunch of change requests 😉 Despite being knowledgeable about our architecture, I couldn’t really make head or tails of the implementation which suffered from the essential and incidental complexities essentially being multiplied instead of added up -always a bad sign in (non-generated) code, as is an almost-exponential effort curve for implementing changes.

So there I was with a few hundred K’s of (not so homogeneous) Java and JSP code serving a Web page with a lot of JavaScript code, plus some change requests for which I had very little idea of implementing in said code base. The first thing I usually do in these circumstances, is to Refactor away. After about a week, this Refactoring hardly made a dint in reducing the total complexity and only increased the quality of the JavaScript code a bit. Time for plan B, which was to mentally throw the existing solution out of the window and think about how a DSL solution (which was plausible given that we already were an MDSD shop) would look.

The Solution

For a couple of days I feverishly but happily worked on implementing a textual DSL (using Xtext, obviously) to capture the screens’ requirements and generate a Web page mockup from it, as that would allow the business analyst to validate her DSL instance functionally on the push of a button (literally). In total, it took me about 1.5 week to implement the DSL, a half-working mockup generator (with the mockups already being quite close to an actual implementation inside the application’s architecture), a DSL instance capturing about 80% of the screen in all its dynamic and purely essential detail plus some additional stuff to be able to interact with the entity model (in UML). When I showed what I had to my colleagues, the business analyst said ‘Please move over, I have a screen to finish!’ which is by far the nicest response I’ve ever had to anything I’ve created. Unfortunately, the project was cancelled before we could progress from here, for reasons beyond our control, but not before having obtained a full GO from the project manager for replacing the existing implementation with a DSL-based one.

The most important thing I learned from this is that I should have taken an interest in what was going on a mere 1m from my desk which would have allowed me to recognize the potential of a DSL to help and get a grip on the complexity of the things at hand at an earlier stage which could have saved a lot of effort and head aches. Interesting is the fact that it took me only a few days to come up with a quite complex and rather mature DSL (having a sub-DSL for Boolean expressions, references of features of an entity model using path expressions, among various other not-so-trivial constructs) using Xtext even though it was my first DSL built with Xtext version 0.7.2 using not-so-trivial scoping -future posts will discuss and detail some of the features mentioned. This means that the ramp-up of a complete DSL+generator solution in these or similar circumstances can be measured in weeks rather than in months or longer. Also, progress was quite linear and number and complexity of features -no exponential curve or 20%-80%-rule in sight.

---

Addendum

Ron Kersic had already argued that ‘incidental complexity’ is a much better nomer than ‘accidental complexity’ since the primary semantics of incidental is ‘happening in connection with or resulting from something more important’ while that of accidental is ‘occurring by chance, unexpectedly, or unintentionally’. Although I agreed with him, ‘accidental complexity’ is a reasonably established concept…until Fred Brooks chose to forego on it and use ‘incidental complexity’ instead. So, I happily replaced all occurrences of ‘accidental’ with ‘incidental’ in the body of the blog. And thanks again to Ron for pointing this out 🙂