A Generic Application Language

This blog is a follow-up to a previous one with, as promised, some of my thoughts on a Generic Application Language (GAL). This is about what I personally would like to see in a modeling/programming language and its tooling, based on my experience actually building applications. As a consequence, the following will be extremely opinionated and in places downright rude. Deal with it or go away! 😉

I’ll start with pitching a few terms which I would have apply to the GAL. Some of these are purely oriented towards functionality, some are technical, some none of the previous. Here goes…

TL;DR: this explains what a language for application development and its IDE should look like

The tooling actually supports software development

Many IDEs are essentially not much more than glorified file tree navigators that open files in editors, with a lot of micro services tacked on top of it. This is great for configurability, but that also means that you actually have to configure it. It typically takes an inordinate amount of work to marry up your team’s/company’s workflow (version control, reviews, Agile, etc.) and the particular architecture of your code base to the IDE. So much even, that I’ve never seen it done to any degree of intimate satisfaction – wait, that sounds wrong, doesn’t it? Usually, this results in many manual actions and/or lengthy and cumbersome build cycles, hindering feedback/validation for the developer. (In all fairness, IDEs on the level of IDEA/IntelliJ provide reasonably smooth sailing these days.)

So, some features I want to see (roughly in that order):

1. Truly smooth integration (and configuration of, if/when required) with workflow, including Continuous Build and Deployment.
2. Live coding everywhere. Point to some code, give it some input, see what happens, debug it. No mucking about with making tests, providing all dependencies by hand  or running the entire application, setting a breakpoint and hoping for the best. Upgrade a live coding session to a test afterwards – unit or integrated, whatever you wanna call it; I don’t care.
3. Being able to reason about the code base within the tool itself. Run a query that’s language- and context-aware on your code base. Define code conventions and  architecture that way. Do impact analyses. Verify assumptions and invariants. Also: these things should be part of the code base itself, versioned and all.
4. Advanced and automatic Refactoring suggestions. If you have two sub classes with identical members, surely an IDE should be able to inform you of that and offer help? Recognise an Anti-Pattern as soon as it crops up. The possibilities are effectively endless, especially when you get to add your own wizard thingies.

Static typing

I know a lot of people experience pronounced localised vascular throbbing because of thinking dynamic languages are intrinsically way better than statically-typed ones, but I’m entirely willing to call those out as innophobic, pseudo-religiously insane bastards (m/f). Static typing helps. A lot. No amount of TDD can compensate for the lack of API discovery, content assist and avoiding stupid-but-hard-to-detect bugs – all things made possible by having a real type system. Studies show it, so get over it.

Sure, you need the following to be in place:

1. A healthy type system. This means reified generics. It does, however, not mean full recursiveness. Because that’s called Reflection-induced Hell in actual practice. Because someone will use it – probably yourself, thinking you’re smart. Same goes for meta programming: easy to abuse, hard to maintain by the rookie once the meta cowboy has moved on to fresher pastures, makes reasoning about your code base nigh impossible. ‘Nough said.
2. A good type inference engine. Reduce syntactic noise: no-one wants another Java. Scala and Xtend already show the way.
3. Polymorphism. Comes quite naturally for dynamic languages but is actually not so hard for statically-typed ones. (Xtend puts it on top of Java essentially effortlessly, e.g.)
4. Tags/tagged values/annotations. A lightweight, orthogonal way of tagging parts of your code and adorning them with extra info. Crucial to those LISP-like macros I’m going to be talking about later.

As long as you can keep the type system reasonable (Scala not being a good example), this is something that’s somewhat tedious and not trivial but also not frightfully difficult. Once you have these you can start thinking about using proof assist software such as Coq could help to reason about null safety, arguments validity, reachability and such. This ties in naturally with points 3 and 4 of the previous section.

Integration of front- and backend

Most applications today are Web applications. Typically, this means that front- and backend run on platforms that are technologically effectively disjoint: e.g. HTML5 vs. JEE. Equally typically, front- and backend are blissfully unaware of their respective details and any change to either requires patches up the other (1) by hand and (2) in several places.

To add insult to injury: if the architecture is set up with this in mind, the code on either end will look quite similar which begs the question why you can’t transpile from say, Java to JavaScript. (If it isn’t: well, good luck with the mess you got yourself in…) Also: tests are confined to their platforms, so these will not alert you to the fact that you’ve missed something.

Surely, in this day and age we can create a language that unifies both aspects using static typing. Google’s Web Toolkit got this, as does Safely-Typed JavaScript. Oh, this also means you could do tests that cross over that Great Divide.

Projectional editing

If you want to play around with a rectangular grid of characters, go play WordFeud™ (or Scrabble™). If you think free form text is the best thing since sliced bread, break out and dust off your grandfather’s type writer and bash out a novel.

In order to make my case, let’s consider changing exactly one character in the entire code base and assume that your entire application is not immediately breaking either because it doesn’t compile anymore or because it plainly blows up in your or someone else’s face when running it. Then, in decreasing order of likelihood:

1. You’re replacing whitespace with whitespace. This means you’re either a fascist or you’ve got a severe case of OCD. Go see either a criminal court or an episode of tBBT.
2. It’s inside a comment. Why is that comment there? And why isn’t it statically checked? In any case, probably endemic of a bigger problem.
3. You’re commenting out code. Planning to commit that, ‘guv?
4. It’s inside a text that’s purely for human consumption. Is it internationalised? Are you secretly writing a novel after all? If so, better go and search for that Underwood.
5. It’s inside a magic value. Hopefully you made that fact explicit. Otherwise, good luck with detecting and fixing the bugs that will inevitably result from that magic.
6. It’s inside dead code (provided the code even compiles). Why is it still there? Why didn’t your IDE alert you to that?
7. You’re just lucky. Well, for now at least. You’ll have some Heisenbugs later on, don’t worry.

So, why going to the trouble of give your developers the freedom to type in essentially random characters in random places, slapping them on the wrist through validation or trying to make up for the mess by means of syntax checking, highlighting, content assist, etc. after the fact? What more is an IDE than an application that visualises a code base and provides editability and other micro services with a defined behaviour on it? Is editor behaviour then not a matter of UX design? Even the command line guys are exploring that  direction.

(Another disadvantage of the textual format is that it leads to code layout wars – see also point 1 above. This applies less to Go and Python, and not at all to Brainf*ck, though.)

Oh, and you could even consider text to be a projection of the original code’s AST to some extent.

Once you’ve got projectional editing working as such, the following features are fairly easy to pull off (and although not impossible, extremely difficult to do for purely textual notations):

1. Multiple projections. The same AST can be visualised in different ways, with altering behavior of the micro services.
2. Great syntax and layout. Once you get started, there’s no end to how good you can make code look. Goodbye to mono-spaced fonts and CGA-era colour schemes.
3. Language embedding. Really difficult for the textual case – I know of only one tool that’s able to do that and it’s pretty much academic in every sense. Much, much easier for projectional editing. One especially important use case for this is LISP-like macros/templates. More on that below.
4. Context-aware editing. Got to put in a date here? Date picker! Same for colours, easy. This often obviates the need for language embedding.

Creating DSLs with macros/templates

I already alluded to this in a previous post, but this requires more explanation and examples. The general idea here is to be able to create code from code, in a way that allows you to gradually move from the level of abstraction that you want to the level that you (or the platform) need. (I guess you could also call this “desugaring”.) I’m going to talk about it in another post, because this post is long enough as it is and I think the idea deserves its own podium.

Functional programming

This is a bit of a no-brainer these days, luckily. Many GPLs are already firmly going that way, with 1st-class support for immutability, closures (I guess “lexically-scoped coroutines” to some), lazy evaluation, etc.. To me the most important take-aways from FP are (in that order!):

1. A good expression sub-language that also innately covers querying and comprehending of collections(/streams), is extremely powerful – essential, even.
2. Expressions are easier than statement (block)s. “Everything is an expression” is easy to achieve. Expressions can easily be upgraded to closures.
3. Pure functions are easy to reason about.
4. Pure functions are trivial to parallelise. In fact, it’s pretty much the only productive way to do actual concurrency/parallelism.

(Note that there’s a distinction between expressions and functions.) In fact, you can view an(y) application as a function of a stream of inputs to a stream of outputs/effects. This notion might not help matters when applying it to the application as a whole, but it certainly can when applied to individual layers of it. E.g., a UI is for the most part a function of application (its data) and client-specific state (which screen, etc.).

Things that are less important to me:

1. Monads. If you need them to make software development easier, you’re missing the point. That a lot of things are easier to reason about once you realise they’re monads, does not imply that you need to forcefully make everything a monad before they could be made to work. Use/apply them where it makes sense, don’t where it doesn’t. “But: side effects!”, you say. “Where?”, I say. It’s extremely useful to make side effects explicit. So much so that you should really consider creating a 1st-class language construct (with decent, non-leaky semantics) for that, instead of providing your developers with a cabinet full of Hello Kitty!™-themed bandage to plaster them over with.
2. While we’re at it, and pretty much for the same reasons: category theory. I know Erik Meijer does magical things with it, but face it: he’s pretty much the only guy that knows how to do that in real life. Again: use/apply it where useful, don’t where it doesn’t and certainly don’t make it a prerequisite for joining ye’olde Clubb.
3. Immutability as an opt-in concept. I see more value in explicitly specifying lifecycle and mutability only where it’s intrinsically relevant (e.g., in the data model). In expressions, immutability is the default and lifecycle irrelevant.
4. Whatever you can name as “super important, enabling concept that can only be found in Our Tool™”. (With the exception, of course, of anything intrinsic about the GAL ;)) I’ll be eternally puzzled by product descriptions along the lines of: “Our Tool™ allows business users to do X. That’s because we’ve built it according to vary-vague-and-academic-principle Y!” Huh? Interest in X and understanding of Y is mutually exclusive, so who should be enthused by this pitch, exactly? Exactly…

Reactiveness

Hey, there’s Erik Meijer again! 🙂 To me, reactiveness is several things at the same time:

• A philosophy.
• An extension of FP: everything is a stream of things and functions thereof.
• An implementation detail of FP: there’s an obvious mapping between pure functions on data to reactive streams of “delta events” on that data.

The latter two points especially pertain to modeling UIs – more on that in a later blog post (perhaps).

 

An incovenient dichotomy: specific vs. generic

Recently, I had a bit of a realisation: the LISPers of this world have a point.

But before explaining that, a question: how much of your application is really specific to your business domain? The meat of the realisation is that the answer typically is: not all that much.

This is because by the nature of applications and how they are embedded in their environments, they are not purely descriptions of the business domain but rather transformations of that domain interspersed with more “mundane” details like UX (design, style, layout, navigation, etc.), authentication, authorisation, validation, persistence, external interfaces, internationalisation, concurrency (these days: asynchrony), error handling, etc. So, however you measure the size of your application’s code base (lines of code, zipped megabytes, etc.), either a very small portion of it will actually represent your business domain authoritatively or the details of the business domain are duplicated and spread out across the entire code base.

Consider as an example a Web application to sell financial products. Such products typically come with a lot of details and business rules, such as applicability (essentially validations on who can buy a product), their actual semantics (i.e., a specification of what should happen when someone buys that product), etc., adorned with “stuff” that exists primarily for people to read. The majority of the Web application does not pertain to any of that, but is rather about guiding prospective customers in some way to the product they want/need, providing general information about the company, points of contact, etc. As for the financial products themselves: the semantics will only be executed on a mainframe backend, while the applicability would translate into some kind of questionnaire checking for that.

So, on the one hand modeling the financial products provides all kind of benefits since you can derive the questionnaire and semantics from the model. Having a DSL to do that modeling makes sense since each financial business domains typically use very specific concepts and jargon. On the other hand, creating a Web application is hardly domain-specific in any way: the number of concepts involved is quite large, but they hardly vary in essence over a broad range of applications. Of course, how these concepts are applied in a specific application should be quite restricted by means of a uniform/house style and an application architecture.

In my experience, it does not pay to create app-specific DSLs capturing this generic range of concepts. (*gasp* Did I just say: “Don’t build DSLs.”?! Yes, I did.) It is simply a lot of work: you have to determine which concepts to capture, taking into account possible extensions, changes to the technology stack, etc. You typically also need an underlying type system and powerful expressions sub-language. Then you need a generator or interpreter tying in with the underlying architecture, technology stack and application environment. This is not trivial stuff and certainly not for the general developer, even with the capabilities of modern language workbenches.

Instead, creating applications should be done through a generic application language which supports raising levels of abstractions essentially continuously. And this is where LISP comes in: not that LISP is this GAL (or should be), but we could, and should, be inspired by the concept of macros. (I say levels of abstraction, since I also realised that there is not one level to abstract over. Rather, at various points in your code there are various axes to abstract over.)

A lot of constructs in general purpose languages are essentially about raising that level of abstraction, or can be used in that way. E.g., (non-polymorphically called) methods/functions are just code blocks with a name to them, inheritance is another way to avoid duplicating code, etc. These constructs force you to raise the level of abstraction by reasoning from the GPL constructs towards the intended, or naturally-suited level of abstraction.

LISP-style macros, on the other hand, work in the opposite direction: they are effectively templates transforming some code into code that exhibits a lower level of abstraction. Provided that you can chain such transformations together to arrive at code that’s actually executable, you can vary the level of abstraction incrementally and essentially continuously. Note that I say “code”, not “data”: making a distinction there would force us to choose where data ends and where code starts – not only is that a hard topic, it’s also not relevant for our purposes!

To go back to our example: the financial products could be modeled as GAL code which is then transformed into parts of the Web application. You can view the modeling GAL code as an internal DSL. The GAL tooling could even help to expose this internal DSL as an external DSL.

So, is LISP this GAL? No. LISP is a very nice language but no amount of tooling is going to be able to general developer to become productive with it while also keeping the LISP community itself on board. In a next blog, I’ll try to say more about what this GAL and its tooling should look like.

In conclusion: specific vs. generic is a false dichotomy. Rather, we need to be able to raise levels of abstraction in a meaningful way, i.e. incrementally and essentially continuously. That way, we can leverage genericity while still embracing specificity.