Object algebras in Xtend

I finally found/made some time to watch the InfoQ video shot during Tijs van der Storm‘s excellent presentation during the Dutch Joy of Coding conference 2014, on object algebras. Since Tijs uses Dart for his code and I’m a bit of an Xtend junkie, I couldn’t resist to port his code. Happily, this results in code that’s at least as short and readable as the Dart code, because of the use of Xtend-specific features.

Object algebras…

In my understanding, object algebras are a way to capture algebraic structures in object-oriented programming languages in a way that allows convenient extensibility in both the algebra (structure) itself as well as in the behavior/semantics of that algebra – at the same time, even.

Concretely, Tijs presents an example where he defines an initial algebra, consisting of only integer literals and + operations. This allows you to encode simple arithmetic expressions inside of the host programming language – i.e., as an internal DSL. Note that these expressions result in object trees (or ASTs, if you will). Also, they can be seen as an example of the Builder Pattern. For this initial algebra, Tijs provides the typical print and evaluation semantics. Next, he extends the algebra with multiplication and also provides the print and evaluation semantics for that.

All of this comes at a cost. In fact, two costs: a syntactical one and a combinatorial one. The syntactical cost is that “1 + 2” is encoded as “a.plus(a.lit(1), a.lit(2))” – at least in the Dart code example. Luckily, Xtend can help here – see below. The combinatorial cost (which seems to be intrinsic to object algebras) is that for every combination of algebra concept (i.c.: literal, plus operation, multiplication operation) and semantics (i.c.: print, evaluation) we need an individual class – although these can be anonymous if the language allows that.

Despite the drawbacks, object algebras do the proverbial trick in case you’re dealing with object trees/builders/ASTs in an object-oriented, statically-typed language and need extensibility, without needing to revert to the “mock extensibility” of the Visitor Pattern/double dispatch.

…in Xtend

…it looks like this. Go on, click the link – I’ll wait 🙂 Note that GitHub does not know about Xtend (yet?) so the syntax coloring is derived from Java and not entirely complete – most notably, the Xtend keywords def, override and extension are not marked as such.

To start with the biggest win, look at lines 80 and 142. Instead of the slightly verbose “a.plus(a.lit(1), a.lit(2))” you see “lit(1) + lit(2)”. This is achieved by marking the function argument of type ExpAlg/MulAlg with the extension keyword. This has the effect that the public members of ExpAlg/MulAlg are available to the function body without needing to dereference them as “a.”. In general, Xtend’s extension mechanism is really powerful especially when used on fields in combination with dependency injection. In my opinion, it’s much better than e.g. mucking about with Scala’s implicit magic, precisely because of the explicitness of Xtend’s extension.

Another win is the use of operator overloading so we can redefine the + and * operators in the context of ExpAlg/MulAlg, even usual the actual tokens: see lines 18 and 105. Further nice features are:

• The use of the @Data annotation on classes to promote these to Value Objects, with suitable getters, setters and constructors generated automatically. Unfortunately, the use of the @Data annotation does not play nice with anonymous classes which were introduced in Xtend 2.6. So in this case, the trade-off would be to have less explicit classes versus more code in each anonymous class. In the end, I chose to keep the code close to the Dart original.
• No semicolons 😉
• Parentheses are not required for no-args function calls such as constructor invocations; e.g., see lines 39, 45 and 90.
• Nice templating using decidedly non-Groovy syntax that is less likely to require escaping and also plays nice with indentation; see e.g. line 39.

All in all, even though I liked the Dart code, I like the Xtend version more.

Addendum: now with closures

As Tijs himself pointed out on Twitter, we can also use closures to do away with classes, whether they are explicit or anonymous depending on your choice of implementation language or style. This is because closures and objects are conceptually equivalent and concretely because the Xtend compiler does three things:

• It turns closures into anonymous classes.
• It tries to match the type of the closure to the target type, i.e.: it can coerce to any interface or abstract class which has declared only one abstract method. In our case that’s the print and eval methods of the respective interfaces.
• It declares all method arguments to be final to match the functional programming style. As a result, the parameters of the factory methods are effectively captured by the closure.

(Incidentally, I’ve made examples of this nature before.)

The resulting code can be found here. It completely does away with explicit and anonymous classes apart from the required factory classes, saving 40 lines of code in the process. (The problem with the @Data annotation naturally disappears with that as well.) Note that we have to make explicit that the closures take no explicit arguments, only arguments captured from the scope, by using the “[| ]” syntax (nothing before |) or else Xtend will infer an implicit argument of type Object – see e.g. line 31.

A slight drawback of the closure approach is that it not only seals the details (i.e., the properties’ values – this is a good thing) but also hides them and that it limits extensibility to behavior that can be expressed in exactly one method. E.g., to do introspection on the objects one has to define a new extension: see lines 124-. Note that this make good use of the @Data annotation after all: both the constructor and a useful toString method are generated.

Using Xtend with Google App Engine

I’ve been using Xtend extensively for well over a year – ever since it came out, basically.

I love it as a Java replacement that allows me to succinctly write down my intentions without my code getting bogged down in and cluttered with syntactic noise – so much so that my Xtend code is for a significant part organized as 1-liners. The fact that you actually have closures together with a decent syntax for those is brilliant. The two forms of polymorphic dispatch allow you to cut down on your OO hierarchy in a sensible manner. Other features like single-interface matching of closures and operator overloading are the proverbial icing on the cake. The few initial misgivings I had for the language have either been fixed or I’ve found sensible ways to work around them. Over 90% of all my JVM code is in Xtend these days, the rest mostly consisting of interfaces and enumerations.

One of the things I’ve been doing is working on my own startup: Más – domain modeling in the Cloud, made easy. I host that on Google App Engine so I’ve some experience of using Xtend in that context as well. Sven Efftinge recently wrote a blog on using Google Web Toolkit with Xtend. Using Xtend in the context of GWT requires a recent version of Xtend because of the extra demands that GWT makes on Java code (which is transpiled from Xtend code) and Java types in order to ensure it’s possible to transpile to JavaScript and objects are serializable. However, when just using Xtend on the backend of a Google App Engine, you don’t need a recent version. However, you do need the “unsign” a couple of Xtend-related JAR files because otherwise the hosted/deployed server will trip over the signing meta data in them.

To use Xtend in a Google Web Application, do the following:

1. After creating the Web Application project, create an Xtend class anywhere in the Java src/ folder.
2. Hover over the class name to see the available quickfixes. Alternatively, use the right mouse click menu on the corresponding error in the Problems view.
3. Invoke the “Add Xtend libs to classpath” quickfix by selecting it in the hover or by selecting the error in the Problems view, pressing Ctrl/Cmd 1 and clicking Finish.
4. At this point, I usually edit the .classpath file to have the xtend-gen/ Java source folder and Xtend library appear after the src/ folder and App Engine SDK library entry.
5. Locate the com.google.guava, org.eclipse.xtext.xbase.lib and org.eclipse.xtend.lib JAR files in the plugins/ folder of the Eclipse installation.
6. Unsign these (see below) and place the unsigned versions in the war/WEB-INF/lib/ folder of the Web Applcation project.

Now, you’re all set to use Xtend in a GAE project and deploy it to the hosted server. In another post, I’ll discuss the benefits of Xtend’s rich strings in this context.

Unsigning the Xtend libs

The following (Bourne) shell script (which kinda sucks because of the use of the cd commands) will strip a JAR file of its signing meta data and create a new JAR file which has the same file name but with ‘-unsigned’ postfixed before the extension. It takes one argument: the name of the JAR file without file extension (.jar).

#!/bin/sh
unzip $1.jar -d tmp_jar/
cd tmp_jar
rm META-INF/ECLIPSE_.*
cp MANIFEST.MF tmp_MANIFEST.MF
awk '/^$/ {next} match($0, "(^Name:)|(^SHA1-Digest:)") == 0 {print $0}' tmp_MANIFEST.MF > MANIFEST.MF
# TODO  remove empty lines (1st clause isn't working...)
rm tmp_MANIFEST.MF

zip -r ../$1-unsigned.jar .
cd ..
rm -rf tmp_jar/

# Run this for the following JARs (with version and qualifier):
#    com.google.guava
#    org.eclipse.xtend.lib
#    org.eclipse.xtext.xbase.lib

Note that using the signed Xtend libs (placing them directly in war/WEB-INF/lib/) isn’t a problem for the local development server, but it is for the (remote) hosted server. It took me a while initially to figure out that the cryptic, uninformative error message in the logs (available through the dashboard) actually mean that GAE has a problem with signed JARs.

And yes, I know the unsign shell script is kind-of ugly because of the use of the cd command, but hey: what gives…

A trick for speeding up Xtend building

I love Xtend and use it as much as possible. For code bases which are completely under my control, I use it for everything that’s not an interface or something that really needs to have inner classes and such.

As much as I love Xtend, the performance of the compilation (or “transpilation”) to Java source is not quite on the level of the JDK’s Java compiler. That’s quite impossible given the amount of effort that has gone into the Java compiler accumulated over the years and the fact that the team behind Xtend is only a few FTE (because they have to take care of Xtext as well). Nevertheless, things can get out of hand relatively quickly and leave you with a workspace which needs several minutes to fully build  and already tens of seconds for an incremental build, triggered by a change in one Xtend file.

This performance (or lack thereof) for incremental builds is usually caused by a lot of Xtend source  interdependencies. Xtend is an Xtext DSL and, as such, is aware of the fact that a change in on file can make it necessary for another file to be reconsidered for compilation as well. However, Xtend’s incremental build implementation is not (yet?) always capable of deciding when this is the case and when not, so it chooses to add all depending Xtend files to the build and so forth – a learned word for this is “transitive build behavior”.

A simple solution is to program against interfaces. You’ve probably already heard this as a best practice before and outside of the context of Xtend, so it already has merits outside of compiler performance. In essence, the trick is to extract a Java interface from an Xtend class, “demote” that Xtend class to an implementation of that interface and use dependency injection to inject an instance of the Xtend implementation class. This works because the Java interface “insulates” the implementation from its clients, so when you change the implementation, but not the interface, Xtend doesn’t trigger re-transpilation of Xtend client classes. Usually, only the Xtend implementation class is re-transpiled.

In the following I’ll assume that we’re running inside a Guice container, so that the Xtend class is never instantiated explicitly – this is typical for generators and model extensions, anyway. Perform the following steps:

1. Rename the Xtend class to reflect it’s the implementation class, by renaming both the file and the class declaration itself, without using the Rename Refactoring. This will break compilation for all the clients.
2. Find the transpiled Java class corresponding to the Xtend class in the xtend-gen/ folder. This is easiest through the Ctrl/Cmd-Shift-T (Open Type) short cut.
3. Invoke the Extract Interface Refactoring on that one and extract it into a Java interface in the same package, but with the original name of the Xtend class.
4. Have the Xtend implementation class implement the Java interface. Compilation should become unbroken at this point.
5. Add a Guice annotation to the Java interface:

   @com.google.inject.ImplementedBy(...class literal for the Xtext implementation class...)
   

Personally, I like to rename the Xtend implementation class to have the Impl postfix. If I have more Xtend classes together, I tend to bundle them up into a impl sub package.

Of course, every time the effective interface of the implementation class changes, you’ll have to adapt the corresponding interface as well – prompted by compilation errors popping up. I tend to apply this technique only as soon as the build times become a hindrance.

Groovy-type builders and JSON initializers in Xtend

One of the nicer features of dynamic languages like Ruby, Groovy, etc. is the possibility to easily implement builders which are constructs to build up tree-like structures in a very succinct a syntactically noise free way. You can find some Groovy examples here – have a special look at the HTML example. Earlier, Sven Efftinge has written a blog on the implementing the same type of builders using Xtend. My blog post will expand a little on his post by providing the actual code and another example.

The main reason that builders come naturally in dynamic languages is that metaprogramming allows adding “keywords” to the language without the need to actually define them in the form of functions. In the case of HTML, these “keywords” are the tag names. Statically-typed languages like Java or Xtend do not have that luxury (or “luxury” as the construct can easily be misused) so we’ll have to do a little extra.

The HTML example

You’ll find Sven’s original example reproduced in HtmlDocumentExample . Note that because of the point mentioned above, we need to have compile-time representations of the HTML DOM elements we’re using. Sven has written these manually but I’m afraid that I’m lazy to do that so I opted for a generative approach. Apart from that, the example works exactly the same, so I’ll refer to his original blog for the magic details – some of which I’ll re-iterate for the JSON example below.

Generation

Some basic HTML DOM element types are provided by the BaseDomElements file. Note two nice features of Xtend 2.3+: one file can hold multiple Xtend classes and the use of the @Data annotation as a convenient way to define POJOs – or should those be called POXOs? 😉 The POJOs for the other DOM elements are generated by the GenerateDomInfrastructure main class: note they are generated as POXOs which are then transpiled into Java. After running the GenerateDomInfrastructure main class and refreshing the Eclipse project, the  HelpDocumentExample class should compile.

A JSON example

You can find the JSON example in JsonResponseExample. For convenience and effect, I’ll reproduce it here:

class JsonResponseExample {

    @Inject extension JsonBuilder

    def example() {
        object(
            "dev"        => true,
            "myArray"    => array("foo", "bar"),
            "nested"     => object("answer" => 42)
        )
    }

}

To be able to compile this example, you’ll need my fork of Douglas Crockford’s Java JSON library with the main differences being that it’s wrapped as an Eclipse plug-in/OSGi bundle and it’s (as properly as manageable) generified. In addition to the GitHub repo, you can also directly download the JAR file.

As with the HTML example, the magic resides in the line which has a JsonBuilder Xtend class Guice-injected as an extension, meaning that you can use the functions defined in that class without needing to explicitly refer to it. This resembles a static import but without the functions/methods needing to be static themselves. The JsonBuilder class has two factory functions: object(..) builds a JSONObject from key-value pairs and array(..) builds a JSONArray from the given objects.

The fun part lies in the overloading of the binary => operator by means of  the operator_doubleArrow(..) function which simply returns a Pair object suitable for consumption by the object(..) factory function. This allows us to use the “key => value” syntax demonstrated in the example. Note that the => operator has no pre-existing meaning in Xtend – the makers of Xtend have been kind enough to provide hooks for a number of such “user-definable” operators: see the documentation.

I’m still trying to find a nice occasion to use the so-called “spaceship” or “Elvis” operators: now that would be positively…groovy 😉

Postscript

I failed to notice earlier that Xtend already has an -> operator which does exactly the same thing that our => operator does. Also, the “Elvis” ?: operator already has a meaning: “x ?: y” returns x if it’s non-null and y otherwise. This makes for a convenient way to set up default values. You’ll find both overloading definitions in the org.eclipse.xtext.xbase.lib.ObjectExtensions class.

Polymorphic dispatch in Xtend

Polymorphic dispatch (or http://en.wikipedia.org/wiki/Multiple_dispatch or multimethods as its also called) is a programming language construct which chooses a code path based on runtime types instead of types that are inferred at compile. The poor man’s method of achieving such behavior would be to litter your code with prose like this:

if( x instanceof TypeA ) { (x as TypeA).exprA }
else if( x instanceof TypeB ) { (x as TypeB).exprB }
else ...

Xtend 2.x offers two much better constructs to do polymorphic dispatching:

1. Through the use of the dispatch modifier for function defs – this construct is Xtend’s “official” polymorphic dispatch.
2. Through the use of the switch statement and referring to types instead of cases.

These are better than the poor man’s method because they are declarative, i.e.: they express intent much more clearly and succinctly. Though both constructs have a lot in common, there are some marked differences and Best Practices for safe guarding type safety which I’ll discuss in the blog.

The example

Consider the following Xtend code – note that the syntax coloring is lacking a bit, but WordPress doesn’t fully understand Xtend – …yet…. Also note that since Xtend2.3 you can have more than one Xtend class in one file.

class CommonSuperType { ... }
class TypeA extends CommonSuperType { ... }
class TypeB extends CommonSuperType { ... }
class TypeC extends CommonSuperType { ... }
class UnrelatedType { ... }
class Handler {

  def dispatch foo(TypeA it) { it.exprA }
  def dispatch foo(TypeB it) { it.exprB }

  def bar(CommonSuperType it) {
    switch it {
      TypeA: it.exprA
      TypeB: it.exprB
    }
  }
}

For simplicity’s sake, let’s assume that exprA and exprB both return an int. The Xtend compiler generates two public methods in the Java class Handler – one for foo, one for bar. Both of these have the same signature: int f(CommonSuperType), where f = foo or bar. In addition, for each foo dispatch function, Xtend generates a public method with signature int _foo(t), where t=TypeA or TypeB – note the prefixed underscore. The actual polymorphic dispatch then happens in “combined” foo(CommonSuperType) method, actually through the previously demonstrated poor man’s method.

By the way: a user-friendly way to inspect the “combined” method is the Outline which will group the dispatch functions belonging together under the combined signature.

Note that the foo and bar method ends up with CommonSuperType as the type of its parameter. This is because CommonSuperType is the most specific common super type of TypeA and TypeB – deftly implied by the name – and Xtend infers that as the parameter’s type for the “combined” method. In general, Xtend will compute the most specific common super type across all dispatch functions, on a per-argument basis. In case of the bar method we declared ourselves what the parameter type is.

As demonstration, add the following code to the Handler class and see what happens:

  def dispatch foo(TypeC it) { it.exprC }

(Assume that exprC again returns an int.)

The generated foo and bar methods are functionally nearly identical, the difference being that foo explicitly throws an IllegalArgumentException mentioning the unhandled parameter type(s) in its message, in case you called it with something that is a CommonSuperType but neither of TypeA nor of TypeB. The bar method does no such thing and simply falls through the switch, returning the appropriate default value: typically null but 0 in our int-case. To remedy that, you’ll have to add a default case which throws a similar exception, like so:

  def bar(CommonSuperType it) {
    switch it {
      TypeA: it.exprA
      TypeB: it.exprB
      default:
        throw new IllegalArgumentException("don't how to handle sub type " + it.^class.simpleName)
    }
  }

In case you already have sensible default case, you’re basically out of luck.

Potential mistakes

Both approaches have their respective (dis-)advantages which I’ll list comprehensively below. In both cases, though, it’s relatively easy to make programmers’ mistakes. The most common and obvious ones are:

1. The parameter type of the “combined” foo method is inferred, so if you add a dispatch function having a parameter type which does not extend CommonSuperType, then the foo method will wind up with a more general parameter type – potentially Object. This means that the foo method will accept a lot more types than usually intended and failing miserably (through a thrown IllegalArgumentException) on most of them. This is especially dangerous for public (which happens to be the default visibility!) function defs.
2. Xtend will not warn you at editor/compile time about the “missing” case TypeC: it’s a sub type of CommonSuperType but not of TypeA nor of TypeB. At runtime, the bar method will simply fall through and return 0.
3. The return type of the combined method is also inferred as the most specific common super type of the various return types – again, potentially Object. This is usually much less of a problem because that inferred type is checked against the parameter type of clients of the combined method.

This shows that these constructs require us to do a little extra to safe guard the type safety we so appreciate in Xtend.

Advantages and disadvantages of both constructs

We list some advantages and disadvantages of both constructs. Advantages of the dispatch construct:

• Provides more visual code space. This is useful if the handling of the separate types typically needs more than 1 line of code.
• Explicit handling of unhandled cases at runtime.

Disadvantages of the dispatch construct:

• Automagically infers parameter types of the “combined” method as the most common super types. In case of a programmer error, this may be (much) too wide.
• Takes up more visual code space/more syntactic noise.

Advantages of the switch construct:

• Takes up less visual code space. This is useful if the handling of the separate types doesn’t need more than one line of code.
• It’s a single expression, so you can use it as such inside the function it’s living in. Also, you can precompute “stuff” that’s useful for more than one case.

Disadvantages of the switch construct:

• Fall-through of unhandled cases at runtime, resulting in an (often) non-sensical return value. You have to add an explicit default case to detect fall-through.

Mixing polymorphic and ordinary dispatch

Since Xtend version 2.3, you are warned about dispatch functions having a compatible signature as a non-dispatch function and vice versa. As an example, consider the following addition to the Handler class:

  def foo(TypeC it) { it.exprC  }
  def foo(UnrelatedType it) { it.someExpr }

Here, exprC again returns an int, but someExpr may return anything. Note that both functions are not of the dispatch persuasion.

The first line is flagged with the warning “Dispatch method has same name and number of parameters as non-dispatch method”, which is a just warning in my book. However, this warning is also given for the second line, as well as for the first two foo functions. (Note that the warnings are also given with only one of these extra functions present.) In that case, it’s not always a helpful warning but it does riddle your code file with warnings.

To get rid of the warnings, I frequently make use of the following technique:

• “Hide” all dispatch functions by giving them (and only these) an alternate name. My personal preference is to postfix the name with an underscore, since the extra _ it’s visually inconspicuous enough to not dilute the intended meaning. Also, give them private visibility to prevent prying eyes.
• Create an additional function with the same signature as the “combined” method for the dispatch functions, calling those.

The net result is that you get rid of the warnings, because there’s no more mixture of dispatch and non-dispatch functions with compatible signatures. Another upshot is that the signature of the “combined” method is now explicitly checked by the additional function calling it – more type safety, yeah! Of course, a disadvantage is that you need an extra function but that typically only is one line of code.

In the context of our example, the original two foo functions are replaced by the following code:

  def foo(CommonSuperType it) { foo_ }

  def private dispatch foo_(TypeA it) { it.exprA }
  def private dispatch foo_(TypeB it) { it.exprB }

Path expressions in entity models, revisited

Some 15 months ago, I wrote a blog detailing how to implement path-like expressions in Xtext DSLs using a custom scope provider. At lot has changed in the Xtext ‘verse since then, and I was triggered to update that blog by a comment on it. Also, the blog seems to be one of my more popular ones and I can refer to it every now and then on the Eclipse Xtext forum.

Most of what I wrote then which wasn’t particular to the ‘Xtext Domain-Model Example’ is still true: the scope provider API hasn’t changed (apart from IGlobalScopeProvider) and the way that scoping is triggered from the generated parser is fundamentally the same. However, the domain model example itself has changed a lot: it now serves to show (off) a lot of features that were introduced with Xtext 2 and Xtend(2). In particular, it relies heavily on Xbase by extending the grammar from Xtype (a sub language of Xbase) and extending the custom scope provider (in Xtend) and validator implementations (in Java) from their Xbase versions, meaning the DSL gets a lot of functionality essentially for free.

This also means that the grammar has changed enough from the original to make it rather delicate to adapt for my original intentions. In particular, the notions of Attribute and Reference have been clobbered together into the notion of Property which directly references a JVM type. To adapt the example I’d have to rely on classes like JvmFeatureDescriptionProvider in order not to re-invent the wheel, but I fear that bringing all that extra machinery is getting in the way of the idea I was trying to get across.

So, instead of that, I’ve opted for the easy way by starting from the original grammar and custom scope provider implementation, porting those over to Xtext 2.1(.0 or .1) and adapting those. In the process, I’ve tried to make the most of the latest, newest features of Xtext 2.x and especially Xtend: all custom code is done in Xtend, rather than Java.

For your convenience, I packed everything in ZIPs and uploaded them to my GitHub repository: complete Eclipse projects and only the essential files – I’m assuming you’ll find out how to interpret the latter in an Xtext context. For even more convenience, I exposed three of the essential files through GitHub’s gist feature: see below, in the running text.

It should be clearly discernible what I’ve added to the grammar to implement the path expressions. I’ve also added a PathElement convenience super type, just to make life a little easier (read: more statically-typed) on the Java/Xtend side of things.

I rewrote the scope provider implementation completely in Xtend. To be able to replace DomainModelScopeProvider.java completely with DomainModelScopeProvider.xtend, you’ll have to add “generateStub = false” to the configuration of the scoping generator fragment in the MWE2 workflow file – otherwise, you’ll get two generated DomainModelScopeProvider classes after running the worklow. Note that the implementations of the scope_Reference_opposite and scope_PathTail_feature are much more succinct and readable than they originally were, because of Xtend’s collection extensions and built-in polymorphic dispatch feature.

Also note that I made use of a separate Xtend file DomainModelExtensions implementing derived properties of model elements in a domain model. Xtend’s extension mechanism allows me to use these extensions transparently with no boilerplate at all apart from the extension declaration.

I also re-implemented the validator in Xtend. Because the Java validator generator fragment doesn’t have a generateStub parameter like the scoping fragment, we have to use a bit of boilerplate in the form of the DomainModelJavaValidator Java class extending the DomainModelXtendValidator Xtend class.

Lastly, I’ve implemented a couple of simple unit tests in Xtend using Xtext 2’s new JUnit4-ready unit testing capabilities. Note that it’s quite simple to do parameterized tests on a fairly functional level: see the ScopingTest Xtend class (in the .tests project) for details.

Using Xtext’s Xtend(2) language

The 2.0 release of Xtext also saw the introduction of the Xtend language. “Wait a minute, didn’t that already exist in Xtext 1.0 and openArchitectureWare 4.x before that?” I hear you saying. Indeed, the new language is should actually be called Xtend2 but since it’s meant to replace both Xtend(1) and Xpand, the makers have opted to drop the 2 suffix and assume that you’d know the difference. In any case, the languages differ in the file extensions used: .xtend for Xtend(2) and .xpt/.ext for Xpand/Xtend(1). Xpand and Xtend(1) are still part of the Xtext distribution, apparently for convenience, backwards compatibility and easy of migration, although there’s no support on these languages and execution engines any more. I also noted that the Xtext generator still relies heavily on Xpand/Xtend(1).

I’ve been using Xtend (i.e., Xtend2, but I’m going to drop the ‘2’ from now on as well) for some time now as a replacement for Xpand/Xtend -and even JSP- and I wanted to share my experiences and impressions -mostly good- with you and discuss a number of differences between Xpand/Xtend and Xtend.

The good

Xtend provides a decidedly functional programming-flavored abstraction over Java. Xtend files are compiled down to Java source code on-the-fly (courtesy of an Xtext builder participant – more on that later). The pros of this approach are performance, integration and debuggability using ye ole’ Java debugger. The generated Java code is fairly readable and 1-to-1 with the original Xtend code so tracing back to that is not really difficult, although full traceability would have been a boon here. I’ve never gotten into the groove of Xtend(1)’s custom debugger, preferring println()-style debugging over it. Compilation also means that you can refer to the compiled Java classes from outside of Xtend. It’s even possible to use Xtend as a type-safe replacement language for JSP.

The rich strings are by far the biggest improvement over the old Xpand/Xtend combination: the intelligent white space handling is nothing short of brilliant. They bring templates into the functional realm: a template is now “just” a function, so you can freely mix templates and “regular” code as it makes sense. Don’t forget that a juxtaposition of rich strings evaluates all but only returns the last, though – there’s no automagical concatenation.

It’s extremely easy to hook into the Xtext builder mechanism using a builder participant. In fact, this is the out-of-the-box situation so you only have to open the generated <MyDSL>Generator.xtend and implement the doGenerate function to have your UI language plug-in automagically fire up that generation on projects with the Xtext nature. Since the Xtext builder  works in an incremental manner, the generation is only triggered for the files which have been touched or which (transitively) depend on it, whereas it used to be “the whole shebang”. If you factored and modularized your language in a sensible way, this means that turnaround for the users of your generation is much, much quicker.

The extension mechanism works just as in Xpand/Xtend. Both the dispatch and create semantics are just a bit more general than their Xpand/Xtend counterparts which is good. I also like the as-syntax for casts: because of type inference, the end of an expression feels like a better place for the cast and it support the usual thought process (“Oh wait, now I have to cast this too!”) better as well.

The bad^H^H^Hnot entirely fortunate

To be fair: I’ve only found a few things less fortunate and they are definitely not show stoppers – they all have reasonable workarounds or are quite minor to begin with. But, since this is the interWebs I’m going to share them with you regardless 😉 You might run into them yourself, after all.

The biggest thing is that Xtend derives its type system directly from the Java type system. Xpand/Xtend had a type system which allowed you to compose various type systems, with the EMF and JavaBeans type systems being the most widely used after the built-in type system which came with a whole lot of convenience for the common base and collection types. In Xtend, you essentially only have an improved JavaBeans type system so you’ll have to rely on what Java and its libraries offer you or else you’ll have to knock it out yourself.

In particular, the following useful features of Xpand/Xtend are missing:

• «EXPAND … FOREACH…» construct. This means that you find yourself typing«FOR item : items»«item.doSomething»«ENDFOR»quite often, especially since it’s often quite natural to factor out the contents of the loop. The workaround for this consists of a utility function-replacement using closures but the result is slightly less aesthetically pleasing than it used to be.
• Syntactic sugar (shorthand) for property collection. “collection.property” was Xpand/Xtend shorthand for “collection.collect(i|i.property)“. In Xtend neither are possible and you’ll have to use .fold or .reduce to achieve the same and the equivalent code is nowhere near as readable as its Xpand/Xtend counterparts.
• Using this as a parameter name. It was perfectly alright to use this as a parameter name in the definitions of Xtend functions and Xpand DEFINEs. It was even OK to say «FOREACH items AS this»…«ENDFOREACH». Since “property” is shorthand for “this.property” (it still is) this allowed you to create extremely succinct code. The User Guide mentions that it should be possible to re-define this but I couldn’t get that to work, so you’ll have to qualify all object access with the name of the parameter or variable.

On the IDE side of things, I miss Java’s automatic ordering and cleaning of imports. Also, the content assist on Java types in the Xtend editor doesn’t do the “captical trick” where “DCPF” expands to “DefaultConstructionProxyFactory”, e.g..

Lastly, Xtend doesn’t offer control over visibility of features in the generated Java and also doesn’t support abstract, static or (non-default) constructors. This has led me to use Java for building APIs and implementing the typical abstract/support part of it and Xtend for the “final” implementation – which tends to benefit most from the succinct syntax and features such as closures and rich strings.

The ugly

Again: I’ve only found a few of these and none of them show stoppers.

First of all, the compilation to Java does not always yield compiling code. (This is in full compliance with Joel Spolsky’s Law of Leaky Abstractions, since Xtend is an abstraction of Java.) Although the User’s Guide mentions that uncaught, checked exceptions thrown from your code are automatically wrapped in unchecked exceptions, this is not the case: instead, they are added to the throws-clause of the containing methods (corresponding to an Xtend function). This can break an override function or it can wreak havoc on code using this function. I’ve found myself writing a lot of try–catches to cope, which detracted from the otherwise quite succinct syntax quite a bit. Obviously, I would have had to write these anyway if I were using Java, but that’s not the point of Xtend, I think. (To compound the matter: you can’t explicitly declare a throws-clause in Xtend.)

Also, generic types involving wildcards (‘?’) are not water tight, although it’s fair to say this is a major problem with the Java type system in general and often enough extremely hard to get right. Not using wildcards is almost always possible, so that’s the obvious workaround.

Conclusion

All in all and despite my few, slight misgivings, Xtend is quite an improvement over Xpand/Xtend. I’d heartily recommend to start using it, both as a replacement to Xpand/Xtend (and JSP) as well as to Java itself.