Archive for the ‘DSLs’ Category

Deploying plugins using ANT

April 3, 2011 3 comments

Whenever you’re developing DSLs in the form of Eclipse plugins, you’ll have to come up with a means of deploying these plugins to the Eclipse’s of your language’s users. At several times, I’ve used a simple ANT script to do just that:

<?xml version="1.0"?>
<project name="DSL deployment" default="deploy">

	<target name="check" unless="eclipse.home">
		<echo message="Property {eclipse.home} not set (run this ANT script inside the same JRE as Eclipse)." />

	<target name="deploy" depends="check" if="eclipse.home">
  	<echo message="Deploying plugin JARs to Eclipse installation (${eclipse.home})" />
    <copy todir="${eclipse.home}/dropins">
      <fileset dir="${basedir}/lib/plugins" />
  	<echo message="Please restart Eclipse to activate/update the plugins." />


You simply put all the plugins to be deployed in the lib/ directory of the containing Eclipse project and run the ANT script inside the same JRE as Eclipse, using the settings on the JRE panel of the ANT Run Configuration. The script checks for this and will signal if it’s not ran in the proper way. After deployment, you’ll have to (have) Eclipse restarted to activate the plugins. The plugins are placed in the dropins directory rather than the plugins directory which allows you to easily distinguish them from “regular” plugins.

This setup has the advantage that you can have Eclipse export the plugins to the lib/ directory of the containing Eclipse project, by pointing the Export wizard to that directory on the Destination tab. In case of errors, the file gets dumped in lib/ directory as well.

Categories: DSLs, The How

Open-sourcing some DSLs

March 31, 2011 1 comment

In the past two months, I’ve re-visited several technical domains I’ve dealt with in the past and implemented DSLs for these (in Xtext 1.0.y) which might be somewhat generically useful.¬†I’m open sourcing these on Github under the MIT license, in the hope that these may prove useful to folks as examples what you can do with DSLs in general and Xtext in particular and how you can achieve that.

Feel free to pull the repo and play around with it. Be sure to drop me a line in case you found it useful (or not, in which case: why?). Do consider though that I’m providing this as-is and that I don’t intend to provide anything remotely resembling serious support on it – unless you’re willing to hire me, of course ūüėČ

Categories: DSLs, The How, Xtext

More on pre- and postfix operators in Xtext

March 24, 2011 3 comments

In the previous blog, I entirely glossed over the aspect of associativity of operators. I turns out that the unitary minus operator we constructed is right-associative and the signum operator is non-associative. In general, prefix operators can either be right-associative, meaning the operations are grouped from right to left, or non-associative, meaning that the operator can’t bind with operators of the same precedence (including itself). Postfix operators can either be left-associative (exemplified by “x++–“) or non-associative.

We already ascertained the right-associativity of the unitary minus by means of the last line in the unit test in the previous post: if unitary minus were non-associative, then “- -2” would trigger a parse error and not be equal to 2. We can check the non-associativity of the signum operator with the following test (still as part of the CalculatorTest class), which asserts that parsing “-2s s” yields a parse error:

    public void test_associativity_of_signum() throws Exception {
		getResourceAndExpect(new StringInputStream("module test  -2s s;"), 1);

(I’ll explain about the extra space in the expression string “-2s s” in a minute.)

It’s not difficult to make our prefix operator non-associative: we simply call the grammar rule UnitaryMinus call the rule for the next precedence level instead of calling itself.

UnitaryMinus returns Expression:
    Signum | ({UnitaryMinus} '-' expr=Signum);

We can test this by trying to parse “–2” and expecting a (1) parse error.

Making the postfix operator left-associative is equally simple: we replace the ?-cardinality with a *-cardinality.

Signum returns Expression:
    PrimaryExpression ({Signum.expr=current} 's')*;

We can test this by checking whether “2s s” is parsed (and equals 1).
(The following “obvious” solution “obviously” introduces left-recursion:

Signum returns Expression:
    Signum ({Signum.expr=current} 's')?;

As usual with implementing left-recursive grammars with LL parser technology, this is circumvented with tree writing, in the case of Xtext using actions.)

The only problem with this grammar is that it doesn’t parse something like “-2ss”. The reason for that is that the sub string “ss” is lexed as an ID token instead of a sequence of ‘s’ terminals. By introducing an extra hidden token (whitespace or comments) in between, we force the lexer to produce a sequence of ‘s’ terminals, but that means we’re bothering the DSLs users with it. A better solution would be to choose a character for the postfix operator that doesn’t clash with the ID terminal rule (the ŌÉ character would’ve been a rather obvious choice, in this case) or rewrite the ID terminal rule to not match the regexp /s+/.

Categories: DSLs, The How, Xtext

Pre- and postfix operators in Xtext

March 21, 2011 4 comments

While re-visiting some Xtext DSLs I made earlier, I came across one which some ambiguities (reported by ANTLR) due to the use of a prefix operator in an expressions sub language. Fixing that stymied me enough to warrant this blog. (It’s actually a sort of addendum to Sven Efftinge’s excellent blog on expression parsing. I’ll do a blog later on to recap everything.)

For the sake of minimizing effort for all parties involved, we’ll extend the Arithmetics example that’s shipped with Xtext with a unitary minus as prefix operator and the signum function (which returns -1, 0, or 1 depending on whether the operand is negative, zero or positive) as postfix operator. (I chose the signum function because it’s an instance method of the BigDecimal class which is used for the interpreter.)

If you’ve had a good, hard look at the grammar of the Arithmetics example or Sven’s blog, you might have realized that the patterns there amount exactly to implementing a classical recursive-descent parser, right in Xtext’s grammar definition language. The rules of the expression language form a (strictly-ordered) sequence, each of which start of¬†calling the next rule in the sequence before trying to match anything else (and doing a tree rewrite with the result of the rule call). The net effect is that the sequence order equals the precedence¬†order, with the first rule corresponding to the lowest precedence and the last rule in the sequence corresponding to the highest level, typically consisting of the ubiquitous parentheses and possibly other things like literals, variable references and such.

We’re going to extend that pattern to deal with pre- and postfix operators as well. The relevant section of the Arithmetics grammar consists of lines 46-52 of the Arithmetics.xtext file:

Multiplication returns Expression:
    PrimaryExpression (({Multi.left=current} '*' | {Div.left=current} '/') right=PrimaryExpression)*;

PrimaryExpression returns Expression:
    '(' Expression ')' |
    {NumberLiteral} value=NUMBER |
    {FunctionCall} func=[AbstractDefinition] ('(' args+=Expression (',' args+=Expression)* ')')?;

We’re going to add two rules, called UnitaryMinus and Signum, in between the Multiplication and PrimaryExpression rules, so we have to change the Multiplication rule:

Multiplication returns Expression:
    UnitaryMinus (({Multi.left=current} '*' | {Div.left=current} '/') right=UnitaryMinus)*;

Matching the unitary minus prefix operator is simple enough:

UnitaryMinus returns Expression:
    '-' expr=UnitaryMinus;

Since this rule always consumes at least one token, the ‘-‘ character, from the input, we can recursively call UnitaryMinus without causing left-recursion. The upshot of this is that ‘–37’ is parsed as -(-(37)). Unfortunately, the rule (as it is) would break the sequence of rules, so that we’d lose the higher levels of precedence altogether. To prevent that, we also call the next rule, Signum, as an alternative:

UnitaryMinus returns Expression:
    Signum | ({UnitaryMinus} '-' expr=UnitaryMinus);

(We need the {UnitaryMinus} action here to make sure the Xtext generator generates a corresponding type in the Ecore model which holds the parsed info.)

Implementing the postfix operator is a matter of calling the next rule (PrimaryExpression) and performing a tree rewrite in case the postfix operator can be matched:

Signum returns Expression:
    PrimaryExpression ({Signum.expr=current} 's')?;

This is all that’s required for the grammar. Now, we only have to fix the Calculator class and add a couple of test cases in the CalculatorTest class. The Calculator class uses the PolymorphicDispatcher class I wrote about earlier which means we just have to add methods corresponding to the new UnitaryMinus and Signum types:

	protected BigDecimal internalEvaluate(UnitaryMinus unitaryMinus, ImmutableMap<String,BigDecimal> values) {
		return evaluate(unitaryMinus.getExpr(),values).negate();
	protected BigDecimal internalEvaluate(Signum signum, ImmutableMap<String,BigDecimal> values) {
		return new BigDecimal( evaluate(signum.getExpr(),values).signum() );

We add a couple of test cases, specifically for the new operators:

    public void test_unitary_minus_and_signum() throws Exception {
        check(-1,	"1 + -2");	// == 1 + (-2)
        check(1,	"-1 + 2");	// == (-1) + 2, not -(1 + 2)
        check(-1,	"-3.7s");	// == -(3.7s) == -1
        check(0,	"1 + -7s");	// == 1 + -(7s) == 1 + -(1) == 0
        check(2,	"--2");		// == -(-2)

Now go out and add your pre- and postfix operators to your DSL today! ūüôā

Tips and tricks for Xpand

January 17, 2011 5 comments

I’ve been using Xpand (in combination with Xtend) for some four years now and it still is my favorite (target language-agnostic) templating engine. As all programmers, I picked up a few habits which I’ll conveniently rebrand as “tips and tricks” ūüôā

Separating templates from logic

Xpand is closely tied with Xtend: not only do they share an expression sub language, but they’re also each other’s perfect buddies in crime since Xtend (as a separate language) allows you to factor out re-used or intricate expressions into a separate file, so the Xpand template only needs to call (hopefully aptly-named) functions in a separate Xtend file. As a bonus, this approach allows you to use the cached keyword so that commonly used, but expensive expressions have to be evaluated only once as well as offering a more convenient way of documenting functionality (see also XtendTools).

Xtend libs

Xtext ships with a number of standard libraries which can be quite helpful, especially the io library. These libraries are all contained in the org.eclipse.xtend.util.stdlib plugin so be sure to add this one to the plugin dependencies (in the META-INF/ file). I’ll highlight a few of the libraries (in alphabetical order):

  • counter.ext: provides counters inside templates (if you really need these -most of the time you don’t and you simply didn’t think hard enough about it, yet ;));
  • crossref.ext:¬†getReferencingObjects(EObject eObject) compiles a list of all objects (in the current Resource) referencing eObject;
  • elementprops.ext: provides access to System properties;
  • io.ext: provides console logging;
  • naming.ext: provides common naming, such as computing a qualified name from the containment hierarchy;
  • properties.ext: provides access to properties injected into the workflow using the¬†org.eclipse.xtend.util.stdlib.PropertiesReader workflow component;
  • uid.ext: provides common UID functionality/generation.

I’d advise anyone to go over these libraries by looking at the .ext files, checking out the (somewhat sparse)¬†documentation and do some ‘finger exercises’ using them.

Polymorphic dispatch and the use of a sentinel

As most things in the Xtext universe, Xpand features polymorphic dispatch, i.e.: in case of an overloaded DEFINE block, Xpand decides which one to call based on the actual (runtime) type of the arguments. This is extremely useful but also presents a slight inconvenience in case of a type hierarchy (as opposed to a number of types having only (E)Object as common super type). As an example, consider the hierarchy where Apple and Orange are types having Fruit as common, direct super type. Now, when you write DEFINE blocks for Apple and Orange with common name ‘squeeze‘ and you call squeeze from someplace you’re dealing with Fruits, the Xpand editor will complain that it “Couldn’t find definition squeeze for type metamodel::Fruit”.¬†The Xpand editor is not able to see that you’ve covered all essential cases, even if Fruit is defined as abstract. The only way to deal with this, is to write a third DEFINE block which acts as a¬†sentinel to cover the Fruit case (har, har).

However, this sentinel can be put to good use by mixing in some fail-safe in case your meta model changes. E.g., if you add Banana to the type hierarchy of Fruit without writing a separate DEFINE block for it, the sentinel block will be executed but obviously it cannot produce meaningful output on its own. That’s why I tend to put in some error logging in the DEFINE block, like so:

¬ęDEFINE squeeze FOR Type¬Ľ
¬ę( "no (specific) DEFINE block 'squeeze' defined for type " + ).error()¬Ľ

Here, I use the error function from the io.ext library mentioned above which you include as follows:

¬ęEXTENSION org::eclipse::xtend::util::stdlib::io¬Ľ

Now, you’ll get a meaningful error message in the console when executing the template, indicating that Banana is missing a (specific) DEFINE block. (Of course, you could just as well factor the inner expression of the sentinel block out into a separate Xtend file.)

Readable template versus readable output

One of the choices you have when creating a template is whether to make the template readable exclusive-or to make the output readable, i.e. nicely formatted. My advice: choose the former (readable template), especially if there’s a beautifier for the output format. For a lot of the more common output formats, such a Java and XML, Xpand beautifiers have been written. Have a look here (navigate to the ‘PostProcessor’ section) to learn more about them and how to configure these. You might not be an instant fan of the beautification offered out of the box, but it generally beats the trouble of getting the template output nicely formatted. Moreover, this choice fits my earlier reasoning the meta software (such as code generators and templates) is software as well, so the readability argument is very much in scope here.

Getting the template output nicely formatted is frankly a pain in the buttocks, for a number of reasons: (1) Xpand constructs often seem to add whitespace unintentionally and (2) it’s hard to get indentation correct with “recursive content”. In fact, the Xpand constructs do not add whitespace: it’s just that everything between ‘¬Ľ’ and ‘¬ę’ (while inside a FILE statement of course) is part of the template text and ends up in the output. This means that something like

¬ęFOREACH entities AS e¬Ľ
entity ¬ę¬Ľ;

will produce a newline in front of each entity in the output because of the newline after the first ‘¬Ľ’. Xpand allows you to add a minus sign just before that ‘¬Ľ’ which has the effect that the whole uninterrupted sequence of¬†whitespace characters (i.e., everything the next non-whitespace character) is gobbled up so it doesn’t produce anything in the output. This might help a lot getting unwanted whitespace out of the way, but it certainly doesn’t help with producing the correct indentation so the ‘-‘ is a two-edged sword.

Again, in general I’d go for a readable template unless there are very pressing reasons (such as the lack of a beautifier) to make the output look nice as-is. In some cases, it might even be worthwhile to combine a readable template with a custom-made parser for the target language and a (separate) serializer/pretty-printer/beautifier for that target language. (In fact, this is how the XML beautifier arose as part of openArchitectureWare.)¬†This adds a step to the pipeline, but it also adds some validation of the templates’ output. Xtext could very well be a perfect fit for this case, as you quite probably can even re-using the existing meta model.

Categories: MDSD, The How, Xpand

DSLs “versus” software engineering

January 13, 2011 8 comments

(New Year’s resolution: blog more frequently ;))

One aspect of using DSLs for software development which seems to be a bit underplayed (IMHO) is the role of good-ole software engineering, by which I happen to mean the practice of creating software through a systematic approach involving analysis, design, implementation and testing in a controlled and predictable manner. It seems to me there’s something of a sentiment or expectation that with DSLs you don’t have to do software engineering anymore, as if everything that makes creating software difficult somehow instantly disappears when using DSLs to model the software to as large a degree as is desirable and productive.

There are two main reasons for using DSLs:

  1. Empowering domain stakeholders (and other participants) by establishing an ubiquitous language for communication and providing them (and disciplines downstream) with dedicated tooling for that language;
  2. Separating the essential complexity (the what and why) from the incidental complexity (the how) by focusing the language on the former and hiding the latter “somewhere else”. (This also means that the software model does the same with less “code”.)

So, how does software engineering come into play then? Well, as I see it, there are two sides to this equation: (1) the DSL itself and the way it’s used and (2) the “meta software”, i.e. the software (parser, editor, tooling, etc.) which brings the DSL to life.

Engineering a DSL

To me, the fundamental value of software engineering is the set of concepts such as¬†Separation of Concerns, Loose Coupling, Strong Coherence, (Unit) Testing etc., which allow us to create quality software. I simply define software quality somewhat non-conventially as “it works as required plus it can be productively changed in case the requirements change”. (It’s interesting to see that Kent Beck defines the concepts Loose Coupling and Strong Coherence in terms of how change spreads through a code base.) A DSL can easily be said to “work” if the two advantages mentioned above are realized: stakeholder empowerment and separating essential from incidental complexity -essentially another incarnation of Separation of Concerns. Unfortunately, it’s almost impossible to make this S.M.A.R.T. so you’ll have to rely on your craftsmanship here.

The most obvious aspect of DSL design which contributes directly to the change part of quality is: modularization. This aspect has two directions: (1) how do you distribute different aspects across parts of the language and (2) how can you cut up the entire domain into manageable pieces. Both of these directions benefit directly from the application of concepts like¬†Separation of Concerns,¬†Loose Coupling and¬†Strong Coherence. As an example, consider a (vertical) DSL for Web application development which would typically address data, screen and interaction modeling: Do you create a separate DSL for the data model? Can you divide that up as well? Do you separate screens and interaction/flow? Do you take the use case as unit of granularity? Etcetera… The answers to all these questions are “It depends…” and you’ll have to address these time and time again for each situation or change to that.

But software engineering on the DSL side doesn’t stop there: the DSL instance, i.e., the model that’s built using the DSL must be viewed as software as well -after all, it’s the center piece for the software creation. E.g., as soon as you can modularize your model, you’ll have to think about how to divide the DSL instance into several pieces. Questions which come into play here: How large should each piece be? What kind of inter-piece dependencies do I want to allow or minimize? (This already depends on how modularization is realized on the language level.) How does the versioning system you use affect these decisions? Again,¬†Separation of Concerns,¬†Loose Coupling and¬†Strong Coherence are key concepts to keep in mind here.

You also might want to think about a way to (unit) test the instance. A famous example is business rules: it is very valuable to be able to test the execution of such rules in different scenarios to validate that the results are what your domain stakeholders are expecting. How you code such tests depend (as ever) on the situation: sometimes it’s better to code them against an the business rules’ execution engine (which is something different than testing that execution engine itself!), sometimes you enhance the DSL (or probably better: create a separate DSL) for this purpose.

Engineering the meta software

By “meta software” I mean all the software which is involved with the DSL and which is not directly contributing to the realization of requirements/business value. This ranges from parser definitions (when using a Parser Generator), parser implementations, model validators and model importers to code generators or model interpreters/execution engines. It’s important to realize that this software in the traditional sense as well -not just a bunch of “utility scripts” lying around. In fact, your meta software has to be at least as good as “regular” software since it typically has a large impact on the rest of the software development because the model does more in the same amount of “code”. Among other things, this warrants that you create automated tests for all components, that these tests are part of the continuous integration (automated build) and that everything is checked in into a versioning system. It also warrants that you design the meta software really well, e.g. with an eye towards Separation of Concerns, both inside components as well as across components. It’s advisable to choose a set of tools/frameworks which integrate well and offer as much IDE and compile-time support as possible, to make meta software development as productive, error- and care-free as possible. (I once did a project in which an UML model was fed directly into a combination of a dynamic language and StringTemplate: in the end you get used to it, but it’s painful all the way…)

If the DSL changes, then it might happen that the current model (DSL instance) breaks and must be repaired -depending on the amount of breakage you could do that manually or expend the effort to devise an automated migration facility. This also means that it should be clear at all times which version of the DSL a model requires: it’s usually best to explicitly version both the DSL as well as the model but you might be able get away with an incremental push from a DSL development branch to the modeling branch. In the latter case, you really need to make sure that you tag the meta software together with releases of the software in order to be able to go back in history reliably.

Using a DSL has a kind of¬†a two- (or even multi-) step nature: the DSL instance is parsed first and then fed to either a code generator or a model interpreter. So, if you change the DSL you’ll probably have to change the second step as well. In fact, changes typically “flow backwards”: changes to or enhancement of the code generator or model interpreter often require a change to the DSL as well, e.g. in the form of a new language construct. This¬†phenomenon is sometimes called¬†coupled evolution. Again, having automated tests for all individual components/steps are really necessary to avoid running into problems. In case you use a model interpreter, it’s usually quite easy to write (unit) tests against that.

Change “versus” code generation

In case you have a code generator things are typically more difficult because (1) the generation often is target language-agnostic since most template engines (such as StringTemplate and Xpand) simply spit out “plain text” and (2) usually there’s non-generated code (either framework code or code behind the Generation Gap) as well which has to be integrated with the generated code. I’ve found that in these cases it’s extremely useful to use a relatively small reference application for the meta software discipline. Such a target application would consists of a smallish DSL instance/model which does, however, touch all language constructs in the DSL (especially the new ones) and a fair amount of combinations of those and non-generated code consisting of framework code, hand-crafted code behind the Generation Gap and –mui importante– unit tests to validate the reference application. As always, the process of generating from the reference model, the building of the reference application from generated + non-generated code and running of the unit tests should be automated to be able to verify the correctness of the meta software reliably and quickly.

Something which is very useful in this scenario, is traceability, i.e. being able to see where generated code came from. In particular, which template produced a particular piece of code and what model element was the primary input for that template. Realizing a modest but already quite useful form of traceability is to generate comments with tracing information along with the real code. This is reminiscent of logging, also because care must be taken that the tracing information is succinct without overly “littering” the actual code.

Wrapping up

I hope I’ve been able to make the case that “old-fashioned” software engineering has a definite merit in the DSL scenario. Of course, there’s a lot more to say on this topic. Personally, I’m hoping that the upcoming book on DSL Engineering by Eelco Visser and Markus Voelter treats this really well.

Polymorphic dispatch in Java using Xtext

December 14, 2010 4 comments

One of the very nice features of both Xtend and Xpand is polymorphic dispatch (also called multiple dispatch, at times). For the truly uninitiated: if you are dealing with an overloaded function/method, polymorphic dispatch is the mechanism that chooses which one to actually call based on the runtime types of the arguments. Note that most compiled and/or statically-typed languages (such as Java) determine which implementation to call at compile-time. The consequence of this is that if you want behavior to depend on runtime types, you’re forced to use a lot of

        if( obj instanceof SomeSubtype ) {
            doSomethingWith((SomeSubtype) obj);
    private void doSomethingWith(Somesubtype subObj) {

in your code.

On the other hand, several dynamic languages (like Groovy and Scala) feature polymorphic dispatch. We also see this feature used in several places in Xtext runtimes, m.n. in sub classes of AbstractDeclarativeScopeProvider and DefaultDeclarativeQualifiedNameProvider. This magic is provided courtesy of the org.eclipse.xtext.util.PolymorphicDispatcher class. Since that class itself has no runtime dependencies on the rest of the Xtext runtime framework, we can use it ourselves outside the context in which it is normally used. This blog will explain how to do that.

I’m going to re-use the¬†path expressions in entity models example and re-implement the scoping required. Please check that blog to see that the implementation of the scope_PathTail_feature method was rife with instanceof‘s and explicit casts. To remedy that we’re going to the PolymorphicDispatcher. The class has a number of constructors and static factory methods, but the easiest to use is the one with signature createForSingleTarget(String methodName, Object object) which returns a PolymorphicDispatcher object that does dispatches polymorphically to methods with the given methodName inside the given object. (Alternatively, you could use createForSingleTarget(String methodName, int min, int max, Object object) to limit the number of parameters -e.g., in our case: min=max=1.)

Now we only need an object which holds the methods to dispatch to. For that, I usually create a separate (top-level) class whose sole responsibility it is to compute “something” using polymorphic dispatch, e.g. a scope. This class¬†holds all the methods to dispatch to as private methods and exposes one public method to actually trigger the computation. This has a number of advantages:

  • It provides a clear separation between “polymorphic code” and plain Java code. The dispatch methods can’t be called by accident and
    forced to think about the computational context. It find it also encourages a more functional way of thinking.
  • The private methods aren’t actually used anywhere so they generate “…never used locally” warnings. By endowing the separate class with a @SuppressWarnings( "unused" ) annotations (only) those warnings disappear.

In the case of the path expressions, our code eventually looks like this:

As you can see, the number of instanceof‘s and explicit casts has reduced to 0. For simplicity I’ve use an inner class called PolymorphicContext here so it can access the structuralFeatures(Entity) method (which is reused verbatim from the earlier post), but in general, I’d recommend using a top-level class. The scope_PathTail_feature method is now reduced to a simple call to the computeScope method of the PolymorphicContext class. The¬†computeScope method, in turn, simply uses the single PolymorphicDispatcher object to polymorphically dispatch to methods called “scope” on this object and produces a suitable IScope object from the result. The first two scope methods correspond to the two outermost if( context instanceof …) statements in the original code. The other scope methods correspond to further such instances of if-statements.

Note that I re-use the dispatcher object for that as well: again, this is mostly for the sake of simplicity. In practice I’d recommend either using a separate class or a separate dispatcher object. What you’d choose, depends on the granularity. In this case, it all fits in relatively comfortably in one class but if you accumulate dispatching methods, you’re probably better off separating concerns as well as possible.

This example also demonstrates some disadvantages of this approach:

  • There’s some boilerplate code involved, so this approach only makes sense if there’s a fair number of dispatch targets. Already in the example, there are more lines of code in this implementation than in the regular Java one which doesn’t help overview and clarity.
  • This type of code is generally a bit harder to read and understand in an imperative manner than a regular, well-laid out implementation. Again, you should consider whether your code will really benefit from this approach.


This blog demonstrates the (relatively simple) use of polymorphic dispatch in Java through Xtext’s PolymorphicDispatcher class, combined with something of a Best Practice. There’s a lot more the¬†PolymorphicDispatcher class than I’ve shown here, which unfortunately is quite undocumented. In my opinion, the best way to find out more is by digging into the code, both of the class itself as clients of it, and then imitating those usage patterns in your own code.

P.S.: Be sure to also check out Sven Efftinge’s blog entry on polymorphic/multiple dispatch.

Categories: MDSD, The How, Xtext

Getting alternative cross-references to work with existing EPackages

December 8, 2010 7 comments

Consider the following, recurring scenario: In your Xtext DSL you want to cross-reference either the type (rule)¬†This or the type¬†ext::That with the latter coming from an existing EPackage, i.e. an Ecore model. Naturally, you’d be inclined to say the following:

    refThis=[This] | refThat=[ext::That]

Unfortunately, Xtext (or rather: ANTLR) now complains during the DSL generation that

Decision can match input such as "RULE_ID" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input

The reason for that is that both cross-references use the same token (by default a string returned by the ID terminal rule) as the name to use for the look-up, so the parser can’t distinguish between the two, based solely on the token under consideration alone. Note that if the parser generator were a bit smarter, it could have simply tried to resolve both alternatives using the scoping providers and see which one returned first.

In case you’d wanted to cross-reference This and Other, both being type rules in the grammar itself, the solution would be to introduce a “convenience super type” like this:

CrossReferencableThingy: This | Other;

Unfortunately, if you try this with ext::That instead of Other, you’ll get a basic parsing error because alternatives in a type rule must be callable rules as well whereas That is “just” a type coming from some Ecore model.

Even if the Xtext grammar definition language allowed such a thing, we’d have a problem: a rule with alternatives like this normally translates into an EClass which should act as a super type for both of the alternative types. But that would mean that the ext::That type would have to be altered to (also) point to¬†CrossReferencableThingy, which is impossible since ext::That is in an EPackage that is considered to be read-online as far as the Xtext generator is concerned.

The solution

The solution is really quite simple. The trick is to realize that ultimately instances of both This and ext::That are EObject-s. Therefore, it’s OK to say:

MyCrossReference: ref=[ecore::EObject];

(Of course, now you’ll have to import the Ecore EPackage as well.) This means that you could reference everything, which is not what we want so we’re going to use scoping to limit the possibilities by actually doing what we said earlier about the parser being smarter.

IScope scope_MyCrossReference_ref (Model model, EReference eRef) {
    List<EObject> crossRefTargets = new ArrayList<EObject>(model.getThings());
    Iterable<That> allThat =
            model.getExternalModel().getStuff(), That.class);, allThat);
    return org.eclipse.xtext.scoping.Scopes.scopeFor(crossRefTargets);

The reason I have the scoping called at the model’s root (of type Model) is that it’s usually quite easy to reach all This and (presumably) all ext::That from the root without having to traverse the Semantic Model (obviously, I’m reading Fowler’s DSL book!) upwards. Note also that I use the Google Collections Iterables class to do some of the tedious, heavy lifting for me. Alternatively, I could have used org.eclipse.xtext.EcoreUtil2#typeSelect.

Categories: DSLs, MDSD, The How, Xtext

Xtext workshop material

October 15, 2010 Leave a comment

Recently, my collaegue Peter Vermeulen and I conducted two workshops -1 “Beginner”, 1 “Advanced” level- on how to create DSLs with Xtext. Currently, we’re making the material available on Google Code free to use.

The GC repository contains¬†both the reference implementation of a relatively simple and small, yet feature-full DSL for the workshops as well as the slide and mockup material (in the .notes project). The workshops followed the format of “couple of slides, bit of talk, lots of exercise”: of the 3 hour workshops at least 2 hours went to our attendees working hands-on on the DSL (grammar + a bit of customization), either individually or in pairs and assisted by the two of us. It turned out that the amount of material in the slides was a tad optimistic for most attendees although it was too shallow for those who already were steeped in language engineering.

The DSL concerns the combination of an entity model and a reasonably simple Web application definition and features:

  • use of cross-references with custom scoping implementation
  • some validations, one of which is detection and flagging of duplicates using the 3-arg error method
  • chaining/path expressions (also see one of my other posts)
  • arithmetic expressions
  • a custom formatter
  • a generator which generates POJOs and HTML pages

The notes also contain a series of mockups which we fed piece-meal to our attendees and which adds one specific concept or a group of simple and congruous concepts at a time.

I hope you find this material useful and do feel free to provide us with feedback on it (e.g. in the comments).

Categories: DSLs, MDSD, The How, Xtext

Scoping and content assist using Xtend on steroids

October 13, 2010 1 comment

Gosh, it has been almost a month since my last blog…about time to get my act together and publish some more. Luckily, there’s enough material in the pipe line ūüôā

By the way: I changed the theme as the previous one was a bit dark and not wide enough to comfortably fit bits of “real” code. I also have started using WordPress’ built-in code tags to show the code as it easier to copy code from that than from screen shots and making the screen shots is often quite a lot of work. The down side to all that is the lack of syntax highlighting for non-standard languages such as Xtend.

I often find that a lot of my scoping and content assist logic can be declaratively expressed in Xtend quite comfortably while the equivalent Java implementation takes up rather more space. One possible solution would be to use JVM-based languages like Groovy or Scala to express such logic since these language allow you to use more or less the same syntax as Xtend.

But the solution I’m going to explore here is to actually use Xtend to implement some scoping. One thing to take into account here is that Xtend is interpreted so it will always be at least somewhat slower and potentially a lot slower. However, I find that in most circumstances Xtend expressions can quite easily be tuned to have the same runtime complexity (in the algorithmic sense) as an equivalent Java implementation. To get that last extra bit of ‘oompf!’ I’m also going to caching for the scoping.

The example

We’re going to use the Domain-Model example shipped with Xtext since version 1.0.0. The scoping implementation for the ‘opposite’ feature of the Reference type is as follows:

Now, even using Google Collections this looks rather ungainly. So, we’ll trim it down to a more succinct and completely functional expression using a bit of pseudo-code which happens to look remarkably like a piece of Xtend code ūüėČ

$features = ((Entity) ref.type.referenced).features   // line 29
$references1 = $features.typeSelect(Reference)        // line 30
$references2 = $ input | ref.eContainer == input.type.referenced )   // lines 31-34

(I use $-prefixes to indicate temporary variables.) Seeing that none of the right-hand side expressions have any side effects, we can roll this expression into the following (Xtend) “one liner” spread over three lines for the sake of expos√©:

((Entity) ref.type.referenced).features
       .select( input | ref.eContainer == input.type.referenced )

This also makes the semantics of this scoping logic a lot clearer: “for this Reference, which is a feature of entity A (which equals ‘ref.eContainer’), pointing to entity B, select all Reference features of entity B which point to entity A”.


Now that we have our Xtend expression we can put it in an Xtend file called Scoping.ext residing next to the DomainModelScopeProvider Java class file:

import domainmodel;

List[Reference] scope_Reference_opposite(Reference ref) :
   ((Entity) ref.type.referenced).features
      .select( input | ref.eContainer == input.type.referenced );

Note that we don’t want to have this Xtend function cached by prepending its definition with the cached keyword as that would mean that the editor would always return the scoping reflecting the initial state, even after the contents changes.



To use this Xtend file, we’re going to make use of the org.eclipse.XtendFacade class which is the work horse behind the Xtend workflow component one usually encounters.¬†(This class resides in a plugin which is by default an (indirect) dependency of any Xtext project.) First we have to obtain an instance of the XtendFacade in the DomainModelScopeProvider which we’ll do by adding a no-args constructor and instance field to the class’ innards:

private XtendFacade xtendFacade;

public DomainmodelScopeProvider() {
  xtendFacade = XtendFacade.create("org::eclipse::xtext::example::scoping::Scoping");
  xtendFacade.registerMetaModel(new EmfRegistryMetaModel());

We obtain an XtendFacade object by pointing to a particular Xtend file, after which we add the EmfRegistryMetaModel meta model implementation which simply ensures that all meta models previously registered with the EMF Registry are included in Xtend’s internal type system. Note that the location of the Xtend file follows the Xtend convention instead of the file system convention, and that you have to leave out the ‘.ext‘ extension.


Now we can replace the existing implementation of the scope_Reference_opposite method with the following one:

public IScope scope_Reference_opposite(final Reference ref, EReference eRef) {
  @SuppressWarnings( "unchecked" )
    List result = (List)"scope_Reference_opposite", ref);
  return scopeFor(result);

We need the cast to List<Reference> because XtendFacade#call has Object as return type. Furthermore, I follow Joshua Bloch’s Item 24 from Effective Java (Second Edition) to suppress the unchecked cast warning.



Finally, we’re going to add some sensible caching to this implementation. Note that in this case, we don’t expect any serious performance hit because usually/hopefully entities typically do not have a huge number of features and the scoping logic is linear in the number of features of the entity pointed to by the reference for which the scoping is invoked. But since the execution of Xtend is typically somewhat slower than the Java counterpart, we might as well show how to.

For the caching, we’re going to use the org.eclipse.xtext.util.IResourceScopeCache interface which has a default implementation which is automagically wired up by the Guice DI container and which has the behavior that it evicts the cache as soon as the containing resource is changed. It’s also the best result one can hope to achieve in general: we could do better by listening to changes on the referenced entity, but that makes the implementation quite a bit hairier.

First, we add an appropriate field to the class:

private IResourceScopeCache scope_Reference_opposite_cache;

Then, we beef up the implementation of scope_Reference_oppositemethod to use the cache:

public IScope scope_Reference_opposite(final Reference ref, EReference eRef) {
	return scope_Reference_opposite_cache.get(ref, ref.eResource(), new Provider<IScope>() {
			public IScope get() {
				@SuppressWarnings( "unchecked" )
					List<Reference> result = (List<Reference>)"scope_Reference_opposite", ref);
				return scopeFor(result);

Note that the previous implementation of this method has simply moved into the get method of anonymous inner class implementing (lines 4-6).


Under the rug

We have swept quite some details under the rug, most notably the exception handling: the code obviously works with this particular grammar definition, but as soon as the grammar def starts¬†to change, then this code is not very robust, because of the following two implicit (in the sense that you most likely don’t get any immediate warnings from the IDE as soon as they break) assumptions in the code:

  1. ref.type.referenced (with ref being a Reference) is assumed to be an Entity, because of the cast in the scoping Xtend expression;
  2. the scope_Reference_opposite function returns a List of References.

As soon as these assumptions don’t hold any more, the scoping simply stops working and the only signs of that you’re likely to see outside of that are stack traces in the console of a launching Eclipse instance.

Furthermore, both the Java as well the Xtend implementation could be optimized slightly:

  1. The typeSelect and select calls could be rolled into one select invocation which also checks type. This saves creation and populating of 1 list and running over it.
  2. The value ‘ref.eContainer‘ could be stored in a local variable to avoid unnecessary dereferencing. This is more important for the Xtend implementation, as the JVM probably inlines that anyway for the Java implementation.

In this case, both optimizations are not worth the effort but for more intricate scoping logic they can make your editor experience just that bit better.

Categories: DSLs, MDSD, The How, Xtext