Groovy for Domain-Specific Languages, Second Edition

All of the teaching in college in those days revolved around the general-purpose languages. I recall sitting in class and being taught about the "two" types of programming language: machine language, and high-level languages. Both were types of general-purpose languages, in which you could build any type of application, but each language had its own strengths and weaknesses. The notion of a DSL was not yet considered as part of the teaching program. Nor was the idea that anyone other than a cadre of trained professional programmers (hackers) would ever write programs for computers. These days, the word "hacker" has bad connotations of being synonymous with virus writers and the likes. In those days, a good "hack" was an elegant programming solution to a hard problem and being called a hacker by one's peers was a badge of pride for most programmers.

The high-level programming language you used defined what type of an application programmer you were. COBOL was for business application programming, Fortran was for scientific programmers, and C was for hackers building Unix and PC software. Although COBOL and Fortran were designed to be used in a particular business domain, they were still considered general-purpose languages. You could still write a scientific application in COBOL or a business application in Fortran if you wanted to. However, you were unlikely to try any low-level device driver development in COBOL.

Although it was possible to build entire applications in assembly language (and many people did), high-level languages, such as C, BASIC, and COBOL, were much better suited to this task. The first version of the world-beating spreadsheet Lotus 1-2-3 was written entirely in 8086 assembly language, and ironically, it was the rewrite of this into the supposed high-level language C that nearly broke the company in the late 1980's.

Languages such as C and C++ provide the low-level functionality in a high-level language, which enabled them to be used across a much greater range of domains, including those where assembly was utilized before. These days, Java, C# and C++ compete with each other like the Swiss Army knives of general-purpose languages. There are almost no application domains to which these languages have not been applied, from space exploration, through to enterprise business systems, and mobile phones.

Programs such as Lotus 1-2-3 and its precursor VisiCalc revolutionized people's view of who would program computers. A whole generation of accountants, financial analysts, scientists, and engineers came to realize that they can develop sophisticated turnkey solutions for themselves, armed only with a spreadsheet and a little knowledge of macros. Spreadsheet macros are probably one of the first DSLs to find their way out of the cloisters of the IT community and into the hands of the general business user.

Around this time, there was also much media attention paid to the new 4GL (fourth-generation language) systems. 4GLs were touted as being hugely more efficient for developing applications than traditional high-level languages, which then became known as third-generation language (3GL). From the hype in the media at the time, you would be forgiven for thinking that the age of the professional programmer was coming to an end and that an ordinary business user could use a 4GL to develop their own business applications. I viewed this claim with a degree of healthy skepticism—how could a non-programmer build software?

Like DSLs, 4GLs were, generally speaking, targeted at particular problem spaces, and tended to excel at providing solutions in those narrow target markets. The sophistication of most applications in those days was such that it was possible to build them with a few obvious constructs. 4GLs tended to be turnkey environments with integrated tools and runtime environments. You were restricted by the environment that the 4GL provided, but the applications that could be built with a 4GL could be built rapidly, and with a minimal amount of coding.

4GLs differ from our modern understanding of a DSL. We generally think of a DSL as being a mini language with a particular purpose, and they do not generally impose an entire runtime or tool set on their use. The best DSLs can be mixed and matched together, and used in conjunction with a general-purpose programming language, such as C++ or Java, to build our applications.

My own introduction to the concept of DSLs came about in 1986 when I joined Computer Products Inc. (CPI) as a software engineer. In this case, the DSL in question was sophisticated enough to enable the stakeholders to develop large parts of a running system.

CPI developed a process control system, which was primarily sold to chemical and pharmaceutical industries. It was a genuinely distributed system when most process control systems were based on centralized mini or mainframe computers. It had its own real-time kernel, graphics, and a multitude of device drivers for all types of control and measurement devices. But the most innovative part of the system, which excited customers, was a scripting language called EXTended Operations Language (EXTOL). EXTOL was a DSL in the purest sense because it drew the domain experts right into the development process, as originators of the running code.

With EXTOL, a chemical process engineer or chemist could write simple scripts to define the logic for controlling their plant. Each control block and measurement block in the system was addressable from EXTOL. Using EXTOL, a process engineer could write control logic in the same pseudo English that they used to describe the logic to their peers.

The following script could be deployed on a reactor vessel to control the act of half-filling the vessel with the reactant from VALVE001:

drive VALVE001 to OPEN
when LEVELSENSOR.level >= 50%
drive VALVE001 to CLOSED

This was an incredibly powerful concept. Up to this point, most process control systems were programmed in a combination of high-level languages on the main process system, and relay logic on PLCs in the plant. Both tasks required specific programming skills, and could not generally be completed by the chemists or chemical engineers, who designed the high-level chemical processing undertaken at the plant. I recall a room full of white-coated chemists at one plant happily writing EXTOL scripts, as we commissioned the plant.

The proof of the pudding is always in the eating, and I don't recall a CPI engineer ever being called upon to write a single line of EXTOL code on behalf of a customer. Given an appropriate DSL that fit their needs, our customers could write all of the code that they need themselves, without having to be programmers.

This shows the power of DSLs at their best. At this extreme end of the spectrum, a DSL becomes a programming tool that a domain expert can use independently, and without recourse to the professional programmer. It's important to remember, however, that the domain experts in this case were mostly process engineers. Process engineers are already well accustomed to devising stepwise instructions, and building process flows. They will often use the same visual representations as a programmer, such as a flow chart to express a process that they are working on.

When devising a DSL for a particular domain, we should always consider the stakeholders who need to be involved in using it. In the case of EXTOL, the DSL was targeted at a technical audience who could take the DSL and become part of the system development process. Not all of our stakeholders will be quite as technical as this. But, at the very least, the goal when designing a DSL should be to make the DSL understandable to nontechnical stakeholders.

It's an unfortunate fact that with many DSLs, especially those based on XML, the code that represents a particular domain problem is often only legible to the programming staff. This leads to a disconnect between what the business analysts and domain experts define, and what eventually gets implemented in the system. For instance, a business rule is most likely to be described in plain English by a business analyst in a functional specification document. But these rules will most likely be translated by developers into an XML representation that is specific to the particular rules engine, which is then deployed as a part of the application. If the business analyst can't read the XML representation and understand it, then the original intent of the rule can easily be lost in translation.

With language-oriented programming, we should aim to build DSLs that can be read and understood by all stakeholders. As such, these DSLs should become the shared living specification of the system, even if in the end they must, by necessity, be written by a programmer with a technical understanding of the DSL.

In Java, as in other statically-typed languages, variables must first be declared with a type before they can have a value assigned to them. In Groovy, type can be left to be determined at the time of assignment. Groovy supports both static and optional typing as follows:

String str1 = "I'm a String"
def str2 = "I'm also a String"

Both variables str1 and str2 are of the type String. The late binding of the type in the Groovy-style assignment allows for a much less verbose code.

One of the great bugbears of the Java language is the cumbersome interfaces required for list and map manipulation. Groovy adds native support for all of the Java collection types through a very intuitive and readable syntax. The following code:

def authors =  [ 'Shakespeare', 'Beckett', 'Joyce', 'Poe' ]
println authors
println authors[2]

Produces this output:

[Shakespeare, Beckett, Joyce, Poe]
Joyce

Maps are also declared with ease:

def book = [ fileUnder: "Software Development",
         title: "Groovy for DSL" , author: "Fergal Dearle"]
println book
println book['title']
println book.title

This produces the following output:

[fileUnder: Software Development, title: Groovy for DSL, author: Fergal Dearle]
Groovy for DSL
Groovy for DSL

Operator overloading is a powerful feature of the C++ language. Java inherited many of the features of the C++ language, but operator overloading was significantly left out. Groovy introduces operator overloading as a base language feature.

Any Groovy class can implement a full set of operators by implementing the appropriate corresponding method in the class. For example, the plus operator is implemented via the plus() method.

Groovy builds regular expression handling right into the language via the =~ operator and matcher objects. The following example creates a regular expression to match all multiple occurrences of the space character. This creates a matcher object from this expression and applies it to a string by using the replaceAll method:

def lorem = 
"Lorem ipsum dolor sit amet, consectetur adipisicing elit"
println lorem
def matcher = lorem =~ " +"
def removed = matcher.replaceAll(" ")
println removed

Optional typing means that variable type annotations are optional. This does not mean that variables have an unknown variable type. It means that the type will be determined at run time based on the value that gets assigned to the variable. All of the following are legal syntax in Groovy:

int a = 3 
def b = 2
String t = "hello" 
def s = 'there'

Trailing semicolons at the end of statements are optional. The only time that you explicitly need to use a semicolon in Groovy is to separate statements that occur on the same line of code, as shown in the first and third lines in the following code:

int a = 3; int b = 4;
def c = 2 
def d = 5; def e = 6

Method call parentheses are also optional when the method being invoked has passed some parameters. We saw earlier, with closures, that we can invoke a closure through its reference as if it were a method call. When invoking a closure in this way, we can also drop the parentheses when passing parameters, as shown in the following code:

println( a );
c = 2
print c
printit = { println it }
printit c

These make for a much looser programming style, which is closer to the scripting syntax of Ruby or Python. This is a big benefit when we are using Groovy to build DSLs. When our target audience is nontechnical, being able to drop parentheses and semicolons will make our code much more legible. Consider the following example, where we have two methods, or closures, to get an account by ID and then credit the account with some funds:

Account account = getAccountById( 234 );
creditAccount( account, 100.00 );

With optional types, such as parentheses and semicolons, this can be used to write code that is far more legible to our target audience:

account = getAccountById 234
creditAccount account, 100.00

There are a number of builder classes built in Groovy. There are markup builders for HTML, XML, Ant build scripts, and for Swing GUI building. Markup builders allow us to write code to build a tree-based structure directly within our Groovy code. Unlike API-based approaches for building structures, the tree-like structure of the resulting output is immediately obvious from the structure of our Groovy markup code. Consider the following XML structure:

<?xml version="1.0"?>
<book>
   <author>Fergal Dearle</author>
   <title>Groovy for DSL</title>
</book>

In Groovy markup, this XML can be generated simply with the following code fragment:

def builder = new groovy.xml.MarkupBuilder()
builder.book {
   author 'Fergal Dearle'
   title 'Groovy for DSL'
}

At first glance, this looks like strange special case syntax for markup. It's not! The structure of this code can be explained through the use of closures and the optional syntax that we've discussed in this chapter. We will go into this in great detail in Chapter 5, Groovy Closures, but it is interesting at this point to see how the clever use of some language features can yield a powerful DSL-like markup syntax.

Breaking down the preceding code a little, we can rewrite it as:

def builder = new groovy.xml.MarkupBuilder()
def closure = {
   author 'Fergal Dearle'
   title 'Groovy for DSL'
}
// pass a closure to book method
builder.book( closure)
// which can be written without parentheses
builder.book closure
// or just inline the closure as a parameter
builder.book {
…
}

In other words, the code between the curly braces is in fact a closure, which is passed to the book method of MarkupBuilder. Parentheses being optional, we can simply declare the closure inline after the method name, which gives the neat effect of seeming to mirror the markup structure that we expect in the output.

Similarly, author and title are just method invocations on MarkupBuilder with the optional parentheses missing. Extending this paradigm a little further, we can decide to have author take a closure parameter as well:

def builder = new groovy.xml.MarkupBuilder()
builder.book {
   author {
                first_name 'Fergal'
                surname 'Dearle'
         }
   title 'Groovy for DSL'
}

This will output the following nested XML structure:

<?xml version="1.0"?>
<book>
   <author>
              <first_name>Fergal</first_name>
              <surname> Dearle</surname>
   </author>
   <title>Groovy for DSL</title>
</book>

The method calls on MarkupBuilder start off by outputting an opening XML tag, after which they invoke the closure if one has been passed. Finally, the XML tag is properly terminated before the method exits. If we analyze what happens in sequence, we can see that book invokes a closure that contains a call to author. Additionally, the author tag contains a closure with calls to first_name, surname, and so on.

Before you go to the Groovy documentation for MarkupBuilder to look for the book, author, and surname methods in MarkupBuilder, let me save you the effort. They don't exist. These are what we call pretend methods. We will see later in the book how Groovy's metaprogramming features allow us to invoke methods on closure that don't really exist, but have them do something useful anyway.

Already, we are seeing how some of the features of the Groovy language can coalesce to allow the structuring of a very useful DSL. I use the term DSL here for Groovy builders because that is essentially what they are. What initially looks like special language syntax for markup is revealed as being regular closures with a little bit of clever metaprogramming. The result is an embedded or internal DSL for generating markup.