Grammars

Perl 6 has brought out an extremely useful and powerful mechanism to accomplish regexes—grammars.

Grammars are a mini-language inside Perl 6 that allows you to describe the rules of other languages (including Perl 6 itself). With grammars, it is quite easy to create a parser, a translator, or a compiler of a domain-specific language (DSL) or a programming language, or even a parser that can work with human languages.

In this chapter, we will be learning Perl 6's grammars by way of creating a compiler for the subset of Perl 6. The following topics will be covered in this chapter:

• Creating a grammar
• Elements of grammars—rules and tokens
• The TOP rule
• Whitespace handling
• Parsing texts
• Using actions
• Using an abstract syntax tree (AST)

This chapter assumes that you are familiar with regexes. If you have not read Chapter 11, Regexes, now is the right moment...

Creating a grammar

Like regexes, grammars define some rules to extract information from given text. A typical application for regexes is finding fragments in text chunks and splitting them into meaningful pieces, for example, finding an email or checking if its format is correct. Grammars have a bigger goal—their task is often to read the whole text and understand all its content. For example, if a grammar is applied to a code source written in some programming language, the grammar must check its validity and create the syntax tree of the program. This difference is still a convention—grammars can parse small text sections just as regexes can be used for analyzing big text sections.

The syntax for grammars in Perl 6 is like defining a class. A grammar starts with the grammar keyword:

grammar G {
}

This grammar is empty and cannot be applied to a text. We have to...

Using rules and tokens

Grammars in Perl 6 offers a very useful way to split the grammar elements into parts. Let us use it to clarify the grammar elements.

The complex regex \s* (.*? ';') ['#' <-[\n]>* ]? contains two parts—the regex to extract a statement and the regex for comments. We are extracting them into separate rules. A single rule describes a small piece of the grammar and can refer to other rules. Examine the following example:

grammar G {
    rule TOP {
        ^ 
            [ <statement> \s* <comment>? ]*
        $
    }

    rule statement {
        .*? ';'
    }

    rule comment {
        '#' <-[\n]>*
    }
}

Now the TOP rule is much clearer and you immediately see that the program is a sequence of statements with optional comments after them (our grammar does not allow a comment without a statement).

So...

Using actions

Grammars by themselves do not just parse the source text and extract data pieces from it. To make the program execute the code, actions are needed. Actions in grammars are pieces of Perl 6 code that are triggered when the grammar successfully parses a rule or a token.

Let us take a look at the variable-declaration rule:

rule variable-declaration {
    'my' <variable>
}

When the grammar finds the sequence my $x in the source text, the rule is satisfied. At this point, you may add an action:

rule variable-declaration {
    'my' <variable> {say 'Declaring a variable'}
}

An action can be a simple alert like this but it also may be much more complex code that will be executed as a reaction to the variable declaration.

To make the action act properly, it needs to know the type of the variable (whether it contains the $ or the @ sigil...

Using abstract syntax tree attributes

Currently, the G grammar only parses the constructions when an integer value is assigned to a variable:

$x = 100;

Let us see how to add support for the following assignments:

$x = $y;

The rule for parsing constructions like $x = 100 used the following rule:

rule assignment {
    <variable> '=' <value> { . . . }
}

On the right-hand side of the equals sign, we see a value, which we can replace with a more general item, expression. In the end, the expression may be any expression that a language understands, such as 10, $x, 10 + 3, or $x + $y, and more. Let us approach that point step by step. First, introduce the expression rule. The problem is that we have to return the value of the expression to the action that makes an assignment.

To keep temporary values, Perl 6 grammars offer the attributes of the abstract syntax tree...

Using the actions class

The more complex a grammar becomes, the more complex the actions. Almost every rule or token in our current grammar has an action. Even if most of the actions are just one or two lines of code, the fact that the Perl 6 code is mixed with the grammar language makes reading the code difficult. Formatting the code also becomes a difficult task, as you need to add more spaces to indent the code properly. In this section, we will see what Perl 6 offers to tackle this issue.

All actions may be moved to a separate class. Thus, the complete grammar contains a grammar itself and an actions class. Correspondence between the grammar rules and tokens and the actions is achieved by simply giving the same names to the methods of the action class. Let us convert our grammar to use the split approach.

First of all, create a class for actions and pass it to the parser:

...

The whole program

We achieved a lot by learning about grammars in Perl 6. Just imagine the program we created can parse another program written in Perl 6!

There is still much room for improvement but you will definitely be able to do that. For example, you can start with implementing support for arrays.

Here is the full code of the compiler that we created in this chapter:

grammar G {
    rule TOP {
        ^ 
            [ <statement> \s* <comment>? ]*
        $       
    }

    rule statement {
        [
            | <variable-declaration>
            | <assignment>
            | <say-function>
        ]
        ';'
    }

    rule comment {
        '#' <-[\n]>*
    }

    rule variable-declaration {
        'my' <variable>
    }

    token variable {
        <sigil> <identifier>
    }

    token sigil {
        &apos...

Summary

In this chapter, we discussed grammars, the leading new feature of Perl 6. Grammars allow building parsers for your own domain-specific languages and are already built-in to the language, so no external modules are required to start using them.

Using the example of a compiler of the subset of Perl 6, we created a grammar and looked at its elements—rules and tokens. Later we updated the grammar with actions and finally moved the actions into a separate class to make the code more maintainable and clean.

In the next chapters, we will talk about concurrent, reactive, and functional programming in Perl 6.