+ LuaMacro

1 files changed, 1010 insertions, 0 deletions

diff --git a/Tools/LuaMacro/readme.md b/Tools/LuaMacro/readme.md
new file mode 100644
index 0000000..e86bbfb
--- /dev/null
+++ b/Tools/LuaMacro/readme.md
@@ -0,0 +1,1010 @@
+## LuaMacro - a macro preprocessor for Lua
+
+This is a library and driver script for preprocessing and evaluating Lua code.
+Lexical macros can be defined, which may be simple C-preprocessor style macros or
+macros that change their expansion depending on the context.
+
+It is a new, rewritten version of the
+[Luaforge](http://luaforge.net/projects/luamacro/) project of the same name, which
+required the [token filter
+patch](http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#tokenf) by Luiz Henrique de
+Figueiredo. This patch allowed Lua scripts to filter the raw token stream before
+the compiler stage. Within the limits imposed by the lexical filter approach this
+worked pretty well.  However, the token filter patch is unlikely to ever become
+part of mainline Lua, either in its original or
+[revised](http://lua-users.org/lists/lua-l/2010-02/msg00325.html) form. So the most
+portable option becomes precompilation, but Lua bytecode is not designed to be
+platform-independent and in any case changes faster than the surface syntax of the
+language. So using LuaMacro with LuaJIT would have required re-applying the patch,
+and would remain within the ghetto of specialized, experimental use.
+
+This implementation uses a [LPeg](http://www.inf.puc-rio.br/~roberto/lpeg.html)
+lexical analyser originally by [Peter
+Odding](http://lua-users.org/wiki/LpegRecipes) to tokenize Lua source, and builds
+up a preprocessed string explicitly, which then can be loaded in the usual way.
+This is not as efficient as the original, but it can be used by anyone with a Lua
+interpreter, whether it is Lua 5.1, 5.2 or LuaJIT 2. An advantage of fully building
+the output is that it becomes much easier to debug macros when you can actually see
+the generated results. (Another example of a LPeg-based Lua macro preprocessor is
+[Luma](http://luaforge.net/projects/luma/))
+
+It is not possible to discuss macros in Lua without mentioning Fabien Fleutot's
+[Metalua](metalua.luaforge.net/) which is an alternative Lua compiler which
+supports syntactical macros that can work on the AST (Abstract Syntax Tree) itself
+of Lua. This is clearly a technically superior way to extend Lua syntax, but again
+has the disadvantage of being a direct-to-bytecode compiler. (Perhaps it's also a
+matter of taste, since I find it easier to think about extending Lua on the lexical
+level.)
+
+My renewed interest in Lua lexical macros came from some discussions on the Lua
+mailing list about numerically optimal Lua code using LuaJIT. We have been spoiled
+by modern optimizing C/C++ compilers, where hand-optimization is often discouraged,
+but LuaJIT is new and requires some assistance. For instance, unrolling short loops
+can make a dramatic difference, but Lua does not provide the key concept of
+constant value to assist the compiler. So a very straightforward use of a macro
+preprocessor is to provide named constants in the old-fashioned C way. Very
+efficient code can be generated by generalizing the idea of 'varargs' into a
+statically-compiled 'tuple' type.
+
+    tuple(3) A,B
+
+The assigment `A = B` is expanded as:
+
+    A_1,A_2,A_3 = B_1,B_2,B_3
+
+I will show how the expansion can be made context-sensitive, so that the
+loop-unrolling macro `do_` changes this behaviour:
+
+    do_(i,1,3,
+        A = 0.5*B
+    )
+
+expands to:
+
+    A_1 = 0.5*B_1
+    A_2 = 0.5*B_2
+    A_3 = 0.5*B_3
+
+Another use is crafting DSLs, particularly for end-user scripting. For instance,
+people may be more comfortable with `forall x in t do` rather than `for _,x in
+ipairs(t) do`; there is less to explain in the first form and it translates
+directly to the second form. Another example comes from this common pattern:
+
+    some_action(function()
+      ...
+    end)
+
+Using the following macro:
+
+    def_ block (function() _END_CLOSE_
+
+we can write:
+
+    some_action block
+       ...
+    end
+
+A criticism of traditional lexical macros is that they don't respect the scoping
+rules of the language itself. Bad experiences with the C preprocessor lead many to
+regard them as part of the prehistory of computing. The macros described here can
+be lexically scoped, and can be as 'hygenic' as necessary, since their expansion
+can be finely controlled with Lua itself.
+
+For me, a more serious charge against 'macro magic' is that it can lead to a
+private dialect of the language (the original Bourne shell was written in C
+'skinned' to look like Algol 68.)  This often indicates a programmer uncomfortable
+with a language, who wants it to look like something more familiar. Relying on a
+preprocessor may mean that programmers need to immerse themselves more in the idioms of
+the new language.
+
+That being said, macros can extend a language so that it can be more expressive for
+a particular task, particularly if the users are not professional programmers.
+
+### Basic Macro Substitution
+
+To install LuaMacro, expand the archive and make a script or batch file that points
+to `luam.lua`, for instance:
+
+    lua /home/frodo/luamacro/luam.lua $*
+
+(Or '%*' if on Windows.) Then put this file on your executable path.
+
+Any Lua code loaded with `luam` goes through four distinct steps:
+
+  * loading and defining macros
+  * preprocessing
+  * compilation
+  * execution
+
+The last two steps happen within Lua itself, but always occur, even though the Lua
+compiler is fast enough that we mostly do not bother to save the generated bytecode.
+
+For example, consider this `hello.lua`:
+
+    print(HELLO)
+
+and `hello-def.lua`:
+
+    local macro = require 'macro'
+    macro.define 'HELLO "Hello, World!"'
+
+To run the program:
+
+    $> luam -lhello-def hello.lua
+    Hello, World!
+
+So the module `hello-def.lua` is first loaded (compiled and executed, but not
+preprocessed) and only then `hello.lua` can be preprocessed and then loaded.
+
+Naturaly, there are easier ways to use LuaMacro, but I want to emphasize the
+sequence of macro loading, preprocessing and script loading. `luam` has a `-d`
+flag, meaning 'dump', which is very useful when debugging the output of the
+preprocessing step:
+
+    $> luam -d -lhello-def hello.lua
+    print("Hello, World!")
+
+`hello2.lua` is a more sensible first program:
+
+    require_ 'hello-def'
+    print(HELLO)
+
+You cannot use the Lua `require` function at this point, since `require` is only
+executed when the program starts executing and we want the macro definitions to be
+available during the current compilation. `require_` is the macro version, which
+loads the file at compile-time.
+
+New with 2.5 is the default @ shortcut available when using `luam`,
+so `require_` can be written `@require`.
+(`@` is itself a macro, so you can redefine it if needed.)
+
+There is also `include_/@include`, which is analogous to `#include` in `cpp`. It takes a
+file path in quotes, and directly inserts the contents of the file into the current
+compilation. Although tempting to use, it will not work here because again the
+macro definitions will not be available at compile-time.
+
+`hello3.lua` fits much more into the C preprocessor paradigm, which uses the `def_`
+macro:
+
+    @def HELLO "Hello, World!"
+    print(HELLO)
+
+(Like `cpp`, such macro definitions end with the line; however, there is no
+equivalent of `\` to extend the definition over multiple lines.)
+
+With 2.1, an alternative syntax `def_ (name body)` is also available, which can be
+embedded inside a macro expression:
+
+    def_ OF_ def_ (of elseif _value ==)
+
+Or even extend over several lines:
+
+    def_ (complain(msg,n)
+      for i = 1,n do
+        print msg
+      end
+    )
+
+`def_` works pretty much like `#define`, for instance, `def_ SQR(x) ((x)*(x))`. A
+number of C-style favourites can be defined, like `assert_` using `_STR_`, which is
+a predefined macro that 'stringifies' its argument.
+
+    def_ assert_(condn) assert(condn,_STR_(condn))
+
+`def_` macros are _lexically scoped_:
+
+    local X = 1
+    if something then
+        def_ X 42
+        assert(X == 42)
+    end
+    assert(X == 1)
+
+LuaMacro keeps track of Lua block structure - in particular it knows when a
+particular lexical scope has just been closed.  This is how the `_END_CLOSE_`
+built-in macro works
+
+    def_ block (function() _END_CLOSE_
+
+    my_fun block
+      do_something_later()
+    end
+
+When the current scope closes with `end`, LuaMacro appends the necessary ')' to
+make this syntax valid.
+
+A common use of macros in both C and Lua is to inline optimized code for a case.
+The Lua function `assert()` always evaluates its second argument, which is not
+always optimal:
+
+    def_ ASSERT(condn,expr) if condn then else error(expr) end
+
+    ASSERT(2 == 1,"damn! ".. 2 .." is not equal to ".. 1)
+
+If the message expression is expensive to execute, then this can give better
+performance at the price of some extra code. `ASSERT` is now a statement, not a
+function, however.
+
+### Conditional Compilation
+
+For this to work consistently, you need to use the `@` shortcut:
+
+    @include 'test.inc'
+    @def A 10
+    ...
+
+This makes macro 'preprocessor' statements stand out more. Conditional compilation
+works as you would expect from C:
+
+    -- test-cond.lua
+    @if A
+    print 'A defined'
+    @else
+    print 'A not defined'
+    @end
+    @if os.getenv 'P'
+    print 'Env P is defined'
+    @end
+
+Now, what is `A`?  It is a Lua expression which is evaluated at _preprocessor_
+time, and if it returns any value except `nil` or `false` it is true, using
+the usual Lua rule. Assuming `A` is just a global variable, how can it be set?
+
+    $ luam test-cond.lua
+    A not defined
+    $ luam -VA test-cond.lua
+    A defined
+    $ export P=1
+    $ luam test-cond.lua
+    A not defined
+    Env P is defined
+
+Although this looks very much like the standard C preprocessor, the implementation
+is rather different - `@if` is a special macro which evaluates its argument
+(everything on the rest of the line) as a _Lua expression_
+and skips upto `@end` (or `@else` or `@elseif`) if that condition is false.
+
+
+### Using macro.define
+
+`macro.define` is less convenient than `def_` but much more powerful. The extended
+form allows the substitution to be a _function_ which is called in-place at compile
+time. These definitions must be loaded before they can be used,
+either with `-l` or with `@require`.
+
+    macro.define('DATE',function()
+        return '"'..os.date('%c')..'"'
+    end)
+
+Any text which is returned will be tokenized and inserted into the output stream.
+The explicit quoting here is needed to ensure that `DATE` will be replaced by the
+string "04/30/11 09:57:53".  ('%c' gives you the current locale's version of the
+date; for a proper version of this macro, best to use `os.date` [with more explicit
+formats](http://www.lua.org/pil/22.1.html) .)
+
+This function can also return nothing, which allows you to write macro code purely
+for its _side-effects_.
+
+Non-operator characters like `@`,`$`, etc can be used as macros. For example, say
+you like shell-like notation `$HOME` for expanding environment variables in your
+scripts.
+
+    macro.define '$(x) os.getenv(_STR_(x))'
+
+A script can now say `$(PATH)` and get the expected expansion, Make-style. But we
+can do better and support `$PATH` directly:
+
+    macro.define('$',function(get)
+        local var = get:iden()
+        return 'os.getenv("'..var..'")'
+    end)
+
+If a macro has no parameters, then the substitution function receives a 'getter'
+object. This provides methods for extracting various token types from the input
+stream. Here the `$` macro must be immediately followed by an identifier.
+
+We can do better, and define `$` so that something like `$(pwd)` has the same
+meaning as the Unix shell:
+
+    macro.define('$',function(get)
+       local t,v = get()
+       if t == 'iden' then
+          return 'os.getenv("'..v..'")'
+       elseif t == '(' then
+          local rest = get:upto ')'
+          return 'os.execute("'..tostring(rest)..'")'
+       end
+    end)
+
+(The getter `get` is callable, and returns the type and value of the next token.)
+
+It is probably a silly example, but it illustrates how a macro can be overloaded
+based on its lexical context. Much of the expressive power of LuaMacro comes from
+allowing macros to fetch their own parameters in this way. It allows us to define
+new syntax and go beyond 'pseudo-functions', which is more important for a
+conventional-syntax language like Lua, rather than Lisp where everything looks like
+a function anyway. These kinds of macros are called 'reader' macros in the Lisp world,
+since they temporarily take over reading code.
+
+It is entirely possible for macros to create macros; that is what `def_` does.
+Consider how to add the concept of `const` declarations to Lua:
+
+    const N,M = 10,20
+
+Here is one solution:
+
+    macro.define ('const',function(get)
+        get() -- skip the space
+        local vars = get:idens '='
+        local values = get:list '\n'
+        for i,name in ipairs(vars) do
+            macro.assert(values[i],'each constant must be assigned!')
+            macro.define_scoped(name,tostring(values[i]))
+        end
+    end)
+
+The key to making these constants well-behaved is `define_scoped`, which installs a
+block handler which resets the macro to its original value, which is usually `nil`.
+This test script shows how the scoping works:
+
+    require_ 'const'
+    do
+      const N,M = 10,20
+      do
+         const N = 5
+         assert(N == 5)
+      end
+      assert(N == 10 and M == 20)
+    end
+    assert(N == nil and M == nil)
+
+
+If we were designing a DSL intended for non-technical users, then we cannot just
+say to them 'learn the language properly - go read PiL!'. It would be easier to
+explain:
+
+    forall x in {10,20,30} do
+
+than the equivalent generic `for` loop. `forall` can be implemented fairly simply
+as a macro:
+
+    macro.define('forall',function(get)
+      local var = get:iden()
+      local t,v = get:next() -- will be 'in'
+      local rest = tostring(get:upto 'do')
+      return ('for _,%s in ipairs(%s) do'):format(var,rest)
+    end)
+
+That is, first get the loop variable, skip `in`, grab everything up to `do` and
+output the corresponding `for` statement.
+
+Useful macros can often be built using these new forms. For instance, here is a
+simple list comprehension macro:
+
+    macro.define('L(expr,select) '..
+        '(function() local res = {} '..
+        '  forall select do res[#res+1] = expr end '..
+        'return res end)()'
+    )
+
+For example, `L(x^2,x in t)` will make a list of the squares of all elements in `t`.
+
+Why don't we use a long string here? Because we don't wish to insert any extra line
+feeds in the output.`macro.forall` defines more sophisticated `forall` statements
+and list comprehension expressions, but the principle is the same - see 'tests/test-forall.lua'
+
+There is a second argument passed to the substitution function, which is a 'putter'
+object - an object for building token lists. For example, a useful shortcut for
+anonymous functions:
+
+    M.define ('\\',function(get,put)
+        local args = get:idens('(')
+        local body = get:list()
+        return put:keyword 'function' '(' : idens(args) ')' :
+            keyword 'return' : list(body) : space() : keyword 'end'
+    end)
+
+The `put` object has methods for appending particular kinds of tokens, such as
+keywords and strings, and is also callable for operator tokens. These always return
+the object itself, so the output can be built up with chaining.
+
+Consider `\x,y(x+y)`: the `idens` getter grabs a comma-separated list of identifier
+names upto the given token; the `list` getter grabs a general argument list. It
+returns a list of token lists and by default stops at ')'.  This 'lambda' notation
+was suggested by Luiz Henrique de Figueiredo as something easily parsed by any
+token-filtering approach - an alternative notation `|x,y| x+y` has been
+[suggested](http://lua-users.org/lists/lua-l/2009-12/msg00071.html) but is
+generally impossible to implement using a lexical scanner, since it would have to
+parse the function body as an expression. The `\\` macro also has the advantage
+that the operator precedence is explicit: in the case of `\\(42,'answer')` it is
+immediately clear that this is a function of no arguments which returns two values.
+
+I would not necessarily suggest that lambdas are a good thing in
+production code, but they _can_ be useful in iteractive exploration and within tests.
+
+Macros with explicit parameters can define a substitution function, but this
+function receives the values themselves, not the getter and putter objects. These
+values are _token lists_ and must be converted into the expected types using the
+token list methods:
+
+    macro.define('test_(var,start,finish)',function(var,start,finish)
+        var,start,finish = var:get_iden(),start:get_number(),finish:get_number()
+        print(var,start,finish)
+    end)
+
+
+Since no `put` object is received, such macros need to construct their own:
+
+        local put = M.Putter()
+        ...
+        return put
+
+(They can of course still just return the substitution as text.)
+
+### Dynamically controlling macro expansion
+
+Consider this loop-unrolling macro:
+
+    do_(i,1,3,
+       y = y + i
+    )
+
+which will expand as
+
+    y = y + 1
+    y = y + 2
+    y = y + 3
+
+For each iteration, it needs to define a local macro `i` which expands to 1,2 and 3.
+
+    macro.define('do_(v,s,f,stat)',function(var,start,finish,statements)
+        local put = macro.Putter()
+        var,start,finish = var:get_iden(),start:get_number(),finish:get_number()
+        macro.push_token_stack('do_',var)
+        for i = start, finish do
+            -- output `set_ <var> <value> `
+            put:iden 'set_':iden(var):number(i):space()
+            put:tokens(statements)
+        end
+        -- output `undef_ <var> <value>`
+        put:iden 'undef_':iden(var)
+        -- output `_POP_ 'do_'`
+        put:iden '_DROP_':string 'do_'
+        return put
+    end)
+
+Ignoring the macro stack manipulation for a moment, it works by inserting `set_`
+macro assignments into the output. That is, the raw output looks like this:
+
+    set_ i 1
+    y = y + i
+    set_ i 2
+    y = y + i
+    set_ i 2
+    y = y + i
+    undef_ i
+    _DROP_ 'do_'
+
+It's important here to understand that LuaMacro does not do _recursive_
+substitution. Rather, the output of macros is pushed out to the stream which is
+then further substituted, etc. So we do need these little helper macros to set the
+loop variable at each point.
+
+Using the macro stack allows macros to be aware that they are expanding inside a
+`do_` macro invocation.  Consider `tuple`, which is another macro which creates
+macros:
+
+    tuple(3) A,B
+    A = B
+
+which would expand as
+
+    local A_1,A_2,A_3,B_1,B_2,B_3
+    A_1,A_2,A_3 = B_1,B_2,B_3
+
+But we would like
+
+    do_(i,1,3,
+      A = B/2
+    )
+
+to expand as
+
+    A_1 = B_1/2
+    A_2 = B_2/2
+    A_2 = B_2/2
+
+And here is the definition:
+
+    macro.define('tuple',function(get)
+        get:expecting '('
+        local N = get:number()
+        get:expecting ')'
+        get:expecting 'space'
+        local names = get:idens '\n'
+        for _,name in ipairs(names) do
+            macro.define(name,function(get,put)
+                local loop_var = macro.value_of_macro_stack 'do_'
+                if loop_var then
+                    local loop_idx = tonumber(macro.get_macro_value(loop_var))
+                    return put:iden (name..'_'..loop_idx)
+                else
+                    local out = {}
+                    for i = 1,N do
+                        out[i] = name..'_'..i
+                    end
+                    return put:idens(out)
+                end
+            end)
+        end
+    end)
+
+The first expansion case happens if we are not within a `do_` macro; a simple list
+of names is outputted.  Otherwise, we know what the loop variable is, and can
+directly ask for its value.
+
+### Operator Macros
+
+You can of course define `@` to be a macro; a new feature allows you to add new
+operator tokens:
+
+    macro.define_tokens {'##','@-'}
+
+which can then be used with `macro.define`, but also now with `def_`. It's now
+possible to define a list comprehension syntax that reads more naturally, e.g.
+`{|x^2| i=1,10}` by making `{|` into a new token.
+
+Up to now, making a Lua operator token such as `.` into a macro was not so useful.
+Such a macro may now return an extra value which indicates that the operator should
+simply 'pass through' as is.  Consider defining a `with` statement:
+
+    with A do
+        .x = 1
+        .y = 2
+    end
+
+I've deliberately indicated the fields using a dot (a rare case of Visual Basic
+syntax being superior to Delphi).  So it is necessary to overload '.' and look at
+the previous token: if it isn't a case like `name.` or `].` then we prepend the
+table. Otherwise, the operator must simply _pass through_, to prevent an
+uncontrolled recursion.
+
+    M.define('with',function(get,put)
+      M.define_scoped('.',function()
+        local lt,lv = get:peek(-1,true) --  peek before the period...
+        if lt ~= 'iden' and lt ~= ']' then
+          return '_var.'
+        else
+          return nil,true -- pass through
+        end
+      end)
+      local expr = get:upto 'do'
+      return 'do local _var = '..tostring(expr)..'; '
+    end)
+
+Again, scoping means that this behaviour is completely local to the with-block.
+
+A more elaborate experiment is `cskin.lua` in the tests directory. This translates
+a curly-bracket form into standard Lua, and at its heart is defining '{' and '}' as
+macros. You have to keep a brace stack, because these tokens still have their old
+meaning and the table constructor in this example must still work, while the
+trailing brace must be converted to `end`.
+
+    if (a > b) {
+       t = {a,b}
+    }
+
+### Pass-Through Macros
+
+Normally a macro replaces the name (plus any arguments) with the substitution. It
+is sometimes useful to pass the name through, but not to push the name into the
+token stream - otherwise we will get an endless expansion.
+
+    macro.define('fred',function()
+      print 'fred was found'
+      return nil, true
+    end)
+
+This has absolutely no effect on the preprocessed text ('fred' remains 'fred', but
+has a side-effect. This happens if the substitution function returns a second
+`true` value.  You can look at the immediate lexical environment with `peek`:
+
+    macro.define('fred',function(get)
+        local t,v = get:peek(1)
+        if t == 'string' then
+            local str = get:string()
+            return 'fred_'..str
+        end
+        return nil,true
+    end)
+
+Pass-through macros are useful when each macro corresponds to a Lua variable; they
+allow such variables to have a dual role.
+
+An example would be Python-style lists. The [Penlight
+List](http://stevedonovan.github.com/Penlight/api/modules/pl.List.html) class has
+the same functionality as the built-in Python list, but does not have any
+syntactical support:
+
+    > List = require 'pl.List'
+    > ls = List{10,20,20}
+    > = ls:slice(1,2)
+    {10,20}
+    > ls:slice_assign(1,2,{10,11,20,21})
+    > = ls
+    {10,11,20,21,30}
+
+It would be cool if we could add a little bit of custom syntax to make this more
+natural.  What we first need is a 'macro factory' which outputs the code to create
+the lists, and also suitable macros with the same names.
+
+    -- list <var-list> [ = <init-list> ]
+    M.define ('list',function(get)
+        get() -- skip space
+        -- 'list' acts as a 'type' followed by a variable list, which may be
+        -- followed by initial values
+        local values
+        local vars,endt = get:idens (function(t,v)
+            return t == '=' or (t == 'space' and v:find '\n')
+        end)
+        -- there is an initialization list
+        if endt[1] == '=' then
+            values,endt = get:list '\n'
+        else
+            values = {}
+        end
+        -- build up the initialization list
+        for i,name in ipairs(vars) do
+           M.define_scoped(name,list_check)
+           values[i] = 'List('..tostring(values[i] or '')..')'
+        end
+        local lcal = M._interactive and '' or 'local '
+        return lcal..table.concat(vars,',')..' = '..table.concat(values,',')..tostring(endt)
+    end)
+
+Note that this is a fairly re-usable pattern; it requires the type constructor
+(`List` in this case) and a type-specific macro function (`list_check`). The only
+tricky bit is handling the two cases, so the `idens` method finds the end using a
+function, not a simple token.  `idens`, like `list`, returns the list and the token
+that ended the list, so we can use `endt` to check.
+
+    list a = {1,2,3}
+    list b
+
+becomes
+
+    local a = List({1,2,3})
+    local b = List()
+
+unless we are in interactive mode, where `local` is not appropriate!
+
+Each of these list macro/variables may be used in several ways:
+
+  - directly `a` - no action!
+  - `a[i]` - plain table index
+  - `a[i:j]` - a list slice. Will be `a:slice(i,j)` normally, but must
+  be `a:slice_assign(i,j,RHS)` if on the right-hand side of an assignment.
+
+The substitution function checks these cases by appropriate look-ahead:
+
+    function list_check (get,put)
+        local t,v = get:peek(1)
+        if t ~= '[' then return nil, true end -- pass-through; plain var reference
+        get:expecting '['
+        local args = get:list(']',':')
+        -- it's just plain table access
+        if #args == 1 then return '['..tostring(args[1])..']',true end
+
+        -- two items separated by a colon; use sensible defaults
+        M.assert(#args == 2, "slice has two arguments!")
+        local start,finish = tostring(args[1]),tostring(args[2])
+        if start == '' then start = '1' end
+        if finish == '' then finish = '-1' end
+
+        -- look ahead to see if we're on the left hand side of an assignment
+        if get:peek(1) == '=' then
+           get:next() -- skip '='
+           local rest,eoln = get:upto '\n'
+           rest,eoln = tostring(rest),tostring(eoln)
+           return (':slice_assign(%s,%s,%s)%s'):format(start,finish,rest,eoln),true
+        else
+            return (':slice(%s,%s)'):format(start,finish),true
+        end
+    end
+
+This can be used interactively, like so (it requires the Penlight list library.)
+
+    $> luam  -llist -i
+    Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
+    Lua Macro 2.3.0 Copyright (C) 2007-2011 Steve Donovan
+    > list a = {'one','two'}
+    > = a:map(\x(x:sub(1,1)))
+    {o,t}
+    > a:append 'three'
+    > a:append 'four'
+    > = a
+    {one,two,three,four}
+    > = a[2:3]
+    {two,three}
+    > = a[2:2] = {'zwei','twee'}
+    {one,zwei,twee,three,four}
+    > = a[1:2]..{'five'}
+    {one,zwei,five}
+
+### Preprocessing C
+
+With the 2.2 release, LuaMacro can preprocess C files, by the inclusion of a C LPeg
+lexer based on work by Peter Odding. This may seem a semi-insane pursuit, given
+that C already has a preprocessor, (which is widely considered a misfeature.)
+However, the macros we are talking about are clever, they can maintain state, and
+can be scoped lexically.
+
+One of the irritating things about C is the need to maintain separate include
+files. It would be better if we could write a module like this:
+
+
+    // dll.c
+    #include "dll.h"
+
+    export {
+        typedef struct {
+            int ival;
+        } MyStruct;
+    }
+
+    export int one(MyStruct *ms) {
+        return ms->ival + 1
+    }
+
+    export int two(MyStruct *ms) {
+        return 2*ms->ival;
+    }
+
+and have the preprocessor generate an apppropriate header file:
+
+
+    #ifndef DLL_H
+    #define DLL_H
+    typedef struct {
+            int ival;
+        } MyStruct;
+
+    int one(MyStruct *ms) ;
+    int two(MyStruct *ms) ;
+    #endif
+
+The macro `export` is straightforward:
+
+
+    M.define('export',function(get)
+        local t,v = get:next()
+        local decl,out
+        if v == '{' then
+            decl = tostring(get:upto '}')
+            decl = M.substitute_tostring(decl)
+            f:write(decl,'\n')
+        else
+            decl = v .. ' ' .. tostring(get:upto '{')
+            decl = M.substitute_tostring(decl)
+            f:write(decl,';\n')
+            out = decl .. '{'
+        end
+        return out
+    end)
+
+It looks ahead and if it finds a `{}` block it writes the block as text to a file
+stream; otherwise writes out the function signature.  `get:upto '}'` will do the
+right thing here since it keeps track of brace level.  To allow any other macro
+expansions to take place, `substitute_tostring` is directly called.
+
+`tests/cexport.lua` shows how this idea can be extended, so that the generated
+header is only updated when it changes.
+
+To preprocess C with `luam`, you need to specify the `-C` flag:
+
+    luam -C -lcexport -o dll.c dll.lc
+
+Have a look at [lc](modules/macro.lc.html) which defines a simplified way to write
+Lua bindings in C. Here is `tests/str.l.c`:
+
+    // preprocess using luam -C -llc -o str.c str.l.c
+    #include <string.h>
+
+    module "str" {
+
+      def at (Str s, Int i = 0) {
+        lua_pushlstring(L,&s[i-1],1);
+        return 1;
+      }
+
+      def upto (Str s, Str delim = " ") {
+        lua_pushinteger(L, strcspn(s,delim) + 1);
+        return 1;
+      }
+
+    }
+
+The result looks like this:
+
+    // preprocess using luam -C -llc -o str.c str.l.c
+    #line 2 "str.lc"
+    #include <string.h>
+
+    #include <lua.h>
+    #include <lauxlib.h>
+    #include <lualib.h>
+    #ifdef WIN32
+    #define EXPORT __declspec(dllexport)
+    #else
+    #define EXPORT
+    #endif
+    typedef const char *Str;
+    typedef const char *StrNil;
+    typedef int Int;
+    typedef double Number;
+    typedef int Boolean;
+
+
+      #line 6 "str.lc"
+    static int l_at(lua_State *L) {
+        const char *s = luaL_checklstring(L,1,NULL);
+        int i = luaL_optinteger(L,2,0);
+
+        #line 7 "str.lc"
+
+        lua_pushlstring(L,&s[i-1],1);
+        return 1;
+      }
+
+      static int l_upto(lua_State *L) {
+        const char *s = luaL_checklstring(L,1,NULL);
+        const char *delim = luaL_optlstring(L,2," ",NULL);
+
+        #line 12 "str.lc"
+
+        lua_pushinteger(L, strcspn(s,delim) + 1);
+        return 1;
+      }
+
+    static const luaL_reg str_funs[] = {
+           {"at",l_at},
+       {"upto",l_upto},
+        {NULL,NULL}
+    };
+
+    EXPORT int luaopen_str (lua_State *L) {
+        luaL_register (L,"str",str_funs);
+
+        return 1;
+    }
+
+Note the line directives; this makes working with macro-ized C code much easier
+when the inevitable compile and run-time errors occur. `lc` takes away some
+of the more irritating bookkeeping needed in writing C extensions
+(here I only have to mention function names once)
+
+`lc` was used for the [winapi](https://github.com/stevedonovan/winapi) project to
+preprocess [this
+file](https://github.com/stevedonovan/winapi/blob/master/winapi.l.c)
+into [standard C](https://github.com/stevedonovan/winapi/blob/master/winapi.c).
+
+This used an extended version of `lc` which handled the largely superficial
+differences between the Lua 5.1 and 5.2 API.
+
+(The curious thing is that `winapi` is my only project where I've leant on
+LuaMacro, and it's all in C.)
+
+### A Simple Test Framework
+
+LuaMacro comes with yet another simple test framework - I apologize for this in
+advance, because there are already quite enough. But consider it a demonstration
+of how a little macro sugar can make tests more readable, even if you are
+uncomfortable with them in production code (see `tests/test-test.lua`)
+
+    require_ 'assert'
+    assert_ 1 == 1
+    assert_ "hello" matches "^hell"
+    assert_ x.a throws 'attempt to index global'
+
+The last line is more interesting, since it's transparently wrapping
+the offending expression in an anonymous function. The expanded output looks
+like this:
+
+    T_ = require 'macro.lib.test'
+    T_.assert_eq(1 ,1)
+    T_.assert_match("hello" ,"^hell")
+    T_.assert_match(T_.pcall_no(function() return x.a  end),'attempt to index global')
+
+(This is a generally useful pattern - use macros to provide a thin layer of sugar
+over the underlying library. The `macro.assert` module is only 75 lines long, with
+comments - its job is to format code to make using the implementation easier.)
+
+Remember that the predefined meaning of @ is to convert `@name` into `name_`. So we
+could just as easily say `@assert 1 == 1` and so forth.
+
+Lua functions often return multiple values or tables:
+
+    two = \(40,2)
+    table2 = \({40,2})
+    @assert two() == (40,2)
+    @assert table2() == {40,2}
+
+For a proper grown-up Lua testing framework
+that uses LuaMacro, see [Specl](http://gvvaughan.github.io/specl).
+
+
+### Implementation
+
+It is not usually necessary to understand the underlying representation of token
+lists, but I present it here as a guide to understanding the code.
+
+#### Token Lists
+
+The token list representation of the expression `x+1` is:
+
+    {{'iden','x'},{'+','+'},{'number','1'}}
+
+which is the form returned by the LPeg lexical analyser. Please note that there are
+also 'space' and 'comment' tokens in the stream, which is a big difference from the
+token-filter standard.
+
+The `TokenList` type defines `__tostring` and some helper methods for these lists.
+
+The following macro is an example of the lower-level coding needed without the
+usual helpers:
+
+    local macro = require 'macro'
+    macro.define('qw',function(get,put)
+      local append = table.insert
+      local t,v = get()
+      local res = {{'{','{'}}
+      t,v = get:next()
+      while t ~= ')' do
+        if t ~= ',' then
+          append(res,{'string','"'..v..'"'})
+          append(res,{',',','})
+        end
+        t,v = get:next()
+      end
+      append(res,{'}','}'})
+      return res
+    end)
+
+We're using the getter `next` method to skip any whitespace, but building up the
+substitution without a putter, just manipulating the raw token list.  `qw` takes a
+plain list of words, separated by spaces (and maybe commas) and makes it into a
+list of strings. That is,
+
+    qw(one two three)
+
+becomes
+
+    {'one','two','three'}
+
+#### Program Structure
+
+The main loop of `macro.substitute` (towards end of `macro.lua`) summarizes the
+operation of LuaMacro:
+
+There are two macro tables, `imacro` for classic name macros, and `smacro` for
+operator style macros. They contain macro tables, which must have a `subst` field
+containing the substitution and may have a `parms` field, which means that they
+must be followed by their arguments in parentheses.
+
+A keywords table is chiefly used to track block scope, e.g.
+`do`,`if`,`function`,etc means 'increase block level' and `end`,`until` means
+'decrease block level'. At this point, any defined block handlers for this level
+will be evaluated and removed. These may insert tokens into the stream, like
+macros. This is how something like `_END_CLOSE_` is implemented: the `end` causes
+the block level to decrease, which fires a block handler which passes `end` through
+and inserts a closing `)`.
+
+Any keyword may also have an associated keyword handler, which works rather like a
+macro substitution, except that the keyword itself is always passed through first.
+(Allowing keywords as regular macros would generally be a bad idea because of the
+recursive substitution problem.)
+
+The macro `subst` field may be a token list or a function. if it is a function then
+that function is called, with the parameters as token lists if the macro defined
+formal parameters, or with getter and setter objects if not. If the result is text
+then it is parsed into a token list.