+ Penlight

1 files changed, 621 insertions, 0 deletions

diff --git a/Data/Libraries/Penlight/docs_topics/01-introduction.md b/Data/Libraries/Penlight/docs_topics/01-introduction.md
new file mode 100644
index 0000000..a8bf26a
--- /dev/null
+++ b/Data/Libraries/Penlight/docs_topics/01-introduction.md
@@ -0,0 +1,621 @@
+## Introduction
+
+### Purpose
+
+It is often said of Lua that it does not include batteries. That is because the
+goal of Lua is to produce a lean expressive language that will be used on all
+sorts of machines, (some of which don't even have hierarchical filesystems). The
+Lua language is the equivalent of an operating system kernel; the creators of Lua
+do not see it as their responsibility to create a full software ecosystem around
+the language. That is the role of the community.
+
+A principle of software design is to recognize common patterns and reuse them. If
+you find yourself writing things like `io.write(string.format('the answer is %d
+',42))` more than a number of times then it becomes useful just to define a
+function `printf`. This is good, not just because repeated code is harder to
+maintain, but because such code is easier to read, once people understand your
+libraries.
+
+Penlight captures many such code patterns, so that the intent of your code
+becomes clearer. For instance, a Lua idiom to copy a table is `{unpack(t)}`, but
+this will only work for 'small' tables (for a given value of 'small') so it is
+not very robust. Also, the intent is not clear. So `tablex.deepcopy` is provided,
+which will also copy nested tables and and associated metatables, so it can be
+used to clone complex objects.
+
+The default error handling policy follows that of the Lua standard libraries: if
+a argument is the wrong type, then an error will be thrown, but otherwise we
+return `nil,message` if there is a problem. There are some exceptions; functions
+like `input.fields` default to shutting down the program immediately with a
+useful message. This is more appropriate behaviour for a _script_ than providing
+a stack trace. (However, this default can be changed.) The lexer functions always
+throw errors, to simplify coding, and so should be wrapped in `pcall`.
+
+If you are used to Python conventions, please note that all indices consistently
+start at 1.
+
+The Lua function `table.foreach` has been deprecated in favour of the `for in`
+statement, but such an operation becomes particularly useful with the
+higher-order function support in Penlight. Note that `tablex.foreach` reverses
+the order, so that the function is passed the value and then the key. Although
+perverse, this matches the intended use better.
+
+The only important external dependence of Penlight is
+[LuaFileSystem](http://keplerproject.github.com/luafilesystem/manual.html)
+(`lfs`), and if you want `dir.copyfile` to work cleanly on Windows, you will need
+either [alien](http://alien.luaforge.net/) or be using
+[LuaJIT](http://luajit.org) as well. (The fallback is to call the equivalent
+shell commands.)
+
+### To Inject or not to Inject?
+
+It was realized a long time ago that large programs needed a way to keep names
+distinct by putting them into tables (Lua), namespaces (C++) or modules
+(Python).  It is obviously impossible to run a company where everyone is called
+'Bruce', except in Monty Python skits. These 'namespace clashes' are more of a
+problem in a simple language like Lua than in C++, because C++ does more
+complicated lookup over 'injected namespaces'.  However, in a small group of
+friends, 'Bruce' is usually unique, so in particular situations it's useful to
+drop the formality and not use last names. It depends entirely on what kind of
+program you are writing, whether it is a ten line script or a ten thousand line
+program.
+
+So the Penlight library provides the formal way and the informal way, without
+imposing any preference. You can do it formally like:
+
+    local utils = require 'pl.utils'
+    utils.printf("%s\n","hello, world!")
+
+or informally like:
+
+    require 'pl'
+    utils.printf("%s\n","That feels better")
+
+`require 'pl'` makes all the separate Penlight modules available, without needing
+to require them each individually.
+
+Generally, the formal way is better when writing modules, since then there are no
+global side-effects and the dependencies of your module are made explicit.
+
+Andrew Starks has contributed another way, which balances nicely between the
+formal need to keep the global table uncluttered and the informal need for
+convenience. `require'pl.import_into'` returns a function, which accepts a table
+for injecting Penlight into, or if no table is given, it passes back a new one.
+
+    local pl = require'pl.import_into'()
+
+The table `pl` is a 'lazy table' which loads modules as needed, so we can then
+use `pl.utils.printf` and so forth, without an explicit `require' or harming any
+globals.
+
+If you are using `_ENV` with Lua 5.2 to define modules, then here is a way to
+make Penlight available within a module:
+
+    local _ENV,M = require 'pl.import_into' ()
+
+    function answer ()
+        -- all the Penlight modules are available!
+        return pretty.write(utils.split '10 20  30', '')
+    end
+
+    return M
+
+The default is to put Penlight into `\_ENV`, which has the unintended effect of
+making it available from the module (much as `module(...,package.seeall)` does).
+To satisfy both convenience and safety, you may pass `true` to this function, and
+then the _module_ `M` is not the same as `\_ENV`, but only contains the exported
+functions.
+
+Otherwise, Penlight will _not_ bring in functions into the global table, or
+clobber standard tables like 'io'.  require('pl') will bring tables like
+'utils','tablex',etc into the global table _if they are used_. This
+'load-on-demand' strategy ensures that the whole kitchen sink is not loaded up
+front,  so this method is as efficient as explicitly loading required modules.
+
+You have an option to bring the `pl.stringx` methods into the standard string
+table. All strings have a metatable that allows for automatic lookup in `string`,
+so we can say `s:upper()`. Importing `stringx` allows for its functions to also
+be called as methods: `s:strip()`,etc:
+
+    require 'pl'
+    stringx.import()
+
+or, more explicitly:
+
+    require('pl.stringx').import()
+
+A more delicate operation is importing tables into the local environment. This is
+convenient when the context makes the meaning of a name very clear:
+
+    > require 'pl'
+    > utils.import(math)
+    > = sin(1.2)
+    0.93203908596723
+
+`utils.import` can also be passed a module name as a string, which is first
+required and then imported. If used in a module, `import` will bring the symbols
+into the module context.
+
+Keeping the global scope simple is very necessary with dynamic languages. Using
+global variables in a big program is always asking for trouble, especially since
+you do  not have the spell-checking provided by a compiler. The `pl.strict`
+module enforces a simple rule: globals must be 'declared'.  This means that they
+must be assigned before use; assigning to `nil` is sufficient.
+
+    > require 'pl.strict'
+    > print(x)
+    stdin:1: variable 'x' is not declared
+    > x = nil
+    > print(x)
+    nil
+
+The `strict` module provided by Penlight is compatible with the 'load-on-demand'
+scheme used by `require 'pl`.
+
+`strict` also disallows assignment to global variables, except in the main
+program. Generally, modules have no business messing with global scope; if you
+must do it, then use a call to `rawset`. Similarly, if you have to check for the
+existence of a global, use `rawget`.
+
+If you wish to enforce strictness globally, then just add `require 'pl.strict'`
+at the end of `pl/init.lua`, otherwise call it from your main program.
+
+As from 1.1.0, this module provides a `strict.module` function which creates (or
+modifies) modules so that accessing an unknown function or field causes an error.
+
+For example,
+
+    -- mymod.lua
+    local strict = require 'pl.strict'
+    local M = strict.module (...)
+
+    function M.answer ()
+        return 42
+    end
+
+    return M
+
+If you were to accidently type `mymod.Answer()`, then you would get a runtime
+error: "variable 'Answer' is not declared in 'mymod'".
+
+This can be applied to existing modules. You may desire to have the same level
+of checking for the Lua standard libraries:
+
+    strict.make_all_strict(_G)
+
+Thereafter a typo such as `math.cosine` will give you an explicit error, rather
+than merely returning a `nil` that will cause problems later.
+
+### What are function arguments in Penlight?
+
+Many functions in Penlight themselves take function arguments, like `map` which
+applies a function to a list, element by element.  You can use existing
+functions, like `math.max`, anonymous functions (like `function(x,y) return x > y
+end` ), or operations by name (e.g '*' or '..').  The module `pl.operator` exports
+all the standard Lua operations, like the Python module of the same name.
+Penlight allows these to be referred to by name, so `operator.gt` can be more
+concisely expressed as '>'.
+
+Note that the `map` functions pass any extra arguments to the function, so we can
+have `ls:filter('>',0)`, which is a shortcut for
+`ls:filter(function(x) return x > 0 end)`.
+
+Finally, `pl.func` supports _placeholder expressions_ in the Boost lambda style,
+so that an anonymous function to multiply the two arguments can be expressed as
+`\_1*\_2`.
+
+To use them directly, note that _all_ function arguments in Penlight go through
+`utils.function_arg`. `pl.func` registers itself with this function, so that you
+can directly use placeholder expressions with standard methods:
+
+    > _1 = func._1
+    > = List{10,20,30}:map(_1+1)
+    {11,21,31}
+
+Another option for short anonymous functions is provided by
+`utils.string_lambda`; this is invoked automatically:
+
+    > = List{10,20,30}:map '|x| x + 1'
+    {11,21,31}
+
+### Pros and Cons of Loopless Programming
+
+The standard loops-and-ifs 'imperative' style of programming is dominant, and
+often seems to be the 'natural' way of telling a machine what to do. It is in
+fact very much how the machine does things, but we need to take a step back and
+find ways of expressing solutions in a higher-level way.  For instance, applying
+a function to all elements of a list is a common operation:
+
+    local res = {}
+    for i = 1,#ls do
+        res[i] = fun(ls[i])
+    end
+
+This can be efficiently and succintly expressed as `ls:map(fun)`. Not only is
+there less typing but the intention of the code is clearer. If readers of your
+code spend too much time trying to guess your intention by analyzing your loops,
+then you have failed to express yourself clearly. Similarly, `ls:filter('>',0)`
+will give you all the values in a list greater than zero. (Of course, if you
+don't feel like using `List`, or have non-list-like tables, then `pl.tablex`
+offers the same facilities. In fact, the `List` methods are implemented using
+`tablex` functions.)
+
+A common observation is that loopless programming is less efficient, particularly
+in the way it uses memory. `ls1:map2('*',ls2):reduce '+'` will give you the dot
+product of two lists, but an unnecessary temporary list is created.  But
+efficiency is relative to the actual situation, it may turn out to be _fast
+enough_, or may not appear in any crucial inner loops, etc.
+
+Writing loops is 'error-prone and tedious', as Stroustrup says. But any
+half-decent editor can be taught to do much of that typing for you. The question
+should actually be: is it tedious to _read_ loops?  As with natural language,
+programmers tend to read chunks at a time. A for-loop causes no surprise, and
+probably little brain activity. One argument for loopless programming is the
+loops that you _do_ write stand out more, and signal 'something different
+happening here'.  It should not be an all-or-nothing thing, since most programs
+require a mixture of idioms that suit the problem.  Some languages (like APL) do
+nearly everything with map and reduce operations on arrays, and so solutions can
+sometimes seem forced. Wisdom is knowing when a particular idiom makes a
+particular problem easy to _solve_ and the solution easy to _explain_ afterwards.
+
+### Generally useful functions.
+
+The function `printf` discussed earlier is included in `pl.utils` because it
+makes properly formatted output easier. (There is an equivalent `fprintf` which
+also takes a file object parameter, just like the C function.)
+
+Splitting a string using a delimiter is a fairly common operation, hence `split`.
+
+Utility functions like `is_type` help with identifying what
+kind of animal you are dealing with.
+The Lua `type` function handles the basic types, but can't distinguish between
+different kinds of objects, which are all tables. So `is_type` handles both
+cases, like `is_type(s,"string")` and `is_type(ls,List)`.
+
+A common pattern when working with Lua varargs is capturing all the arguments in
+a table:
+
+    function t(...)
+        local args = {...}
+        ...
+    end
+
+But this will bite you someday when `nil` is one of the arguments, since this
+will put a 'hole' in your table. In particular, `#ls` will only give you the size
+upto the `nil` value.  Hence the need for `table.pack` - this is a new Lua 5.2
+function which Penlight defines also for Lua 5.1.
+
+    function t(...)
+        local args,n = table.pack(...)
+        for i = 1,n do
+          ...
+        end
+    end
+
+The 'memoize' pattern occurs when you have a function which is expensive to call,
+but will always return the same value subsequently. `utils.memoize` is given a
+function, and returns another function. This calls the function the first time,
+saves the value for that argument, and thereafter for that argument returns the
+saved value.  This is a more flexible alternative to building a table of values
+upfront, since in general you won't know what values are needed.
+
+    sum = utils.memoize(function(n)
+        local sum = 0
+        for i = 1,n do sum = sum + i end
+        return sum
+    end)
+    ...
+    s = sum(1e8) --takes time!
+    ...
+    s = sum(1e8) --returned saved value!
+
+Penlight is fully compatible with Lua 5.1, 5.2 and LuaJIT 2. To ensure this,
+`utils` also defines the global Lua 5.2
+[load](http://www.lua.org/work/doc/manual.html#pdf-load) function as `utils.load`
+
+ * the input (either a string or a function)
+ * the source name used in debug information
+ * the mode is a string that can have either or both of 'b' or 't', depending on
+whether the source is a binary chunk or text code (default is 'bt')
+ * the environment for the compiled chunk
+
+Using `utils.load` should reduce the need to call the deprecated function `setfenv`,
+and make your Lua 5.1 code 5.2-friendly.
+
+The `utils` module exports `getfenv` and `setfenv` for
+Lua 5.2 as well, based on code by Sergey Rozhenko.  Note that these functions can fail
+for functions which don't access any globals.
+
+### Application Support
+
+`app.parse_args` is a simple command-line argument parser. If called without any
+arguments, it tries to use the global `arg` array.  It returns the _flags_
+(options begining with '-') as a table of name/value pairs, and the _arguments_
+as an array.  It knows about long GNU-style flag names, e.g. `--value`, and
+groups of short flags are understood, so that `-ab` is short for `-a -b`. The
+flags result would then look like `{value=true,a=true,b=true}`.
+
+Flags may take values. The command-line `--value=open -n10` would result in
+`{value='open',n='10'}`; generally you can use '=' or ':' to separate the flag
+from its value, except in the special case where a short flag is followed by an
+integer.  Or you may specify upfront that some flags have associated values, and
+then the values will follow the flag.
+
+	> require 'pl'
+	> flags,args = app.parse_args({'-o','fred','-n10','fred.txt'},{o=true})
+	> pretty.dump(flags)
+	{o='fred',n='10'}
+
+`parse_args` is not intelligent or psychic; it will not convert any flag values
+or arguments for you, or raise errors. For that, have a look at
+@{08-additional.md.Command_line_Programs_with_Lapp|Lapp}.
+
+An application which consists of several files usually cannot use `require` to
+load files in the same directory as the main script.  `app.require_here()`
+ensures that the Lua module path is modified so that files found locally are
+found first. In the `examples` directory, `test-symbols.lua` uses this function
+to ensure that it can find `symbols.lua` even if it is not run from this directory.
+
+`app.appfile` will create a filename that your application can use to store its
+private data, based on the script name. For example, `app.appfile "test.txt"`
+from a script called `testapp.lua` produces the following file on my Windows
+machine:
+
+    @plain
+	C:\Documents and Settings\SJDonova\.testapp\test.txt
+
+and the equivalent on my Linux machine:
+
+    @plain
+	/home/sdonovan/.testapp/test.txt
+
+If `.testapp` does not exist, it will be created.
+
+Penlight makes it convenient to save application data in Lua format. You can use
+`pretty.dump(t,file)` to write a Lua table in a human-readable form to a file,
+and `pretty.read(file.read(file))` to generate the table again, using the
+`pretty` module.
+
+
+### Simplifying Object-Oriented Programming in Lua
+
+Lua is similar to JavaScript in that the concept of class is not directly
+supported by the language. In fact, Lua has a very general mechanism for
+extending the behaviour of tables which makes it straightforward to implement
+classes. A table's behaviour is controlled by its metatable. If that metatable
+has a `\_\_index` function or table, this will handle looking up anything which is
+not found in the original table. A class is just a table with an `__index` key
+pointing to itself. Creating an object involves making a table and setting its
+metatable to the class; then when handling `obj.fun`, Lua first looks up `fun` in
+the table `obj`, and if not found it looks it up in the class. `obj:fun(a)` is
+just short for `obj.fun(obj,a)`. So with the metatable mechanism and this bit of
+syntactic sugar, it is straightforward to implement classic object orientation.
+
+    -- animal.lua
+
+    class = require 'pl.class'
+
+    class.Animal()
+
+    function Animal:_init(name)
+        self.name = name
+    end
+
+    function Animal:__tostring()
+      return self.name..': '..self:speak()
+    end
+
+    class.Dog(Animal)
+
+    function Dog:speak()
+      return 'bark'
+    end
+
+    class.Cat(Animal)
+
+    function Cat:_init(name,breed)
+        self:super(name)  -- must init base!
+        self.breed = breed
+    end
+
+    function Cat:speak()
+      return 'meow'
+    end
+
+    class.Lion(Cat)
+
+    function Lion:speak()
+      return 'roar'
+    end
+
+    fido = Dog('Fido')
+    felix = Cat('Felix','Tabby')
+    leo = Lion('Leo','African')
+
+    $ lua -i animal.lua
+    > = fido,felix,leo
+    Fido: bark      Felix: meow     Leo: roar
+    > = leo:is_a(Animal)
+    true
+    > = leo:is_a(Dog)
+    false
+    > = leo:is_a(Cat)
+    true
+
+All Animal does is define `\_\_tostring`, which Lua will use whenever a string
+representation is needed of the object. In turn, this relies on `speak`, which is
+not defined. So it's what C++ people would call an abstract base class; the
+specific derived classes like Dog define `speak`. Please note that _if_ derived
+classes have their own constructors, they must explicitly call the base
+constructor for their base class; this is conveniently available as the `super`
+method.
+
+Note that (as always) there are multiple ways to implement OOP in Lua; this method
+uses the classic 'a class is the __index of its objects' but does 'fat inheritance';
+methods from the base class are copied into the new class. The advantage of this is
+that you are not penalized for long inheritance chains, for the price of larger classes,
+but generally objects outnumber classes! (If not, something odd is going on with your design.)
+
+All such objects will have a `is_a` method, which looks up the inheritance chain
+to find a match.  Another form is `class_of`, which can be safely called on all
+objects, so instead of `leo:is_a(Animal)` one can say `Animal:class_of(leo)`.
+
+There are two ways to define a class, either `class.Name()` or `Name = class()`;
+both work identically, except that the first form will always put the class in
+the current environment (whether global or module); the second form provides more
+flexibility about where to store the class. The first form does _name_ the class
+by setting the `_name` field, which can be useful in identifying the objects of
+this type later. This session illustrates the usefulness of having named classes,
+if no `__tostring` method is explicitly defined.
+
+    > class.Fred()
+    > a = Fred()
+    > = a
+    Fred: 00459330
+    > Alice = class()
+    > b = Alice()
+    > = b
+    table: 00459AE8
+    > Alice._name = 'Alice'
+    > = b
+    Alice: 00459AE8
+
+So `Alice = class(); Alice._name = 'Alice'` is exactly the same as `class.Alice()`.
+
+This useful notation is borrowed from Hugo Etchegoyen's
+[classlib](http://lua-users.org/wiki/MultipleInheritanceClasses) which further
+extends this concept to allow for multiple inheritance. Notice that the
+more convenient form puts the class name in the _current environment_! That is,
+you may use it safely within modules using the old-fashioned `module()`
+or the new `_ENV` mechanism.
+
+There is always more than one way of doing things in Lua; some may prefer this
+style for creating classes:
+
+    local class = require 'pl.class'
+
+    class.Named {
+        _init = function(self,name)
+            self.name = name
+        end;
+
+        __tostring = function(self)
+            return 'boo '..self.name
+        end;
+    }
+
+    b = Named 'dog'
+    print(b)
+    --> boo dog
+
+Note that you have to explicitly declare `self` and end each function definition
+with a semi-colon or comma, since this is a Lua table. To inherit from a base class,
+set the special field `_base` to the class in this table.
+
+Penlight provides a number of useful classes; there is `List`, which is a Lua
+clone of the standard Python list object, and `Set` which represents sets. There
+are three kinds of _map_ defined: `Map`, `MultiMap` (where a key may have
+multiple values) and `OrderedMap` (where the order of insertion is remembered.).
+There is nothing special about these classes and you may inherit from them.
+
+A powerful thing about dynamic languages is that you can redefine existing classes
+and functions, which is often called 'monkey patching' It's entertaining and convenient,
+but ultimately anti-social; you may modify `List` but then any other modules using
+this _shared_ resource can no longer be sure about its behaviour. (This is why you
+must say `stringx.import()` explicitly if you want the extended string methods - it
+would be a bad default.)  Lua is particularly open to modification but the
+community is not as tolerant of monkey-patching as the Ruby community, say. You may
+wish to add some new methods to `List`? Cool, but that's what subclassing is for.
+
+    class.Strings(List)
+
+    function Strings:my_method()
+    ...
+    end
+
+It's definitely more useful to define exactly how your objects behave
+in _unknown_ conditions. All classes have a `catch` method you can use to set
+a handler for unknown lookups; the function you pass looks exactly like the
+`__index` metamethod.
+
+    Strings:catch(function(self,name)
+        return function() error("no such method "..name,2) end
+    end)
+
+In this case we're just customizing the error message, but
+creative things can be done. Consider this code from `test-vector.lua`:
+
+    Strings:catch(List.default_map_with(string))
+
+    ls = Strings{'one','two','three'}
+    asserteq(ls:upper(),{'ONE','TWO','THREE'})
+    asserteq(ls:sub(1,2),{'on','tw','th'})
+
+So we've converted a unknown method invocation into a map using the function of
+that name found in `string`.  So for a `Vector` (which is a specialization of `List`
+for numbers) it makes sense to make `math` the default map so that `v:sin()` makes
+sense.
+
+Note that `map` operations return a object of the same type - this is often called
+_covariance_. So `ls:upper()` itself returns a `Strings` object.
+
+This is not _always_ what you want, but objects can always be cast to the desired type.
+(`cast` doesn't create a new object, but returns the object passed.)
+
+    local sizes = ls:map '#'
+    asserteq(sizes, {3,3,5})
+    asserteq(utils.type(sizes),'Strings')
+    asserteq(sizes:is_a(Strings),true)
+    sizes = Vector:cast(sizes)
+    asserteq(utils.type(sizes),'Vector')
+    asserteq(sizes+1,{4,4,6})
+
+About `utils.type`: it can only return a string for a class type if that class does
+in fact have a `_name` field.
+
+
+_Properties_ are a useful object-oriented pattern. We wish to control access to a
+field, but don't wish to force the user of the class to say `obj:get_field()`
+etc. This excerpt from `tests/test-class.lua` shows how it is done:
+
+
+    local MyProps = class(class.properties)
+    local setted_a, got_b
+
+    function MyProps:_init ()
+        self._a = 1
+        self._b = 2
+    end
+
+    function MyProps:set_a (v)
+        setted_a = true
+        self._a = v
+    end
+
+    function MyProps:get_b ()
+        got_b = true
+        return self._b
+    end
+
+    local mp = MyProps()
+
+    mp.a = 10
+
+    asserteq(mp.a,10)
+    asserteq(mp.b,2)
+    asserteq(setted_a and got_b, true)
+
+The convention is that the internal field name is prefixed with an underscore;
+when reading `mp.a`, first a check for an explicit _getter_ `get_a` and then only
+look for `_a`. Simularly, writing `mp.a` causes the _setter_ `set_a` to be used.
+
+This is cool behaviour, but like much Lua metaprogramming, it is not free. Method
+lookup on such objects goes through `\_\_index` as before, but now `\_\_index` is a
+function which has to explicitly look up methods in the class, before doing any
+property indexing, which is not going to be as fast as field lookup. If however,
+your accessors actually do non-trivial things, then the extra overhead could be
+worth it.
+
+This is not really intended for _access control_ because external code can write
+to `mp._a` directly. It is possible to have this kind of control in Lua, but it
+again comes with run-time costs.