Three of Coins » yaclml

Yaclml in pictures, part II: Templating

Maciej — Thu, 21 Jan 2010 14:00:47 +0000

After a short intermission and explanation/excuse, let’s go on interesting stuff, Yaclml; and if you still didn’t read the first part, where I wrote about HTML generation and compared Yaclml to CL-WHO, do so now!

Continuing first part’s focus on comparing Yaclml to Ediware, let’s start by mentioning that CL-WHO doesn’t do templating at all; Edi Weitz wrote a separate library for this: HTML-Template. It provides simple templating support, which turns template of a HTML file (or actually any text file, not limited to HTML) to a closure which, when called, can fill the template with supplied values and output result to a stream. Simple enough, based on Perl’s HTML::Template—which shows in the templating language syntax. The template directives of HTML-Template are embedded in HTML comments.

Yaclml includes the templating feature, and it chose a different path. For me, with a tiny bit of Zope background, their path is a little nicer: they decided to support a Lisp variant of Zope’s Template Attribute Language. TAL is strictly an XHTML and XML templating language, whose directives are special XML tags and attributes, living in a separate XML namespace. I find this approach much more elegant than magic directives embedded in comments. This also makes it possible to use XML/XHTML editing tools (such as Emacs’ nxml-mode and excellent nxhtml-mode built on top of it) to aid in authoring and validation of the code. Supposedly it also plays fine with visual design tools, such as Adobe Dreamweaver, but I can’t confirm that, since I don’t use those.

In this section, I won’t go on comparing Yaclml to HTML-Template, because I didn’t use much of the latter, those two are not as similar as CL-WHO and Yaclml’s HTML generation, and would require non-trivial examples to really see meaningful differences; besides, Yaclml’s TAL is obviously better, and anyone will see it from TAL alone. ;) Seriously, HTML-Template’s documentation is great, it’s a simple package, and you can compare it for yourself. These two packages are similar, main difference being TAL’s focus on XML and XHTML, and syntactic differences are much more of a matter of taste than it is with HTML generation. However, Yaclml, and especially its TAL support, is underdocumented (there was some tutorial over at UCW Wiki, but at the moment it’s inaccessible, and only option to read it is Google’s cache) and it’s really hard to figure out how to use it, especially outside of UCW framework; that’s the hole I’m trying to fill here.

During writing this article, I’ve found that Yaclml is actually somewhat documented, in the qbook format, but the generated, readable docs are nowhere to be found. I’ve rebuilt the HTML files and uploaded them at http://common-lisp.net/~mpasternacki/yaclml-qbook/. These are auto-generated API docs, so it’s not an easy-to-read tutorial, but rather a reference, and some descriptions in there might be dated; however, it may come in handy when you explore Yaclml’s APIs on your own.

Using templates

Let’s start from the basic Hello, World template and see how to render it from Lisp. I will write more about the template language in the next section, but I want to write about the Lisp part first, so that you’re able to run and test more complex examples as you read. Here’s the template:

<html lang="en"
      xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <head>
  <title>Hello, world!title>
  head>
  <body>
    <h1>Hello, World!h1>
    <p tal:content="$param">Parameter will go here.p>
  body>
html>

We can see it’s an XHTML document, which makes it also proper XML, and uses XML namespaces. The namespace tal will refer to the templating language tags and attributes; the tal:content attribute’s meaning is to replace tag’s interior with value of a variable. We’ll get to this later, now we just want to display this.

To render a template from Lisp, we need cooperation of three parts: the generator, the template itself, and the environment. Generator is an object that finds templates by name, and compiles them to efficient closures. Yaclml provides a filesystem generator, which finds templates as files in specified directories (‘roots’), but it’s possible to get templates e.g. from SQL database or from network, by creating a class that would inherit from TAL-GENERATOR or FILE-SYSTEM-GENERATOR. A loaded template is a closure which, when called (with an environment and a generator for finding included templates as arguments) renders the template to *YACLML-OUTPUT*. Environment is mapping from variables used in templates to values.

So, let’s render our template in the simplest way possible:

(defpackage :yaclml-test
  (:use :common-lisp :yaclml))
(in-package :yaclml-test)

(defvar *generator*
  (make-instance 'file-system-generator
                 :root-directories (list *default-pathname-defaults*))
  "A filesystem-based TAL generator that looks for templates only in
  current directory.")

(defvar *environment*
  (tal-env 'param "foo")
  "Simple environment with a single variable.")

(defvar *template*
  (load-tal *generator* "test.tal")
  "Template read from generator.")

;;; Render the template:
(funcall *template* *environment* *generator*)
;;; This yields:
;; 
;;   
;;   Hello, world!
;;   
;;   
;;     Hello, World!
;;     foo
;;   
;;

Here, we define a filesystem generator, a simple environment, then we load the template using generator, and we call it. Simple. Result is rendered into stream defined in YACLML:*YACLML-STREAM* variable (default is T, which makes output go to *STANDARD-OUTPUT*; macros (YACLML:WITH-YACLML-STREAM STREAM &BODY BODY) and (YACLML:WITH-YACLML-OUTPUT-TO-STRING &BODY BODY) can be used to redirect the output elegantly).

There is not much more to say about generators: the only generator type actually provided by Yaclml is FILE-SYSTEM-GENERATOR, which accepts a list of pathnames naming directories that will be searched for templates, and an optional initarg :CACHEP which tells whether to cache already parsed templates.

To get the template closure, we use the generator and LOAD-TAL function. Alternatively, we can compile template directly from file or string, using COMPILE-TAL-FILE or COMPILE-TAL-STRING. To render a template, we simply call resulting closure, passing it an environment and a generator (which is used for finding templates included by a template being called) as arguments.

Finally, we get to environments. These are used to fill in templates; they map variable names used in TAL expressions to values. Environments are lists of binding sets; binding set may be a hash table, an association list, a CLOS object (in this case, key would be a slot name), or anything on which a method for FETCH-TAL-VALUE generic is defined. A new environment can be constructed from key-value pairs using TAL-ENV function, from list of binding sets using MAKE-STANDARD-TAL-ENVIRONMENT, or from two existing environments with EXTEND-ENVIRONMENT. The last of these functions effectively allows to create binding stacks in Lisp code.

Template syntax

As I already wrote, template is plain XML (usually XHTML), and whole logic is done by active tags and attributes. Yaclml maps XML namespaces to Lisp packages (see YACLML:*URI-TO-PACKAGE* variable), so you can easily look up definition of any tag with SLIME (or—if you’re one of those people—using your Lisp vendor’s IDE).

Only package/namespace which actually contains active tags/attributes provided by Yaclml is :IT.BESE.YACLML.TAL, AKA :TAL, attributed to namespace http://common-lisp.net/project/bese/tal/core.

Attributes

More interesting work, and actual logic, lives in TAL’s attributes. Let’s look at those.

`tal:content`, `tal:content-as-is`, and `tal:replace`

These attributes insert into their tag (content) or replace their tag entirely with (replace) with a TAL expression. Plain content and replace escape HTML special chars (<">); content-as-is does not escape anything. We’ve already seen those in action.

`tal:when` and `tal:unless`

These tags are conditionals. They render the tag they belong to (and its content, of course) when a TAL expression is true (tal:when) or false (tal:unless). We’ve seen those too. Unfortunately, there is no if-then-else construct; this would be hard to encode elegantly in XML.

`tal:dolist`

Looping construct. Its TAL expression should return a list of environments. Its tag will be executed with current environment extended by each of environments on the list. Let’s see it:


<ul xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <li tal:dolist="$envlist">
    <span tal:replace="$foo" />
    <em tal:content="$bar">defaultem>
  li>
ul>

(funcall (load-tal *generator* "dolist.tal")
         (tal-env 'envlist (list (tal-env 'foo 1 'bar 2)
                                 (tal-env 'foo 3 'bar 4)
                                 (tal-env 'foo 5)))
         *generator*)
;;; Yields:
;; 
;;   
;;     1
;;     2
;;   

;;     3
;;     4
;;   

;;     5
;;     
;;   
;;

Not most readable or elegant, but expressive and gets the work done. A bit like next attribute…

`tal:let`

This one, added to Yaclml by yours truly (of which yours truly should be slightly ashamed), extends environment for tag content with variables specified as for LET command. This breaks layer separation almost as badly as TAL:LISP, could be implemented way better (e.g. by using xmlns:param="http://common-lisp.net/project/bese/tal/params" namespace and allowing user to use it in all tags, not only tal:include), and is generally evil. Nevertheless, it’s already commited to Yaclml, and should be documented. Please, don’t look at the following example:

<div xmlns:tal="http://common-lisp.net/project/bese/tal/core"
     tal:let="((foo 1) (bar 2) (baz 3))">
  <div tal:replace="$foo"/> <div tal:replace="$bar"/> <div tal:replace="$baz"/>
div>

(funcall (load-tal *generator* "let.tal")
         nil
         *generator**)
;;; Yields:
;; 
;;  1 2 3
;;

`tal:in-package`

This attribute sets current package for TAL expressions within its tag. We’ll talk about expressions in a moment, now—a silly example:

<div xmlns:tal="http://common-lisp.net/project/bese/tal/core"
     tal:in-package=":yaclml">
  <pre tal:content="*uri-to-package*"/>
div>

(funcall (load-tal *generator* "inpackage.tal") nil *generator*)
;;; Yields:
;; 
;;   (http://common-lisp.net/project/bese/tal/core . #<PACKAGE "IT.BESE.YACLML.TAL">) (http://common-lisp.net/project/bese/tal/params
;;  . #<PACKAGE "IT.BESE.YACLML.TAL.INCLUDE-PARAMS">) (http://common-lisp.net/project/bese/yaclml/core
;;  . #<PACKAGE "IT.BESE.YACLML.TAGS">) (http://www.w3.org/XML/1998/namespace . #<PACKAGE "IT.BESE.YACLML.XML">) (http://www.w3.org/1999/xlink/ . #<PACKAGE "IT.BESE.YACLML.XLINK">)
;;

Expressions

Yaclml’s TAL expressions are simply Lisp expressions, with all their upsides and downsides. Current package is one set with tal:in-package or, when none was set, package that was active when template was called. There is one difference, though: readtable is modified. Symbols following the $ prefix are looked up in current environment. That’s all. You can (and should not) use all the power of Lisp in the expressions; remember to quote the double quotation marks (the " sign) as " entities. Yes, this gets unreadable and ugly. Simply, don’t overuse it. When in doubt, move logic to Lisp code.

These rules apply for most Yaclml attributes; unfortunately, not for all of them. The ugly exception is the tal:name-expression attribute of the tal:include tag. This tag, and all plain HTML tags, can include TAL expressions, by surrounding it with ${tal-expression}. That’s where the ugly ${$included}.tal syntax in tal:include example came from.

The second form, @{tal-expression}, expects tal-expression to return a list, and resulting string is concatenation of this list’s elements.

Caveats

Yaclml has some issues to be aware of. Here are those that I know; this is probably not an exhaustive list, but it should be useful anyway. So…

Input TAL templates are read and interpreted as XML files. This means that XML comments are discarded before the interpreter even sees them. This usually is a good thing; however, if someone tries to use conditional comments, there’s a nasty surprise: conditionals are eaten by template engine. In the next part, which will be about extending Yaclml, we’ll learn, how to work around this.

Attributes aren’t interpreted consistently: most TAL attributes accept TAL expressions, except the tal:name-expression attribute of the tal:include tag, which is interpreted as plain HTML attribute and needs escaping expressions with ${…} syntax; and tal:name attribute of the same tag ignores every attempt to use any syntax at all.

TAL expressions need to be valid XML attributes, so Lisp has to be quoted. This is especially annoying when you try to use string literals within expressions. However, this has a simple workaround: don’t pack logic into your expressions. If this is annoying, make sure you’re not cramming controller logic into your view layer inside templates.

Summary

TAL is not perfect, but it is a solid, usable and quite optimized code base. For generating HTML and XML documents, it is way more convenient (for me) than text based approaches, as it enforces well-formedness and (to some extent) validity of generated HTML. It is also extensible, about which I’ll write in the next part—stay tuned!

Yaclml in pictures, part I: HTML generation

Maciej — Sat, 07 Feb 2009 20:41:04 +0000

Everyone and their dog loves Edi Weitz’s Lisp software (unless they don’t use Lisp, that is). Not without reason—Edi’s libraries are solid, well-designed and robust pieces of software. CL-PPCRE is unbeatable (it parses Perl’s regular expressions faster than Perl, when properly compiled, and is more compatible with Perl 5.8 than Perl 5.8 is with Perl 5.6), and Hunchentoot seems to be the best and most hassle-free HTTP server library available for Lisp. Hunchentoot is also most popular server, and many people using it, automatically use Edi’s CL-WHO for HTML output, and HTML-Template for templating, which—I think—are not the best libraries available for this, and I will explain now, why.

I worked for about a year on an UnCommon Web-based application. This was an interesting experience; UCW provides a great way to express complex behaviour in a Web application, as it is continuation-based, which enabled me to code app’s logic as regular control flow, complete with looping, conditions, etc., from time to time presenting user a form and receiving user-supplied values (form presentation was as simple as calling out to a function, which returned values supplied by user). Cool, huh? Currently, UCW seems to be mostly a dead project, and there is an alternative, which may be more interesting—namely, Weblocks—but I didn’t check that out yet. What’s more, UCW, with all its complexity, had very little documentation, and what was avalilable, was mostly outdated, as the framework was constantly evolving at that time. Not so cool, but it forced me to learn to RTFS when needed (with Slime’s M-. it was not as hard as it seems). But I digress; UCW made use of some other projects of Bese (UCW developers seemed to have quite a bad case of NIH), including HTML output and templating library Yaclml—Yet Another Common Lisp Markup Language—which I really loved, and which, in my opinion, surpasses CL-WHO + HTML-Template in many ways. Unfortunately it is almost undocumented, which is what I’ll try to fix in this article, along with comparing Yaclml to its Ediware counterparts.

Formatting HTML output

Edi, when designing CL-WHO, decided to use keyword symbols to represent HTML tags; Bese guys decided to create separate package for tags—it.bese.yaclml.tags, conveniently nicknamed <. Let’s compare a simple example code:

;; CL-WHO
(with-html-output-to-string (s)
  (:ul (:li "foo")
       (:li "bar")))

;; Yaclml
(with-yaclml-output-to-string
  (<:ul (<:li "foo")
        (<:li "bar")))

Looks like a trivial difference, only on visual level (but this level is also important: we see the < sign, which is distinct visually and actually looks like an HTML tag, not a keyword that looks just like any other keyword scattered throughout the code), but it actually has many subtle consequences. First, keywords can’t have a function or macro definition, at least not in a practical way without conflicting with anything. Thus, with CL-WHO syntax, one has to explicitly pass the sexprs through sexp-to-html translation function or macro; escaping to Lisp code from HTML-describing sexprs, and back to HTML is not trivial and requires a separate macro layer to switch back and forth. Using separate package to describe HTML tags makes it possible to implement tags as macros, which was what Bese guys did. As an important side effect, this enables basic HTML validation support, and guards users from their own typos, which CL-WHO is simply unable to do. Yaclml also avoided the stream variable binding (they just use a dynamically bound special variable to hold output stream), so we don’t see this binding contributing to visual noise. Now, let’s see what happens with the code when we wish to use loop to generate list items:

;; CL-WHO
(with-html-output-to-string (s)
  (:ul (dolist (i '("foo" "bar" "baz" 23))
         (htm (:li (esc (princ-to-string i)))))))

;; Yaclml
(with-yaclml-output-to-string
  (<:ul (dolist (i '("foo" "bar" "baz" 23))
          (<:li (<:as-html i)))))

We see now that Yaclml actually includes a whole lot less visual noise. There is an <:AS-HTML macro, which is necessary because tags are implemented as macros, not as functions, which allows for some optimizations, but requires some more thought from user. Yaclml authors decided to ignore value returned by forms included in tag macros, and require them to write directly to YACLML:*YACLML-STREAM*. There are helper macros <:AS-HTML (with shorthand <:AH), which quotes its arguments’ evaluation results to Yaclml output stream, and <:AS-IS (<:AI), which outputs evaluation results directly to the stream, allowing called function to generate its own HTML, but also requiring it to escape anything that needs to be escaped.

Using special variable YACLML:*YACLML-STREAM* instead of a lexical binding (which is what CL-WHO does) has one more benefit, when we want to delegate some parts of HTML generation to separate functions. Let’s compare once again:

;; CL-WHO
(defun item-html/who (val)
  (with-html-output-to-string (s)
    (:li :class "item"
         (esc (princ-to-string val)))))
(with-html-output-to-string (s)
  (:ul (dolist (i '("foo" "bar" "baz" 23))
          (str (item-html/who i)))))

;; Yaclml
(defun  (val)
  (<:li :class "item"
    (<:ah val)))
(with-yaclml-output-to-string
  (<:ul (dolist (i '("foo" "bar" "baz" 23))
          (
The CL-WHO functions include very much noise, which Yaclml manages to avoid.  The visual distinction of tags with the < package alias helps with code readability.  We also use a convention to put formatting-only functions and macros in a separate package, with name starting with <, or to start their names with <.  This convention clearly indicates formatting/visual layer; for simplicity, I used function name starting with <, but in a larger project, I’d use separate package for formatting, nicknamed , so I would use  here.

Could Yaclml HTML generation be better?  Sure thing.  It could step away from macros (at a performance penalty, though), and use functions, which would return and consume HTML fragments (or consume strings, or actually any Lisp values—including closures for lazy evaluation).  This would introduce first-class HTML fragment object, giving us sort of DOM and possibility of passing around and manipulating already generated HTML fragments, and wouldn’t requre the <:AH and <:AI helper macros (or actually would require only one of those).  That seems to be what core-server guys are doing with their DOM programming support.  I don’t use it yet, partly because I don’t need this kind of functionality and I’m more acquainted with Yaclml; partly because I don’t want to buy into core-server as a whole—core-server is a monolith and ripping DOM support out of it would require substantial amount of work—and partly because I use the second part of Yaclml, which I will describe in the following article: HTML and XML templating.

Three of Coins » yaclml

Yaclml in pictures, part II: Templating

Using templates

Hello, World!

Template syntax

Tags

`tal:tal`

`tal:lisp`

`tal:include`

Attributes

`tal:content`, `tal:content-as-is`, and `tal:replace`

`tal:when` and `tal:unless`

`tal:dolist`

`tal:let`

`tal:in-package`

Expressions

Caveats

Summary

Yaclml in pictures, part I: HTML generation

Formatting HTML output

Three of Coins » yaclml

Yaclml in pictures, part II: Templating

Using templates

Hello, World!

Template syntax

Tags

tal:tal

tal:lisp

tal:include

Attributes

tal:content, tal:content-as-is, and tal:replace

tal:when and tal:unless

tal:dolist

tal:let

tal:in-package

Expressions

Caveats

Summary

Yaclml in pictures, part I: HTML generation

Formatting HTML output

`tal:tal`

`tal:lisp`

`tal:include`

`tal:content`, `tal:content-as-is`, and `tal:replace`

`tal:when` and `tal:unless`

`tal:dolist`

`tal:let`

`tal:in-package`