Yaclml in pictures, part II: Templating

Maciej — Thu, 21 Jan 2010 14:00:47 +0000

After a short intermission and explanation/excuse, let’s go on interesting stuff, Yaclml; and if you still didn’t read the first part, where I wrote about HTML generation and compared Yaclml to CL-WHO, do so now!

Continuing first part‘s focus on comparing Yaclml to Ediware, let’s start by mentioning that CL-WHO doesn’t do templating at all; Edi Weitz wrote a separate library for this: HTML-Template. It provides simple templating support, which turns template of a HTML file (or actually any text file, not limited to HTML) to a closure which, when called, can fill the template with supplied values and output result to a stream. Simple enough, based on Perl’s HTML::Template—which shows in the templating language syntax. The template directives of HTML-Template are embedded in HTML comments.

Yaclml includes the templating feature, and it chose a different path. For me, with a tiny bit of Zope background, their path is a little nicer: they decided to support a Lisp variant of Zope’s Template Attribute Language. TAL is strictly an XHTML and XML templating language, whose directives are special XML tags and attributes, living in a separate XML namespace. I find this approach much more elegant than magic directives embedded in comments. This also makes it possible to use XML/XHTML editing tools (such as Emacs’ nxml-mode and excellent nxhtml-mode built on top of it) to aid in authoring and validation of the code. Supposedly it also plays fine with visual design tools, such as Adobe Dreamweaver, but I can’t confirm that, since I don’t use those.

In this section, I won’t go on comparing Yaclml to HTML-Template, because I didn’t use much of the latter, those two are not as similar as CL-WHO and Yaclml’s HTML generation, and would require non-trivial examples to really see meaningful differences; besides, Yaclml’s TAL is obviously better, and anyone will see it from TAL alone. ;) Seriously, HTML-Template’s documentation is great, it’s a simple package, and you can compare it for yourself. These two packages are similar, main difference being TAL’s focus on XML and XHTML, and syntactic differences are much more of a matter of taste than it is with HTML generation. However, Yaclml, and especially its TAL support, is underdocumented (there was some tutorial over at UCW Wiki, but at the moment it’s inaccessible, and only option to read it is Google’s cache) and it’s really hard to figure out how to use it, especially outside of UCW framework; that’s the hole I’m trying to fill here.

During writing this article, I’ve found that Yaclml is actually somewhat documented, in the qbook format, but the generated, readable docs are nowhere to be found. I’ve rebuilt the HTML files and uploaded them at http://common-lisp.net/~mpasternacki/yaclml-qbook/. These are auto-generated API docs, so it’s not an easy-to-read tutorial, but rather a reference, and some descriptions in there might be dated; however, it may come in handy when you explore Yaclml’s APIs on your own.

Using templates

Let’s start from the basic Hello, World template and see how to render it from Lisp. I will write more about the template language in the next section, but I want to write about the Lisp part first, so that you’re able to run and test more complex examples as you read. Here’s the template:

<html lang="en"
      xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <head>
  <title>Hello, world!title>
  head>
  <body>
    <h1>Hello, World!h1>
    <p tal:content="$param">Parameter will go here.p>
  body>
html>

We can see it’s an XHTML document, which makes it also proper XML, and uses XML namespaces. The namespace tal will refer to the templating language tags and attributes; the tal:content attribute’s meaning is to replace tag’s interior with value of a variable. We’ll get to this later, now we just want to display this.

To render a template from Lisp, we need cooperation of three parts: the generator, the template itself, and the environment. Generator is an object that finds templates by name, and compiles them to efficient closures. Yaclml provides a filesystem generator, which finds templates as files in specified directories (‘roots’), but it’s possible to get templates e.g. from SQL database or from network, by creating a class that would inherit from TAL-GENERATOR or FILE-SYSTEM-GENERATOR. A loaded template is a closure which, when called (with an environment and a generator for finding included templates as arguments) renders the template to *YACLML-OUTPUT*. Environment is mapping from variables used in templates to values.

So, let’s render our template in the simplest way possible:

(defpackage :yaclml-test
  (:use :common-lisp :yaclml))
(in-package :yaclml-test)

(defvar *generator*
  (make-instance 'file-system-generator
                 :root-directories (list *default-pathname-defaults*))
  "A filesystem-based TAL generator that looks for templates only in
  current directory.")

(defvar *environment*
  (tal-env 'param "foo")
  "Simple environment with a single variable.")

(defvar *template*
  (load-tal *generator* "test.tal")
  "Template read from generator.")

;;; Render the template:
(funcall *template* *environment* *generator*)
;;; This yields:
;; 
;;   
;;   Hello, world!
;;   
;;   
;;     Hello, World!
;;     foo
;;   
;;

Here, we define a filesystem generator, a simple environment, then we load the template using generator, and we call it. Simple. Result is rendered into stream defined in YACLML:*YACLML-STREAM* variable (default is T, which makes output go to *STANDARD-OUTPUT*; macros (YACLML:WITH-YACLML-STREAM STREAM &BODY BODY) and (YACLML:WITH-YACLML-OUTPUT-TO-STRING &BODY BODY) can be used to redirect the output elegantly).

There is not much more to say about generators: the only generator type actually provided by Yaclml is FILE-SYSTEM-GENERATOR, which accepts a list of pathnames naming directories that will be searched for templates, and an optional initarg :CACHEP which tells whether to cache already parsed templates.

To get the template closure, we use the generator and LOAD-TAL function. Alternatively, we can compile template directly from file or string, using COMPILE-TAL-FILE or COMPILE-TAL-STRING. To render a template, we simply call resulting closure, passing it an environment and a generator (which is used for finding templates included by a template being called) as arguments.

Finally, we get to environments. These are used to fill in templates; they map variable names used in TAL expressions to values. Environments are lists of binding sets; binding set may be a hash table, an association list, a CLOS object (in this case, key would be a slot name), or anything on which a method for FETCH-TAL-VALUE generic is defined. A new environment can be constructed from key-value pairs using TAL-ENV function, from list of binding sets using MAKE-STANDARD-TAL-ENVIRONMENT, or from two existing environments with EXTEND-ENVIRONMENT. The last of these functions effectively allows to create binding stacks in Lisp code.

Template syntax

As I already wrote, template is plain XML (usually XHTML), and whole logic is done by active tags and attributes. Yaclml maps XML namespaces to Lisp packages (see YACLML:*URI-TO-PACKAGE* variable), so you can easily look up definition of any tag with SLIME (or—if you’re one of those people—using your Lisp vendor’s IDE).

Only package/namespace which actually contains active tags/attributes provided by Yaclml is :IT.BESE.YACLML.TAL, AKA :TAL, attributed to namespace http://common-lisp.net/project/bese/tal/core.

Tags

There is only a handful of tags, so let’s start from them.

`tal:tal`

This tag is semantically neutral—meaningless, and this is why it’s useful. It is used whenever we want to do something with templates, but don’t want to introduce HTML/XML-level elements. I usually use it to group together a sequence of tags.

For example: if we include a sub-template, and the included template consists of more than one tag, we need a top-level tag to be well-formed XML, and to e.g. set XML namespace:

<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <h1 tal:content="$title">Tytułh1>
  <div class="container" tal:content="$body" />
tal:tal>

Other example: I need to conditionalize a sequence of list elements in menu:

<ul id="menu" xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <li><a href="/">Main pagea>li>
  <li><a href=/sub">Subpagea>li>
  <tal:tal tal:when="$logged_in">
    <li><a href="/profile">My profilea>li>
    <li><a href="/logout">Logouta>li>
  tal:tal>
  <tal:tal tal:unless="$logged_in">
    <li><a href="/sign-up">Sign upa>li>
    <li><a href="/login">Log ina>li>
  tal:tal>
ul>

`tal:lisp`

This is the tag that you should never, ever use. Seriously. It is the root of all evil. Cause of mixing MVC layers by introducing logic to templates. However, it was included by Yaclml authors, so I feel obligated to cover it. And, when you’re a programmer and create a structure of templates for further use, or library of widgets, it might have some limited use, and might save some keystrokes. But rule of thumb, when it comes to using this tag, is don’t.

OK, you’ve been warned. Now, here’s how this tag works: it simply interprets tag’s content as a TAL expression, which is actually Lisp code sprinkled with a bit of environment magic. That’s it. I don’t want to spoil you, so no examples for this tag; figure it out yourself (there is not much there to figure out anyway).

`tal:include`

This tag allows one to dynamically include templates within templates. It requires either tal:name attribute literally specifying name of included template, or tal:name-expression attribute, which is interpreted as a TAL expression, whose result specifies name of included template:


<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <tal:include tal:name="inc1.tal" />
  <strong tal:content="$included" />
  <tal:include tal:name-expression="${$included}.tal" />
tal:tal>


<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">FOOtal:tal>

Let’s render these two templates:

(funcall (load-tal *generator* "inc0.tal")
         (tal-env 'included "inc1")
         *generator*)
;;; Yields:
;;
;;  FOO
;;  inc1
;;  
;;  FOO
;;

But hey! There’s more!

I didn’t yet mention one special XML namespace, xmlns:param="http://common-lisp.net/project/bese/tal/params". It allows you to pass arguments (environment parameters) to included template, which effectively gives you not only parametrized subtemplates, but also template inheritance, a very powerful tool for organizing your templates. Let’s look into Yaclml’s own test suite and see how it works:


<tal:include xmlns:tal="http://common-lisp.net/project/bese/tal/core"
             xmlns:param="http://common-lisp.net/project/bese/tal/params"
             tal:name="inc4.tal"
             param:baz="$foo">
  <param:bar><tal:include tal:name="inc5.tal"/>param:bar>
  <param:quux><p>xyzzyp>param:quux>
tal:include>


<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  foo is <tal:tal tal:replace="$foo">bad bad badtal:tal>,
  bar is <tal:tal tal:replace="$bar"/>,
  baz is <tal:tal tal:replace="$baz" />, and
  quux is <tal:tal tal:content-as-is="$quux" />.
tal:tal>


<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">BARINCLUDEDtal:tal>

(funcall (load-tal *generator* "inc3.tal")
         (tal-env 'foo "FOOPARAM")
         *generator*)
;;; Yields:
;;  foo is FOOPARAM,
;;  bar is BARINCLUDED,
;;  baz is FOOPARAM, and
;;  quux is xyzzy.

We can pass to subtemplates not only plain parameters, we can pass whole HTML subtrees.

Attributes

More interesting work, and actual logic, lives in TAL’s attributes. Let’s look at those.

`tal:content`, `tal:content-as-is`, and `tal:replace`

These attributes insert into their tag (content) or replace their tag entirely with (replace) with a TAL expression. Plain content and replace escape HTML special chars (<">); content-as-is does not escape anything. We’ve already seen those in action.

`tal:when` and `tal:unless`

These tags are conditionals. They render the tag they belong to (and its content, of course) when a TAL expression is true (tal:when) or false (tal:unless). We’ve seen those too. Unfortunately, there is no if-then-else construct; this would be hard to encode elegantly in XML.

`tal:dolist`

Looping construct. Its TAL expression should return a list of environments. Its tag will be executed with current environment extended by each of environments on the list. Let’s see it:


<ul xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <li tal:dolist="$envlist">
    <span tal:replace="$foo" />
    <em tal:content="$bar">defaultem>
  li>
ul>

(funcall (load-tal *generator* "dolist.tal")
         (tal-env 'envlist (list (tal-env 'foo 1 'bar 2)
                                 (tal-env 'foo 3 'bar 4)
                                 (tal-env 'foo 5)))
         *generator*)
;;; Yields:
;; 
;;   
;;     1
;;     2
;;   

;;     3
;;     4
;;   

;;     5
;;     
;;   
;;

Not most readable or elegant, but expressive and gets the work done. A bit like next attribute…

`tal:let`

This one, added to Yaclml by yours truly (of which yours truly should be slightly ashamed), extends environment for tag content with variables specified as for LET command. This breaks layer separation almost as badly as TAL:LISP, could be implemented way better (e.g. by using xmlns:param="http://common-lisp.net/project/bese/tal/params" namespace and allowing user to use it in all tags, not only tal:include), and is generally evil. Nevertheless, it’s already commited to Yaclml, and should be documented. Please, don’t look at the following example:

<div xmlns:tal="http://common-lisp.net/project/bese/tal/core"
     tal:let="((foo 1) (bar 2) (baz 3))">
  <div tal:replace="$foo"/> <div tal:replace="$bar"/> <div tal:replace="$baz"/>
div>

(funcall (load-tal *generator* "let.tal")
         nil
         *generator**)
;;; Yields:
;; 
;;  1 2 3
;;

`tal:in-package`

This attribute sets current package for TAL expressions within its tag. We’ll talk about expressions in a moment, now—a silly example:

<div xmlns:tal="http://common-lisp.net/project/bese/tal/core"
     tal:in-package=":yaclml">
  <pre tal:content="*uri-to-package*"/>
div>

(funcall (load-tal *generator* "inpackage.tal") nil *generator*)
;;; Yields:
;; 
;;   (http://common-lisp.net/project/bese/tal/core . #<PACKAGE "IT.BESE.YACLML.TAL">) (http://common-lisp.net/project/bese/tal/params
;;  . #<PACKAGE "IT.BESE.YACLML.TAL.INCLUDE-PARAMS">) (http://common-lisp.net/project/bese/yaclml/core
;;  . #<PACKAGE "IT.BESE.YACLML.TAGS">) (http://www.w3.org/XML/1998/namespace . #<PACKAGE "IT.BESE.YACLML.XML">) (http://www.w3.org/1999/xlink/ . #<PACKAGE "IT.BESE.YACLML.XLINK">)
;;

Expressions

Yaclml’s TAL expressions are simply Lisp expressions, with all their upsides and downsides. Current package is one set with tal:in-package or, when none was set, package that was active when template was called. There is one difference, though: readtable is modified. Symbols following the $ prefix are looked up in current environment. That’s all. You can (and should not) use all the power of Lisp in the expressions; remember to quote the double quotation marks (the " sign) as " entities. Yes, this gets unreadable and ugly. Simply, don’t overuse it. When in doubt, move logic to Lisp code.

These rules apply for most Yaclml attributes; unfortunately, not for all of them. The ugly exception is the tal:name-expression attribute of the tal:include tag. This tag, and all plain HTML tags, can include TAL expressions, by surrounding it with ${tal-expression}. That’s where the ugly ${$included}.tal syntax in tal:include example came from.

The second form, @{tal-expression}, expects tal-expression to return a list, and resulting string is concatenation of this list’s elements.

Caveats

Yaclml has some issues to be aware of. Here are those that I know; this is probably not an exhaustive list, but it should be useful anyway. So…

Input TAL templates are read and interpreted as XML files. This means that XML comments are discarded before the interpreter even sees them. This usually is a good thing; however, if someone tries to use conditional comments, there’s a nasty surprise: conditionals are eaten by template engine. In the next part, which will be about extending Yaclml, we’ll learn, how to work around this.

Attributes aren’t interpreted consistently: most TAL attributes accept TAL expressions, except the tal:name-expression attribute of the tal:include tag, which is interpreted as plain HTML attribute and needs escaping expressions with ${…} syntax; and tal:name attribute of the same tag ignores every attempt to use any syntax at all.

TAL expressions need to be valid XML attributes, so Lisp has to be quoted. This is especially annoying when you try to use string literals within expressions. However, this has a simple workaround: don’t pack logic into your expressions. If this is annoying, make sure you’re not cramming controller logic into your view layer inside templates.

Summary

TAL is not perfect, but it is a solid, usable and quite optimized code base. For generating HTML and XML documents, it is way more convenient (for me) than text based approaches, as it enforces well-formedness and (to some extent) validity of generated HTML. It is also extensible, about which I’ll write in the next part—stay tuned!

Three of Coins » programming

Yaclml in pictures, part II: Templating

Using templates

Hello, World!

Template syntax

Tags

tal:tal

tal:lisp

tal:include