Three of Coins » common lisp http://www.3ofcoins.net Just another WordPress weblog Thu, 21 Jan 2010 14:01:34 +0000 http://wordpress.org/?v=2.9.1 en hourly 1 Yaclml in pictures, part II: Templating http://www.3ofcoins.net/2010/01/21/yaclml-in-pictures-part-ii-templating/ http://www.3ofcoins.net/2010/01/21/yaclml-in-pictures-part-ii-templating/#comments Thu, 21 Jan 2010 14:00:47 +0000 Maciej http://www.3ofcoins.net/?p=94 After a short intermission and explanation/excuse, let’s go on interesting stuff, Yaclml; and if you still didn’t read the first part, where I wrote about HTML generation and compared Yaclml to CL-WHO, do so now!

Continuing first part’s focus on comparing Yaclml to Ediware, let’s start by mentioning that CL-WHO doesn’t do templating at all; Edi Weitz wrote a separate library for this: HTML-Template. It provides simple templating support, which turns template of a HTML file (or actually any text file, not limited to HTML) to a closure which, when called, can fill the template with supplied values and output result to a stream. Simple enough, based on Perl’s HTML::Template—which shows in the templating language syntax. The template directives of HTML-Template are embedded in HTML comments.

Yaclml includes the templating feature, and it chose a different path. For me, with a tiny bit of Zope background, their path is a little nicer: they decided to support a Lisp variant of Zope’s Template Attribute Language. TAL is strictly an XHTML and XML templating language, whose directives are special XML tags and attributes, living in a separate XML namespace. I find this approach much more elegant than magic directives embedded in comments. This also makes it possible to use XML/XHTML editing tools (such as Emacs’ nxml-mode and excellent nxhtml-mode built on top of it) to aid in authoring and validation of the code. Supposedly it also plays fine with visual design tools, such as Adobe Dreamweaver, but I can’t confirm that, since I don’t use those.

In this section, I won’t go on comparing Yaclml to HTML-Template, because I didn’t use much of the latter, those two are not as similar as CL-WHO and Yaclml’s HTML generation, and would require non-trivial examples to really see meaningful differences; besides, Yaclml’s TAL is obviously better, and anyone will see it from TAL alone. ;) Seriously, HTML-Template’s documentation is great, it’s a simple package, and you can compare it for yourself. These two packages are similar, main difference being TAL’s focus on XML and XHTML, and syntactic differences are much more of a matter of taste than it is with HTML generation. However, Yaclml, and especially its TAL support, is underdocumented (there was some tutorial over at UCW Wiki, but at the moment it’s inaccessible, and only option to read it is Google’s cache) and it’s really hard to figure out how to use it, especially outside of UCW framework; that’s the hole I’m trying to fill here.

During writing this article, I’ve found that Yaclml is actually somewhat documented, in the qbook format, but the generated, readable docs are nowhere to be found. I’ve rebuilt the HTML files and uploaded them at http://common-lisp.net/~mpasternacki/yaclml-qbook/. These are auto-generated API docs, so it’s not an easy-to-read tutorial, but rather a reference, and some descriptions in there might be dated; however, it may come in handy when you explore Yaclml’s APIs on your own.

Using templates

Let’s start from the basic Hello, World template and see how to render it from Lisp. I will write more about the template language in the next section, but I want to write about the Lisp part first, so that you’re able to run and test more complex examples as you read. Here’s the template:

<html lang="en"
      xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <head>
  <title>Hello, world!</title>
  </head>
  <body>
    <h1>Hello, World!</h1>
    <p tal:content="$param">Parameter will go here.</p>
  </body>
</html>

We can see it’s an XHTML document, which makes it also proper XML, and uses XML namespaces. The namespace tal will refer to the templating language tags and attributes; the tal:content attribute’s meaning is to replace tag’s interior with value of a variable. We’ll get to this later, now we just want to display this.

To render a template from Lisp, we need cooperation of three parts: the generator, the template itself, and the environment. Generator is an object that finds templates by name, and compiles them to efficient closures. Yaclml provides a filesystem generator, which finds templates as files in specified directories (‘roots’), but it’s possible to get templates e.g. from SQL database or from network, by creating a class that would inherit from TAL-GENERATOR or FILE-SYSTEM-GENERATOR. A loaded template is a closure which, when called (with an environment and a generator for finding included templates as arguments) renders the template to *YACLML-OUTPUT*. Environment is mapping from variables used in templates to values.

So, let’s render our template in the simplest way possible:

(defpackage :yaclml-test
  (:use :common-lisp :yaclml))
(in-package :yaclml-test)

(defvar *generator*
  (make-instance 'file-system-generator
                 :root-directories (list *default-pathname-defaults*))
  "A filesystem-based TAL generator that looks for templates only in
  current directory.")

(defvar *environment*
  (tal-env 'param "foo")
  "Simple environment with a single variable.")

(defvar *template*
  (load-tal *generator* "test.tal")
  "Template read from generator.")

;;; Render the template:
(funcall *template* *environment* *generator*)
;;; This yields:
;; <html lang="en">
;;   <head>
;;   <title>Hello, world!</title>
;;   </head>
;;   <body>
;;     <h1>Hello, World!</h1>
;;     <p>foo</p>
;;   </body>
;; </html>

Here, we define a filesystem generator, a simple environment, then we load the template using generator, and we call it. Simple. Result is rendered into stream defined in YACLML:*YACLML-STREAM* variable (default is T, which makes output go to *STANDARD-OUTPUT*; macros (YACLML:WITH-YACLML-STREAM STREAM &BODY BODY) and (YACLML:WITH-YACLML-OUTPUT-TO-STRING &BODY BODY) can be used to redirect the output elegantly).

There is not much more to say about generators: the only generator type actually provided by Yaclml is FILE-SYSTEM-GENERATOR, which accepts a list of pathnames naming directories that will be searched for templates, and an optional initarg :CACHEP which tells whether to cache already parsed templates.

To get the template closure, we use the generator and LOAD-TAL function. Alternatively, we can compile template directly from file or string, using COMPILE-TAL-FILE or COMPILE-TAL-STRING. To render a template, we simply call resulting closure, passing it an environment and a generator (which is used for finding templates included by a template being called) as arguments.

Finally, we get to environments. These are used to fill in templates; they map variable names used in TAL expressions to values. Environments are lists of binding sets; binding set may be a hash table, an association list, a CLOS object (in this case, key would be a slot name), or anything on which a method for FETCH-TAL-VALUE generic is defined. A new environment can be constructed from key-value pairs using TAL-ENV function, from list of binding sets using MAKE-STANDARD-TAL-ENVIRONMENT, or from two existing environments with EXTEND-ENVIRONMENT. The last of these functions effectively allows to create binding stacks in Lisp code.

Template syntax

As I already wrote, template is plain XML (usually XHTML), and whole logic is done by active tags and attributes. Yaclml maps XML namespaces to Lisp packages (see YACLML:*URI-TO-PACKAGE* variable), so you can easily look up definition of any tag with SLIME (or—if you’re one of those people—using your Lisp vendor’s IDE).

Only package/namespace which actually contains active tags/attributes provided by Yaclml is :IT.BESE.YACLML.TAL, AKA :TAL, attributed to namespace http://common-lisp.net/project/bese/tal/core.

Tags

There is only a handful of tags, so let’s start from them.

tal:tal

This tag is semantically neutral—meaningless, and this is why it’s useful. It is used whenever we want to do something with templates, but don’t want to introduce HTML/XML-level elements. I usually use it to group together a sequence of tags.

For example: if we include a sub-template, and the included template consists of more than one tag, we need a top-level tag to be well-formed XML, and to e.g. set XML namespace:

<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <h1 tal:content="$title">Tytuł</h1>
  <div class="container" tal:content="$body" />
</tal:tal>

Other example: I need to conditionalize a sequence of list elements in menu:

<ul id="menu" xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <li><a href="/">Main page</a></li>
  <li><a href=/sub">Subpage</a></li>
  <tal:tal tal:when="$logged_in">
    <li><a href="/profile">My profile</a></li>
    <li><a href="/logout">Logout</a></li>
  </tal:tal>
  <tal:tal tal:unless="$logged_in">
    <li><a href="/sign-up">Sign up</a></li>
    <li><a href="/login">Log in</a></li>
  </tal:tal>
</ul>


tal:lisp

This is the tag that you should never, ever use. Seriously. It is the root of all evil. Cause of mixing MVC layers by introducing logic to templates. However, it was included by Yaclml authors, so I feel obligated to cover it. And, when you’re a programmer and create a structure of templates for further use, or library of widgets, it might have some limited use, and might save some keystrokes. But rule of thumb, when it comes to using this tag, is don’t.

OK, you’ve been warned. Now, here’s how this tag works: it simply interprets tag’s content as a TAL expression, which is actually Lisp code sprinkled with a bit of environment magic. That’s it. I don’t want to spoil you, so no examples for this tag; figure it out yourself (there is not much there to figure out anyway).

tal:include

This tag allows one to dynamically include templates within templates. It requires either tal:name attribute literally specifying name of included template, or tal:name-expression attribute, which is interpreted as a TAL expression, whose result specifies name of included template:

<!-- inc0.tal -->
<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <tal:include tal:name="inc1.tal" />
  <strong tal:content="$included" />
  <tal:include tal:name-expression="${$included}.tal" />
</tal:tal>


<!-- inc1.tal -->
<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">FOO</tal:tal>

Let’s render these two templates:

(funcall (load-tal *generator* "inc0.tal")
         (tal-env 'included "inc1")
         *generator*)
;;; Yields:
;;
;;  FOO
;;  <strong>inc1</strong>
;;  
;;  FOO
;;

But hey! There’s more!

I didn’t yet mention one special XML namespace, xmlns:param="http://common-lisp.net/project/bese/tal/params". It allows you to pass arguments (environment parameters) to included template, which effectively gives you not only parametrized subtemplates, but also template inheritance, a very powerful tool for organizing your templates. Let’s look into Yaclml’s own test suite and see how it works:

<!-- inc3.tal -->
<tal:include xmlns:tal="http://common-lisp.net/project/bese/tal/core"
             xmlns:param="http://common-lisp.net/project/bese/tal/params"
             tal:name="inc4.tal"
             param:baz="$foo">
  <param:bar><tal:include tal:name="inc5.tal"/></param:bar>
  <param:quux><p>xyzzy</p></param:quux>
</tal:include>


<!-- inc4.tal -->
<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  foo is <tal:tal tal:replace="$foo">bad bad bad</tal:tal>,
  bar is <tal:tal tal:replace="$bar"/>,
  baz is <tal:tal tal:replace="$baz" />, and
  quux is <tal:tal tal:content-as-is="$quux" />.
</tal:tal>


<!-- inc5.tal -->
<tal:tal xmlns:tal="http://common-lisp.net/project/bese/tal/core">BARINCLUDED</tal:tal>


(funcall (load-tal *generator* "inc3.tal")
         (tal-env 'foo "FOOPARAM")
         *generator*)
;;; Yields:
;;  foo is FOOPARAM,
;;  bar is BARINCLUDED,
;;  baz is FOOPARAM, and
;;  quux is <p>xyzzy</p>.

We can pass to subtemplates not only plain parameters, we can pass whole HTML subtrees.

Attributes

More interesting work, and actual logic, lives in TAL’s attributes. Let’s look at those.

tal:content, tal:content-as-is, and tal:replace

These attributes insert into their tag (content) or replace their tag entirely with (replace) with a TAL expression. Plain content and replace escape HTML special chars (<">); content-as-is does not escape anything. We’ve already seen those in action.

tal:when and tal:unless

These tags are conditionals. They render the tag they belong to (and its content, of course) when a TAL expression is true (tal:when) or false (tal:unless). We’ve seen those too. Unfortunately, there is no if-then-else construct; this would be hard to encode elegantly in XML.

tal:dolist

Looping construct. Its TAL expression should return a list of environments. Its tag will be executed with current environment extended by each of environments on the list. Let’s see it:

<!-- dolist.tal -->
<ul xmlns:tal="http://common-lisp.net/project/bese/tal/core">
  <li tal:dolist="$envlist">
    <span tal:replace="$foo" />
    <em tal:content="$bar">default</em>
  </li>
</ul>


(funcall (load-tal *generator* "dolist.tal")
         (tal-env 'envlist (list (tal-env 'foo 1 'bar 2)
                                 (tal-env 'foo 3 'bar 4)
                                 (tal-env 'foo 5)))
         *generator*)
;;; Yields:
;; <ul>
;;   <li>
;;     1
;;     <em>2</em>
;;   </li><li>
;;     3
;;     <em>4</em>
;;   </li><li>
;;     5
;;     <em></em>
;;   </li>
;; </ul>

Not most readable or elegant, but expressive and gets the work done. A bit like next attribute…

tal:let

This one, added to Yaclml by yours truly (of which yours truly should be slightly ashamed), extends environment for tag content with variables specified as for LET command. This breaks layer separation almost as badly as TAL:LISP, could be implemented way better (e.g. by using xmlns:param="http://common-lisp.net/project/bese/tal/params" namespace and allowing user to use it in all tags, not only tal:include), and is generally evil. Nevertheless, it’s already commited to Yaclml, and should be documented. Please, don’t look at the following example:

<div xmlns:tal="http://common-lisp.net/project/bese/tal/core"
     tal:let="((foo 1) (bar 2) (baz 3))">
  <div tal:replace="$foo"/> <div tal:replace="$bar"/> <div tal:replace="$baz"/>
</div>


(funcall (load-tal *generator* "let.tal")
         nil
         *generator**)
;;; Yields:
;; <div>
;;  1 2 3
;;</div>

tal:in-package

This attribute sets current package for TAL expressions within its tag. We’ll talk about expressions in a moment, now—a silly example:

<div xmlns:tal="http://common-lisp.net/project/bese/tal/core"
     tal:in-package=":yaclml">
  <pre tal:content="*uri-to-package*"/>
</div>


(funcall (load-tal *generator* "inpackage.tal") nil *generator*)
;;; Yields:
;; <div>
;;   <pre>(http://common-lisp.net/project/bese/tal/core . #&lt;PACKAGE &quot;IT.BESE.YACLML.TAL&quot;&gt;) (http://common-lisp.net/project/bese/tal/params
;;  . #&lt;PACKAGE &quot;IT.BESE.YACLML.TAL.INCLUDE-PARAMS&quot;&gt;) (http://common-lisp.net/project/bese/yaclml/core
;;  . #&lt;PACKAGE &quot;IT.BESE.YACLML.TAGS&quot;&gt;) (http://www.w3.org/XML/1998/namespace . #&lt;PACKAGE &quot;IT.BESE.YACLML.XML&quot;&gt;) (http://www.w3.org/1999/xlink/ . #&lt;PACKAGE &quot;IT.BESE.YACLML.XLINK&quot;&gt;)</pre>
;; </div>

Expressions

Yaclml’s TAL expressions are simply Lisp expressions, with all their upsides and downsides. Current package is one set with tal:in-package or, when none was set, package that was active when template was called. There is one difference, though: readtable is modified. Symbols following the $ prefix are looked up in current environment. That’s all. You can (and should not) use all the power of Lisp in the expressions; remember to quote the double quotation marks (the " sign) as &quot; entities. Yes, this gets unreadable and ugly. Simply, don’t overuse it. When in doubt, move logic to Lisp code.

These rules apply for most Yaclml attributes; unfortunately, not for all of them. The ugly exception is the tal:name-expression attribute of the tal:include tag. This tag, and all plain HTML tags, can include TAL expressions, by surrounding it with ${tal-expression}. That’s where the ugly ${$included}.tal syntax in tal:include example came from.

The second form, @{tal-expression}, expects tal-expression to return a list, and resulting string is concatenation of this list’s elements.

Caveats

Yaclml has some issues to be aware of. Here are those that I know; this is probably not an exhaustive list, but it should be useful anyway. So…

Input TAL templates are read and interpreted as XML files. This means that XML comments are discarded before the interpreter even sees them. This usually is a good thing; however, if someone tries to use conditional comments, there’s a nasty surprise: conditionals are eaten by template engine. In the next part, which will be about extending Yaclml, we’ll learn, how to work around this.

Attributes aren’t interpreted consistently: most TAL attributes accept TAL expressions, except the tal:name-expression attribute of the tal:include tag, which is interpreted as plain HTML attribute and needs escaping expressions with ${…} syntax; and tal:name attribute of the same tag ignores every attempt to use any syntax at all.

TAL expressions need to be valid XML attributes, so Lisp has to be quoted. This is especially annoying when you try to use string literals within expressions. However, this has a simple workaround: don’t pack logic into your expressions. If this is annoying, make sure you’re not cramming controller logic into your view layer inside templates.

Summary

TAL is not perfect, but it is a solid, usable and quite optimized code base. For generating HTML and XML documents, it is way more convenient (for me) than text based approaches, as it enforces well-formedness and (to some extent) validity of generated HTML. It is also extensible, about which I’ll write in the next part—stay tuned!

]]> http://www.3ofcoins.net/2010/01/21/yaclml-in-pictures-part-ii-templating/feed/ 3 Common Lisp, Clojure, and seriousness. http://www.3ofcoins.net/2009/01/30/common-lisp-clojure-and-seriousness/ http://www.3ofcoins.net/2009/01/30/common-lisp-clojure-and-seriousness/#comments Fri, 30 Jan 2009 14:06:16 +0000 Maciej http://www.3ofcoins.net/?p=78 Brian Carper described a few days ago, how Clojure is better (for him) than Common Lisp (actually, SBCL).  I managed to dig through ensuing flame war, but it seems like nobody in the flame war realized (or it wasn’t stressed enough) that original post is actually comparing apples to oranges, a serious language to a toy language.

A language, to be considered serious, needs to be self-sufficient, a serious language can’t be a mere parasite on some host language or environment, and its bus factor can’t be finite.  That translates to just a couple of features:

  • A serious language has to have a defining standard, it can’t be implementation-defined.  A good standard bumps language’s bus factor to aleph null: after a nuclear catastrophe, archæologists of future generations should be able to re-implement the language on any hardware, including the Calculor;
  • A serious language has to be self-hosting, or at least have some self-hosting implementations.  Only then language stops being dependent on other languages.

Leaving aside other important traits (such as having multiple implementations, having a machine code compiler, extensibility, and so on), these two alone are necessary and sufficient for language to be serious; all programing languages not having these traits are just toys that can’t be guaranteed to last.

Toy languages may stay this way (as did Tcl and Perl, I think, and TeX is one-of-a-kind toy language which managed to grow enormously, staying implementation-defined and standardless).  On the other hand, starting a toy language may be just a way to bootstrap a serious language without starting from gathering a commitee, and I think that’s what is currently happening to Python.  I hope that will also happen to Clojure; I looked at it, and I liked what I saw (good concurrency support, decoupling polymorphism from data structures even more than in Common Lisp, a tiny bit more modern syntax, and it feels really well designed), however at the moment it is a toy language, feeding off Java, at very early stage of development.

Toy languages have short-term advantages (they are fast to create, usually simple, they leverage the host environment (libraries, operating system interface), they solve problem at hand quickly), and long-term disadvantages (lack of stability).  Serious languages have short-term disadvantages (multitude of slightly differing implementations to choose from, every single one of which has its own differences and extensions to the standard; implementation/library compatibility matrix; standard features are set in stone and it may be tricky to work around them), and long-term advantages (stability, one can rely on a serious language to be there in one form or another in ten years).

I have nothing against toy languages: they do their job, and do it well.  As a glue language, as a scripting language, as a solution to short-term problems, they’re good enough, and short-term advantages win over long-term disadvantages.  Brian was trying to solve a short-term problem and, not surprisingly, toy language worked better.  But: will the solution still work in ten, twenty years?  Heck, will it work with next release of Clojure, or next release of JVM, or if Rich Hickey (Clojure author) gets hit by a bus?  Brian didn’t need to answer these questions; do you?

When you get to choose a programming language, or any other tool, you may have some personal preferences, but don’t forget to ask yourself the single most important question: What is the problem you’re trying to solve?  The answer to this question determines all further questions and answers.

]]> http://www.3ofcoins.net/2009/01/30/common-lisp-clojure-and-seriousness/feed/ 19 Lisp HTML sanitizer http://www.3ofcoins.net/2008/10/24/lisp-html-sanitizer/ http://www.3ofcoins.net/2008/10/24/lisp-html-sanitizer/#comments Fri, 24 Oct 2008 22:46:44 +0000 Maciej http://www.3ofcoins.net/?p=38 Lately, I was thinking a lot about enabling webapp users to edit rich text easily while staying secure and injection-free.  Until recently, I would just use trane-bb module of CL-Trane, and make users type BBCode inside a textarea, since many users are familiar with it, and I’d be able to easily convert their BB to safe HTML.  However, all JavaScript WYSIWYG editors provide HTML code, which is not that surprising.  I googled around and read a bit on all the issues related with BBCode, Textile and other markup languages, and came to agree with John Atwood (Is HTML a Humane Markup Language?) on HTML being the actually friendly, single markup language.  I was pleasantly surprised to see Bese’s fork of Franz’s phtml actually support HTML sanitizing, and (having contributed quite a bit to Bese a few years ago) not surprised at all that this feature is not actually described or documented anywhere.  So, if you’re worried about accepting HTML (and if you’ve decided to accept HTML from users, you should be worried!), check this out:

darcs get http://common-lisp.net/project/bese/repos/parse-html/

]]> http://www.3ofcoins.net/2008/10/24/lisp-html-sanitizer/feed/ 2