The origins of ODD

I’m moving house this week, which involves packing up thirty years of accumulated junk of various sorts. As a result, every now and then I stumble upon some long lost historic document, like this one. It dates from a lunch that Michael Sperberg-McQueen and I enjoyed at the Lido restaurant Bergen in November 1991. This being a family restaurant, it was equipped with paper table cloths and wax crayons, Norwegian kids for the use of, which Michael and I were quick to reappropriate to our immediate needs, namely some kind of visual representation of the production system we wanted to create for the editing and processsing of the TEI Guidelines, version P2. We knew we were going to write and edit it in some version of TEI SGML; we had faith that anything in SGML could be transformed into anything else. We just had to work out how, and what.

P1 had been produced by some devious hackery that only Michael understood, and more critically which only ran on the mainframe at UIC; we wanted something that would be platform (hardware and software) independe nt. Such was the promise of TEI SGML, after all. Somewhat to our horror, the only reasonable high level programming language in which we were both reasonably competent and for which there were decent implementations on all the machines we collectively used (IBM CMS, VAX VMS, IBM PC, Macintosh…) seemed to be a now largely-forgotten string handling language called Macro Spitbol, so we decided that our production system (what nowadays we’d call a work flow) would have to be written in that. But of course the heart of everything would be a nice author-friendly TEI SGML dialect, for which we optimistically coined the acronym ODD: One Document Does-it-all. ODD files would be parsed by an SGML parser, and its output filtered through a variety of Spitbol processors to create other formats. And that, more or less, is what we did.

On this schematic you can see the basic idea in blue. The big blue circle is the ODD format, from which are generated canonical TEI files (with extension .TIN (for Tiny) or .TEI), RL files (extension .TD), and DTD files, the three little blue boxes. DTD files are of course SGML DTD files, which ois why you see a green line going back from them to validate individual ODD files (I dont know why it’s labelled LB though). “Tiny” files would use a subset of the TEI Lite schema defined back in 1988; RL (later renamed .REF) files would use the TEI vocabulary Michael had developed for reference documentation of individual elements (“TD” for tag documentation). Down the middle you see a list of TLAs in blue which I think must have been attempts to decide on a name for the format (WEB, Joe, LAM, RDF, CSP…), though what they expand to I really don’t remember – what a pity we didn’t choose RDF. Or not. And over on the left in red you see some notes which eventually became the canonical structure of the TEI Guidelines: there is a chapter about the “blort”, containing prose paragraphs; there is a documentation element referencing the blort tag, and there is a parameter entity reference which pulls in the definitions for the blort chapter.

What happened next? Well, we did set up a workflow more or less on this model, and we did use three separate filters written in macro Spitbol (mostly by Michael) which turned our ODD SGML into two flavours of straightforward TEI-lite-like SGML, which we called “P2X” and “REF” and also generated SGML DTD fragments. After experimenting with a generic filter called “tf” (also in Spitbol) to translate the generated TEI files into LaTeX, and dallying with a Canadian tool called Omnimark, we finally settled on a rather swish transformation engine called Balise, which was produced by a French company called AIS. Either way we were able to print the fascicles of P2 in something that not only looked quite nice but also looked just the same whether I printed it in Oxford, or Michael in Chicago. Except for the paper size, of course: ain’t standardisation a marvellous thing.

And what happened to ODD? It turned out to be quite a good idea. We gave a presentation about it at the ACH-ALLC conference in 1994, though I cannot remember what we said and we never got round to writing it up. Michael developed the ideas in the “tag documentation” part quite extensively, and (I believe) used them also in his next job working for the W3C, but the TEI’s ODD stayed more or less unchanged until work started on the TEI’s XML reincarnation, at which time the whole system was re-imagined and redesigned as the lean mean generic schema generation system we know and love today. But that’s another story.

Tweaking the Agora Stylesheets – 1

The AGORA project (this one, not to be confused with this other one nor even this other one again) has defined a very simple TEI XML schema for  scholarly publishing. In this series of blog entries, I report my attempts to process a set of documents which conform to that schema into PDF and other formats, using the TEI stylesheet library.  My environment is a laptop running Ubuntu 10.04, on which I have installed the 5.1.4 release of the tei-xsl package and most of the texlive Ubuntu packages (versions dating from July 2009 according to dpkg).

On the train to London this morning, I  wrote a Makefile which validates each file and, if valid, then processes it using the teitolatex and xelatex commands. This produced something not entirely discouraging, with the  following obvious things to fix:

  • some of my files had  numbered headings and others didn’t. By  default the stylesheets added numbers willy nilly. I need to switch this behaviour off.
  • some of my files used <byline> in the header to indicate the affiliation for an author, like this:
    <byline><docAuthor>Fred Flintstone </docAuthor>
        Euphoria State University, Kansas</byline>.

    By default, the stylesheets clearly have no idea what to do with the text fragment following the <docAuthor>, and therefore spit it out on a page of its own.

I learned at the excellent MUTEC workshop last week that the recommended way of modifying these stylesheets is to set up a new « profile », so I duly visited the directory  /usr/share/xml/tei/stylesheet/profiles and created a new folder there called   /usr/share/xml/tei/stylesheet/profiles/agora (somewhat to my surprise this did not require root access).  I then copied the existing default specifications for each of the target transformations I thought I might use in my Agora work into this folder. Like this:

$cd /usr/share/xml/tei/stylesheet/profiles
$mkdir agora
$cp -r default/latex agora
$cp -r default/docx agora
$cp -r default/oo agora

The directory names (latex, docx, etc.) are not particularly well publicized: I worked out by inspection that « oo » must be the one invoked by the command « teitoodt »… presumably at some point it will be renamed Liboff vel sim.

Anyway, this setup should mean that if I now do e.g.

$teitolatex --profile=agora foo.xml

I should get the same result as I would if I left out the –profile … and so indeed I do. Good.  Time to start messing about.

I take a peek into the contents of my agora/latex folder. It contains just one file, called to.xsl — which presumably controls the conversion from tei to latex. One day maybe some clever person will add a file called from.xsl which does the opposite. Or not.

The file is rather dull: all it does is remind me that the file is copyright TEI Consortium 2008, and that the library it invokes is « distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY ». Fair enough. It also loads the stylesheet at
../../../latex2/tei.xsl but all it does to modify that is set some mysterious parameter called reencode to false. So clearly I am at liberty to add further modifications in this file… or will be once I have  changed permissions on the file.
../../../latex2 (i.e. /usr/share/xml/tei/stylesheet/latex2, sibling of the profiles directory) is the directory with the real biz. It contains files named for most TEI modules, as well as promising looking files like tei-param.xsl. A little sniffing around, and I have discovered the XSL template for procesing the TEI <head> element inside the file core.xsl, which contains the following magic:

<xsl:choose>
<xsl:when test="ancestor::tei:floatingText">Star</xsl:when>
<xsl:when test="parent::tei:div/@rend='nonumber'">Star</xsl:when>
<xsl:when test="ancestor::tei:back and $numberBackHeadings='false'">Star</xsl:when>
<xsl:when test="$numberHeadings='false' and      ancestor::tei:body">Star</xsl:when>
<xsl:when test="ancestor::tei:front and $numberFrontHeadings='false'">Star</xsl:when>
</xsl:choose>

That looks to me suspiciously like there should be a parameter called numberHeadings which I should set to false in order to suppress those pesky generated section numbers. (Of course, I’d have found that out immediately if I’d bothered to read the documentation, but …)

Back in my file profiles/agora/latex/to.xsl, I add the following line

<xsl:param name="numberHeadings">false</xsl:param>

and then regenerate the PDF, using the tweaked stylesheet in my agora profile:

teitolatex --profile=agora aaberge_2007.xml
xelatex aaberge_2007.tex

Bingo! no numbering. This could maybe be easier than it looks…

My second problem is trickier. The challenge and the delight of the TEI is precisely its open-endedness, and so it often happens that something which looks plausible in TEI has no obvious translation in some other markup system, such as LaTeX. In my case, how *should* the <byline> element be processed? A grep through the LaTeX directory shows me that at present there is no template at all for it, so my hands are comparatively untied. My first thought is just to add a template like the following to my file:

<xsl:template match="tei:byline/text()">
\author{<xsl:value-of select="."/>}
</xsl:template>

on the assumption that the bit of text inside the byLine elements might as well be treated in the same way as an author as anything else. But LaTeX is not so liberal: when it finds that I have generated

\title{The Semantic Web in a philosophical perspective}\author{Terje Aaberge}
\author{,
Sogndal, Norway}

it simply ignores the first \author. This suggests that I cannot solve this without learning more about LaTeX than I really want to.

Maybe I can modify the existing template for <docAuthor> to deal with this special case. In the file header.xsl there is a template like this

<xsl:template match="tei:docAuthor">
<xsl:if test="not(preceding-sibling::tei:docAuthor)">
<xsl:text>\author{</xsl:text>
</xsl:if>
<xsl:apply-templates/>
<xsl:choose>
<xsl:when test="count(following-sibling::tei:docAuthor)=1"> and </xsl:when>
<xsl:when test="following-sibling::tei:docAuthor">, </xsl:when>
</xsl:choose>
<xsl:if test="not(following-sibling::tei:docAuthor)">
<xsl:text>}</xsl:text>
</xsl:if>
</xsl:template>

It’s a horrible kludge, but if I insert the following before the final <xsl:if> element, it should make sure I output any following sibling text fragment before outputting the }

<xsl:if test="parent::tei:byline and (following-sibling::text())">
<xsl:value-of select="following-sibling::text()"/>
</xsl:if>

I therefore copy the whole of the <xsl:template> for docAuthor into my to.xsl file, add the above clause, and blow me down it (nearly) works. I had, of course, forgotten to suppress a second appearance of those pesky text fragments caused by the default processing for <byline>.  One more template:

<xsl:template match="tei:byline/text()"/>

fixes that.

Of course, the more I look at this, the less I like it. A much better solution would be to tag the affiliation data as such in the XML source, using an element such as <affiliation> perhaps, and then process it correctly into whatever LaTeX provides for the treatment of such things. But that would, as aforesaid, require some research into what LaTeX can do, as well as changing the Agora schema.

Not a bad way to pass the train journey to Paris, especially when surrounded by kids returning home after the half term hols.