Notes towards a definition of TEI Conformance








Each of the blobs here represents three subtly different things:

  • an ODD : that is, a collection of TEI specifications
  • a formal schema generated from that ODD, and its natural language documentation
  • the set of documents considered valid by that schema.
    The TEI provides TEI All : a set of over 500 uniquely identifiable elements, classes, attributes, etc. and a schema in which they are all permitted. For all practical purposes a user of the TEI must make a selection from this cornucopia, and we call that selection a TEI subset. Of course there are many many possible TEI subsets, each making different choices of elements or attributes or classes, but the sets of documents which each consequent schema will validate all have in common that they will also be considered valid by the schema TEI All.

A user of the TEI may however do more than simply choose a subset of the provided specifications. They may also provide additional constraints for aspects of an encoding left underspecified by the TEI, for example by requiring that attribute values be taken from a closed list of possible values rather than being any syntactically valid token. They may simply change the datatype of an attribute, for example from a string to an integer or a date. They may also provide an alternative identifier for an element or an attribute, for example to change its canonical English name for one from another language. In some cases, attribute value changes are equivalent to a subsetting operation; in others not. Renaming operations never result in a subset: a document in which the element names have all been changed to their French equivalents cannot be validated by an English language version of TEI All. A user of the TEI can also change the content model or the class membership of existing TEI elements, in ways which may or may not be equivalent to a subsetting operation.

We use the term customised subset for all these kinds of personalisation because they result in something which is not necessarily a further subset of the TEI subset concerned, but a further modification of it. In the general case, their conformance with TEI All can be determined only by inspection, and their validation may require some additional processing.

Finally, a user of the TEI is at liberty to define entirely new elements and attributes, and to make such components members of existing TEI classes so that existing TEI elements may refer to them. They may also modify the content models of existing TEI elements to refer explicitly to such new elements. This results in an extended subset, since it contains elements or attributes additional to those provided by the TEI All schema. Such additional components should always be labelled as belonging to a non-TEI namespace. A processor can then determine that these components may be left out of consideration when determining the validity of a document with respect to TEI All.

In additional to these formal considerations, TEI conformance involves attention to some less easily verifiable constraints, specifically the twin requirements of honesty and explicitness. By honesty we mean that elements in the TEI namespace must respect the semantics which the TEI Guidelines supply as a part of their definition. By explicitness we mean that all modifications (i.e. both customized and extended subsets) should be expressed using an ODD to document exactly how the TEI declarations on which they are based have been derived. (An ODD need not of course be based on the TEI at all, but in that case the question of TEI conformance does not arise)

Formally speaking, we can say of a conformant TEI document :

  • it must be a well formed XML document and
  • it is valid against the TEI All schema :
    • without modification (it is a TEI subset), or
    • after deletion of any elements it contains which are not in the TEI namespace including their children, irrespective of namespace (it is a TEI extension), or
    • after application of any canonicalization algorithm specified by its associated ODD (it is a TEI customized subset)

The purpose of these and similar rules is to make interchange of documents easier. They do not however guarantee it, and they certainly do not provide any guarantee of interoperability. Unlike many other standards, the goal of the TEI is not to enforce or impose consistency of encoding, but to provide a means by which encoding choices and policies may be more readily understood, and hence (to some extent) algorithmically comparable.

(Another reason) Why I love the Internet

So, at the recent TEI Conference in Vienna, Elisa and I were indulging in a little mutual admiration on our knowledge of an obscure work entitled Thalaba the Destroyer by the early English Romantic Poet called Robert Southey (rhymes, as any fule kno, with « mouthy »). So when I got back home, I went to look for the volume containing said work which I dimly remembered having on my shelves, in the decrepit-but-too-nice-to-throw-away-section. And sure enough, there it was. The front board has come loose, but the first three openings  look like this:


Having scanned those first few pages, I naturally asked Mr Google what he knew about the matter. And was thus able rapidly to confirm :

  • My copy of Thalaba is the cheap reprint (two volumes in one) published
    by Vizetelly and Beeton in 1853. There is a Google-scanned version of the same edition, available from the Internet Archive. They have included with it a couple of pages of  advertisements for other works published by Clarke Beeton (p 7 and 8) which are missing in mine however.
  • What seems like another copy of the same edition is currently on sale at Abe Books for the startling sum of $199.76. Mine is in poor condition,  which is why it only cost me half a crown back in 1967, when I used to frequent Oxford’s second hand bookshops (there aren’t any to frequent these days).

As you may have noticed above, my copy also contains several signs of its previous owners. As well as the book plate, and the inscription above, there’s a nice message from Aunty Sarah, the donor,  opposite the preface:

front-1and there’s also an intriguing note from « JB » dated some twenty years later, opposite the start of the poem proper.


So… what have we learned? Rosamund was given this book by her aunt, Sarah Brent, in 1860. And in 1882, her husband felt compelled to record his own experience of the Eastern exotic in the same book « We met at Persepolis an Arab maiden of most lovely form and features — she was a dream of beauty never to be forgotten ». What she made of it, one can only conjecture.

But why I love the Internet, is that (pondering these matters after breakfast this morning), it has helped me place these people a little more precisely in time and place. A search for « Rosamund Borrowman » told me that  the 1861 Census shows a person of that name, born 1825 in Kent, married to John Borrowman, born 1830 in Midlothian, residing in Middlesex in 1861. The site where I found this record is pay-walled so no further details available, but that seems reasonably plausible.

And searching for « Rosamund Borrowman John » I was able to find a record of her death. Some industrious volunteers have been surveying the gravestones of a place called Hambledon in Surrey, and there she is:   « Rosamund Vertue the beloved wife of John Borrowman. She died 25th August 1895. Also the above named John Borrowman son of Robert Borrowman born in Edinburgh 3rd April 1830 died at Hambledon 4th July 1906. Also Elizabeth daughter of the above died 22nd October 1932 aged 72 years » It’s all in the spreadsheet.

My next step, obviously, will be to find out where Hambledon is, and whether you can get there by train. Maybe.

Rompons avec le ronron techno-productiviste des institutions!

Back in August or September, I remember bleating anxiously on this blog about having rashly accepted to give a talk on les Humanités Numériques as part of a seminaire « Avenue Centrale » organised by the MSH in Grenoble. I can now report that eventually (in the English sense) I did manage to get some old slides updated and licked into shape, aided by the four hour T-not-so-GV journey to Grenoble last December, and a week or so of being unbearable to myself and everyone else around me. The slides duly appeared on Slideshare, and the folks at Avenue Centrale have even published a nice video and a podcast of me delivering them, but that’s not nearly as interesting  as what actually happened on the day.

Coming to terms with an implausibly pink armchair
Coming to terms with an implausibly pink armchair

On 20th December, after a meeting of the conseil scientifique du MSH Alpes, I found myself esconced in an implausibly pink fauteuil,  clutching a microphone, and ready to go, having delayed the obligatory 30 minutes for bigwigs to turn up, when there was a minor kerfuffle  as the organisers realised that a bunch of scruffy students were busy at the front door handing out an A2 sized pamphlet promisingly titled Humanités Numériques: Gare à la propagande!!! 

The source of the pamphlet, which characterizes me as un petit soldat de la conversion au numérique  des Humanités,  was  subsequently tracked down online by one of the French DH twitterati  (à savoir, Martin Grandjean) within a few minutes of my tweeting this image of it  after the show. Aside from the distribution of the pamphlet, the promised  Action-critique  took the form of three or four extra persons attending my lecture, one of whom also gave a brief speech deploring the industrial and social cost of mass digitization (I think) during the Q&A session. An agreeable though brief debate ensued, none of which sadly seems to have made it to the published version of the video, and we then all adjourned for coffee and horrible sandwiches downstairs, during which I was able to continue to chat amicably with the protesters, though the term seems barely appropriate. I learned that these were actually eco-warriors with concerns about the way big business was driving technology into inappropriate places (There have been somewhat critically received plans to hand out tablets to all school children, in an interesting reprise of the UK Government’s BBC Micro initiative in the 1980s)  On my way out I also tried to take some photos of the activists using my new tablet, which involved much banter and cursing, as I have barely mastered this new device. Out of deference to their desire for anonymity, the photos will have to stay in my personal archive for a few more decades though.

Tidings of this unusual  event caused a (very brief) flurry of excitement on twitter. Frederic Clavert was  a bit peeved to find that his logo had been appropriated for the pamphlet; others were disappointed to find no coherent plan for action in it. And there were also   (tee hee) expressions of extreme jealousy from a few of my DH colleagues — Moi aussi une affiche!  A brief sample of my first significant « moment » in the political history of DH  in France (Marjorie told me that’s what it was) follows.


Data versus Reality

… is not the title of the book I’ve been re-reading this week, though it might well be. Bill Kent’s Data and Reality was first published in 1978, and comes from the heroic age of database design and development, a period when such giants as Astrahan, Chen, Chamberlin, Codd, Date, Nijssen, Senko and Tschritzis were slugging it out over the relative merits of the relational, network, and binary database models and the abstractions they supposedly modelled : a struggle predominantly over terminology and ways of thought since (as Kent shows) almost all of these differently named and passionately advocated models were fundamentally very similar, differing only in the specific compromises they chose when confronted by the messiness of reality. Whether you call it a relation or an object or a record, the globs of storage handled by every database system were still records, combinations of fields containing representatives of perceptions of reality, chosen and combined for their utility in a specific context. The claim that such systems modelled reality in any complete sense is easy to explode; it’s remarkable though that we still need to be reminded, again and again, that such systems model only what it is (or has been) useful for their creators to believe. Kent is sanguine about this epistemological lacuna : “I can buy food from the grocer, and ask a policeman to chase a burglar, without sharing these people’s view of truth and beauty”, but for us, living in an age of massively interconnected knowledge repositories, which has developed almost accidentally from the world of more or less well-regulated corporate database systems, close attention to their differing underlying assumptions should be a major concern. This applies to the differently constructed communities of practice and knowledge which we call “academic disciplines” just as much as it does to the mechanical information systems those communities use in support of their activities.

In its time, Data and Reality was remarkable for introducing the idea that data representations and the processes carried out with them should be represented in a unified way, the basic idea of what we now call object-oriented processing; yet it also reminds of some fundamental ambiguities and assumptions swept under the carpet even within that paradigm. Are objects really uniquely identifiable? “What does ‘catching the same plane every Friday’ really mean? It may or may not be the same physical airplane. But if a mechanic is scheduled to service the same plane every Friday, it had better be the same physical airplane.” The way an object is used is not just part of its definition. It may also determine its existence as a distinct object.

Kent’s understanding of the way language works is clearly based on the Sapir-Whorf hypothesis: indeed, he quotes Whorf approvingly “Language has an enormous influence on our perception of reality. Not only does it affect how and what we think about, but also how we perceive things in the first place”. There is an odd overlap between his reminders about the mocking dance which words and their meanings perform together and contemporaneous debates within the emerging field that Wilks has charmingly characterized as “Good Old Fashioned AI”. And we can also see echoes of similar concerns within what was in the 1970s regarded as a new and different scientific discipline called Information Retrieval, concerned with the extraction of facts from documents. Although Kent explicitly rules text out of discussion (“We are not attempting to understand natural language, analyse documents, or retrieve information from documents”) his argument throughout the book reminds us that data is really a special kind of text, subject to all the hermeneutical issues we wrongly consider relevant only to the textual domain.

This is particularly true at the meta-level, of how we talk about our data models, and the systems we use to manipulate them. Because they were designed for the specific rather the general, and because they were largely developed in commercially competitive contexts, the database systems of the 1970s and 1980s proliferated terms and distinctions amongst many different kinds of entity, to an extent which Kent (like Ockham before him) argues goes well beyond necessity. This applies to such comparatively arcane distinctions as those between entity, attribute, and relationship, or between type and domain, all of which terms have subtly different connotations in different contexts, though all are reducible to a more precise set of simple primitives. It applies also (and here the TEI in me sits up and smirks) to the distinction between data and metadata. Many of the database systems of the eighties and nineties insisted that you should abstract away all the metadata for your systems into a special kind of database variously called a data dictionary, catalogue, or schema, using entirely different tools and techniques from those used to manipulate the data itself. This is a needless obfuscation once you realise that you cannot do much with your data without also processing its metadata. In more recent times, one of the more striking improvements that XML made to SGML was the ability to express a schema and the objects it describes using the same language. Where what are usually called the semantics of an XML schema should be described and how remains a matter which only a few current XML systems (notably the TEI) explicitly consider.

Kent seems to have been a modest and likeable man. He retired in 2000, and died five years later, leaving a legacy of accessible and still provocative papers, most of them available from his website . Like many other pioneers in computer science, his academic qualifications come from unrelated fields (in his case, chemical engineering and maths); like many others he worked long hours for IBM and HP, but achieved fame and intellectual satisfaction outside the corporate world in the development of industry standards and professional associations. Maybe that experience is also what underlies the much quoted paragraphs which end his book:


Unexpected adieux

This sunny Sunday morning sees me setting off for a couple of weeks of TEI workshops, one in Paris, one in Graz. Nothing unusual there, nor in the fact that one is better prepared for than the other. But it has been an unusual week all the same, with two deaths and possibly a new beginning. The deaths first, since they are more difficult to write about. They perturb habitual patterns, making me confront and try to articulate parts of life that are hard to fit into a public blog, yet belong there in the absence of any other personal journal. (I say « public » but doubt that anyone except me reads this).

On Tuesday morning, I received from my friend Guy in Italy a text message saying that his partner Daniela had suffered a stroke and was in a coma; 24 hours later came another announcing her death. It is hard to react adequately to such events at a distance, and particularly so by text message, so I am waiting for a later less painful time to talk to Guy. I learned from a mutual friend that the funeral was yesterday. I dont want to obituarize but Daniela was a very generous and very affectionate person, as well as a fiercely independent one. I am very glad that she did not stay long in her coma, nor return from it badly scarred; I am also very glad that the last time I saw her was at a joyful family occasion in London.

On Saturday evening, yesterday, I received an email informing me of another death, also coincidentally on Tuesday: Chris Sheppard, in whose company I passed my adolescence and early twenties, chasing the same girls, crashing the same teenage parties, growing up to pursue not the same but similar academic careers. Chris was the first person in my school to know where to buy Levi 501s and how to shrink them to fit (in the bath). He introduced me to the works of Raymond Chandler and the collection of cigaratte packets. He was far too cool to take fashionable drugs at Oxford, but was on good terms with those who did. It was largely following his example that I returned to Oxford to take my masters degree in 1969, a year behind him. As graduate students we shared a rented hovel in Stanton St John (chemical toilet, coal fire, wall to wall books) for a year during which Chris taught me almost everything I know about literary scholarship and the love of books, not by precept, but simply by example. I was best man at his wedding back in 1976, but our paths diverged thereafter. At his retirement a couple of years ago, he was head of special collections at Leeds University’s Brotherton Lribrary, where I  emember visiting him and being shown some of his more recondite treasures (a lock of Mozart’s hair, Conan Doyle’s photos of faery folk); I think the last time I saw him in person must have been at a lunch with P.N.O. Pullman some time in the 90s. Now of course that it is too late, I regret bitterly even the dwindling flow of Xmas Card exchanges and the fact that my last email with him was more than six years ago.

As to the new beginning — well, it seems a small thing in this context, but I am now feeling quite positive about the idea of buying a house some distance from the back of beyond in rural France. A specific house, that is, of which perhaps more anon. But for now, I will go back to worrying about tomorrow’s training course at the EPHE in Paris, and that in Graz a week later.

A trip to La France Profonde

So this week I have mostly been not thinking about writing academic papers at all, which may or may not be a good think. Instead I spent the first part of the week tidying up materials for the next TEI training course, which is now pretty well polished, and also for the one after that, which is not. The process of thinking about what materials to use follows a fairly recognisable pattern in which ambitious optimism (I’m going to completely revise this bit, make up something new and exciting, strike out into unknown territory) has to eventually give way to pragmatic opportunism (I’ve got this already, it just needs checking, minor tweaks, translating). When I am preparing two courses which are due within a few weeks of each other, this means that the first course moves on to the second stage while the second one is still rejoicing in the first stage. Which was the case this week. Oh, and about the only thing I have in common with Sam Beckett is that I can no longer say whether my material is originally French translated into English or the reverse, since most of it has been through the process both ways several times.

Aside from that, I spent most of the week on trains, or other forms of transport, on an expedition to La Vergne, returning at the weekend via Nottingham. As follows:

depart arrive
Weds 27 Aug home 0910 on foot/bus
Oxford to Paddington 1001 1130 train
London St P to Paris Nord 1225 1547 eurostar
Paris Austerlitz to La Souterraine 1652 1921 Ic 3655
Thurs 28 Aug
La Souterraine to Gueret 930 1005 ter bus
Gueret to La Vergne 1100 1130 taxi
La V to Bussieres Dunoise nice walk
Bussieres to La Souterraine 1445 1615 taxi
Frid 29 Aug
La Souterraine to Paris A 1038 1318 ic3620
Paris Nord to St Pancras 1513 1640
Kings Cross to Grantham 1719 1827
Grantham to Nottingham 1857 1934
Sund 31 Aug
Nottingham to Oxford 1310 1546

And what have we learned? It’s possible to get to and from La Vergne by train within a day for about £400 pounds return (less if you cash in some Eurostar points). There’s not much happening in La Souterraine , and even less in Bussières-Dunoise. Guéret seems like a decent sized town though and is accessible by train from at least two different directions. Generally speaking, the Creuse is not the back of beyond: it’s behind the back of beyond. There are many cows, and many hills. There are probably no decent restaurants for miles. There used to be a railway to transport potatoes and beef to Paris, but they took it away years ago, and now all that’s left is a rather nice rural track which has the merit of avoiding most of the aforesaid hills. And there’s a lake behind the house, where you can fish but not (allegedly) swim.

Genetic editors, please note

After finishing last week’s entry about my rash commitment to write a book chapter, I secretly vowed to monitor my progress by producing weekly reports here. I then spent the entire week (when not shopping, eating, or sleeping) working on next month’s TEI course in Paris, essentially a revision of the one I gave in May. Almost, but not quite, because half way through the week, I received another reminder of another rashly-made commitment, this time to deliver a public lecture in Grenoble in December. I promptly dashed off the following proposition :

Ceci n’est pas une pipe: l’importance de la modélisation aux humanités numériques

Lou Burnard

Récemment, on a vu emerger de l’ombre de la inter-disciplinarité une discipline nouvelle qui s’appelle les Humanités Numériques (Humanités Digitales en Suisse, Digital Humanities ailleurs). Elle represente d’abord la confrontation, et ensuite l’adaptation aux méthodes et possibilités des technologies nouvelles de l’entreprise intellectuel et scientifique de toute la domaine des sciences humaines. Ces technologies comportent notamment l’informatique, mais aussi de la statistique, de la linguistique computationelle, et de la visualisation des données. Mais en effet cet emergence ne serait qu’une évolution, voire une continuation, d’un débat assez vieux – déjà percéptible au 19ème siècle — qui opposerait les sciences dures aux sciences humaines. Dans cette intervention, je propose que cette opposition semble d’origine plus sociale que méthodologique, et que les méthodes des SHS et les méthodes des sciences dites dures ne sont pas tellement loin l’une de l’autre. C’est la création, l’évaluation, et la manipulation des modèles et des hypothèses qui caractérise tout effort d’élargissement de science, et la modélisation comme processus abstrait donc devrait être au centre de nos disciplines, qu’il s’agit de la modélisation des structures textuels et linguistiques, de la modélisation des procédures informatiques, ou de la modelisation du monde physique.

Né en 1946, Lou Burnard a pris son DEA en littérature anglaise du 19e siecle à Oxford en 1971. De 2002 a 2012, il est Directeur-adjoint aux Services informatiques de l’Université d’Oxford où il s’occupait des applications informatiques dans les domaines des sciences humaines depuis des années, surtout en linguistique de corpus (British National Corpus), en bibliothèque numérique (Oxford Text Archive), et en l’encodage de textes. Actuellement retraité, il est reconnu comme expert dans ces domaines. Il a travaillé en France comme prestateur de services aux agences Adonis et Hum-Num et ailleurs en France: il est membre des Comités Scientifiques des Maisons de Sciences de l’Homme à Caen et à Grenoble.

Cunning, or what, I said to myself : if I have to produce a chapter in English on a topic I know nothing about it, I might as well repurpose it in French and get good value for money. And then, just to be on the safe side, I ran this text by my friend Marjorie, who is a native French speaker amongst many other good qualities, and thus well placed to tactfully remove the many barbarisms in this first draft. I was duly humbled by her response :

Ceci n’est pas une pipe : l’importance de la modélisation pour les humanités numériques
Lou Burnard

Récemment, on a vu émerger de l’ombre de linter-disciplinarité une discipline nouvelle qui s’appelle les Humanités Numériques (Humanités Digitales en Suisse, Digital Humanities ailleurs). Elle représente, pour tout le domaine des sciences humaines, la confrontation puis l’adaptation aux méthodes et possibilités des technologies nouvelles. Ces technologies comprennent notamment l’informatique, mais aussi la statistique, la linguistique computationelle, et la visualisation de données. Mais cette émergence ne serait en fait qu’une évolution, voire une continuation, d’un débat assez ancien – déjà perceptible au 19ème siècle — qui opposerait les sciences dures aux sciences humaines. Dans cette communication, j‘avance l’idée que cette opposition semble d’origine plus sociale que méthodologique, et que les méthodes des SHS et celles des sciences dites dures ne sont pas si éloignées. C’est la création, l’évaluation, et la manipulation des modèles et des hypothèses qui caractérise tout effort d’élargissement de la science, et la modélisation comme processus abstrait devrait donc être au centre de nos disciplines, qu’il s’agisse de la modélisation des structures textuelles et linguistiques, de la modélisation des procédures informatiques, ou de la modélisation du monde physique.

Né en 1946, Lou Burnard a obtenu son DEA en littérature anglaise du XIXe siècle à Oxford en 1971. De 2002 à 2012, il a été Directeur-adjoint du Service informatique de l’Université d’Oxford, où il s’occupait depuis des années de l’applications de l’informatique au domaine des sciences humaines, surtout pour la linguistique de corpus (British National Corpus), les bibliothèques numériques (Oxford Text Archive), et l’encodage de textes. Actuellement retraité, il est un expert reconnu de ces domaines. Il a travaillé en France comme prestataire de services auprès d’Adonis et Huma-Num, et ailleurs en France : il est membre du Comités Scientifiques des Maisons de Sciences de l’Homme de Caen et de Grenoble.

Suitably chastened by this salutary reminder  that my command of the French language is not as perfect as might be wished for, I removed the green ink, and sent it off to Grenoble, from which I rapidly received the following reminder that sometimes less is more :

Le résumé que vous nous avez envoyé est de fait plus important (environ 1300 caractères), je vous propose donc
(pour la version papier uniquement, la version web pouvant elle rester plus développée) de le réduire quelque peu. Seriez-vous
d'accord pour que, par exemple, nous enlevions la partie finale (cf proposition ci-dessous) et les déclinaisons autour du nommage
((Humanités Digitales en Suisse, Digital Humanities ailleurs)  ou préférez-vous le retoucher vous- même ?

Pour brochure : "Récemment, on a vu émerger de l'ombre de l'inter-disciplinarité une discipline
nouvelle qui s'appelle les Humanités Numériques. Elle représente pour tout le domaine des sciences humaines la confrontation, puis
l'adaptation aux méthodes et possibilités des technologies nouvelles. Ces technologies comprennent notamment l'informatique,
mais aussi la statistique, la linguistique computationelle, et la visualisation de données. Mais cette émergence ne serait en fait
qu'une évolution, voire une continuation, d'un débat assez ancien – déjà perceptible au 19ème siècle -- qui opposerait les sciences
dures aux sciences humaines. Dans cette communication, j'avance l'idée que cette opposition semble d'origine plus sociale que
méthodologique, et que les méthodes des SHS et celles des sciences dites dures ne sont pas si éloignée."

That’ll teach me. Maybe.

Metamodelling through : the prolegomena

So back in February I was asked to contribute a chapter to a new book being confected by some top people in the domain of the digital humanities,  an invitation which I naturally accepted with alacrity, and only a small sense of alarm. I admit: I was flattered, though naturally also felt it was about time my eminence was recognised in such a way.

Dashing off an abstract is an easy task, so I did that, and then forgot all about it. Here’s the abstract. Like other such pieces,  it promises much, and even gets mildly polemical towards the end, which seemed to do the trick, as the proposal was, in due course, accepted.


Where do metamodels come from and how do they survive?
Lou Burnard

There is a very old joke about standards which says "Standards are a
 good thing because there are so many to choose from". Like many old
 jokes, this plays on an internal contradiction (the structuralist
 might say "opposition") in its topic. Standards are, on the one hand,
 of most benefit to the extent that they reflect and facilitate
 diversity ; on the other, they are of necessity managed or even
 imposed by a centralising authority. This contradiction is
 particularly noticeable when the process of standardisation has been
 protracted because the technologies concerned are only gradually
 establishing themselves. We see this tension even in consumer
 electronics where there is a financial market-driven imperative to
 establish standards as rapidly as possible; but the same tension
 underlines the gradual evolution of ways of thought via communities of
 practice into de facto and (eventually) "real" standards. This article
 explores the evolution of standards for data modelling methodologies
 with regard to this tension. It considers some significant early
 experiments with the application of data modelling techniques to
 humanities research data (Manfred Thaller; J-C Gardin) and discusses
 to what extent some researchers simply adopted technical standards
 emerging in the wider data processing community (relational databases,
 information modelling), while other communities strove to define their
 own models (AI, language understanding systems). It will present in
 some detail the theoretical model (metamodel) underlying the Text
 Encoding Initiative's approach to standardisation and ask the question
 whether, over time, all such community-based efforts are forced
 further towards convergence and away from diversity. The TEI currently
 maintains a balance between "do it like this" and "describe it like
 this" schools of standardisation; in the long run, it therefore risks
 being superceded by advocates of the latter who distrust the former,
 or advocates of the former, who are impatient with the latter. 
Oxford, 1 Mar 2014

Summer came and summer is now going, and this particular bird is coming home to roost. I received last week a polite reminder that my manuscript should be delivered by the end of the current month, should conform to a defined house style, and would I please sign in blood the form I was sent back in April assigning my rights in this non-existent work to non-existent publishers Snipcock and Tweed ? Naturally I replied at once pleading for a stay of execution (but ignoring the rights assignment question) which was graciously accorded, somewhat to my surprise, even unto mid October. So now I really have little excuse not to find out what grand idea this abstract is abstracted from, really ought to get down to doing the research it grandly promises to summarise, and write the wretched piece. If only I didn’t have all those other more interesting (or less interesting but more urgent) things to do.

Well, let’s see., I plan to use this blog as a record of the painful process, just so that in years to come I can look back and see where it all went horribly wrong. At least no-one is likely to find me here.




What does the textual scholar require of computer science?

[Here’s the text of the talk I gave under the above title at the Fifth Postdamer I-Science-Tag « Digital Humanities Meets Information Science » on 19  March. I haven’t revised it properly yet — there’s nothing like reading a text aloud for making you aware of the places where it’s wandering off into the land of waffle — but here it is anyway] 

A recent diatribe by Roger Scruton (‘Scientism in the Arts and Humanities’, in New Atlantis, Fall 2013) has got me thinking about that old chestnut “what is the digital humanities”. Scruton argues passionately and persuasively against what he terms “scientism” – the pretension to scientific method – in the humanities, reserving particular disdain for the notion of “research” in the humanities as the term is currently used by cross-disciplinary “xxxx-studies” in humanities departments across the English-speaking world. He points out that “research” in the sciences is concerned with the establishment by scientific method of evidence to support or refute a pre-existing hypothesis about the world, whereas in the Humanities, it is applied to just about kind of activity that may add to the sum of cultural knowledge at our collective disposal, or may simply act as a substitute for such knowledge. I was struck by his unashamedly Arnoldian appropriation of the term “culture” and what follows represents some further thoughts along similar lines.


If science aims to deepen our understanding of “the world”, and the humanities to deepen our understanding of “culture”, we do need to find a definition for culture which goes beyond simply saying (as Scruton does) that it is about the “I and I” (probably not so much a hint of Rastafarian influence as an insistence on the subjectivity of cultural thinking), though it is true that any account of culture which ignores its effects on the individual cultural consumer will be sadly deficient. The laws of physics operate whether we know about them or not; the same cannot be said of cultural norms. And yes, of course, culture, particularly “high culture”, is a social and political construct, reflecting or reacting against the social and political power structures of the context in which it is articulated, and thus seemingly entirely contextual and contingent. But such naïve cultural relativism simply ignores the effectiveness with which the very contingency of culture also reveals, often unconsciously, its context, enabling us to construct hypotheses around the social and political norms concerned, and to assess it with reference to a wider context. The pre-occupations of human culture have not changed so much over the centuries, though different reactions to (say) birth, sexual partnership, time, death, and the construction of society are readily discernible, as are different reactions to those reactions. It seems to me that a study of culture, in the sense for which the Germans used to use the term geistwissenschaft , is a study of human reactions to, and constructions of, the world, and of those constructions. I further suggest that the relative merits of the various possible explanations it offers may be assessed in the same way as we evaluate purely scientific explanations.


A scientific explanation is valued according to the effectiveness with which it provides evidence in support of a hypothesis. If however the hypothesis is very general, for example that there is a single elegant principle governing the behaviour of space, mass, and time, it may not readily be identifiable as a hypothesis. When Eco says (in Interpretation and overinterpretation, 1992) that we value Copernicus’ model of the Universe more than Ptolemy’s not only because the former explains aspects left mysterious by the latter, but also because Copernicus enables us to understand the reasoning behind Ptolemy, he is not simply applying a humanistic perspective, exercising the hermeneutic meme to rhetorical effect, but demonstrating that evaluation always proceeds in the same way, whether we are considering the motions of the planets, or the relative merits of 19th century pulp fictions. For cultural objects do exist in the real world, and the cultural readings which confer “cultural” status upon them are also phenomena of the real world. Hence there is nothing inherently implausible about using scientific methods to gain some understanding of their behaviour, and how they function,

We should not however fall into the trap of supposing that in applying such methods to generate “scientistic” descriptions we have exhausted all there is of value in understanding a cultural object, a work of art. The history of a cultural object includes the history of its status considered as a work of art, but its meaning goes beyond the aggregation of perceptions of it as manifested by recorded opinion. Some of those perceptions may be ill-conceived or unhelpful, failing the Eco test of greater explanatory power for example, or other conceptual norms. To read King Lear solely as a political argument about kingship ignores the greater resonance of what it has to tell us about family life. To read Hamlet solely as an instance of the vogue for “revenge” tragedies that seems to have occurred on the English stage around the turn of the 16th century seems similarly wide of the mark. Contemporary African readers of Dickens’ Great Expectations sometimes reduce it to a fable in which Pip’s innocent life as a village dweller is corrupted by wealth and social class as soon as he moves to the town. Such a reading is one which Dickens might have recognised, and which the text certainly licences, but historically-minded critics may still feel that there is something wrong with implicitly equating the experience of a 20th century upwardly-mobile African villager and that of an imagined member of the 19th century rural poor. (Even so, a judgement we might consider inappropriate on the grounds of anachronism when applied to a specific cultural product – for example, the use of racist or sexist terms in early 20th century writings – is surely appropriate when applied to the context in which such writings are created or delivered; indeed, the writings constitute essential evidence warranting such judgements.

Consider, for example, linguistics. Language is surely the archetypal manifestation of a cultural object, almost a metaphor for culture itself (we talk about the “language of art”, we say that paintings and poets “talk to us” in a particular way, we even talk of a “vernacular” architecture). Over the last few decades, it has become increasingly clear that new technologies have facilitated a new perspective on the ways languages are used, hence how they change, and even perhaps what fundamentally they are. Corpus linguistics emphasizes the performative aspects of language, seeking to identify recurrent, possibly unconscious, regularities of usage, patterns which demand an explanation. Some have even claimed that no linguistic structures exist beyond such regularities of usage and the patterns associated with them, that there is no such thing as “grammar” analogous to the laws of physics in the real world. Even so, some explanation has to be found for these patternings. It is not necessary to subscribe to atavistic Chomskyan theories about innate grammar to seek explanations for them in terms of some general (and falsifiable) hypotheses about how languages function, for example to explain language variation and change by reference to a principle that innovation must always show itself first as deviation and is frequently associated with an assertion of group identity, that language users always value mutual comprehension above formal coherence or adherence to predefined norms, and so on.

So we should avoid depending only on a scientifically-derived and statistically-justified assessment of the facts of cultural reception. The history of digital humanities is dotted with the corpses of over-enthusiastic systematisers, from T.C. Mendenhall’s “characteristic curves of composition”  to J.F. Burrow’s reduction of Jane Austen’s style to vectors of frequency data (Computation into Criticism, OUP, 1987).  This is not, of course, to say that statistical stylometry has nothing to tell us; just that it can only ever be a means to an end. The most scientific of stylometricians will always use the objective evidence revealed by their analysis in support of an entirely subjective judgment, be it about authorship or about style. As Stanley Fish did not quite say “There is always a text in this class”: in all such judgements, the constructed text, the reading of the evidence, is the end result of the research, whether it is obtained by meticulous statistical methods or good old fashioned introspection. And I tend to agree with Arnold, and with Scruton, that constructing such readings and transmitting them is actually the purpose of the Humanities.


A reading is however by no means the same kind of thing as a model.




When I give introductory talks about the purpose and nature of text encoding, I often use the following schema to represent the distinction:

In its usual context, this schema is meant to show several things:

  1. The process of transforming resources (cultural objects such as books, paintings, historical documents etc.) into digital form is always a form of re-presentation, abstraction, reduction, reinterpretation, or encoding. Or, one might say, reading.
  2. The results of that transformation into digital form can be analysed and re-interpreted, automatically giving rise to an enriched version of that reading, which in turn can continue to be enriched by analysis in a kind of virtuous hermeneutic circle
  3. The process of encoding, and the process/es of analysis, must however be informed by the same abstract model

Perhaps this is merely a long winded way of saying that you cannot get more out of a system than you put into it, but it does suggest that the conceptual model underlying a set of readings is a different kind of thing from any of those readings and operates at a higher level of abstraction.

I freely confess that my ideas about what the humanities are or should be were formed during a distant epoch: the end of the 1960s. And my ideas about what computer science is or should be were formed during one that feels even more remote : the end of the 1970s. I have also lived long enough in the hinterland between the two disciplines to see how the intellectual territory laid claim to by either has evolved.

In the 1960s, the discipline associated with the study of English literature, at least as I experienced it at Oxford University, was going through one of its periodic fits of self doubt. At other universities, these were tumultous times, as the waves of what was to become known as Theory (with a capital T) began to sweep away the Arnoldian consensus that works of art existed independently from their creators and consumers and were invested by an innate cultural value. Even at Oxford, traditionally skeptical about such French (or worse Cambridge) vulgarity as the search for theory, it was advisable to be aware of such theoreticians as Beardsley and Wimsatt, and to be able to shoot down the intentionalist and affective fallacies. We agreed that an understanding of its author’s intention (as far as these could be determined) did not exhaust the meaning of a work of literature, any more than did an itemisation of its effects as recorded by its readers. We felt obscurely that we needed to place works within their historical and social context, to assess the extent to which they deviated from or confirmed reader expectation at different times, but we lacked the tools to do that, other than by the painstaking process of reading and remembering many, many books. We lacked an abstract model of how literature functioned or what it was, and did not know how to construct one.


The computer science I encountered at the end of the 1970s, by contrast, seemed obsessed by ways of representing knowledge, and constructing models. The entity-relationship model was succeeded by the Codasyl network model, which was blown away by the relational model, just as the giant mainframes began to be blown away by distributed networks of “personal” computers. Under the influence of large amounts of money and requirements for increasingly complex centralized information systems, these modelling techniques naturally evolved into methodologies such as SSADM (Structured Systems Analysis and Design Method, a set of standards developed in the early 1980s for systems analysis and application design widely used for UK government computing projects) . It is easy to poke fun at that expansive pre-web era, in which modish re-brandings of essentially identical techniques succeeded each other with confusing regularity, always with extravagant claims of advanced capabilities just around the corner, in the “next generation” architecture. The next generation when it actually arrived in the nineties was distributed, decentralized, and almost entirely uninterested in all of the effort which the database designers and conceptual modellers of previous generations had put into trying to construct and impose a federated approach to the representation and storage of knowledge. (Which is why we now see the reinvention of logic programming in the form of linked data: but that is a different story.)


Nevertheless, like many others at the time, I found that the tools and techniques of computer science, though they might be described in terms of a particular jargon, and though their field of application might seem entirely alien, still had something to offer the humanist. Could it be that an abstract model for the way that texts and documents function – which I take to be the essential business of the humanities – might be expressed using the same language as that used to model the data flows and processing requirements of East Midlands Gas?

It seemed clear to me that texts and documents should be described from at least three perspectives:



as physical objects with a visual representation; as linguistic objects made up of words and phrases drawn from some kind of linguistic system; and as intensional objects with reference to real world objects, events, or entities. Most computer systems of the time tended to prefer one or other of these aspects. A word processor would help you produce nice printed copies of your documents; an information retrieval system would help you investigate their language; a database would help you describe what they were about. Systems which crossed these frontiers, enabling you to control the appearance of particular words used to describe felonious transactions in a court record, for example, were harder to find, and usually required to be custom-built, with many compromises along the way.

With the arrival of markup languages such as SGML in 1987 and XML a few years later, it became possible at last to describe a document in a detailed way independently of whichever of these three aspects was to predominate in its processing, and hence in a way that facilitated all of them equally. And with the arrival of the Text Encoding Initiative around the same time, an extraordinary adventure in document modelling got underway. Much has been written about the TEI (not all of it by me) and its significance; my favourite comment is that whatever else we may say of the TEI Guidelines, as Basil Bunting said of Pound’s ‘Cantos’, “they resemble the Himalayas: you can ignore them if you like – but you will have to go an awfully long way round.” The TEI’s relevance to the present paper is that it represented the first and so far only time that scholars from across the humanities disciplines were succesfully corralled into achieving some kind of consensus about the “significant particularities” of the documents they studied. The TEI was (and perhaps remains) a unique exercise in inventorising the components of the models underlying research in the humanities, from the disparate points of view of lexicographers, linguists, critical editors, manuscript scholars, historians, literary scholars, and librarians. To find an abstract language adequate to represent such divergent perspectives within a single framework we naturally sought to apply data modelling techniques inherited from computer science, expressed not in UML or SQL but using the new text-friendly features of SGML. The rest, as they say, is history.

If the success of the TEI shows us that the modelling techniques inherent to computer science could successfully be imported and made to function within the humanities paradigm, it seems reasonable to conclude by asking whether this is a unique instance of such synergy.

So, parodying Monty Python, What did computer science ever do for us humanists?  Unsurprisingly perhaps, the things that working textual scholars seem to most appreciate about the impact of information technologies on their working practices are all things that computer science as a discipline tends to take for granted. When asked a version of my working title for this piece, an English professor of my acquaintance replied:

I think I owe the discipline a great deal… the advantages of on-line ordering before a visit to the British Library (say), and (from home) easy access to bibliographical and biographical information when preparing a book or essay ms for the press. I’ve regularly used Google to track down unattributed quotations which might otherwise have taken me ages to locate; I also use the electronic databases ECCO and EEBO, although I think the interfaces and general tractability have some way to go. I ought to add, I think, the sheer convenience of being able to assemble large and complex texts–such as editions—electronically, where relevant information comes to hand over a period of time. I only wish word-processing had been available when I completed my Ph.D. in 1979… Finally, I assume that without information science there could be no email, and without email I think that academic exchange as we know it might grind to a halt.”

This reply perhaps demonstrates how deeply embedded information science has become. The aspects selected by my colleague – networked access to information resources both of the kind traditionally held by libraries and of the kind traditionally embodied in one’s peers – constitute a change in the knowledge infrastructure, the context in which work is done. There is much to be added if we are to give an adequate account of that infrastructure: about the politics of open source access, about the alleged democratising (or to use the French word vulgarisation) of access to cultural resources, about all the ways in which the Internet has transformed our ways of knowing about the world, and the world that we know about. “Never before have so many people known so little about so much” … but these changes are driven more by commercial and social imperatives than they are by the interplay of academic disciplines which is my subject.

My colleague’s reference to word processing also hints at a more subtle change in the way that work is done itself. Of course writing on a word processor is only superficially like writing on a typewriter, just as a typewriter is only superficially like a quill pen. The extent of quantitative change in going from a machine in which making corrections is an expensive and limited process to one in which documents are never finished such is their fluidity and plasticity really does approximate to a qualitative change. In the 90s this occasioned anxiety about apparently fundamental shifts in the very nature of scholarly communication, even the thinking process itself, induced by the spread of new technologies. A couple of decades later, in a seemingly entirely fragmented, and decentred world, drowning in media which seem to be dominated by twitter and sound bites, we do well to remember that there is a positive side to this transformation.

In placing first the availability of digital resources, however, I think my colleague hits the mark exactly. The challenge for computer science has always been to find better tools for coming to terms with information glut, whether in the form of paper archives or millions of digitized books. The success of Google may have suggested to some that the indexing and cataloguing techniques associated with classical information retrieval were entirely superannuated. But that model of the document as witness to a mode of expression, a particular discourse, suggests that such a view is premature. Indexing techniques are beginning to take on new and more sophisticated clothing, their function rebranded as text mining or text modelling. If it is the words that conspire to form the meaning of a text, we should be able to able to formulate new, more coherent, and better informed hypotheses about that meaning on the basis of their relative co-occurrences and absences in the immense bodies of digital text now at our disposal. To quote Ted Underwood, “The notion that documents are produced by discourses rather than authors is alien to common sense, but not alien to literary theory.” As we do so, the availability for the first time of massive quantities of digital text structured and organized in terms of our traditional models of text and textuality (rather than their purely visual properties) will enable us to make richer (and thus more explanatory) models against which to judge the salience of individual works, and in terms of which to categorise their context. Rather than looking for the proverbial needle in a haystack, we should start considering why hay is such a good home for it. I do not know whether that is a notion that computer science has fully assimilated as yet.

Poster Slamming

At the TEI 2013 member conference today, I had the pleasure of participating in the « Poster Slam », a well established TEI ritual in which each poster-presenter is given one minute (and one slide) to introduce the topic of their poster as a means of persuading people to come to it. Preferably in verse. This year, Syd made the fatal mistake of allowing presentations in languages other than English, providing they were accompanied by a translation. So Nicolas Larrousse and I naturally presented the following poem:

Le Tageur et L’Archiviste

Le tageur ayant tagué tout l’été
Se trouva embarrassé l’avenir étant arrivé.
Pas un seul petit morceau
d’explication claire de ses travaux

Ze tagger having tagged all summer long
Found himself embarassed when the future has arrived
Not one little bit of explanation survived for all his efforts

Il alla chercher des avis malins
Chez l’archiviste son copain
Le priant de lui prêter
De la sémantique pour tout regler.

E went to ask some tricky advice from his friend ze archivist
Begging im to lend him some semantics to sort sings out

Les archivistes ne sont pas créateurs
C’est là leur moindre défaut.
Que faisiez-vous au temps chaud ?
Dit-il à cet emprunteur.

But archivists are not creators, that’s their smallest problem
What did you do during the fine weather?
He asked the borrower

– Nuit et jour à tout venant
Je taguais, ne vous déplaise.
– Vous taguiez ? j’en suis fort aise.
Eh bien! transformez maintenant.

Day and night I was tagging for anyone, if you dont mind
You were tagging? Oh that’s fine. So now you can do transformations!

Nicolas did the English bits, and I did the French bits, under the inspiration of the late great C. Trenet.

Further adventures with ODDs

This post is mostly an aide-memoire, since how to do the ODD things I want to do is not very well documented in the TEI as such.

First challenge

I have an ODD which was produced by webRoma some time ago and which (naturally) uses the traditional « exclude » syntax. I want to convert this to the new « include » format and also to ensure that it won’t get any of the new elements added to P5 since it was first defined.  I proceeds as follows:

  1. I look at the source of my ODD and I see the comment Roma inserted in the <sourceDesc> « created by ROMA on Monday 21st June 2010 »
  2. I go to the list of releases on the TEI sourceforge site to find which release of P5 must have been in use at that date. Judging by the dates here, it is probably release 1.6 I want
  3. whichversionBuried away in the standard release of the TEI Stylesheets there is a a cool utility for converting an « exclusion » ODD into an « inclusion » one. It’s called tools/odd2nuodd.xsl and I run it like this:
saxon -p defaultSource= 
myOldODD.xml tools/odd2nuodd.xsl > myNewODD.xml

Note the inclusion of the 1.6.0 release number as the source directory to be used when the stylesheet starts looking for TEI definitions.

Second challenge

I have two or more new style ODDs and I want to compare their use of the TEI to assess their interoperability. So far, I only have an approximation to an answer for this, in part because I am too lazy to improve the scripts I hacked together for it last time, in part because it’s actually a rather ill-defined problem, and hence hard.

The approximation goes as follows:

  1. Run an XSL transformation on each ODD in turn, appending the results to a big text file listing element names and what happened to them in which ODD;
  2. Run a perl script (ouch) on the results of (1) to produce a summary table which starts like this:
<row role='label'><cell>Element</cell><cell>lodel</cell><cell>tei</cell><cell>sc
<!-- 2 --><row><cell><ref target='
/html/ref-TEI.html'>TEI</ref></cell> <cell>change</cell> <cell>use</cell><cell>2
<!-- 2 --><row><cell><ref target='
/html/ref-ab.html'>ab</ref></cell> <cell>use</cell> <cell>use</cell><cell>2</cel
<!-- 2 --><row><cell><ref target='
/html/ref-abbr.html'>abbr</ref></cell> <cell>use</cell> <cell>use</cell><cell>2<

OK, work is needed in this area. But it’s a start.




TEI++ : une formation avancee

Cet été, j’étais invité par le consortium CAHIER, en partenariat avec les consortium Corpus Ecrits et IRCOM,  d’organiser un atelier dite « TEI avancée » sur quatre journées.  Je leur ai propose une mode d’opération divisée en deux parties (présentation, travaux pratiques) et une organisation selon trois axes:

  1. modélisation des ressources et séléction des traits signifiants
  2. encodage et explicitation TEI des structures modelisés
  3. exploitation et analyse des ressources structurés

Je leur avais aussi propose de partager les  travaux de formation avec quelques experts francais.  La formation s’est tenue à l’Institut de Linguistique Française à Paris du 19 au 22 novembre 2012.

Voici en sommaire (anglophone, désolé) ce qui s’est enfin passé …

Day 1

Proceedings began on the fifth floor of the ILF, a nice light room but not quite big enough for the 18 or so participants. This was not altogether bad, since the consequent huddling together encouraged emergence of a cohesive group identity, which I also tried to encourage by getting the participants to place themselves on an improvised graph along two intersecting axes : literarature vs linguistics, and researchers vs support staff. Most people turned out to be in the bottom right quadrant, i.e. ingénieur + linguistique, but there was also a smattering in the littéraire + scientifique box, to say nothing of two sociologues who insisted on positioning themselves in the middle of the littéraire vs linguistique axis. The rest of this first introductory session was given over to a rapid review of some fundamentals of encoding, and a sampling of the websites of half a dozen real TEI projects around the world, which might have gone better had I rehearsed it better, but got the message across that quite a few very different projects were doing seriously cool stuff with the TEI.

After coffee, I introduced them very quickly to a spot of data analysis, using as vehicle the celebrated postcard archive of M Marcel Virgolos, and Lauranne then took over for a refresher on using Oxygen. They marked up a postcard or two, and reviewed commonly used TEI tags for each of some pre-selected texts : a French novel, a poem, and a play. Most students completed all of these, mastering most of the key features of the Oxygen XML Editor quite rapidly, but I think we did not allow enough time for this session, given the mix of abilities present.

Lunch in the form of a large cardboard box containing a “plateau” of cold cuts, salads, bread roll, plastic cutlery etc. duly appeared and was despatched. Thus strengthened, I embarked on an all-singing all-dancing overview of all the TEI modules, and what you can do with some of them. That took about an hour, but helped motivate Lauranne’s traditional exercise on using Roma to make a schema by reduction from TEI-ALL which followed it. By the end of the day, everyone seemed quite comfortable with the idea of pérsonalisation de schéma, and reasonably convinced that they might find what they wanted to mark-up somewhere, somehow in the TEI.

Day 2

On this and subsequent days we were were displaced to a much better (because bigger) teaching room. I began the day in seriously magisterial mode by explaining (many of) the components of the ODD language, and why you might care to know about it. This was quite punishing, both for me and for some of the less technically-minded participants, but no-one visibly fell asleep. For the subsequent practical, Lauranne had prepared a script in which participants submitted their own texts to analysis by OddByExample, generating a personalised ODD. The majority of course had not come with their own texts, or had texts not in TEI P5, so they ran the exercise on a rather inadequate sample of the Virgolos corpus instead. With a bit more prep, I think this could be a really fun exercise and an excellent way of getting people to learn ODD properly. It also revealed a bug in OddByExample which Sebastian graciously fixed overnight

Lunch being cardboard plateaux once more, I went for a stroll round the nearby Parc de Montsouris to see some of the gray autumnnal paris daylight while it was available. I then droned on for well over an hour on the subject of the TEI Header, which I am now selling as being “metadata for the rest of us”. They made me do it. The exercise we adopted was the French version of creating an ms description for the W. Owen ms, last seen in Berne, This is quite useful for the purpose, especially in combination with the following exercise on marking up a transcription of that manuscript ; time permitting however I would have instead preferred to use a different French manuscript for both. If I had one.

Day 3

Wednesday I had carefully billed as “journee des guest stars” since the idea was to make other celebrated French TEI enthusiasts share in the work. So we began with a presentation about TEI recommendations for dealing with named entities and their names given by Alexandre Gefen. Since the room contained more than a few French linguists this gave rise immediately to a heated debate on the philosophical basis and nature of nominal reference. The exercise in marking up names was a little under-prepared and one of the students insisted on asking the Emperor’s New Clothes question (why bother?), which was answered by another participant citing (at length, and with enthusiasm) the work of Nicole Dufournaud inter alia, which was nice. Since Alexandre had to leave early, I filled the gap by moving my brief overview of tools options up, giving a plug for Sebastian’s stylesheets, and letting them experiment with OxGarage, which they loved.

For lunch we went to the brasserie down the road, which was a much much better idea than the plateaux. Everyone got very jolly and there was a fair amount of shouting. Our second invited expert of the day was Bertrand Gaiffe, from ATILF, who delivered an excellent pair of lectures about encoding of oral and linguistic data respectively, also involving a fair amount of interaction and discusion, but not much actual tagging, since TEI interfaces for the appropriate tools remain elusive, best efforts of the LingSig notwithstanding. .

Day 4

The final day began with a presentation on the various TEI orthodoxies concerned with the editing of primary resources given by our third invited expert : Alexei Lavrentev from ICAR. Participants were then offered the choice of doing either the reverse transcription exercise or the visual encoding exercise from Berne; both options were taken up, though I was too busy sorting out the website to see how far they got with either.

After another nice brasserie lunch (roast duck), I spent about 15 minutes showing how to use TEIBoilerplate, which went down remarkably well, “Génial” they cried, as they saw all that tricky encoding in John’s demo file being rendered beautifully by Safari (France is still largely land of the Mac). The rest of the afternoon, was devoted to a more ambitious TEI-savvy piece of software : txm, from the textometrie project at Lyon. Alexei showed us what it was, and demonstrated how to make it sit up and do tricks with the Graal and Brown corpora, which participants had pre-installed. He also showed it working with a selection of literary texts prepared for use throughout the workshop.


I think this workshop worked much better than it deserved to (I always think that). All the participants seemed very happy at the end, and several of them said they had learned more than they expected to. I think the organisation of the programme made good sense, and the balance of exposition and exercise was approximately right, though we probably didn’t do enough to make the practicals consistent and relevant. A few parts suffered from lack of preparation, and I think we could have done more to get a single case study working throughout the course of the four days, in addition to the various more specialised materials we introduced. But next time we’ll definitely get everything right. Thanks are due to the participants, the organisers, and all my co-formateurs, especially Lauranne for calming me down at moments of high anxiety.
All the materials used in the workshop are available in PDF starting from Dedicated TEI hackers may also be interested in the XML sources of the presentations which are available from my svn repository at

Digital Palaeography meets Optical Glyph Recognition in Rouen

HDDA2012 (“Historical Documents in the Digital Age”)  at the University of Rouen turned out to be unusual (for me at least) in a number of respects. Firstly it was organised as part of a project (“DocExplore”)  funded under the Interreg framework of the EU, and hence attended by people from both sides of the channel, rather than being exclusively French. As a consequence the presentations were in both English and French, with apparently quite successful simultaneous translation, though I did not test this for more than a a few minutes. Secondly, I didn’t have to explain to anyone what the TEI was, and why it might be interesting; everyone seemed to know all about that already, even the informaticiens. And thirdly, there was no-one else from Adonis present, so it fell to me to ask the man from the Archives Nationales why they did not provide an OAI feed into Isidore as well as into Europeana (they’re planning to).

There were about eighty attendees, most of whom survived the full day and a half of invited presentations/round tables. There was a bit of audience interaction, but not much, and surprisingly perhaps only a couple of desultory tweeters, one of which doesn’t count since it was me. There was however plenty of time for old-fashioned face to face discussion over lengthy pauses for sustenance in Rouen’s rather nice Maison de l’Universite. As far as I can tell there were roughly equal numbers of archivistes and informaticiens, but they did not mix a great deal.

Proceedings were kicked off with two very good “state of the art” summaries of what’s going on in the way of cultural heritage digitization in France by J F Moufflet from the Archives de France, and Matthieu Bonicel from the BNF. I particularly liked the latter because of his optimism about using the technology to break down the walls between the silos of digital artefacts being created everywhere, pointing to evidence from maybe half a dozen great projects previously unknown to me. Both of these speakers pushed all the right buttons about open public access and accountability, transparency and integration of resources, respect for standards etc. thus making quite a contrast with the following speaker, from the archive of Canterbury Cathedral, who found herself having to explain why they’d made a deal with Satan in the form of to get their parish records database online, thus perhaps revealing the very different business models in which archivists operate on either side of the channel.

The second session was given over to tools for transcribing and indexing all those lovely digital images. Stephane Nicolas from LITIS, the Rouen team responsible for software development, laid out clearly the challenges and advantages of integrating transcription and images. Two rather more technical presentations followed: one from Franck Lebourgeois which felt a bit like a graduate seminar about the mathematical basis of OCR, and another from Marcal Rusinol from a Spanish lab about vision processing techniques for word recognition or (as it seems it is called in the trade) “word spotting”.

The last session of the day was billed as being about digital paleography proper, and was divided appropriately between two contributions from palaeographers (Elizabeth Lalou from Rouen, and Marc Smith from the Ecole Nationale des Chartes) and two computer engineers (Veronique Eglin from LIRIS and Richard Guest from Kent). The former group clearly understood the potential the technology offered to address some long standing difficulties in the treatment of e.g. allographic variation or the use of frequency statistics in the definition of “writing style” ; the latter group maybe had a harder job in making explicit just what the state of those particular arts currently is.

The second day I arrived a bit late, for some rather odd discussions, again revealing extraordinary differences in attitude on either side of the Channel, about the “ludique” use of IT in cultural heritage applications, i.e. how to make cool exhibits in museums. It began with a moderately dreadful intervention from a professional French developer of such things, but was rescued by a man from the British Library called Clive Izard who gave a historical survey of the BL’s flirtations with technology, from the days of the Information Access programme (which, I may say, was one of the funders of the BNC) up to the present, third, generation of the “Turning the Pages” application. He was followed by another excellent (and splendidly named) speaker Clotilde Vaissaire-Agard from Le Havre, who reminded us about the need to place the scholar at the centre of the picture (I was reminded of a former OUC S colleague’s plaintive cries of “What about the users”?) . She also endeared herself to me forever by citing the manuscriptorium project (remember Enrich?) as an outstanding example of what the technology facilitated by making it possible to share metadata and digital resources across institutional boundaries for the benefit of manuscript scholarship.

The final session though labelled as con cerning that old war horse “Is there such a thing as Digital Humanities”, actually contained three very good and complementary talks intimately concerned with the themes of the conference. Alison Wiggins from Glasgow’s Bess of Hardwick project gave a convincing account of their attempt to ground the project in practical user-focussed concerns (she cited Claire Warwick et al’s Lairah as one of their inspirations); Dominique Stutzmann from IRHT raged, with ample evidence, against the lack of decent interfaces in transcription software; and finally Alixe Bovey from Kent gave a well illustrated overview of the strengths and limitations of various interfaces developed for interacting with the physicality of medieval sources. She concluded by lamenting, in the way people do, the absence of smell associated with digital images, and the mismatch between the haptics of the touchscreen and the codex. I was more impressed by Alison’s comment that it was more useful to know what kind of paper Bess of Hardwick wrote to the Queen on than it was to be able to reproduce it.

The price of a new laptop, or My love affair with Ubuntu

I think the first unix system I installed on a laptop was release 0.8 or thereabouts of Knoppix. You could take an innocuous -looking CD, stick it into a crusty old PC, tweak its bios to boot from CD, and bingo, you were running a real linux without having touched the Windows hard disk itself. How cool is that? OK, it took the best part of a day to load Knoppix from the CD into memory, so I pretty soon found the button for copying Linux itself onto the hard disk of my elderly laptop, but the “Live CD” concept had much to recommend it. You didn’t have to drop the Windows security blanket, and you could go on using your Windows filestore. That must have been around the year 2001 or so : at OUCS we started using Knoppix as the basis for a series of TEI give-aways at workshops and conferences – at first on CD, then on USB sticks, as the technology improved.

And then came Ubuntu. I think my first real Linux laptop was a Thinkpad on which I installed the first of an entire menagerie, from Warthog in 2004, to the Quetzal I installed yesterday. Yesterday also, I switched allegiance from Lenovo to Samsung, and installed the Quetzal on a series 9. It looks a bit like a Mac Airbook, but it’s not evil.

To get a new laptop in 2004, I had to write a one page justification for the boss, fill in numerous forms, send them off to the University’s approved supplier, and wait a few weeks for the machine to arrive. Then it would take a few minutes to unpack the machine, and at least 3 days to get Linux installed on it, much of it involving me pestering smarter people with better things to do.

In 2012, it took me just a few minutes to click on a button and order a new laptop, which was delivered to my house in about 24 hours. It took rather more than a few minutes to get it out of the packaging, but installing Linux took a couple of hours max. I downloaded an ISO image from the Ubuntu website. I made it into a bootable USB key using some software recommended by some other website. I stuck the key into the side of my new laptop. I tweaked its BIOS (just like the old days) to boot from the USB drive. And everything Just Worked. OK, I had to make difficult decisions like what language I use, what is my name, what graphic did I want to represent me, and did I want to wipe out the Windows 78 partition on this machine, so maybe a little more than an hour or two later, and that’s it. Wifi works, sound and graphics work, wireless mouse works (good, as my fingers don’t understand trackpads) … I can even imagine getting used to the Unity interface (which seems to be a bit more stable than it was under Pangolin).

And that, dear reader, is when I realise that the real price of a new laptop is yet to be paid. OK, I expect to have to install some favourite bits of software to get my familiar working environment back (digikam, subversion, emacs, oxygen, chrome, dropbox …) : that doesn’t take long. I can remember how to re-configure thunderbird to collect my mail: what I’d forgotten is just how long it takes to get nice new fresh copies of all the old mail which was sitting gathering dust on the IMAP server. I know how to check out all the stuff that actually matters from the Sourceforge and Googlecode TEI repositories : don’t underestimate how long that takes either. It took me over an hour just to remember how to re-set my password on the OUCS subversion repository.

My goal for today was to be able to rebuild TEI P5 from source, and crank out nice PDF slides from TEI source. (that’s the sort of thing I do every day, to be honest). On top of what I already had, I needed to download and install Oxygen XML editor, Chrome, and Dropbox. I needed to install nice new packaged 64-bit versions of jing, trang, onvdl, rnv, tei-emacs, openjdk-jdk, latex-beamer, texlive-xetex, and texlive. That all went smoothly, except that the packaged version of rnv was a 32 bit one, and so I had to rebuild it from source. Same problem with Acrobat Reader, but there is, of course, no option to rebuild that from source, so I will probably have to live without Mr Adobe’s fine products for a while.

Conclusion? Nothing surprising I suppose: installing your own system is still a good feeling, seductive enough for enough people that other people put lots of effort into making it much simpler to achieve. Hats off to the unsung labourers in the Canonical salt mines, and the open source community generally, who just go on doing what they have always promised they would, keeping the faith.  It’s a relief that manufacturers like Samsung let them get away with it.

And now, it is the evening of my first full day with psammead and I think another glass of wine is in order. If I could only learn how to use this wretched trackpad…

Teaching TEI in Bern

At the invitation of Bénédicte Vauthier from the University of Bern (whose indefatigable enthusiasm, fund raising expertise, and hard work organizing the event, are all hereby gratefully acknowledged) I spent the last week teaching the French strand of a four and a half day long trilingual TEI training session, aimed primarily at people working on the encoding of modern primary manuscript sources, in collaboration with Christof Schoech and Alejandro Bia, who provided the German strand, and Elena Pierazzo who provided the Italian one. We did try a bit to co-ordinate our efforts so that the material covered in each strand was the same, though our methods of teaching, and some of the materials used naturally varied. The course also included a full day of invited presentations about a wide range of Swiss projects and a series of invited plenary presentations. The first day was hosted by the Swiss National Archive who showed us some of their collection, which includes a splendid exhibit of authors’ typewriters. More details of the whole event are, or will be, available elsewhere; there are also a few photos here. But mostly this blog entry just reports on the French teaching strand, and may be of most interest to others undertaking similar quixotic enterprises.

Each of the three strands contained five sessions, each held in a different teaching room (the French one equipped with PCs, the German and Italian ones with Macs) and combining a 45-60 minute presentation with a 60-90 minute scripted hands-on exercise. The programme, also common to all three strands, was as follow:

  1. Introduction to basic ideas of encoding, distinction between document and text, refresher on XML. In the practical session, students were shown how to use oXygen to create a simple document from scratch, using just the TEI- bare schema.
  2. Introduction to TEI, focussing on how to tag basic components of a text, brief overview of some commonly encountered TEI elements, using French examples. Practical session using  TEI Lite as schema. In the first part, students used oXygen to add tags to a plain text version of a sonnet by duBellay; in the second, they used oxGarage to convert a Word file containing a scene from Jarry’s Ubu Roi into TEI, and then used oXygen to improve its tagging.
  3. The TEI Landscape with a slide or two on each of the 23 chapters in the Guidelines, followed by a quick guide to using the TEI website as a means of exploring them. In the practical session, students first use oXygen to create a tei_all file, and then use Roma to create a schema which is better constrained to the encoding of modern mss.
  4. TEI encoding of editorial interventions and simple manuscript transcription. In the practical; students reverse-engineer a rather heavily edited version of the start of a Wilfred Owen manuscript into a more diplomatic TEI transcription in oXygen, using the schema they made in the previous tp (which is also used for the rest of the course).
  5. TEI Header, focussing on its use as a repository for metadata of importance to librarians and archivists, but also to scholars and end-users. In the practical session following students create a full msDesc for a manuscript letter, using oXygen to tag the various parts of a plain text catalogue entry with which they are supplied.
  6. TEI documentary editing, starting with facsimiles and zones, and moving on to include sourceDoc, line, mod, metamark, and other recent additions to the TEI family, notably change and @change. In the practical session, students used Inkscape to identify various zones within a page from a Durenmatt ms, transferred these to a basic TEI transcription of the same file, and then ran an XSLT conversion to combine these into an SVG file in which the transcript and the graphic were merged in such a way that a piece of javascript could display the changes documented in the TEI independently

Archives of all the materials I used for these sessions are available for download in PDF form (Exercices; Talks).  The TEI XML source is also available, under the usual CC-BY licence, from the MEET subversion repository.

In the final session, students were invited to apply their experience to their own materials, samples of which they’d been instructed to bring with them. Most of them set to immediately and started tagging, which was quite encouraging. I was able to do some hand-holding with the one or two students who had somehow failed to understand what was going on earlier, and the others all started banging away in earnest.

Some further comments on the sources I used for each of these talks, and some things I found it necessary to change follow….

  1. The pattern and content for Session 1 is pretty much traditional now. Even so, I managed to find some typos to correct in the exercise script — the keyboard shortcuts for the Windows, Mac, and Linux versions of oXygen are all subtly different, and my script is indecisive as to which it is using. I noted, as always, that it’s hard for beginners to understand that « I want a new line here » really means « I want to close this paragraph and start a new one ».
  2. I put the talk for session 2 together using some old talks introducing TEI Lite. I included a bit of the Punch exercise for old times’ sake, but mostly it derives from a very old talk introducing TEI Lite, spiced up with French examples taken from those in the Guidelines. Some of these are not tagged in quite the way I would have expected, or don’t demonstrate things as clearly as they might, so the whole lot probably needs to be reviewed for consistency. The first part of the exercise for this session went very well, with everyone suitably impressed by the effect of tweaking the CSS to render the document in « Mode Auteur ». The second part was less successful (though not everyone got to it through lack of time) largely because of improvements to oxGarage since I last revised the script; however, I remain convinced that this is the right place to introduce the possibility of starting from a Word document.
  3. The « TEI from chap 1 to chap 23 » talk in session 3 is quite a stretch to do in 40 minutes, especially if one is easily distracted, but remains useful since it motivates the following Roma practical. It’s essential to get across the notion of TEI modularity and this helps students see the point immediately. I based the tutorial on the  script used at this year’s Oxford Summer School, which I translated more or less faithfully, just removing some of James’s jocularities.
  4. The two talks on textual editing (sessions 4 and 6) with TEI were both derived from one I gave in 2011. The initial part, about encoding critical apparatus, seemed a bit redundant (especially for an audience including e.g. Jean-Louis Lebrave)  since we didn’t proceed to use the tags it introduces, and also because I spent too long on it; though it does provide some useful background. In the practical session, both Christof and I decided to use the Wilfred Owen examples used at this year’s Oxford Summer School, though with some  revisions. In the first exercise on transcription,  rather than making sense of an arbitrary non-XML transcription, students had to work out for themselves what was going on in the manuscript image, using the edited transcript provided purely for reference. They found this initially difficult, but once they’d realised that simply cutting and pasting would not be good enough, they became quite enthusiastic. During this exercise I also learned the interesting linguistic fact that the word raturer can be used for any kind of deletion, whereas the word biffer means specifically a deletion carried out by striking through something with a single line. This means that if I revise this exercise again (as I probably will) I shall be providing @rend values in French. (The word « stroked » used throughout the Oxford material seems to me just wrong, by the way. We’re not in the business of cuddling our texts. Au contraire).
  5. Session five’s talk on the TEI Header was more or less unchanged (except for a few typos) from the version I gave earlier this year, itself translated from plenty of other older versions. I think that providing the context for a <msDesc> is useful; the talk also provides an opportunity to talk about named entities, a topic which was otherwise missing in our programme. The practical was again based on one from this year’s Oxford Summer School, though again with some modification — notably that the manuscript described surely contains two <msItem>s, not one.
  6. For session 6, I used the second part of the textual editing talk more or less unchanged: the underlying concepts when defining zones seemed to go well, particularly once I’d admitted to being nul en math. The practical session was an entirely new one, made by translating Christof’s German version, which he produced on the basis of some notes from Elena, describing how she’d presented her Proust prototype at a conference in Australia earlier this year. In the event, one of the nicest moments of the entire course was seeing one of my students punching the air in excitement after having successfully opened the SVG output from this exercise in Internet Explorer. There was a small hiccup when I realised that the version of Inkscape we were using was in German, not English which made configuring it correctly rather tricky, but otherwise this all went far more satisfactorily than I expected. Which was, as they say, nice.