Multiple authorship

As noted previously Nicoll’s Handlists are organized by author name, which makes them manageable, but also can be seriously misleading. In particular, where a play is to be credited to more than one author, Nicoll’s practice is to repeat the information about the play in a second slightly degenerate entry, thus inflating the number of entries in the Handlist. Here for example is the “main” entry for a play co-authored by A’Beckett and Lemon:

<entry type="Bsq.">
<author>A'BECKETT, GILBERT ABBOTT</author>
<title>The Knight and the Sprite </title>
<note type="perf">Strand, M. 11/11/1844</note>
<note>L.C. 9/11/1844.</note>
<note type="auth">[Written in collaboration with M. LEMON]</note>

And here is what I have unkindly termed the “degenerate” entry for same:

<entry type="Bsq.">
<author>LEMON, MARK</author>
<title>The Knight and the Sprite </title>
<note type="perf">Strand, M. 11/11/1844</note>
<note>See G. A. A BECKETT.</note>

I assume Nicoll’s rationale for this redundancy is to make it easier to find everything written by a given author when flipping through the pages of a printed volume. But this makes much less sense in a digital resource. What we would rather see (I think) is an entry which makes explicit its multiple authorship: like this

<entry type="multi">
<author>A'BECKETT, GILBERT ABBOTT</author>
<author type="also">LEMON, MARK</author>
<title>The Knight and the Sprite </title>
<note type="perf">Strand, M. 11/11/1844</note>
<note>L.C. 9/11/1844.</note>
<note type="auth">[Written in collaboration with M. LEMON]</note>

(Note that to get there I have had to rethink the way I encode Nicoll’s genre tags, initially by moving them to an element of their own rather than using the @type attribute of the <entry> element. And note also that I am preserving those arguably redundant <note type=”auth”> elements so I can tell if something goes wrong)

I have spent the last week or two slowly making this possible. Slowly because I am slow, but also because it is not entirely straightforward to translate the string “M. LEMON” (as given in the note in the main entry for A’Beckett) into “LEMON, MARK”, which is the handle used on other main entries for the distinguished editor of Punch. (The same would apply, of course, if I decided to use the note within the degenerate entry to effect the join: I would then have to map “G.A. A. BECKETT” to “A’BECKETT, GILBERT ABBOTT.” ) And these are easy cases: Nicoll’s canonical format for names can get quite complicated. Consider, for example, “YORKE, ELIZABETH, Countess of HARDWICKE” or “ADDISON, Captain (later Lieutenant-Colonel) HENRY ROBERT” … Anyway, I made the job easier for myself by extracting from the entries a lookup table mapping name components (as given by notes within main entries) to canonical full names: like this

<author f="49">
<str>LEMON, MARK</str>

This all worked quite satisfactorily for the 1800-1850 entries, for which there are only 58 additional name entries to handle, though getting to the point of being reasonably confident in that number took much longer than you might think, involving as it did quite a lot of OCR error correction.

However, things got much more challenging when I looked into the 1850-1900 entries. Firstly, there are many more entries to deal with: 1299 cases of “collaboration” . Secondly, some cases (34 to be exact) use an abbreviated form like this:

<entry type="P.">
<author>BYAM, MARTIN </author>
<title>The Babes in the Wood </title>
<note type="perf">R.A. Woolwich, 14/12/57.</note> L.C.
<note type="auth">[Written in collaboration with F. GRAHAM and W. T. VINCENT.]</note>

This main entry will need to get two additional author elements, one for “F.GRAHAM” and one for “W.T. VINCENT”, not just one – which means revising my simple-minded XSLT script yet again. And it will also have to handle notes like this without too much fuss:

<note type="auth">[Written in collaboration with A. R. SMITH, F. TALFOURD and W. P. HALE.]</note>

The script does a good job of alerting me to cases where Allardyce has apparently nodded, and named as a collaborator someone who does not appear anywhere in the rest of the Handlist. This happens precisely once in the 1800-1850 volume, but seemingly many times more in the later volume. However, on examination, many of these discrepancies are a consequence of my cavalier editing praxis. Things like the kinds of quotation marks used to flag up pseudonyms, or whether or not surnames can contain spaces, return to bite me. Others are caused by OCR failures – occasionally lines seem to have just dropped out.

And, further to keep me on my toes, I have now discovered that there are three cases in which Nicoll gives up entirely on this painstaking method of documenting multiple authorship. The first concerns 18 titles to be attributed to the pseudonymous “Richard Henry”: these all appear once only under “HENRY, RICHARD”, like this

<entry type="Bsq.">
<title>Lancelot the Lovely; or, The Idol of the King </title>
<note type="perf">(Aven. 22/4/89).</note> L.C.
<note type="music">[Music by J. Crook.]</note>

None of these 18 titles is listed, however, under NEWTON, nor indeed under BUTLER. A further, and apparently disjoint, batch of titles is listed under “NEWTON, H. CHANCE (“RICHARD HENRY”), I think I am going to pretend I haven’t noticed them. Likewise this one:

<entry type="D.Sk.">
<author>GORDON-CLIFFORD, E. and H. </author>
<title>A Black Dove </title>
<note type="perf">P’s. H. Kew, 12/9/94.</note>

Handlists made handier

I have been down a deep deep rabbit hole for the last week or two trying to get my XML-tagged versions of Allardyce Nicol’s two Handlists into shape. Here is an interim report.

What is an entry?

One problem has to do with the actual content of Nicoll’s Handlists. What exactly do they list? Although this is essentially a record of performances, it is organized very much by author. There are, for example, about 30 entries which don’t refer to any specific play, but are merely there to indicate the preferred form of an author’s name: like this one

<entry>“LAWRENCE, SLINGSBY.” See G. H. LEWES</entry>

Multiple authorship is also a problem. An entry like this one is straightforward enough:

<entry type="D."><author>ANDERSON, JAMES R. </author><title>The Robbers </title><note type="perf" >D.L. 21/4/51</note>. L.C. D.L. 26/12/45.</entry>

Inter alia, this tells us that there was a performance of a drama called “The Robbers” at Drury Lane theatre on 21 April 1851, and that the author of the piece is recorded to be James R. Anderson.

But there are also entries like this one:

<entry type="F."><author>ATWELL, E. </author><title>A Stuffed Dog </title><note type="perf">Park. H. Camden Town, 2/11/89</note>. See J. A. KNOX.</entry>

This one tells us that “A Stuffed Dog”, written by E. Atwell, was making them roar at the Park Theatre in Camden Town in November 1889. If we look for Mr J. A. Knox, we find another entry, apparently for the same performance:

<entry type="F."><author>KNOX, J. ARMORY </author><title>A Stuffed Dog </title><note type="perf" >Park H. Camden Town, 2/11/89, copy.</note>. L.C. [Written in collaboration with E. ATWELL .]</entry>

On the face of it, if I want to determine how many Farces are listed in the Handlists, for example to determine the waxing or waning of this particular type of performance over time, I need to be wary of cases like this one, where a single farce has multiple authors, and therefore gives rise to multiple entries: both these entries refer to a single performance, so should only be counted once.

How serious a problem is this? Out of 25 thousand- plus entries, (25,632 to be exact) I find that there are 1346 entries containing the word “collaboration” and 1738 containing the word “See ”. Most, but not all, of them point to a collaboration entry which references the same performance of the same play. There are only two entries in which the word “See ” really appears as part of a title, but there are maybe a dozen or more other types of cross references, for example to plays renamed or whose authorship Nicoll has resolved. The number of cross references which go nowhere or to an entry which documents a different title or performance is unknown, but not zero. I spent some time trying to check automatically but did not finish: other bits of the rabbit hole (like checking and fixing OCR errors in the dates) seemed more useful.

Although ostensibly organized by author, about half the entries in the Handlists record performances for which there is no author. (10,374 out of 25,662 entries to be precise). Usually, but not always, distinct entries are given for the same title performed on different occasions or at different venues – but occasionally an entry will provide a list of performances: like this

<entry type="C.O."><author>MANCHESTER, G. </author><title> The School Girl </title><note type="perf">Grand, Cardiff, 2/9/95; Stand. 14/10/95</note>. L.C. [Music by A. Maurice.]</entry>

I have not checked, but I suspect that this happens when a play has its first performance in the provinces: in this example, we may conjecture that “The School Girl” went down well enough in Cardiff for the management to risk bringing it to the Standard in Shoreditch a month later.

What sort of play is this?

Nicoll thoughtfully provides lists of the abbreviated codes he uses to indicate “the nature of the play itself” for both the Handlist 1800-1850 and the Handlist 1850-1900. There are a few codes present in only one or other of the two lists (B.O. for Ballad Opera is in the earlier list only, for example). An investigation of his usage of these codes over the two hand lists indicates the justice of the warning he also provides that “these designations are in no way final, and are often indefinite”. The two lists propose a total of 87 different codes including some very general categories (D for drama, F for Farce, P for Pantomime etc.) as well as many more nuanced classifications such as “Military Drama”, “Operatic Drama”, “Poetic Drama” “Romantic Comedy”, “Romantic Comedy Drama”, and “Romantic Drama”. Nicoll further remarks “Where possible, the designation employed in the original bills has here been followed” – so we should take these characterisations as indicative of the language in which Victorian Theatre chose to describe itself, not as a formally organized taxonomy.

I did some counting up of the actual usage of these codes, and found a further 23 codes not specified in either list. However, most of these are used very infrequently : less than 10 times for all except two of them. The exceptions are the emdash which is used for 15 entries that describe non performance items such as published collections of plays and the code “Bsq. O.” which is used for 22 entries, all of them presumably “burlesque operas”.

For what it’s worth, here’s a colourful camembert to show the distributional statistics of these categorisations. Of the 110 different codes used, more than half (64) are used fewer than 10 times. Or, to put it another way, the top six codes between them account for 17,242 out of the total nunber of 25,632 entries – over 67%.; the top eight codes (labelled in the picture) account for more than three-quarters of the whole population.

My next project will be to see if these proportions change significantly over time.

Categorising Lacy

 Collecting the data

Allardyce Nicoll’s monumental “History of English Drama 1660-1900” (Cambridge University Press, 1955) has two volumes containing “hand lists” of theatrical titles produced in the first and second halves of the 19th century respectively. The list in Volume 4, covering 1800-1850, can be downloaded from CUP’s site in PDF format if you have an appropriate institutional login; it even has a DOI:

I downloaded and worked on the PDF version (handlist18001850.pdf) with the following workflow:

  • Generate DOCX version using Abby (thanks, HumaNum) (handlist18001850_hnOCR.docx)
  • Process this with `docxtoTEI` (thanks TEI) (handlist18001850_hnOCR.xml); some hand edits too to get rid of unnecessary clutter such as forme work and simplify the subsequent processing
  • Process this with `addMilestones.xsl` to delimit entries and author sections
  • Process output with `writeList.xsl` to produce structured list of all titles (entries1800-1850.xml)
  • Process with `addKey.xsl` to select entries mentioning Lacy and add a magic key derived from the title (LacyEntries1.xml)
  • Process this with `checkEntryList.xsl` to determine whether there is a match for this magic key in the current Lacy Catalog file, and add an attribute @matched indicating the result (LacyEntriesChecked1.xml)

The text of Volume 5 covering titles from 1850 to 1900, is not available in digital form from CUP for some unknown reason. I did however discover in the Internet Archive a less than perfect scan of it, provided by the Digital Library of India ( I used a grubby perl script (``) to process the fairly unreliable OCR plain text version of this to produce output compatible with that produced by `addMilestones.xsl`, and fed this into the toolchain listed above to produce LacyEntries2.xml and LacyEntriesChecked2.xml


– entries1800-1850.xml has 8495 titles of which 456 (53.6%) mention Lacy; of these, 307 could be matched, and 149 not.

– entries1850-1900.xml has 17202 titles of which 754 (43.8%) mention Lacy, of these, 454 could be matched and 300 not.

Failing a bit better

I next tried a different way of checking for matches, viz. the ever reliable unix utility `comm` applied to a pair of text files, one containing all the values for title/@n in the LacyCatalog and the other containing all the values for entry/@n in the tweaked Nicoll entry lists. First time round, unsurprisingly this gave different results: 537 matches in all, with 652 values unique to the Nicoll-derived list, and 961 unique to the Lacy catalogue.

I noticed however that both files had some duplicate entries, which were causing confusion in the matching process, though unsurprisingly, the most frequent cause of disagreement seemed to be OCR error or inconsistent formatting.

I manually added suffixes to handle the discrepancies I noticed in @n values. The author’s surname was used to distinguish identically titled but different works; the suffix “-bis” for actual repetitions.

Removing the duplicate entries and some of the more egregious OCR errors gave more plausible values : 760 matches in all, with 429 values unique to the Nicoll-derived list, and 738 unique to the Lacy catalogue. This was further improved by checking the 23 titles in Nicoll’s list which indicated that the Lacy publication used a different title. The output from comm after this modification gave me 772 matched titles, with 410 unique to Nicoll, and 726 unique to Lucy.
saxon ~/Public/Lacy/newcatalog.xml listKeys.xsl > lacyKeys.txt
saxon lacyEntries.xml listEntryKeys.xsl > nicollKeys.txt
comm --total lacyKeys.txt nicollKeys.txt

This exercise also revealed that there are a couple of genuine double entries in the Lacy Catalogue. Henry Byron’s “Bluebeard” extravaganza appears both in volume 19.3 (L0273) and in volume 49.9 (L0729); likewise Selby’s “Witch of Windermere” appears both in volume 84.4 (L1248) and in volume 3.6 (L0036).

After more tweaking, I decided to declare victory with 753 matched entries, and 436 not matching. Nicoll’s catalog includes entries for Lacy titles from volumes after 100, which I do not include in this project, so I expected some matches to fail.

Categorizing the titles

One of the reasons for being interested in Nicoll’s Handlists is that they assign each title a code such as ‘D’ for drama. How useful are these codes as a way of categorizing the contents of LAE?

Nicoll’s category codes are at once very delicate and very vague. Clearly, they are derived from the way the piece describes itself on its title page (if it says it’s an “extravaganza” then that’s what it is), but at the same time these often ornate descriptions have clearly been rationalised to make up a smaller number of unique category labels than simply using the words of the title would. But only to a degree: “burletta” and “burlesque” are distinguished, with roughly equal numbers of each, but there are plenty of titles which describe themselves as “a burlesque burletta”. In an attempt to further simplify these descriptions, I decided to map Nicoll’s codes to just four main classes: Comedy, Drama, Musical, and Spectacle — even though items frequently cross these very broad categories: should “comic drama” go under “comedy” or “drama”, for example? “Musical” is particularly problematic since many comedies include songs, as does almost anything classed as a “Spectacle”: my intention was to limit it to pieces clearly operatic or at least operetta-ic .

Undeterred, I tried my own experiment in classification, based on the words appearing on the titlepages. I used the following regexp to collect categorizing strings :


and concatenated all the matching strings for each title. Prefixed with the number of acts in the play, this became the value for a @type attribute on each catalogue entry. For example, a one act play whose subtitle contained the phrase “A Burlesque Burletta” would be given the category label “1_BurleBurle”. A script `listTypes.xsl` counted up unique category labels (disregarding the number of acts) producing a list that begins:
Burle (40)
BurleBurle (3)
BurleBurleOpera (1)
BurleDrama (7)
BurleExtravagan (21)
BurleExtravaganComicPantomime (1)

(Note that the counts are for the whole string of categorizers: the count for “Burle” does not include the count for “BurleBurle”)

In many cases, it’s easy to map these category labels to the four basic categories identified above: all of the above would count as “Spectacle” — except perhaps “BurleDrama” and “BurleBurleOpera” (the latter however proves to be a mistake in segmentation: the “opera” part of the characterization concerns the source, not the play itself.)

There are 57 titles which lack any of these categorizing substrings, many of them preferring humorous variants on traditional title page discourse, such as “An Original Irish Stew” or “A new and original, aerial, floreal and conchological fairy tale (Of which the most striking feature is borrowed from the Countess D’Aulnois)“. These I (regretfully) left to one side for the moment.

My first (rough) simplification produced the following figures for all 1500 titles in Lacy’s Acting Edition:

Categorizations by subtitle, for all 1500 titles

These relative proportions for these gross categories seem to be much the same as those derived from the more finely-grained Nicoll analysis of half the number of titles:

Categorizations by Nicoll, for Lacy titles only

The story looks rather different however when we count up the categories for all 15,021 categorized entries in the two Nicoll handlists, not just those for Lacy titles:

Categorizations by Nicoll, all titles

It has to be said that these counts are all fairly unreliable — aside from encoding problems caused by flakey OCR and my post-processing, Nicoll’s records count performances of the same title as different items where a title is of unknown authorship: this has the effect (probably) of inflating the counts for some categories such as Pantomimes. And of course my crude four part classification really needs much more thought. But putting all that to one side, it does seem that in selecting titles for his Acting Edition, Lacy preferred comedy over drama.

ELTeCTiT : ELTeC Titles in Translation

(I haven’t posted here since last October. I expect it’s lockdown keeping me quiet. But this morning I did manage to dream up quite an interesting research proposal, which I post here for now.)

The ELTeC corpora were designed explicitly and deliberately [ref design criteria] to exclude translated works. [quote] Despite this principled design decision, it seems self-evident that analysis of the mechanisms and results of the cross-cultural dispersal of the novel across Europe – which emphatically is within the scope of the ELTeC project – depended largely if not entirely on the availability of works in translation. It seems probable that the spread of the novel as a popular form was largely determined by the success of particular works, or classes of work, in translation; in particular, we may surmise, those works which responded to common social problems and common cultural trends. Novelists in the traditions from which those works sprang influenced novelists working in entirely different cultural milieux; just as writers raised in other traditions may be presumed to have influenced the development of what we now perceive as a unified European culture by providing easily assimilated versions of the exotic.

Although some multi-lingual expertise was undoubtedly prevalent during this period, the availability of translated versions of novels must have been essential to this diffusion, in both directions. But even basic data about the scale and scope of translations over the period covered by the ELTeC (1840 to 1920) is hard to find [refs needed] being largely diffused across national library catalogues which vary in the extent to which such works are associated with their originals, and rarely give any indication of the exact pedigree of any translation. It seems probable for example that translations into some target languages (say Romanian) would have started not from the version in the original language (say English) but from some other more accessible L2 (say French), but this is hard to determine without substantial research into individual titles and authors. Even harder to find or quantify is any information about the linguistic skills or preferences of a novel’s intended or actual readership. While it is highly probable that the languages of the great imperial powers (English, French, German) would be widely understood in those countries directly under the political or cultural influence of those powers, the extent to which they would be considered appropriate vehicles for reading for pleasure is less clear.

There are many theoretical and formal difficulties associated with any investigation of the relationship between a source and its translation, particularly (perhaps) for works of a literary nature. Translation, like speech itself, is one of the more inexplicable human behaviours. It ought not to be possible, and yet it is done, apparently more or less successfully, every day. [For an entertaining and accessible discussion, written from the perspective of a professional translator, see David Bellos “Is that a fish in your ear?” (2011)] We do not propose to address any such issues in this project, though we may provide some indicative data points to help us others do so. Our goals are more modest. Each completed ELTeC corpus already provides us with a sample of novel production in a given language within a given time frame, hopefully more or less well balanced with respect to date, size, authorship, and impact. We propose to enrich this list of titles with bibliographic data about all translated versions published within a short period (say 15 years) of their first appearance, recording for example the target language, the translated title, date and other details of publication, the translator’s name, and (where this can be determined) the source of the translation. This data will of course be provided in an open format compatible with existing ELTeC deliverables.

Amongst other research questions which availability of this data should address, we identify at least the following:

  • To assess “impact” or “persistence” of titles, the ELTeC corpora rely on a simple reprint count. Do translation counts complement or contradict this classification?

  • Are translation counts statistically correlated with any of the ELTeC classification criteria? That is, where a given collection shows an imbalance for a given criterion, is this also reflected in the translation count?

  • What patterns are discernible in the L1/L2 pairings manifested by our data: for example, which languages are most frequently translated into for each source language?

  • Is there any correlation between stylistic properties of a given group of sources and the languages into which they are translated? Crudely speaking, are romances more often translated into romance languages?


Where translated texts are available in digital form, it would be easy also to provide an ELTeC encoded version, using existing production pipelines. At this stage in the project, it is impossible to say whether this will be feasible on a sufficiently large scale to constitute true parallel ELTeC corpora: it would in any case require significant investment of time and effort from the existing ELTeC partners, whereas the collection of metadata can be done more simply.

Lou Burnard

February 2021

A tale of precision and recall

Back in the day when “text retrieval” was a thing, I remember learning the difference between precision and recall, and the need for a philosophical attitude to the fact that an optimal search has to maximize both these fairly incompatible factors. I now realise how much this whole ATCL exercise has been about that fact. My earlier efforts to identify ATCL titles in the catalogues of existing digital archives involved comparison on the basis of a manufactured key, algorithmically derived from each resource by the same process, which seemed a good compromise. This method also seemed necessary because of the limited facilities some of those resources offered for querying and manipulating the results of queries. With the availability of the wonderful “” service neither of these constraints applies – but the difficulties of balancing precision and recall have not gone away.

Here are the steps I am jumping through:

1. Generate a list of queries, one for each title in ATCL which doesn’t yet have any digital copy

2. Using CURL, send the queries off to the server and get back an XML representation of the results, including catalogue information and a link to the digital version

3. Process the results to check that this is actually the title we are looking for, and then extract the link to add to my atcl-links database

As of today, my query list has 7791 items. The NLS server doesn’t seem to mind dealing with several thousand CURL requests in rapid succession: it takes about ten minutes to run and dutifully sends me back a fat file containing a fairly straightfoward XML representation of the data.

This is fortunate since I am finding it difficult to decide how exactly to construct my query with maximal precision (to avoid false positives) and maximal recall (to avoid missing any) Most titles contain lots of words, most of which are preserved in most catalogues, so an exact word match for the full title is a good start. There are however still a few problems: punctuation and articles sometimes disappear; some titles appear more than once; some titles are very short, and thus generate many false positives. Quite a few titles have the BTAO problem – that tendency of Victorian publishers to improve the title of a new work by adding to it the formula “By The Author Of [insert previously successful titles by this author]” which results in multiple titles containing the same (irrelevant) string. What’s a good filter to cut down the noise from such things? My first thought was to require that the author’s name should be included; my second was to use the date of publication.

The problem with using the author’s name of course is that that it isn’t necessarily present on the title page, and therefore not necessarily present in the title field of the catalogue record. Many novels are anonymous; many authors published under a pseudonym. The ATCL has done a great job of rounding up and normalising authors, grouping under a single entry all variations of an author’s names. Using this it would be possible to find all the works of “Isabella Harwood” whether published under her name or the more usual pseudonym of “Ross Neil”, by increasing the recall of my “creator” search to allow for either name, but I haven’t yet done that. Instead, for my first experiment, I just use the main ATCL surname of the author, and resign myself to less recall, but more precision.

Running my 7791 queries like this

curl “

gets me a total of 6503 results saying “nothing doing”, and 1288 for which there is one or more matching record. I anticipate multiple hits for each title, since there are multiple editions, and of course most of these catalogues list works by volume rather than by work. A very large number of hits usually indicates a problem: for example, there is a novel with the title “Arthur” by Christiana Jane Douglas. Searching just for “title: arthur AND creator:douglas” gets many many titles containing the word “arthur”, some of them editions of the Morte D’Arthur, edited by James Douglas, and others being numerous editions of Crimean War memoirs by one Douglas Arthur Reid. But 1288 hits is not too big a list to refine further.

My second experiment searches for the full title as above, but filters by date of publication. This produces slightly different numbers: there are now 6143 “nothing doing” responses, and 1648 with at least one hit. More interesting perhaps is that I can now compare the two result sets and see which titles are not found by either query – by hypothesis these are genuinely not available, because they don’t exist in the OpenTexts database – and which are found by one but not the other. There are 659 records not found by the search-with-author queries but found by the search-with-date option, whereas there are only 299 records not picked up by the search-with-date query but found with the search-with-author option. Looking down that list very quickly, I see that in most cases the disparity in dates is because the digitized copy is of a later edition of the same work, and this starts me wondering how much later an edition has to be before I decide it’s not satisfactory. The ideal might be to include only digitizations of the first edition, but an edition produced a year or two later is probably fine. Some of these texts have a long and complicated publishing history in which distinguishing the edition is quite critical; others were reprinted once or twice and then disappeared forever.

I am now leaning to the view that the way forward is to maximize recall, simply by combining the 299 records missed by the search-with-date strategy with the rest, and then to pass those results through another filter to improve its precision. This filter would check, for example, whether the publication details for each candidate match, or are within an acceptable range. But it’s very pleasing to note that I have now identified at least one digital version for 13,769 of the 19,912 titles in ATCL, i.e. 69%. Now, if I could only persuade the British Library to be a bit less secretive…

Counting the books: yes, there’s more

My efforts to find links to digitized versions of all the titles in ATCL made one huge methodogological leap forward last week, and is now poised on the brink of another.

Going through the titles I had managed to extract from a rather uncooperative Google Books interface last week, I noticed that rather a lot of them were marked as “not available” for some reason: more precisely although my 11,104 searches, each corresponding to an entry in ATCL for which I had not yet found a digitized version, had succeeded in identifying 2186 previously unseen titles, they had also thrown up 3885 titles which Google considered inaccessible, presumably for copyright reasons, and 5033 of which it flatly denied any knowledge. Yet when I looked up a few of these same titles (whether allegedly “inaccessible” or “non-existent”) in SOLO – the Bodleian’s wizard student-friendly query interface to its catalogue– there they were, page images downloadable in PDF, no sweat.

Now, amongst other delights, SOLO allows quite rich facetted searching, so it is easy to formulate a query like “find me all titles classed as fiction published in London or Scotland between 1830 and 1900, which have also been digitized by Google”, which made me think for a few moments that my work was now done. But as with many other classy library interfaces, SOLO stops short of allowing a mere automaton to carry out any searching: you have to sit at a keyboard and type, though it will grudgingly allow you to save and download the results of your query … provided it contains no more than 50 (FIFTY!) hits. Which (as I politely pointed out to the harassed librarian on online-chat duty last week), is almost entirely useless for my purposes.

Then I remembered that Real Librarians Do It With Z39.50 and dusted off my YAZ skills. The Bodleian, like all real libraries, has a perfectly good Z39.50 interface, which is not only entirely unbothered by a succession of several hundred queries but also happy to send back directly as many full catalogue entries for the hits as you can (err) handle. The only catch is that the queries have to be expressed in some antique syntax called PQN (Prefix Query Notation) and the results come back in MARC 11. I cut my programming teeth on Fortran IV, so these ancient tongues scare me a lot less than, say, JSON. I turned my list of queries unsatisfied by Google Books into PQN, fired them at library.ox, and put the kettle on for a nice cup of tea. PQN is not very discriminating, or not in my hands at any rate, and my queries massively overgenerated. But once my 11297 results had passed through through a couple of utilities (yaz-marcdump to produce marcxml, and my very own `marctotei` to identify and fillet the relevant records) I had a set of 780 CPF format records to add to the ATCL database list, and the tea wasn’t even cold (774 once I’d weeded out some duplicates and mismatches).

A natural question is: can we do the same trick with the British Library? Or any other library offering a Z39.50 interface? In principle, yes. But of course the Bodleian’s use of MARC fields may not be entirely the same as everyone else’s, and so the script I wrote to fillet the results of a query may need fine tuning. For example, the BL does not seem to use Marc code 856 (which I rely on) at all: its digital texts are stored in something called the Digital Store, and their identifiers there don’t seem to map directly to anything like a URL. And while I was thinking about that, something unexpected happened.

A tweet arrived, alerting me to the existence online of the “” search engine: a search interface to a much more ambitious and much more comprehensive view of the world’s digital resources, namely the Global Digitised Dataset Network (GDD Network), originally a research project into the feasibility of creating a global catalogue of digitised texts. At the end of this project’s first funding year it has made available not only a nice search interface but also (applause) the underlying complete dataset. The latter looks a bit like the HT snapshot dumps I have processed before, though it is missing quite a few useful fields, such as type of text, place of publication, etc. And the nice search interface so far has only limited functionality: nice if you are exploring the data, and really quite annoying if you know exactly what you want to find. On the bright side, it allows you to download the results of the query as a CSV file and even has a sort of API, apparently supporting Lucene-style queries to be passed in via a URL to a SOLR-indexed version of the data. This could well be the answer…

Counting the Books contd.

A couple of days ago I reported here on some imbalance in the representation of male and female novelists in current digital archives, written while I was still trying to persuade the Google Books server to do tricks for me. I can now report further progress. After 21 iterations, I did finally manage to confect a complete list of all the ATCL titles freely available from Google Books. Putting this together with data already stored in ATCL for Google, I have now identified 2823 ATCL titles in Google Books, which brings the total number of known digitizations up to 11,510: 58% of all available titles. This seemed good enough pretext to revisit the summary table I produced last time, so here it is again in a new and hopefully slightly more comprehensible form:

Revised counts

As might have been predicted, with more data the situation becomes more nuanced. Note first that the percentages of available titles which get digitized (column “%dig”) apparently decreases as the actual number of texts available for digitization (column “All”) increases, suggesting that the more titles you have available the less likely you are to deal with any one of them. Only tentative conclusions are warranted, since we are lacking so much data for the later part of the century. That said, comparing the columns M-dig and F-dig suggests that throughout the century, digitizers are consistently and disproportionately more likely to go for a male-authored text. Even in titles from the 1850s, where there are substantially more female authored titles available than male (778 as opposed to 595), the proportion of them which get digitized is still lower than the proportion of male authored titles (79% as opposed to 84%). In the 1880s, 55% of titles are explicitly female-authored, as opposed to 41% male (the remainder being unspecified); yet the male authors are still sampled for digitization at a far higher rate (59% as opposed to 37%).

My previous accusations of sexism amongst the digitizers en masse thus vindicated I next considered the practice of individual archives. The following table shows the numbers of ATCL titles I found in each of five major archives, and the proportions attributed to male and female authors in each.

Hathi Trust5655356820226563%36%
Google Books28231138158010540%56%
British Lib51042742225211054%44%
Digitization choices by archive

Overall, the balance is comparable with that shown in the previous table: a small preference for male as opposed to female authored titles (54% to 44%). But this is perhaps concealing a marked variation in practice amongst the archives. At one extreme, Project Gutenberg has nearly three times as many male authored titles as female, while at the other Google Books actually has significantly more female authors than male (56% as opposed to 40%). In between is the British Library Microsoft collection, which matches exactly the proportions for all the archives combined.

Irrespective of gender, how much variation is there in the holdings of these archives? Here’s a frequency distribution showing how many ATCL titles are available from 1, 2, or more archives (note that I cannot distinguish how many of these are actually copies of the same digital version).

Archive overlap: how many titles are available from how many archives?

Encouragingly, this suggests that there is little overlap amongst the holdings of the main digital archives: 63% of all 11,511 digitized titles listed occur in only one archive, 87% in one or two.

Which titles get digitized most frequently? This is hard to tell, for several reasons. Some archives list multi-volume titles as multiple copies; some archives list items simply copied from other archives. For my Google Books listing I excluded titles which were already listed by another archive. But for what it’s worth, here, in no particular order, are the fifteen titles listed as available from all five archives I looked at:

  • Caine, Hall (1853-1931). The Deemster: A Romance (1887)
  • Caine, Hall (1853-1931). A Son of Hagar: A Romance of Our Time (1887)
  • Hamerton, Philip Gilbert (1834-1894). Wenderholme: A Story of Lancashire and YorkshireEdinburgh: Blackwood 1869
  • Collins, Wilkie (1824-1889). Antonina: or, The Fall of Rome. A Romance of the Fifth Century London: Bentley 1850
  • Collins, Wilkie (1824-1889). The Woman in White London: Sampson Low1860
  • Eliot, George (pseud.) (1819-1880). The Mill on the Floss. Edinburgh: Blackwood 1860
  • Dickens, Charles (1812-1870). Oliver Twist: or, The Parish Boy’s Progress. London: Bentley 1838
  • Eliot, George (pseud.) (1819-1880). Middlemarch: A Study of Provincial Life. Edinburgh: Blackwood 1872
  • Gaskell, Elizabeth Cleghorn (1810-1865). Mary Barton: A Tale of Manchester Life. London: Chapman and Hall 1848
  • Grant, James (1822-1887). The Romance of War: or, The Highlanders in Spain London: Henry Colburn 1847
  • Dickens, Charles (1812-1870). Barnaby Rudge: A Tale of the Riots of ‘EightyLondon: Chapman and Hall 1841
  • Dickens, Charles (1812-1870). Bleak HouseLondon: Bradbury and Evans 1853
  • Wood, Mrs. (-). It May be True: A Novel. London: T. C. Newby 1865
  • Oliphant, Margaret (1828-1897). Harry Jocelyn. London: Hurst and Blackett 1881
  • Ouida, (pseud.) (1839-1908). Folle-Farine. London: Chapman and Hall 1871

No, it makes no sense to me either. I expected to see Charles Dickens and George Eliot and Mrs Gaskell on the list, but Hall Caine and Philip Hamerton? Clearly one needs to be very careful in interpreting this data.

See previous bloggage for details of how the numbers were obtained. Supporting data and scripts have been updated in my github repo.

An experiment in counting the books

A couple of years ago I spent some time trying to determine which of the titles in the wonderful “At the Circulating Library” (ATCL) database were freely available online in digital form. This was for largely pragmatic reasons to do with building the ELTeC English language collection: other blog entries describe the method I used and some preliminary results. It’s not as easy as you might suppose to download reliable catalogue information from most digital libraries, nor is it always readily tractable when you do. After some experimentation, I hit on the idea of creating a magic key, a kind of fingerprint, derived from the title and author name as specified, which could then be matched against keys in the same format derived from ATCL entries.

More recently, it occurred to me that this data might also provide some interesting numbers to contribute to current debates about digitization priorities. Exactly why some titles make it to Project Gutenberg, or the HathiTrust, or the Internet Archive and others don’t is not a question to which simple un-nuanced answers are likely or even (maybe) possible, but we should still ask them. Those responsible for the digitization efforts of major libraries are a little coy about the principles on which books are chosen for digitization, or even whether they actually have explicit selection policies, for some reason. I assume that there is a difficult tightrope walk between on the one hand practical but purely adventitious matters such as the relative locations of volume and scanner, the size and state of the volume, the time of day, the temperament of the scanner operator etc.) and on the other principled criteria aiming to ensure a balance of say titles by female and male authors, or high and low brow, date of production, longevity of readership, and so on. It would be surprising if the choices were completely unrelated to characteristics of the population being sampled, or totally failed to reflect the cultural priorities of the scanning operation; the same uncertainties apply, of course, to the collection being sampled for digitization itself.

Anyway, I recently read an interesting article by Allen Riddell and Troy Bassett (“What library digitization leaves out”; preprint available from  which reports that in the data they looked at – the comparatively small sample of surviving English novels published in 1836 and 1838 – shorter books, and books with male authors are disproportionately more likely to be digitized. I naturally wondered whether this applies equally well across the whole of the 19th century.  Which is what led me to revisit my efforts of two years ago. But first, here are the results.

There are 19,912 titles in the current ATCL database. Of these, 9152 (46%) have authors identified in the database as male, 9809 (49%) are identified as female, and 951 (4%) are identified as unknown. These relative proportions are rather different if we look at titles with at least one digital surrogate, of which there are in total 9099 (45%). Of these 9099 digitized texts, we find 5221 (57%) are of male authorship, 3718 (41%) of female authorship, and 160 (2%) are unsexed.

Look at that again. Although there are actually more titles available for digitization from female authors than for male, the number that actually gets digitized is significantly smaller (if, like me, you think a gap of 16 percentage points is pretty significant). Hmmm. These counts of course derive from the whole period covered by ATCL, from 1800 to 1900, so I also calculated them for each decade, only to find that the proportions and their imbalance remain fairly consistent across the century. And this despite huge changes in the numbers: for the last decade of the century ATCL lists nearly 6000 titles, a six-fold increase on (for example) the fourth decade. What percentage of those titles were digitized? In both decades, over 51%. And what proportion of those digitized titles were male-authored ? In both decades, 62%. There is some variability across the decades, but the basic picture remains the same

One possible explanation might be that titles with unknown or unsexable authorship (e.g. the ubiquitous “Anonymous”) are more likely to have been female, and that hence we are not seeing all the truly female authors. But even were this the case (after all, why should we not equally well hypothesize that male authors might be bashful or crave secrecy?), the proportions for books ostensibly male-authored with respect to books ostensibly not male-authored (i.e. those classed as either F or U by ATCL) remain stubbornly higher than the proportions for books definitely not male-authored. And indeed, the same mutatis mutandis is true for the ostensibly-female to ostensibly-not-female ratio.

Here’s a table showing the raw counts:

Decade All “Male” “Female” “U” A-dig M-dig F-dig U-dig
19912 9152 9809 951 9099 5221 3718 160
1830s 482 256 174 52 250 164 85 1
1840s 1037 543 422 72 538 334 202 2
1850s 1483 595 778 110 718 347 358 13
1860s 2341 1019 1093 229 1015 540 456 19
1870s 2866 1189 1514 163 1300 642 633 25
1880s 4126 1693 2287 146 1765 945 782 38
1890s 5979 2995 2863 121 3092 1929 1103 60


And here’s another showing the percentages:

Decade Ad% M% Md% F% Fd% U% Ud%
45.70% 45.96% 57.38% 49.26% 40.86% 4.78% 1.76%
1830s 51.87% 53.11% 65.60% 36.10% 34.00% 10.79% 0.40%
1840s 51.88% 52.36% 62.08% 40.69% 37.55% 6.94% 0.37%
1850s 48.42% 40.12% 48.33% 52.46% 49.86% 7.42% 1.81%
1860s 43.36% 43.53% 53.20% 46.69% 44.93% 9.78% 1.87%
1870s 45.36% 41.49% 49.38% 52.83% 48.69% 5.69% 1.92%
1880s 42.78% 41.03% 53.54% 55.43% 44.31% 3.54% 2.15%
1890s 51.71% 50.09% 62.39% 47.88% 35.67% 2.02% 1.94%


In an ideal world, you’d expect the percentages for titles with male authors (M%)  and for digitized titles with male authors (Md%)  to be roughly the same, right?  Think on… And feel free to download the csv file behind these tables for your own experimentation.

One should always suspect the data, so I make no excuse for the following detailed blow by blow account of how I got these numbers. Full gruesome details, including the scripts mentioned below, are available from

The basic method was to download a complete catalogue of relevant titles available from each target digital library, and then try to match them with records in the ATCL. For Google Books, which does not seemingly provide a complete catalogue online, I tried a different method, discussed further below.

I started by downloading the latest (June 2020) dump of the ATCL database, and converting it to a basic TEI XML format. I then did much the same for the holdings of five digital libraries with good holdings of 19th century novels: the Hathi Trust, the British Library, the Internet Archive, Project Gutenberg, and Google Books. As a control, and for testing purposes, I also looked at a few smaller collections, notably the Victorian Women Writers Project at Indiana University and the (now defunct) University of Adelaide “ebooks” repository. I wanted to provide something similar to John Mark Ockerbloom’s lovely Online Books Pages at but more precisely tied in to ATCL.

Hathi Trust makes available a monthly dump of their entire collection as a huge tab-delimited file. Working with the most recent dump, dated September 1 2020, I used a simple minded perl script `hathiProcess.prl` to parse this file and select from it only freely-available English language books published in Great Britain between 1800 and 1920; an  XSLT stylesheet `htConv.xsl` then converted the results to the common project format (CPF).

The British Library website makes available an Excel spreadsheet providing metadata for the titles from their collection which were digitized some time ago by the Microsoft Books project I downloaded this, converted it to TEI with `csvtotei` and converted the result to CPF, (selecting just the 19th century titles) with `blConv.xsl`.

Project Gutenberg makes available several versions of its catalogue data. I worked with the most recently updated one, which is a vast archive of unbelievably verbose RDF files. Despite its complexity, this data doesn’t include any publication data for the source texts concerned (unsurprising really), though it does provide birth and death dates for the authors. To cut down the numbers a little, I rejected titles whose authors were not born during the 19th century, and also those which specified a MARC relator field “edt” (to cut out non-original editions). Once I had remembered how on earth to handle a gazillion tiny files of RDF (I did this back in 2018 ), I used the `gutConvRDF.xsl` script to process them all to CPF, and concatenated the results into a single file.

The Internet Archive, so far as I can see, doesn’t have any generally available or downloadable catalogue, though it does have a really good query interface. The method I used for attacking Google Books would presumably work equally well (or equally badly; see below) in this case, but I haven’t tried it. Instead I just used a predefined collection called `19thcennov` which someone at UIC Urbana Champaigne thoughtfully created back in December 2008. This gave me 7828 XML records which were easily converted to CPF using `iaConv.xsl`.

The common project format files all consist of TEI <bibl> elements with either an @xml:id attribute or an <idno> specifying the identifying code for this item in the relevant repository, e.g. ‘ia:foreignersnovel03pric` identifies the Internet Archive’s digitization of volume three of Eleanor Price’s novel “The Foreigners” . Each <bibl> also has an @n attribute supplying the magic key for the title, which is confected as follows:

  • remove the full stop following Mr or Mrs in any title containing one
  • take the substring of the title up to the first occurrence of one of the punctuation marks . , : ; or /
  • concatenate this with the author’s last name
  • convert to lower case and remove all punctuation characters and spaces

So, for example ATCL lists a work with the title “The Foreigners: A Novel” attributed to author “Eleanor C. Price”. The same work appears in the Internet Archive list, but with the author “Price, Eleanor C. (Eleanor Catherine)” and the title “The foreigners : a novel”. Despite the differing strings, both will get the same magic key “theforeigners|price”. This method is far from bullet proof, but it’s serviceable.

For Google Books, as noted above, there is no readily downloadable catalogue. But there is an API, which in a moment of madness I thought it might be cool to learn how to use. A day of poking around led me to a neat python script some helpful person had written to look up ISBN numbers (hat tip to AO8’s treasury , which I mercilessly hacked to my own purposes. My version reads a file of URL-encoded search requests like this “inTitle:the+inTitle:foreigners+inAuthor:Price”, fires them at the Google API, and processes the returns into a rudimentary bibl or a comment lamenting the absence or unavailability of the item in question. The file of search requests is rather long (one for each title in ATCL for which I have not yet found any digital version – a total of 11,203 ) so I make the program sleep for a while after firing off about 40 consecutive requests, to help the Google server catch up. Despite this considerate behaviour on my part, it did not take Mr Google long to decide that my program (or my IP address) was a threat, and then to start returning unco-operative HTTP messages like 503 (“Service Unavailable”) and 429 (“Too many requests”). The API Help pages confirm that Google considers “using an app, program or script to perform a large number of searches in a short time” prima facie justification for temporarily blocking the IP address in question; though it’s not clear what exactly is meant by “large” (more than 100?) and “short” (less than a minute?) in that phrase. Furthermore, when I search using my specially-minted API key, there seems to be a hard limit of 1000 queries per day in any case: so this job is not going to be finished very quickly. Still, I do now have an extra 1517 records to show for two day’s work.

Once I’ve created all these lists, I run the merger.xsl script to add <ref> elements to the ATCL-TEI file I created in the first step. This makes for some redundancy, for two reasons: firstly, for most of the archives a three volume novel is likely to get a separate entry for each volume; secondly, for many titles, there exist multiple digitizations – which may (or may not) derive from the same source. The following table shows for each archive the number of records selected for processing, the number of references to ATCL titles found, and the number of titles affected. Note that I haven’t yet done any de-duplication to remove overlaps.

British Library 62015 9920 5104  
Hathi Trust 460070 18891 5655  
Internet Archive 7829 4691 1655  
Project Gutenberg 38338 2880 2275  
Google Books ? 1517 1517  

I haven’t made available the CPF files for each archive, nor the final merged TEI version of the ATCL dump, since this is not really my data to share. But I have made available a file called atcl-links.csv, which is a spreadsheet with a row for each ATCL title digitized in one or more publicly available digital collection, mapping its ATCL identifier to its identifier in each repo. I’ll  update these as and when the data improves.

Can we trust the ATCL database?

I’ve been enthusiastic about the database behind the At the Circulating Library website ever since I discovered it nigh on three years ago. Troy Bassett, its creator, deserves much respect and lots of credit both for the work put into creating it and maintaining it, and for his generous open minded policy of making the data itself freely available in snapshot form for nerds like me to play with. I think of the ATCL as the nearest thing we are ever likely to get to an authoritative catalogue of the 19th century novel, and use it as such in other work which I’ll report on later. But (you guessed there was going to be a but) just how reliable is its coverage? Does it cover each decade of the 19th century equally well, or are there gaps?

When asked, Troy assures me that he’s fully aware that there are gaps. He estimates that the true size of the database should be well over 25,000 titles, rather than its current 18,000. He and his team are slowly and meticulously filling the gaps, by hoovering up and checking data from resources such as published catalogues and bibliographies, or online collections like Proquest’s NSTC. It’s tedious and fiddly work, ideally suited to a small farm of graduate students, should you have such a resource at your disposal.

My interest in ATCL is to use it as a reference point when assessing the coverage of other resources, in particular the catalogues of digital libraries. Take, for example, the current Hathi Trust catalogue. This lists 17,433,331 titled volumes in all; 235,035 of them being volumes published in the UK between 1800 and 1900. Removing obvious duplicates (Hathi catalogues each volume of a three-decker separately, usually though not always) brings this down to an only slightly more manageable 129,817 titles. How good is its coverage of 19th century novels? The difficulty of course is to winnow out just the entries which are novels. As I suggested on my blog back in June 2018, the word “novel” in a title turns out to be a very good indicator (other good words include “or” and “tale”, “history” etc), so I extracted from the HT records just those titles containing the word “novel” for further investigation: there are 953 of them.

In my earlier work, I’d just assumed that if a title in HT didn’t appear in ATCL then it wasn’t a novel. But how complete is ATCL? A good way of finding out might be to work the other way round: first get a list of things which are definitely (probably) novels — and then check to see whether  they also appear in ATCL.

On my first pass, 284 of my 953 “novels” (i.e. titles containing that word) did not appear in ATCL, which surprised me. But this was mostly a result of my matching procedure being insufficiently robust to cope with the amount of variation in cataloguing practice within the HT records and their occasional divergence from ATCL practice. I spent a happy day or two going through the list of delinquents by hand, taking the opportunity to discard 19th c. reprints of earlier works, translations from other languages (mostly scandalous Zola), and a handful which were clearly not novels at all (e.g. “Photographic amusements : including a description of a number of novel effects obtainable with the camera / by Walter E. Woodbury.”) – all of which brought the number down to 191. I then fixed the “magic keys” used to identify each title in ATCL as necessary, all of which brought me to a manageable 40 titles apparently missing from ATCL, with which to torment Troy Bassett and his team. 40 titles missing from a sample of 953 is pretty good; even 191 missing is not so bad in this always very approximate work. I remain persuaded that the ATCL is a reliable approximation to a representative sampling of the 19th century novel.

A brief report on my attempts to understand the make-up and cataloguing of EEBO-TCP

Q. What exactly is EEBO-TCP made of?

A. (digital) copies of (microfilm) copies of works.

Q. “Works”?

A. Well, no. Specific copies of works, held in libraries. What I think FRBR calls “instances”.

Q. FRBR makes me think of unhappy cats. Can you give an example?

A. Consider Jeremy Taylor’s 1668 page-turner “XXV sermons preached at Golden-Grove being for the winter half-year beginning on Advent-Sunday until Whit-Sunday”. Two copies of this book from the British Library’s collection, as printed by one E.Tyler for book-seller Richard Royston, have been microfilmed, one as image set 199641 and one as image set 45789. Both copies are of the same book and therefore have the same identifier (T410) in Wing. Check ’em out at and and see if you can tell them apart (I can’t, in this case) And yes, EEBO also has images of two different copies of the 1655 edition of this book: same author, title , and publisher, but completely different bibliographic entity, with a different Wing identifier (T409). And probably many others. He was big in the 17th century, that Jeremy Taylor.

Q. So?

A. Perhaps the primary key to the EEBO catalogue should be the image sets, since each catalogue entry concerns one set of images. However, given the choice, no-one would prefer a title like “microfilm no 42” to one like “amusing title of work”. Consequently, Proquest provides a unique identifier for each bibliographic work, which is suffixed with an image set identifier if there are multiple copies of the same work (which happens a lot), or if the work exists in multiple volumes which have been scanned separately (fortunately, less frequent). And, not to be outdone, TCP also provides a unique identifier, but in their case it identifies a specific copy of a specific work. The URLs I quoted above behave similarly: they uniquely identify a specific copy of a specific work.

Q. Wait, how many identifiers are there now?

A. Sticking with the 1668 third edition of Taylor’s XXV Sermons, we have two image sets (identifiers 45789 and 199641). Since these correspond with a single bibliographic entity, also known as Wing 610, Proquest gives them the same catalogue number (10772247). (You may wonder why they don’t just use the Wing number, unless you are a librarian). But we can’t have two entries in our catalogue with the same identifier, so Proquest appends the image set identifier to the catalogue number for one of them (I don’t know how they decide which, nor why they don’t do it for both). TCP, as noted above, just give each distinct item a distinct identifier: in this case the transcription of image set 45789 is in TCP text A64140, and the other is in TCP text B30404. All clear now?

Q. There must be an easier way of doing this

A. Yes. Amongst a raft (or maybe a coracle) of other improvements this year when Proquest moved the online access to EEBO to a new platform, they also introduced a new identifier called a “GOID”, which is essentially a unique number for every entry in the catalogue. That’s what I used in the URLs quoted above. Hoorah! We can now access anything in the EEBO catalogue using a simple numeric code, just like we can in the TCP subset of it.

Q. What about the TCP identifiers though?

A. Alas, these are currently not included in the dataset which underlies the Proquest online catalogue. As noted here earlier, I am working on a super TEI-compliant version of said dataset, and that will assuredly include the TCP identifiers. More on that anon.

My humble thanks to Paul Schaffner for patiently explaining all this to me. Any residual errors are mine, not his.


There you are idly scrolling through Twitter when someone announces a tempting resource that’s just crying out for TEI-ification. And there goes the afternoon and most of the evening.

Anyway, creating my EEBO Bibliography in TEI was insultingly easy. I just grabbed the Excel spreadsheet (34 Mb) thoughtfully created by those lovely people at Protext, and even more thoughtfully publicized by the even more lovely Heather Froelich on twitter this morning. I opened it up in Open Office. I exported it as a CSV file (85 Mb) and munged same through the standard TEI cvstotei stylesheet to generate 175 Mb of TEI compatible data. Then I sat down to consider how to make it actually TEI conformant, i.e. how to make the title of a bibliographic entry appear not as the content of <cell n=”5″> but as the content of a …. <title>. As you might suppose, defining the right mapping was easy for some things, but less so for others of the 17 cells in each of the 146,323 rows of the spreadsheet. There’s a table to show the mapping I decided on at the end of this blog, for those unwilling to read my pellucid XSLT code which actually uses it.

The resulting TEI file isn’t quite complete because it doesn’t have a TEI Header, needed to define the prefixes I use to save space in the URLs, but (at 120 Mb) it’s too big for github. And it’s now available at for your xpathing pleasure.

Here’s the table. The whole spreadsheet is a <listBibl> and each row becomes a <bibl>. I like simple solutions. I’m not proud of the <note type=”foo”>s, but that’s the best I could think of without getting far too complicated.

1MARC identifier@xml:id : prefixed by eebo:
2Image set identifier@facs : prefixed by eeboIs:
3Publication type@type (always either Book or Issue)
4Collection <series>
6 Author<author>
7Publication Date<pubDate>
9Country name<pubPlace>
10Publication language@xml:lang gives ISO code equivalent; text goes in a<note type=”langNote”> c
11Accession number<idno>
12Source Library<note type=”sourceLibrary”>
13Full text imageif “Y”, <note type=”transcriptType”> contains “image”
14Full textif “Y”, <note type=”transcriptType”> contains “text”
15USTC Classification<note type=”keywords”>
16Release dateToo boring to include
17 URL@ref with prefix proquest:
Mapping EEBO spreadsheet fields to bits of a TEI <bibl>

Counting the books

As a follow up to my previous rather excited posting I have finally got round to actually trying to count how many copies of each title selected for the ELTeC English collection various prestigious national libraries hold. Here’s how I have operationalised the need for some kind of metric approximating to the persistence or canonicity of a given title.

First I run a little XSLT script against the corpus to create a file full of lines like the following:

f @and @attr 1=1003 sinclair @attr 1=4 "modern flirtations"
set marcdump ENG18410.usmarc
show all

This means:

      • find records in which the author field contains “sinclair” and the title contains the words “modern” and “flirtations”.
      • send the output to a file called ENG18410.usmarc
      • display all the results from that query

Creating this query automagically is not without problems. Including words like “the” or punctuation like the question mark is ill advised. Some records include subtitles in their “titles” but most don’t. When the records do contain subtitles they may result in false hits: see further below.

Next I throw this at a z3950 server and go make myself a cup of tea while it chunters away. As noted in my previous posting, getting Z3950 access to a library in question is mostly just a matter of knowing the address of the server and its port, the name of a database, and sometimes (as with the British Library) also wheedling a login and password. The reason I use the recondite syntax above for my query input, and the reason that I accept the results in usmarc 21 format is … that’s what every z3950 server I have looked at so far promises to provide. Some have other exotic options for query or for output, but nothing else is universally guaranteed to work.

Returning with my cup of tea, I now have a bunch of inscrutable marc21 records tidily filed away. I wasted the best part of an evening yesterday trying but failing to find a simple online tool which would convert them into marcxml or indeed anything readable, but the best I could come up with was a perl utility called marcdump. Here’s the start of the output it gives me for ENG18410.usmarc

LDR 00535nam a2200181uu 4500
001 006812208
005 20100212180700.0
008 040420s1841 xx || 000 ||eng
019 u _aG11034382
040 _aUk
082 04 _a823
100 1 _aSinclair, Catherine,
245 10 _aModern flirtations :
_bor, A month at Harrowgate /
_cCatherine Sinclair. Vol. 1.
260 _a[S.l.] :
336 _atext
337 _aunmediated
338 _avolume
852 41 _aBritish Library

Exciting stuff, eh. The useful bit here is the publication date, which appears as subfield _c of field 260 here (sadly, there are other possibilities), and even more useful the following, which appears at the end of the output file:

Recs Errs Filename
----- ----- --------
4 0 ENG18410.usmarc

Tis but a matter of moments to grep through these files and extract a list of record counts for each title, together with a list of publication dates.

Furthermore, and much to my relief, the counts do seem to reflect my initial expectations as to which titles would be highly rated and which not. The top ten titles in my 90 are (drumroll)…

94 ENG18860 Hardy: The Mayor of Casterbridge
106 ENG18531 Yonge: The Heir of Redclyffe
135 ENG18621 Braddon : Lady Audley’s Secret
143 ENG18481 Dickens: Dombey and son
148 ENG18610 Eliot: Silas Marner
152 ENG18530 Dickens: Bleak House
157 ENG18540 Dickens : Hard Times
168 ENG18480 Thackeray: Vanity Fair
298 ENG18471 Bronte: Wuthering Heights
664 ENG18652 Carroll: Alice in Wonderland

Nearly all of these would figure on any list of long-lasting 19th c English novels. An eyebrow night be raised by some in the English department about the appearance of Yonge and Braddon, but the explanation is simple: both ladies (or their publishers) were very fond of including the phrase “by the author of ‘Most Famous Title’ ” on the title of their less famous works, and I have not yet worked out how to remove such imposters as “Work you’ve never heard of (by the author of Most Famous Title) ” from the results of a search for “Most Famous Title”.

Another eyebrow might be raised at the frequency distribution of the scores found: there is a very long tail, with nearly two-thirds of my 90 titles scoring 20 or less, while the top scorers, as shown above, score very much more. To some extent, this is explained by the crudity of my search technique, which will include musical adaptations, commentaries, versions for the use of slow readers, study notes, etc etc provided that “Most Famous Title” appears in the title somewhere. This worries me less, since the existence of such things is surely also testimony to the salience of the title in question. This factor does however have an inflationary effect on the scores, so that titles which don’t benefit from it appear lower than might be expected. “Middlemarch” for example – widely regarded as amongst the greatest English novels of the period, but not subject to – scores only 77, ahead of Sherlock Holmes debut novel “The sign of four” (72) but behind George Eliot’s closest rival for the depiction of provincial life Mrs Gaskell’s “Mary Barton” (82).

But these scores should not be subjected to such close scrutiny. If we are looking for a proxy metric for the “impact factor” of these works, it’s not implausible to be guided by the numbers of different editions of them that have accumulated in our great national libraries. If we say that a score of less than (say) 20 suggests a low impact, and anything above (say) 50 a high one, we should not go too far wrong.

So far I have tested this procedure only on the British Library’s collection. An obvious next step is to try a different English-language library (COPAC springs ro mind) to check that the ranking is not too widely different. And then to try out a different language: the BNF also has a z3950 server so I plan to subject the French collection to the same treatment.

Hoorah for standards (and librarians)

Librarians do it with Z3950. Of course, I knew that all along. I just didn’t know what it implied for the non-librarian.  I have been wondering for some time how to get at the bibliographic riches of our national and university library catalogues without using one of the artfully constructed snazzy web interfaces they all seem to have set up in the interests of “usability”. Which interfaces don’t usually include any means of answering such questions as “List dates of publication for all editions of this title which you hold”. BUT (tl:dr) most library catalogues also run a server which exposes their data via an antique API called “Z3950”, originally developed by the Library of Congress and others, apparently so that they could steal catalogue records from each other without the pain of sending round a man with a truck.

When I say antique, believe me I am not joking. Some Z3950 servers I have looked at (eg the Bodleian) will deliver records in funky modern formats like SGML, but the only format you can be sure of getting is US MARC 21. This is an old school binary format, remarkably similar to the ones I used to have to deal with when debugging IDMS databases or alien magnetic tape formats back in the 1970s. You have pointers and variable length fields and a mix of character data and binary and you really don’t want to hack this stuff in Python, honest. Which is why there are dozens of snazzy interfaces and convertors and alternative syntaxes listed in bewildering plenitude on the Library of Congress website. Some of them cost money. Most of them are for Windows only. I took an instant dislike to nearly all of them… until I discovered YAZ.

YAZ is a properly constructed library of utilities offering all sorts of features I don’t claim to understand; what I liked is that it also provides a small number of command line utilities which do things like access a z3950 server, search for records, save the results, and convert from US Marc to something a tad more readable. And it’s open source, and free, and runs on linux, just like that, no need to install lots of bloatware as well. Sudo apt install yaz* and I’m done.

Now, let’s say I want a list of the publication data for all copies of Disraeli’s Sybil held at the British Library. Simple. On the unix command line I type:

$ yaz-client -u xxxxxxxx

(the row of xs there is my authentication string, which I promised the BL I wouldn’t share with anyone as a condition of getting access to their server for nothing. The rest is common knowledge though – see

The yaz client operates a bit like telnet used to: at each prompt I can type special commands like “find” and “show” and see the results. By default it uses something called Prefix Query Notation, or PQN, though I could also use something called Common Command Language (CCL) “an international standard query language often used in libraries” according to Wikipedia.

Z> find @and @attr 1=1003 disraeli @attr 1=4 sybil

Don’t you just love that PQN syntax? It’s like typing in Latin. It means Find records with “disraeli” in marc field 1003 and “sybil” in marc field 4. Who needs high falutin nonsense like Xquery?

Then I can type:

Z>show 1+100

to see the first 100 records (it’s obvious, surely, that “n+m” must mean “up to m records following record number n”)

Even better, I can type

Z>set_marcdump wibble

and have a copy of  all the marc records returned saved to the file wibble. Why should I do that? Because I can then process that file to something readable with another little command line utility called marcdump:

$yaz-marcdump wibble | grep 204

which will convert wibble to something more readable, and pick out all the lines containing publication data! Yes, you do have to know what those Marc field labels mean, Cynthia.

And yes, there are features for batch mode operation so I could run my commands from a file, save the results, tidy them up and put my own super-simplistic interface on the whole business. I can see this is going to keep me happily occupied for quite a while.

Building the ELTeC : the quest continues

My quest for a reasonably reliably complete list of 19th c. English novels complete with indication of available electronic versions thereof has made a large leap forward as a result of the generosity of others.

Firstly, Professor Troy Bassett of Purdue University Fort Wayne, whom God Preserve, tells me that he will now start distributing regular extracts from his database at in CSV form for my (and others’) mungeing pleasure. I spent some time testing this out, using Libre Office to convert the CSV into an XML form which I could then reprocess with yet another xslt stylesheet. This worked reasonably well (eventually) and I was able to create a new version of this data in TEI, all 15,682 records of it. To merge in identifiers for VWWP, Internet Archive, Gutenberg, and Google texts I used a simple minded stylesheet which tries to match on the magic keys I created earlier, as discussed in previous blog postings. This worked, though not exactly with blistering speed, and I produced a second version of the Bassett TEI file with added references. Of those 15,682 titles, only 3,430 have at least one digital version and there are a total of 3,813 digital versions in all.

Secondly, and with some trepidation, I approached the sacred grove of Big Data and sent a plea for help to the nice people at Hathi Trust, that extraordinary repository of 16 million or so books in digital form. You can download a full list of their holdings in a nice simple processable form from : the catch is that their monthly snapshots contain 16 million (and counting) records derived, I think automatically, from contributing libraries’ catalogue systems, not all of them entirely consistent in their metadata usage, and certainly none of them clearly identifying which titles might be considered to be novels or have female authors. I did some initial poking around with a perl script and ascertained that I could whittle the 16,208,265 entries in the May 4th version of the file down to a mere 1,253,950 by selecting only those which are publically acccessible, have a publication date between 1849 and 1820, are books in English, and are not US government documents. These counts are inflated by the fact that we are counting volumes here, rather than works.

Meanwhile, a helpful person on the Hathitrust help desk had got in touch pointing me to the existence of various selections created by other researchers before me, notably Ted Underwood’s file filtered_fiction_metadata.csv [available on Github at]
which lists 93,708 volumes of fiction, published between 1800 and 2007. “It focuses on fiction, but will include works in translation. It’s also not restricted to ‘the novel’; short stories and even folk tales are included. But we have tried to exclude fiction aimed explicitly at a juvenile audience across the whole timeline.” Underwood points out that “There is no authoritative master list of all English-language fiction from 1800 to the present. So scholars who want to do research at scale have to construct their own list.” which is true enough, though I wonder if he’s aware of Bassett’s work. Anyway, I wrote more scripts to process his CSV file into TEI, adding my magic keys to enable me to link his data with the titles in the Bassett database. All of which meant that I could provide Bassett with a list of Hathtrust links as well as those already provided for Gutenberg et al.

Bassett’s data includes some titles published only in the US, which I would like to exclude, as well as some “aimed explicitly at a juvenile audience” (the two sets of course have nothing in common, except that I would like to exclude them both). More annoyingly, it stops in 1901. I therefore went back to thinking about how to extract comparable data from the Hathifiles for the period 1901-1920. The most recent Hathi file (dated 1 June) has entries for 16,370,821 volumes. I tweaked my perl script to extract records for books in English published between 1900 and 1921, as before, tweaking it also to suppress duplicate titles, which gave me a more manageable but still daunting total of 348,232 titles.

The question is, as ever, how to pick out the novels from the chaff? The obvious answer is to apply some basic text analytic tools. I have a list of 15k titles which the obliging Bassett has already identified as novels. I have 348,232 titles extracted from the Hathitrust file. If I had to scan through them choosing the titles which are most likely to be novels, which words would catch my eye? Which words in fact appear significantly more frequently in the first list than their frequency distribution in the second would lead you to suppose? This sort of question is surely what computer linguists deal with ten times a day before breakfast.

I install Lawrence Anthony’s excellent AntConc program from and battle through its interface a bit. Here are the top twenty “keywords” for the Bassett title list, when I use the aforesaid Hathitrust titles list as the “base corpus” for comparison. The underlying statistic (need you ask) is log likelihood (4 term); the threshold being p < 0.05 (+Bonferroni), i.e. the defaults.

#Keyword Types: 611
#Keyword Tokens: 47261
#Search Hits: 0
1 2693 + 15387.79 0.0556 novel
2 8559 + 10677.59 0.0763 a
3 1969 + 6571.89 0.0388 or
4 1262 + 6402.77 0.0266 tale
5 1426 + 4104.1 0.0283 story
6 721 + 2599.46 0.0152 romance
7 2195 + 1957.72 0.0327 s
8 471 + 1072.51 0.0098 stories
9 364 + 1058.08 0.0077 love
10 837 + 1019.33 0.016 life
11 355 + 1001.66 0.0075 tales
12 9268 + 875.37 0.0384 the
13 272 + 773.82 0.0058 adventures
14 163 + 594.6 0.0035 daughter
15 224 + 591.09 0.0048 lady
16 217 + 546.42 0.0046 woman
17 150 + 484.95 0.0032 wife
18 570 + 478.07 0.011 other
19 221 + 422.18 0.0047 little
20 131 + 395.37 0.0028 miss

This seems like pretty good evidence that simply looking for titles which contain the words “novel”, “tale”, “story”, “romance” or – this one a surprise – “or” might do pretty well. It also suggests that the Bassett novel is quite concerned about women, but that’s another story.

Reassured, I re-tweak my Perl script to extract bibliographic records as before, but this time only if their title contains one of the words “novel”, “tale”, “story”, or “romance”. A glance at the results shows that this approach cuts down the number of titles to a more manageable 16,908 but still has many false positives (“My life and the story of the Gospel hymns and of sacred songs and solos” for example) . A more sophisticated approach is needed, clearly. Time to make a cup of tea.

Building the ELTeC : stage 0

Problem: if the ELTeC is supposed to represent in some sense the full range of novel production in a given language (EN in my case) for a given time slot (1850 to  1920, it says here) how do you find out what the population actually is before starting to sample it?

Enter at this point a wonderful database: Bassett, Troy J. At the Circulating Library: A Database of Victorian Fiction, 1837-1901. Victorian Research Web. [accessed 2018-02-09] ( I say “wonderful” advisedly: I have found nothing comparably complete and usable in a week of scratching around the internet.

According to Bassett’s technical notes, “This website was written on a Macintosh using a MySQL database and PHP.” Consequently its contents are pretty consistently organized and tagged, and consequently screen scraping and munging into a different format are both pretty easy: like this

 for number in {1850..1901}
 echo "$number "
 wget "$number" -O $number.html ; tidy -asxml -n --new-empty-tags image $number.html | saxon - bassetConv.xsl > $number.xml
 exit 0

The `bassetConv.xsl` stylesheet generates a TEI format bibl entry for each of the 15,000 plus titles, like this:

<bibl xml:id="11169" n="acounselofperfection|Malet">
<author n="576">Lucas Malet.</author>
1 vol.  London: Kegan Paul.</bibl>
The numbers are the identifiers used by the mySQL database, so they should be unique. The @n attribute on the bibl element is a key I generate for matching purposes (see later).

I would rather like to know how many of these titles are available in
digital form, and from where. The current database has links to
Google Books (500 or so, it claims: I haven’t worked out how to find
them without downloading the entire database) but nothing else so far
as I can tell.

How to accomplish this?

My best plan so far is to extract  lists of keys or fingerprints, derived from the cataloguing information supplied by each online repository  and then start looking for overlaps with Bassett. My assumption is that for any relevant title not to exist in Bassett would be rather remarkable. Some initial experiments suggest that the first few words of the title combined with the surname of the author is a reasonable approximation to a fingerprint for each title, though obviously not

So far I have looked at the following four online repositories:

1. Victorian Women Writers Project

This has 75 titles identified as fiction, and all in good TEI format; the following grabs the catalogue info for them

Life is too short to process all the resulting HTML: in any case I am sure if I wanted a proper XML catalogue my friends at Indiana would give me one. So I manually separate out the useful chunk of HTML: and mangle it through a simple XSLT stylesheet (`vwwpConv.xsl`), to produce a file of entries like this:

<bibl xml:id="VAB7046" n="daphne|Ward">
<title>Daphne, or Marriage a la Mode</title>
<author>Ward, Humphry, Mrs., 1851–1920</author>
<publisher>London; New York: Cassell, 1909. 315 p.</publisher>

Note the cunningly constructed @n attribute supplying a key which, all things being equal, should match an entry in the Bassett database, if there is one.

2. The ebooks@adelaide project

This is a quirky site hosted by the University of Adelaide library which makes many texts available in a well structured XHTML format which is easy to munge into TEI. For info about this apparently little known but rather splendid project see
Selecting relevant titles for our purposes is not so easy, so I grabbed a readable version of their entire catalogue in HTML from the website, and then grepped through it for potentially useful entries, resulting in a file full of lines like this:

<li><a href="/a/abbott/edwin/flatland/">Flatland: a romance of many dimensions / Edwin A. Abbott [1884]</a></li>

along with a lot of less useful lines like this

<li><a href="/a/aristotle/meteorology/">Meteorology / Aristotle; translated by E. W. Webster</a></li>

A perl script was the quickest way of munging this into minimal TEI entries like this:

<bibl><title>Flatland: a romance of many dimensions </title><author>Edwin A. Abbott </author> <date>1884</date><ref>/a/abbott/edwin/flatland/</ref></bibl>

I am not sure that there is much wheat of this kind in this chaff however: I will come back to it later.

3. And then there’s Gutenberg

Yes, the Gutenberg Project also has a catalogue: a huge one, which you
can download as an incomprehensible RDF file or a plain text
monster. I took the latter option, and hacked out of it 49639 lines
like this:

<title>100 Desert Wildflowers in Natural Color</title><author>Natt Noyes Dodge</author><idno>54631</idno>

I won’t rehearse here the manifold  inconveniences of using Gutenberg texts. But it would be useful to come back to this and see how many of the Bassett titles are available here. As a first approximation, I first reduced the title list to just the first part of the title and the author’s surname, with no spaces or punctuation, and then used the invaluable unix  comm command to identify common lines in the two files. Obviously, this procedure is vulnerable to typos and inconsistent editorial practices, which are not uncommon, but in my first experiment I found 984 items with identical titles and authors in both the Gutenberg title list and the Bassett title list.

4 Internet Archive

This wonderful collection has a good search interface, spoiled somewhat by the unreliability of the data. I used it to grab all the entries for a collection called “19thcennov” which looked promising. Sorted by descending date, the very first item had a date of “1983” which, on inspection, turned out to be a typo for “1883”, so a good thing I didn’t use “date” to limit my search. OTOH, the second item in the list was something called “Mathematics in urban science, V : Catastrophe theory” published by the “Monticello, Ill. : Council of Planning Librarians”. No matter: at least the output is in XML, and the text identifiers used by the IA bear a striking resemblance to those I thought I had invented. My search gives me 7829 items which look like this:

<str name="creator">North, William, d. 1854</str>
<str name="date">1847-01-01T00:00:00Z</str>
<str name="identifier">impostororbornwi03nort</str>
<str name="language">eng</str>
<str name="publisher">London : T.C. Newby</str>
<str name="title">The impostor, or, Born without a conscience</str>
<str name="volume">3</str>

which I then munge into

<bibl xml:id="impostororbornwi01nort" n="theimpostor|North">
<author>North, William, d. 1854</author>
<title>The impostor, or, Born without a conscience</title>
London : T.C. Newby</bibl>

The IA gives each volume a separate catalogue entry, so these 7829 entries boil down to only 2739 unique keys which I can compare with the Bassett keys. On a first pass through I identify 1235 matches, i.e. nearly half. I also identify lots of ways of improving the matching procedure, but for the moment this seems like it might be useful. Though I am not sure that a success rate of around 10% in identifying matches is altogether worth shouting from the rooftops about.

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search