Non classé – Foxglove

The Lacy Workflow

This summer has seen the establishment of a fairly routine procedure addressing the quixotic task of producing a TEI conformant version of Lacy’s Acting Edition. I can comfortably now do one play a day, or 365 a year, which means I should have done the lot within the next four years, assuming I don’t go mad or find something more interesting to do. For the avoidance of doubt, and in case I forget what I am doing, here’s a fairly complete account of the current workflow.

The final validated version of each file is stored in the repository named Lacy/TEI and has the form Lxxx.xml (or LxxxR.xml)

Intermediate versions are stored in the work folder Desktop/G3, in one of the subfolders Pending (currently being worked on), Done (finished with), or Laters (No longer pending). Each file is named as follows:

Lxxx.txt plain text version as downloaded from Hathitrust or similar;
Lxxx-final.xml initial tagged version of the same produced by an LLM, so far only Gemini 3;
Lxxx-ed.xml manually corrected version, checked against a specific source text

Processing proceeds like this:

Download plain text version of a volume.
Run splitVols.pl to produce individual .txt files, one per play.
Select play, according to the following arcane principles:
- there should be at least three titles from each volume: run saxon catalogue.xml volFreq.xsl to see how well this criterion is being met;
- the proportion of plays by a given author should be roughly the same in the TEI subset and in the whole LAE: run saxon catalogue.xml balanceCheck.xsl to see how we’re doing on this one;
- other arbitrary predilections not summarised here.
Casefix: as we are not even attempting to reproduce the typographic luxuriance of these originals, the use of uppercase is merely annoying. We run perl Scripts/caseFixer.pl Lxxxx-final.xml > Lxxxx-ed.xml, or (inside Desktop/G3/Pending) sh caseFix.sh Lxxx to normalize case throughout.
Clean up and correct the following systematic errors (inter alia) using oXygen:
- use of hyphen vs emdash;
- placement and numbering of page breaks;
- placement and punctuation of stage directions;
- tagging of titlepage, cast lists, and other front matter;
- assign a gender to each <role>;
- tag italicized phrases in speech using <hi>;
- remove redundant <p> tag surrounding a sequence of <l> in verse texts;
- add <spGrp> around musical numbers;
- correct occasional hallucinations, misplaced tags, wild speculations etc.
Run saxon doPartix.xsl Lxxx-ed.xml (or choose the doPartix scenario from Oxygen menu). This produces a file Partix/Lxxxx-ed.xml containing a draft <particDesc> derived from the <speaker> elements in the text. Correct this file (or the contents of <speaker> elements) as documented in this blog posting.
Run sh enrichHdr.sh Lxxxx : a script which:
- rewrites the TEI Header to include this particDesc and other metadata from the catalogue.xml;
- adds who attributes to each <sp>, pointing to the appropriate <person> element;
- validates the resulting file against the current Dracor schema.
The most likely problem at this stage will be <sp> elements with missing or defective who attributes: these have to be fixed by hand. When all is well, update the catalogue.xml to show that a TEI version of this title is now available (by adding subtype='TEI' to its corresponding div type='work'), move the validated file from subfolder Dracor to Lacy/TEI, and push the changes to the github repo
Regenerate the public listing of the whole archive at lb42.github.io/Lacy/report.html by running python3 Scripts/refreshRepo.py and pushing associated changes to the lb42.github.io repository.
Put the kettle on.

Tom Taylor confirms the importance of Lacy’s authors

When in 1863 Tom Taylor wrote a brief introduction for a printed edition of his runaway box office hit The Ticket of Leave Man, he seems to have been primarily concerned to defend himself against the charge of having simply translated or adapted its French original. “As much has been said apropos of this drama, on the subject of originality in play writing, I wish to submit here a few remarks on this matter….” he writes before rehearsing the familiar argument that he had always been explicit about the sources of his inspiration, and that, like Shakespeare, he had felt free to re-present plots and stories written first by others in his own style and using his own ingenuity. So far, so predictable; though indeed the argument is a fair one. But he goes on to make a much stronger claim for the theatre of his day, rebuffing the common wisdom about the decline of the theatre:

“Lastly, I may express my belief, however startling the avowal may be thought, that there has been no period, for the last two centuries, in which invention and activity have been more conspicuous in the dramatic field than during the thirty or forty years which include the epoch of such dramatists as Miss MITFORD*, SHERIDAN KNOWLES*, BULWER LYTTON, JAMES WHITE, JERROLD*, BROWNING, G. DARLEY, SEARLE, MARSTON*, HORNE*, LOVELL*, TROUGHTON*, BELL, Mrs. GORE, SULLIVAN*, PEAKE*, POOLE*, HOOK, PLANCHE*, CHARLES* and GEORGE DANCE, the MORTONS**, MARK LEMON*, BUCKSTONE* , SELBY*, FITZBALL*(who, whatever may be the literary quality of his plays, has given evidence of genuine romantic invention). BERNARD*, COYNE*, OXENFORD*, SHIRLEY BROOKS*, WATTS PHILLIPS*, and those peculiar products of our own time, the burlesque writers, like the Brothers BROUGH*, and Messrs. BYRON* and BURNAND*.

T. TAYLOR. LAVENDER SWEEP, Wandsworth, June, 1863.”

An interesting thing to note about this alleged pantheon, apart from the robust way in which Taylor uses it to assert the “invention and activity” of his contemporaries and colleagues, is perhaps that the vast majority of these dramatists are (like Taylor himself) republished in Lacy’s Acting Edition: indeed, only Bulwer, White, Darley, Searle, Bell, and Mrs Gore – 6 out of the 30 names – are missing from it. This confirms (if confirmation were needed) the extent to which the LAE was perceived (at least by some) as the embodiment of Victorian theatre of a certain “epoch”.

A Productivity Boost

Today the total number of Lacy Plays in DRACOR format has exceeded my original goal of 150 (10% of the whole LAE) and stands at 155 titles. This is almost entirely due to a much improved and streamlined workflow, depending on cautious use of the Gemini LLM to do what it’s good at (text recognition and structuring) while restraining its tendency to summarize, tell lies, lose bits of the text etc. I am now fairly confident I can produce a usable TEI version of any play in a day or less. I use a four stage pipeline:
• extraction of plain text
• addition of basic TEI tagging
• proof reading and correction
• improving and standardizing header metadata

Text extraction

The “Victorian Plays Project” produced modern PDF files of about 300 Lacy titles, derived from a copy of the LAE digitized by the Birmingham Library. These files were (apparently) carefully proof read, and extracting the text from them therefore seemed like a good first step. In my original workflow, the OCR step was done by Abbyy, with an output in Word, which I subsequently converted to TEI by means of a home-grown set of XSLT transformations and much fiddling about. Most of the 300 VPP titles were converted to Word format in a batch process run over a few days back in 2023 (thanks Huma Num). I subsequently realised that I could produce a usable text file from the Word output using Libre Office directly, and let Gemini do the hard stuff. This would in any case be necessary for the 1000+ texts for which no VPP PDF file existed. These were available as PDF images from Hathi Trust, faithfully reproducing the wonky typography of the original and entirely uncorrected. Initial experiments suggested that a lot of work would be needed to train Abbyy to do a decent job of OCRing these, or to correct its untrained output. However, for many of my texts, Hathi Trust or the Internet Archive (or both) also offered a “plain text” version which was completely adequate. In some cases, this was derived from a source later than the VPP would have used, but those multiple copies of a given title I looked at all seemed to be faithful reprints, apparently derived from the same plates, showing the same misprints, but just varying in their state of preservation or the fidelity of their digitization.

Basic TEI tagging

Gemini Pro is quite well informed about the possibilities of TEI markup. It knows about stage directions and speeches and speakers and even the difference between prose and verse. For the most part it does well at introducing appropriate tagging to mark them. As previously noted, its activities need to be constrained to what it is good at, and limited to manageable amounts of data. After a fair number of false alarms and other nonsense, I managed to produce a set of “prompts” that did more or less what I wanted, and to embed this into a python script to hide the whole nasty business away. Full disclosure: a text version of the script is here (but it won’t work for you until you add your own API key to it, and rename it). For a text with identifier 999, the script is invoked by saying “python L0999” and provided there is a line somewhere in a file called “L0999.txt” that says “Act 1” or similar, this usually gives me a file optimistically called “L0999-final.xml” to move on to the next stage.

Proof-reading

This is where the time goes. I open the XML file in oXygen, and find the page images corresponding to the LAE edition of it. I then go through methodically checking that all the text is there and in the right order and that any occasional weirdness in the original has not been corrected. I usually also have to check for page break tags, since these are not always recognised by Gemini, and for highlighted phrases, since my prompts do not request them (and I don’t want them when they appear within a stage direction anyway). And I also usually have to normalize speaker names as mixed case, since the small caps used in the source are variously transcribed in all upper case or a random mixture of cases. And distinguish hyphens from em dashes. And fix end of line hyphenation. And join together speeches fragmented by a page break or stage direction. Much, if not quite all, of this is painlessly handled by oXygen’s basic editing tools, of course. I do some more specialised checking of the front matter (is the title page complete? Are first performance details correctly tagged? Are genders indicated for each role in the castList? Are miscellaneous things like costume lists, setting notes, etc. properly tagged? And (and this is the important bit) I read the text carefully. Oh what fun.

Metadata

Once the TEI text is reasonably stable, I enhance its TEI Header, using a script imaginatively named “enrichHdr.sh”. Two ancillary files are used for this: a “participant description”, the creation of which is documented at https://foxglove.hypotheses.org/1063; and the Lacy bibliography visible at https://lb42.github.io/Lacy/lacyCatalogue.html .The script copies the source description for the play from the full Lacy bibliography (which is the master copy), along with the participant description. It then adds pointers from each speech in the play to its associated speaker. The result is then validated against the current Dracor schema.

Plays-per-month

Here’s a hastily produced graphic, showing how productivity has ramped up since switching to this new pipeline. Note that the month being tracked here is that in which a title first acquires a TEI header, not that in which it was last updated; this would not be very interesting, since all the headers get checked and updated as a batch every now and then. It’s therefore unclear how to determine which update counts as “finishing” a title; however, the rate at which new titles are appearing is clearly going up…

Taking my pick

The availability of my new pipeline encourages me to think again about which titles to choose. This month, having noticed that the number of titles taken from some volumes far exceeds the number from others, I did a little checking, and found that of the 100 volumes, only four have supplied more than 4 or more TEI titles so far; only 10 provide 3 titles, and 24 volumes supply 2. Which means that a whopping 61 (nearly two-thirds) contribute only one title. There is also one sad volume (78) which has not yet been sampled at all, but I’m currently working on that.

Wasting my Tokens

I seem to have spent most of 2026 so far prompting Gemini to do the right thing with my Victorian plays. As previously mentioned, our working relationship got off to a rocky start, largely due to my unrealistic expectations of its capacity, and its reluctance to explain (or even admit to) the implications of its capacity limitations. A lot of time was wasted in prompts requesting it to process the whole of a PDF file, which received glowingly positive and entirely mendacious responses, along the lines of “Here is the whole play in TEI XML”, followed by a richly encoded version of the first ten pages, or a richly encoded (but entirely mendacious) summary of the text, or something like it. I don’t now recall the moment at which I realised I was (as it were) talking to the over-worked idiot bot on the helpdesk, rather than the system designer, nor how I finally managed to get my prompts put through to something more helpful, but some time in mid January, the conversation did become a bit more productive.

Firstly, I established that my rather complicated prompts would deliver the goods for a shortish play (such as a farce) without problem – but only for five or six of them in a row, after which rate limits set in, and the model would be unable to deliver more than a page or two.

Next, I decided to simplify the task by starting from a plain text transcription rather than a PDF file, thus removing the complication of an initial OCR step. Plain text versions of at least some Lacy plays are not hard to track down on the Internet, so I used those for my initial experiments. For the rest – well, producing a plain text version from PDF is a largely well understood operation these days, requiring no special LLM magic, or so I hope. We shall see.

Then (and maybe this was the break through moment), I started conversing with Gemini via its API rather than by typing idiotic prompts into the web interface. This meant writing a script in python, a language I detest; fortunately Gemini has no such scruples, and is indeed designed (apparently) to guide unwilling amateurs through the thorny thickets of coding, It even helped me to set the run time environment up properly, using something called anaconda, a tool I have never understood.

My current script (it’s in the repo) uses gemini-2.5-flash, and operates piecemeal on the text. First it creates a minimal header, for which it needs to extract only a title and author from the first 2000 characters of the input file. Constructing a proper TEI header is something I do later, using other resources. Next it splits the text up into chunks, using a simple-minded regex search for strings such as “ACT I” as delimiter. Everything in the first such chunk is to be tagged using front matter tags such as <titlePage>, <castList> etc. Everything in the second and subsequent chunks is to be tagged using body tags such as <div>, <sp>, <stage>` etc. The script makes one API call for each chunk identified and appends its output to a local file. The API calls for front and body matter use different “system instructions” to prioritize what the LLM will encode in its output, which also simplifies the overall task.

This approach means I can produce a usable version of a plain text in a matter of minutes. The time needed for proof reading, correction against the PDF original, and validation against the Lacy schema will of course depend on how faithfully the plain text version reproduces the PDF, but rarely takes much longer than reading the play attentively.

What can go wrong? The method depends entirely on the reliability of the chunking algorithm: if there are too many divs or none at all, the results will not be much use. And of course, if the text transcription is some distance from the printed copy text, there will be corresponding a lot of hand editing to do. But so far, the results are encouraging. As of today, there are 124 titles in the TEI repository; 11 of which were added in January 2026, and 12 in December 2025; which compares well with the 10 added for the whole of 2024. It remains to be seen if this rate keeps up when starting from PDF. Just for fun, here’s a hastily prepared graphic showing PPM (play-per-month) productivity so far.

I have no idea what underlies the peak in the winter of 2023: probably a lot of low-hanging fruit.

Representativeness and how to check it

As the number of TEI encoded titles in Digital Lacy is about to reach a milestone, it seems a good moment to consider its representativenessity. So I have been dusting off my utterly decayed spreadsheet knowledge, and doing some counting. I can compare (fairly easily) the following characteristics for all plays in the LAE:

– first performance date

– size

– genre

Other possibly useful categories might include

– popularity as measured by the number of performances or revivals

– text type (translation, adaptation, burlesque etc.)

– resource usage (size of cast, effects, music etc.)

Reliable and consistent data covering these is however rather more difficult to compile.

First performance date

This table shows the number of titles claiming a first performance date for each decade between 1820 and 1880 in each of the complete LAE, the subset of it included in the VPP, and the subset of it available in TEI format. It shows that there are 1512 titles in the LAE, of which only one was first performed in the 1880s, a decade barely represented in the LAE. Moving back in time, however, 99 titles (or 6.6% of LAE titles) have a premiere dating from the 1870s. The table allows us to compare this with the equivalenmt figures for VPP (which has a total of 335 titles) and TEI (total so far of 99). We can see that the TEI sample (at 5%) is close to the Lacy proportion, while the VPP sample (2.3%) really should have included more titles from this decade.

	total	1880s	1870s	1860s	1850s	1840s	1830s
LAE	1512	1	99 (6.6%)	442 (29%)	406(27%)	186(12%)	164(11%)
VPP	335	1	8 (2.3%)	115 (34%%)	122(36%)	43(13%)	28(8.3%)
TEI	99	0	5%	20%	35%	13%	12%

For each decade, except the 1860s, the TEI proportion is close to the proportion of titles found in the whole Lacy corpus, which is encouraging. It does appear however that we need to add one or two more titles from the 1860s – and maybe remove some from the 1850s.

For titles premiered up to and including the 1820s, the situation is slightly different:

1820s	1810s	1800s	18th c.
91(6%)	32(2%)	16(1%)	15 (1%)
11(3%)	2(0%)	0 (0%)	3 (0%)
2	2	0	4

These figures suggest that VPP has made a determined effort to exclude older titles; the TEI however seems to over-represent titles from the 18^th century, some of which might perhaps be removed.

Size

As an approximation for size, we use the number of pages in the printed edition, allocating each title a value of Large if it has more than 45 pages, Medium (25-44 pages), or Small (fewer than 25 pages).

Calculated in much the same way as above, the following table shows some discrepancies in the the TEI sample:

	total	large	medium	small
LAE	1512	209 (13.8%)	547(36.1%)	755 (49.9%)
VPP	335	50 (14.9%)	125 (37.3%)	160 (47.7%)
TEI	99	14	26	59

The TEI sample has too many small texts, and not enough medium sized ones. (I admit it: the short texts are generally funnier and easier to encode rapidly)

Genre

Deciding on a genre for each play is a tricky matter, which I explored in this blog some time ago (see https://foxglove.hypotheses.org/828 and https://foxglove.hypotheses.org/876 ) Amongst other problems, the terminology used by the theatre (and hence the plays) is not stable. It’s clear that broad categories such as “drama”, “comedy” etc. fragment into particular niche terms (“melodrama”, “equestrian drama”, “burlesque”, “burletta” etc.) which then go in and out of fashion. A classification derived from formal properties (presence or absence of songs, use of verse, presence or absence of traditional characters, etc.) might well be preferable but is hard to implement (there’s a boot strapping issue here). So I have fallen back (again) on my own simplistic 15-category taxonomy. Which produces a table like this:

	Total	FARCE	DRAMA	COMEDY	SPECTACLE	BURLESQUE
LAE	1512	461 (30.49%)	339 (22.42%)	190 (12.57%)	121 (8.0%)	86 (5.69%)
VPP	335	93 (27.76%)	109 (32.54%)	25 (7.46%)	31 (9.25%)	27 (8.06%)
TEI	99	36	21	11	7	5

MELODRAMA	COMEDIETTA	BURLETTA	OPERA	TRAGEDY	PANTO	BALLET
75 (4.96%)	75 (4.96%)	50 (3.31%)	47 (3.11%)	38 (2.51%)	11 (.73%)	2 (.13%)
17 (5.07%)	11 (3.28%)	10 (2.99%)	1 (.3%)	4 (1.19%)	3 (.9%)	0 (.0%)
6	2	4	1	4	1	0

This, to my surprise, shows that the proportion of TEI titles to LAE titles is more or less identical (i.e. they differ by about 1%) for all but a few categories, notably (and unsurprisingly) FARCE which is over-represented in TEI, as is (unexpectedly) TRAGEDY, while COMEDIETTA is mystifyingly under-represented.

Conclusion

The astute reader may have been fretting somewhat that I have presented as percentages values based on a total of only 99, rather than 100. This is because I am about to add the 100^th title – and will of course be making sure that this one, at least, contributes to a balancing of the corpus. On the basis of the foregoing, I will need to select a title which is a COMEDIETTA, of medium length, and was first seen on stage during the 1860s. A quick Xpath shows me that there are 17 candidates: I narrow this down to six by adding the requirement that the title also appears in the VPP subset.

As we say in Xpath `div[@type='work' and ends-with(@ana,'COMEDIETTA') and starts-with(@ana,'M') and bibl[@type='printSource']/idno[@type='vpp']] … and then start browsing to choose one by an author as yet unrepresented, or with an interesting subtitle, as usual.

Hunting down the dramatists

Of the 332 distinct author names attested on the title pages of Lacy’s Acting Editions, 30 remain stubbornly invisible elsewhere. In some cases there just isn’t enough information to identify them (the “Captain Arbuthnot” credited with the authorship of the burlesque L’Africaine; or, The Belle of Madagascar played at Covent Garden in October 1865 is a case in point) ; in others the name is just too common (“G. Smith” anyone?). But that still leaves 302 authors to investigate.

All of these works have been catalogued in some library somewhere however, which means (I naively supposed) that some cataloguer will have chosen a VIAF code for their authors. It seemed worthwhile therefore investing effort in trawling through the VIAF lists to pick out a code for each of my authors. No doubt ChatGPT could have done it quicker; in any case, at the end of the trawl, I had VIAF numbers for 300 of my authors – and only two mysteries. Firstly, George Marsh, author appropriately enough of “Lost and Found” and two other Lacy titles, who is listed as a member of the Dramatic Authors Society but is otherwise hitherto untraceable; secondly, George Roberts, who contributed seven titles under the stage name of “Robert Walters” including “Under the Rose” and “An Ample Apology”, but is listed only by the Era Almanac and in the ATCL database, described below.

Of the 300 authors anointed with a VIAF code , after a few days fighting with SPARQL (tx Stuart) I was able to endow 214 with a Wikidata identifier, which I had rather hoped would serve as the key to all manner of other useful things. It seems odd that over a quarter of the authors known to VIAF (86 of them to be precise) are otherwise unknown to Wikiworld, but there you are.

As might be expected, many of the authors (176 at the last count) have entries in Wikipedia. Some of these (but by no means all) provide links to other, perhaps more respectable, sources, notably the Dictionary of National Biography, in both its (wikisource-hosted) original and (paywall-protected) OUP versions. I was able to harvest links to ODNB entries for 144 of my authors via Wikidata; finding links to the Wikisource version was done by hand, which took a day or two. At the end of this part of the exercise, I had found ODNB biographies for 147 authors and DNB entries for 127. Unsurprisingly there were 22 authors with an entry in ODNB but not in DNB; less so, there was at least one case of a DNB author apparently purged from the ODNB, though present (admittedly as an appendage to the entry for his wife) in the original DNB. Where did you go wrong, Edward Stirling (1809-1894)?

I also checked all my author names against Boase’s Modern English Biography, which is a lot easier to search, and found 99 of them there. Interestingly, 30 of these were authors unknown both to ODNB and DNB.

The long-running theatrical newspaper The Era published in its annual Almanac volumes for 1868 and 1869 a set of short biographies of (then) living dramatic authors, ostensibly written by Edward Ledger. These brief notices provide an interesting contemporary perspective on 53 of my authors; 15 of them unknown to ODNB.

Impressively detailed exhaustive bibliographies and biographical notices for 27 of my authors are provided (behind a paywall) by Gale’s Dictionary of Literary Biography. In the absence of any such thing as an agreed canon of dramatic writers, it is difficult to see on what basis authors have been chosen for this particular collection. It cannot be anything to do with the number of titles they produced, since some of the most prolific of Lacy’s authors are conspicuously absent. Nor can it be related to the fame of the writer as witnessed by the number of other reference works mentioning them, since such well-known figures as Edmund Yates, Mark Lemon, G.H. Lewes, and even J.R. Planche are also missing.

I also sought traces of my authors in some less obvious quarters. The ATCL database (http://www.victorianresearch.org) is a wonderful resource documenting the production of the victorian novel, in the widest sense; 51 of my dramatists were not averse to churning out a few novels too, and hence feature in it; 6 of them are unknown to ODNB – indeed, one of them (Captain George Hatton Colomb) is listed by ATCL and nowhere else. ATCL also provides an interesting list of novelists who also wrote dramas; not all of these were Lacy authors, of course. It also indicates when the author concerned has a more detailed notice elsewhere.

Another wonderful resource, the Curran index (https://www.curranindex.org/) documents the wider world of Victorian journalism; it enabled me to determine that at least 66 of my dramatists also earned a living by contributing articles to periodicals indexed by Curran et al.

And finally, a footnote on the Dramatic Authors’ Society, which was “was the only organisation in the nineteenth century specifically devoted to the needs of dramatists. “ (Stephens) . Founded in 1834 after the Dramatic Authors Act became law, the DAS collected and redistributed fees on behalf of its members. All but two of its original 23 members were Lacy authors. During its fifty year life, the Society published lists of its members and their works with an indication of associated fees, for the use of provincial theatres. Putting together such lists as I have found, it seems that at least 91 of my authors are or were at some time DAS members.

The data thus far collected is feeding into a web page I am constructing: if you want to see its prototype version, check out the web site. And if you have information to fill my gaps, please get in touch by commenting on this page!

Lacy’s Lost Authors

If we consider Lacy’s Acting Edition as a microcosm of Victorian theatre (albeit with some biases), the authors of its nearly 1500 titles seem to be worth investigating a little. Who wrote the plays that Lacy felt confident he could sell? What sort of people were they? Where did they spring from? John Russell Stephens’ definitive historical survey The profession of the playwright (CUP 1992) tells the story at a macro level, backed up by all manner of bibliographical and biographical pointers which I have plundered shamelessly to build up my own micro-level survey of Lacy’s people. To begin with something basic : how many of these authors were there? How much did each one write? And, critically, are they known to other biographical sources?

By my reckoning, the LAE cites 333 identifiable authors, excluding obvious spooks such as “John Doe”, or the moderately prolific “Anon”, but including co-authors, translators, and adapters if they are credited on a title page. Pseudonymous authors are counted only once, usually under their real name, where this is known. Of these 333, nearly all (296) now have a VIAF identifier, which seems like a fairly reliable authority file for this kind of material. By reliable, I mean that (so far as I have been able to check) the VIAF heading cites someone with the same name, either as author of a work with the same title, or with birth and death dates which seem plausible taking into account the date of its first performance.

For nearly half of these authors (188), I have tracked down some sort of online biography, mostly in Wikipedia, which of course links to other sources as well. Other sources (notably Boase and the original DNB) have yet to be pillaged systematically, but it seems unlikely that they will add many more names to those already found in Wikipedia. I have however added links to 52 of the brief biographical notes by Edward Ledger originally published in the Era Almanac: these provide a contemporaneous rather than retrospective view of their subject. Putting all this together has enabled me to add dates to 226 entries, rather more than two-thirds of the authors.

There remain, of course, unresolved mysteries. For about 30 authors, I still know nothing beyond the titles and first performance information given in the Lacy Catalogue. Here’s a summary list in the forlorn hope that someone else might be able to make a suggestion…

W0012: | Arbuthnot, Captain | ARBUTHNOT, Captain L1005 : L'Africaine; or, The Belle of Madagascar (Covent Garden 21 Oct 1865) W0079: | Danvers, H. | DANVERS, HENRY L0392 : A Conjugal Lesson (Royal Olympic Theatre3 July 1856. Licenced for performance (BL ms LCP_52960_A)) L0394 : A Fascinating Individual; or, Too Agreeable by Half (Royal Olympic Theatre 9 June 1856. Licenced for performance (BL ms LCP_52959_F)) W0092: | Edwards, Pierrepont | EDWARDS, PIERPONT L1195 : Honour Before Wealth; or, The Romance of a Poor Young Man (Wallack's Theatre, New York) W0094: | Ellis, George | ELLIS, GEORGE L0123 : Harlequin, Cherry and Fair Star; or, The Green Bird, the Dancing Waters, and the Singing Tree! (Royal Princess's Theatre 27 Dec. 1852) W0108: | Gilbert, H.P. | GILBERT, H. P. L1037 : The Court of Lions; or, Granada Taken and Done For (Unknown n.d.) W0112: | Gott, Henry | GOTT, HENRY L1224 : The Wizard of the Moor (Royal West London Theatre) W0345: | ‎Hale, William Palmer | HALE, WILLIAM PALMER L0056 : Godiva; or, Ye Ladye of Coventrie and Ye Exyle fayrie (Strand Theatre7 July 1851) W0123: | Hancock, William | HANCOCK, WILLIAM L0794 : John Smith (Royal Strand Theatre13 Jan. 1862. Licenced for performance (BL ms LCP_53011_B)) L0912 : "Stolen - £20 Reward" (New Royalty Theatre26 Dec. 1863. Licenced for performance (BL ms LCP_53028_B)) L0913 : Margate Sands (Royal Strand Theatre 10 Jan. 1864) L1152 : Mr Scroggins; or, Change of Name () W0124: | Hardwicke, Pelham | HARDWICKE, PELHAM L0177 : A Bachelor of Arts (Royal Lyceum Theatre23 Nov. 1853. Licenced for performance (BL ms LCP_52944_B)) W0132: | Hermann, Charles | HERMANN, CHARLES L0178 : Uncle Tom's Cabin (Theatre Royal Manchester 1 Feb. 1853) W0146: | Irwin, Edward | IRWIN, EDWARD L0442 : King O'Toole's Goose; or, The Legends of Glendalough (Queen's Royal Theatre, Dublin 24 March 1856) W0147: | James, Walter | JAMES, WALTER L1472 : A Return Ticket to the International Exhibition (Royal St. James's Theatre11 August 1862) W0186: | Lynch, Thomas J. | LYNCH, T. J. L0092 : The Rose of Ettrick Vale; or, The Bridal of the Borders (Adelphi 29 Oct 1829) W0194: | March, George | MARCH, or MARSH, M. GEORGE L1322 : Lost and Found (Royal Alexandra Theatre, Liverpool 27 June 1870) L1327 : Who's the Heir? (Theatre Royal, Sunderland 14 Oct. 1870) L1433 : The Shepherd of Cournouailles (St G. Liverpool 28 Apr 1879) W0199: | Martin, William | MARTIN, WILLIAM L1014 : Chang-Ching-Fou! Cream of Tartar; or, The Prince, The Princess, and The Mandarin (Luton 11 April 1864) W0216: | Morton, Edward | MORTON, THOMAS, Jr., L0232 : The Eton Boy (Theatre Royal, Drury Lane 29 Oct. 1842) L1056 : The Windmill (Theatre Royal Drury Lane 25 Jan. 1842) W0237: | Phelps,S. Jr | PHELPS, S., Jr. L0136 : The Obstinate Family (Theatre Royal Sadler's Wells21 Feb. 1853. Licenced for performance (BL ms LCP_52938_A)) W0238: | Phillips, Frederick | PHILLIPS, FREDERIC LAURENCE L0429 : A Bird in the Hand is Worth Two in the Bush (Royal Surrey Theatre 19 Jan. 1857) W0239: | Phillips, L. | PHILLIPS, L. L0686 : Marianne, The Vivandiere; or, The Mystery of Twenty Years (Great National Standard Theatre 3 Feb. 1851) W0263: | Rice, Charles | RICE, CHARLES L0255 : The Three Musketeers; or, The Queen, the Cardinal, and the Adventurer (Manchester 2 Aug. 1850) W0267: | Robertson, William L0213 : Faust and Marguerite () W0274: | "Rowe, Nicholas" | ROWE, NICHOLAS L1095 : The Orange Girl (Royal Surrey Theatre24 Oct. 1864) W0286: | Smith, G. | SMITH, G. L0877 : Make Your Wills (Theatre Royal, Haymarket1836. Licenced for performance (BL ms LCP_53022_AA)) W0294: | Spencer, George | SPENCER, GEORGE L1472 : A Return Ticket to the International Exhibition (Royal St. James's Theatre11 August 1862) W0305: | Thomas, J. | THOMAS G. L0324 : The Young Widow (Royal Adelphi Theatre1 Nov. 1824) W0319: | Walker, C.E. | WALKER, C. E. L0678 : The Warlock of the Glen (Theatre Royal, Covent Garden 2 Dec. 1820) W0325: | Welstead, Henry | WELSTEAD, HENRY L0384 : Giralda; or, The Invisible Husband (Royal Olympic Theatre 12 Sept. 1850) W0329: | Wilkins, John H. | WILKINS, JOHN H. L0137 : Civilization (City of London Theatre10 Nov. 1852. Licenced for performance (BL ms LCP_52935_T)) L0174 : The Egyptian (City of London Theatre18 April 1853. Licenced for performance (BL ms LCP_52938_DD)) W0331: | Williams, Arthur | WILLIAMS, ARTHUR L1435 : "Leave it to me" (Royal Surrey Theatre26 Dec. 1870) W0341: | Young, A. | YOUNG, A. L0309 : The First of May (Theatre Royal Sadler's Wells 26 Oct. 1849) W0344: | Younge, A. | YOUNGE, A. L0302 : A Village Tale (Theatre Royal Sadler's Wells 1 April 1850)

Going public(ish)

Despite silence here, I’ve been busy on all things Lacyesque for the last few weeks, getting ready to wheel out the first ever English-language presentation of what I now realize should be referenced as “The Digital Lacy Project” if it is to be taken seriously in the grooves of academe. So last week I gave said presentation, 20 minutes of introduction to the various ways in which 19^th c drama as attested by Lacy’s Acting Edition has been passed on to us, along with some possibly dubious statistics about its representativeness. The occasion was a bijou, one might even say “boutique”, conference to launch the Punch Pocket Books Project at the University of Caen Normandie, a funded collaboration between said University and others, see website. From my perspective, this was a great opportunity to revisit the afore-mentioned grooves from the perspective of the unattached researcher, rather than the annoying technical expert. I got to hear a couple of excellent plenaries and a range of research presentations, plus, as a bonus, participated in a practical workshop on pocket-book construction. The ambience was laid back and the company was genial, even though the technical infrastructure was occasionally hostile (nobody seemed to know how to make zoom behave properly in the classy new amphitheatre). I read my script in 22 minutes exactly, and flashed through my prepared slides without incident. Nobody visibly fell asleep; and there were a few encouraging reactions. I even got invited out for a post-conference drink, which was nice.

In preparation for this event, I started writing at least three different papers: one on the varieties of format in which the components of the LAE have been transmitted to us; one on the vexed issue of representativeness; and one focussing on the burlesque. To boil all these down to a 20 minute presentation was an interesting challenge; I expect to recycle some of the inevitable rejectamenta on this platform anon.

Another Fine Mess…

As previously mentioned, I have been trying to mangle Allardyce Nicoll’s Handlists into a tractable database for what seems like forever. Here’s the latest and hopefully last update.

Some of the entries are just disambiguating cross references: these are (or should be) marked as eType=’note’. Some of them are partial entries including a reference to another entry which may or may not contain the same data: these are (or should be) marked as eType=’ref’. This classification of entries was carried out by the addAtts script early on in the pipeline; the same script also added a magic key for each entry to facilitate matching up Lacy and Nicoll entries, but ignored entries with eType=’ref’ for some reason. I did not notice this gaffe till later, much later, after I had spent weeks on the next stage of the pipeline, (the clever bit of matching up Lacy and Nicoll entries, which involved a lot of manual intervention)

Here’s what I did to fix that blunder…

saxon -xi entries.xml addAttsAgain.xsl > oops.xml (run a corrected version of addAtts script (renamed addAttsAgain) to generate a file of corrected entry elements for the entries of eType ref, now renamed as eType=part).
saxon oops.xml addWhen.xsl > oops2.xml (run the existing addWhen script to add a @when for these new entries)
saxon allEntries.xml attributePatch.xsl > temp.xml (run attributePatch script to produce an improved version of allEntries.xml. )

The text is now quite intelligently tagged, and there is a (non TEI) schema to describe its markup. I need to do more on its documentation, but there is an ODD.

Dates

Round about now, I realised that @when values were missing for many titles, and were mostly not in ISO format, which matters, partly because I can now use my ODD-defined schema to validate the file but mainly because it would be nice to sort entries correctly by date. So I embarked on the long tedious process of dating the entries a bit more consistently.

Nicoll represents dates in one of three different ways.

Where the full date of a performance or a license is known, it is given as DD/MM/YY or (occasionally) DD/MM/YYYY. This is easy to identify and extract to the @when attribute for the entry
Where the date is only partial, it may appear in the form MM/YYYY. This is more problematic.
Where the only date available is that of a publication, it will usually be in the form YYYY, possibly in brackets. I wrote a script to extract these to the @when attribute too.

There are quite a lot of OCR errors to correct (I instead of 1, u instead of 11, s instead of 5, redundant blanks or nonexistent punctuation, and so on). Many of these could be fixed with regexp search and replace. I also found cases where the end of a printed line had simply been ignored, which were more difficult to detect.

Eventually, I have plausible dates for as many as possible of the datable entries, in one or other of the three formats specified. I run another script to convert them all to a kosher iso format i.e. YYYY, YYYY-MM, or YYYY-MM-DD, and then validate. A surprisingly large number trip at this last hurdle, mostly because of a previously unspotted OCR error, but this does throw up five cases which can only be attributed to lax proof reading at Cambridge University Press. These five include obvious nonsense like “32/2/1822” given as the date for the Drury Lane performance of Edward P. Knight’s “The Veteran Soldier”, and more tangled cases such as “29/2/1823” given as the date for a performance at the Adelphi of Moncrieff’s “The Secret”. Sorry, Allardyce, but 1823 was not a leap year, so this cannot be true. Moreover, according to the Adelphi Calendar (https://www.umass.edu/AdelphiTheatreCalendar/auth.htm) , on 28 Feb 1823, the theatre was dark for Lent… and the same source is stubbornly silent on the existence of a play of this title and authorship anywhere. So someone is mistaken.

Just to put those peccadillos into perspective : by my reckoning, there are now 24,351 distinct entries in the Nicoll database, of which 24,301 are apparently now correctly dated . Fifty are genuinely undated; five have impossible datings. A pretty good error rate.

Multiple authorship

As I may have remarked before, the entries in Nicoll’s Handlists are of quite a few different types. Some of them are just cross references, supplying the name under which a pseudonym has been indexed but not documenting any particular performance; others (quite a few) are partial entries, associating a performance or publication for one author with an entry for the same performance or publication listed under the name of the “main” author in a collaboration. Nicoll supplies the following definition: “I have adopted the principle of placing the main entry of any particular play under the name of that author whose name appeared first in the play-bill, newspaper advertisement or review from which information regarding authorship was obtained”. For my purposes however, all these additional entries simply inflate the number of performances etc. (by a factor of nearly 10%) and are unnecessary for a resource in digital form. I therefore tag them differently, and process the multi-author entries so that all the authorship information is accessible in the same place. For example, here is the “main” entry for a play with multiple authorship:

<entry when="1897-09-06" xml:id="N08816" eType="multi" n="ohsusannah_AMBIENT">
  <class group="FARCE">F.C.</class>
  <author>AMBIENT, MARK </author>
  <title>Oh! Susannah! </title>
  <perf>Eden, Brighton, 6/9/97; Roy. 5/10/97.</perf>
  <lic>L.C. </lic>
  <bib>French </bib>
  <note type="auth">[Written in collaboration with A. ATWOOD and R. VAUN .]</note>
 </entry>

The Handlist also contains two fragmentary entries, one for each of the two co-authors:

<entryFrag when="1897-09-06" xml:id="N08946" eType="part" n="ohsusannah_ATWOOD">
<author>ATWOOD, ALBAN </author>
<title>Oh! Susannah! </title>
<perf>Eden, Brighton, 6/9/97.</perf>
<note>See M. AMBIENT.</note>
</entryFrag>

<entryFrag when="1897-09-06" xml:id="N19381" eType="part" n="ohsusannah1_VAUN">
<author>VAUN, RUSSELL </author>
<title>Oh! Susannah 1 </title>
<perf>Eden, Brighton, 6/9/97.</perf>
<note>See M. AMBIENT.</note>
</entryFrag>

I wrote a script to combine all these to produce a new multi-author entry, like this:

<entry when="1897-09-06" xml:id="N08816" eType="multi" n="ohsusannah_AMBIENT">
  <class group="FARCE">F.C.</class>
  <author>AMBIENT, MARK </author>
  <author type="also">ATWOOD, ALBAN </author>
  <author type="also">VAUN, RUSSELL </author>
  <title>Oh! Susannah! </title>
  <perf>Eden, Brighton, 6/9/97; Roy. 5/10/97.</perf>
  <lic>L.C. </lic>
  <bib>French </bib>
  <note type="auth">[Written in collaboration with A. ATWOOD and R. VAUN .]</note>
 </entry>

Note that e.g. “R. VAUN” now appears as “VAUN, RUSSELL”. Achieving that particular coup de main involved quite a lot of XSLT juggling before I found a 99% successful solution.

In the process, I found only the following five cases where the author name referenced by Nicoll was hard to find in the Handlists.

“TAIT” (but I found him in the errata list for vol 4)
“PINCROFT” Confusingly, this exists as a pseudonym for BANERO J.M. , which is also the name of the main author. Something wrong there: Nicoll nodded.
“MOUNTJOY” No other sign of this pseudonym.
“Corri” must be Clarence Collingwood Corri, who supplied the music for George Sims’ 1899 farce In Gay Piccadilly. Dan Leno was in it.
“CARGILL, G.B.” I have not yet found any other sign of this co-author.

The next challenge

It is definitely time to revisit the Lacy catalogue. What’s to do with the 196 catalogue entries for which no corresponding entry has shown up in either of Nicoll’s Handlists? Are they all old stuff, first published or performed long before 1800, and therefore reasonably omitted from the Handlists? Or are there some rogue components amongst them? Time will tell. Meanwhile, here’s the current state of affairs, according to my reportCounts script.

Today there are 24351 entries in this file, of which ....
24350 are classified
22995 are plain old entries 
some of which are unclassified
1356 multi-authored entries
1750 fragmentary entries constructed from cross references
30 fragmentary entries constructed from notes
51  notes, xrefs,  and comments
18784  with perf data 
15896  with lic data 
4158  with bib data 
1169 have been matched out of 1300 Lacy references
1327 catalogue entries linked to a Nicoll entry out of 1511

391 entries have a bib referencing Lacy but no @matches
493 have a female author

Multiple authorship

As noted previously Nicoll’s Handlists are organized by author name, which makes them manageable, but also can be seriously misleading. In particular, where a play is to be credited to more than one author, Nicoll’s practice is to repeat the information about the play in a second slightly degenerate entry, thus inflating the number of entries in the Handlist. Here for example is the “main” entry for a play co-authored by A’Beckett and Lemon:

<entry type="Bsq.">
<author>A'BECKETT, GILBERT ABBOTT</author>
<title>The Knight and the Sprite </title>
<note type="perf">Strand, M. 11/11/1844</note>
<note>L.C. 9/11/1844.</note>
<note type="auth">[Written in collaboration with M. LEMON]</note>
</entry>

And here is what I have unkindly termed the “degenerate” entry for same:

<entry type="Bsq.">
<author>LEMON, MARK</author>
<title>The Knight and the Sprite </title>
<note type="perf">Strand, M. 11/11/1844</note>
<note>See G. A. A BECKETT.</note>
</entry>

I assume Nicoll’s rationale for this redundancy is to make it easier to find everything written by a given author when flipping through the pages of a printed volume. But this makes much less sense in a digital resource. What we would rather see (I think) is an entry which makes explicit its multiple authorship: like this

<entry type="multi">
<class>Bsq</class>
<author>A'BECKETT, GILBERT ABBOTT</author>
<author type="also">LEMON, MARK</author>
<title>The Knight and the Sprite </title>
<note type="perf">Strand, M. 11/11/1844</note>
<note>L.C. 9/11/1844.</note>
<note type="auth">[Written in collaboration with M. LEMON]</note>
</entry>

(Note that to get there I have had to rethink the way I encode Nicoll’s genre tags, initially by moving them to an element of their own rather than using the @type attribute of the <entry> element. And note also that I am preserving those arguably redundant <note type=”auth”> elements so I can tell if something goes wrong)

I have spent the last week or two slowly making this possible. Slowly because I am slow, but also because it is not entirely straightforward to translate the string “M. LEMON” (as given in the note in the main entry for A’Beckett) into “LEMON, MARK”, which is the handle used on other main entries for the distinguished editor of Punch. (The same would apply, of course, if I decided to use the note within the degenerate entry to effect the join: I would then have to map “G.A. A. BECKETT” to “A’BECKETT, GILBERT ABBOTT.” ) And these are easy cases: Nicoll’s canonical format for names can get quite complicated. Consider, for example, “YORKE, ELIZABETH, Countess of HARDWICKE” or “ADDISON, Captain (later Lieutenant-Colonel) HENRY ROBERT” … Anyway, I made the job easier for myself by extracting from the entries a lookup table mapping name components (as given by notes within main entries) to canonical full names: like this

<author f="49">
<s>LEMON</s>
<w>MARK</w>
<str>LEMON, MARK</str>
</author>

This all worked quite satisfactorily for the 1800-1850 entries, for which there are only 58 additional name entries to handle, though getting to the point of being reasonably confident in that number took much longer than you might think, involving as it did quite a lot of OCR error correction.

However, things got much more challenging when I looked into the 1850-1900 entries. Firstly, there are many more entries to deal with: 1299 cases of “collaboration” . Secondly, some cases (34 to be exact) use an abbreviated form like this:

<entry type="P.">
<author>BYAM, MARTIN </author>
<title>The Babes in the Wood </title>
<note type="perf">R.A. Woolwich, 14/12/57.</note> L.C.
<note type="auth">[Written in collaboration with F. GRAHAM and W. T. VINCENT.]</note>
</entry>

This main entry will need to get two additional author elements, one for “F.GRAHAM” and one for “W.T. VINCENT”, not just one – which means revising my simple-minded XSLT script yet again. And it will also have to handle notes like this without too much fuss:

<note type="auth">[Written in collaboration with A. R. SMITH, F. TALFOURD and W. P. HALE.]</note>

The script does a good job of alerting me to cases where Allardyce has apparently nodded, and named as a collaborator someone who does not appear anywhere in the rest of the Handlist. This happens precisely once in the 1800-1850 volume, but seemingly many times more in the later volume. However, on examination, many of these discrepancies are a consequence of my cavalier editing praxis. Things like the kinds of quotation marks used to flag up pseudonyms, or whether or not surnames can contain spaces, return to bite me. Others are caused by OCR failures – occasionally lines seem to have just dropped out.

And, further to keep me on my toes, I have now discovered that there are three cases in which Nicoll gives up entirely on this painstaking method of documenting multiple authorship. The first concerns 18 titles to be attributed to the pseudonymous “Richard Henry”: these all appear once only under “HENRY, RICHARD”, like this

<entry type="Bsq.">
<author>“HENRY, RICHARD" [RICHARD BUTLER and H. CHANCE NEWTON] </author>
<title>Lancelot the Lovely; or, The Idol of the King </title>
<note type="perf">(Aven. 22/4/89).</note> L.C.
<note type="music">[Music by J. Crook.]</note>
</entry>

None of these 18 titles is listed, however, under NEWTON, nor indeed under BUTLER. A further, and apparently disjoint, batch of titles is listed under “NEWTON, H. CHANCE (“RICHARD HENRY”), I think I am going to pretend I haven’t noticed them. Likewise this one:

<entry type="D.Sk.">
<author>GORDON-CLIFFORD, E. and H. </author>
<title>A Black Dove </title>
<note type="perf">P’s. H. Kew, 12/9/94.</note>
</entry>

Handlists made handier

I have been down a deep deep rabbit hole for the last week or two trying to get my XML-tagged versions of Allardyce Nicol’s two Handlists into shape. Here is an interim report.

What is an entry?

One problem has to do with the actual content of Nicoll’s Handlists. What exactly do they list? Although this is essentially a record of performances, it is organized very much by author. There are, for example, about 30 entries which don’t refer to any specific play, but are merely there to indicate the preferred form of an author’s name: like this one

<entry>“LAWRENCE, SLINGSBY.” See G. H. LEWES</entry>

Multiple authorship is also a problem. An entry like this one is straightforward enough:

<entry type="D."><author>ANDERSON, JAMES R. </author><title>The Robbers </title><note type="perf" >D.L. 21/4/51</note>. L.C. D.L. 26/12/45.</entry>

Inter alia, this tells us that there was a performance of a drama called “The Robbers” at Drury Lane theatre on 21 April 1851, and that the author of the piece is recorded to be James R. Anderson.

But there are also entries like this one:

<entry type="F."><author>ATWELL, E. </author><title>A Stuffed Dog </title><note type="perf">Park. H. Camden Town, 2/11/89</note>. See J. A. KNOX.</entry>

This one tells us that “A Stuffed Dog”, written by E. Atwell, was making them roar at the Park Theatre in Camden Town in November 1889. If we look for Mr J. A. Knox, we find another entry, apparently for the same performance:

<entry type="F."><author>KNOX, J. ARMORY </author><title>A Stuffed Dog </title><note type="perf" >Park H. Camden Town, 2/11/89, copy.</note>. L.C. [Written in collaboration with E. ATWELL .]</entry>

On the face of it, if I want to determine how many Farces are listed in the Handlists, for example to determine the waxing or waning of this particular type of performance over time, I need to be wary of cases like this one, where a single farce has multiple authors, and therefore gives rise to multiple entries: both these entries refer to a single performance, so should only be counted once.

How serious a problem is this? Out of 25 thousand- plus entries, (25,632 to be exact) I find that there are 1346 entries containing the word “collaboration” and 1738 containing the word “See ”. Most, but not all, of them point to a collaboration entry which references the same performance of the same play. There are only two entries in which the word “See ” really appears as part of a title, but there are maybe a dozen or more other types of cross references, for example to plays renamed or whose authorship Nicoll has resolved. The number of cross references which go nowhere or to an entry which documents a different title or performance is unknown, but not zero. I spent some time trying to check automatically but did not finish: other bits of the rabbit hole (like checking and fixing OCR errors in the dates) seemed more useful.

Although ostensibly organized by author, about half the entries in the Handlists record performances for which there is no author. (10,374 out of 25,662 entries to be precise). Usually, but not always, distinct entries are given for the same title performed on different occasions or at different venues – but occasionally an entry will provide a list of performances: like this

<entry type="C.O."><author>MANCHESTER, G. </author><title> The School Girl </title><note type="perf">Grand, Cardiff, 2/9/95; Stand. 14/10/95</note>. L.C. [Music by A. Maurice.]</entry>

I have not checked, but I suspect that this happens when a play has its first performance in the provinces: in this example, we may conjecture that “The School Girl” went down well enough in Cardiff for the management to risk bringing it to the Standard in Shoreditch a month later.

What sort of play is this?

Nicoll thoughtfully provides lists of the abbreviated codes he uses to indicate “the nature of the play itself” for both the Handlist 1800-1850 and the Handlist 1850-1900. There are a few codes present in only one or other of the two lists (B.O. for Ballad Opera is in the earlier list only, for example). An investigation of his usage of these codes over the two hand lists indicates the justice of the warning he also provides that “these designations are in no way final, and are often indefinite”. The two lists propose a total of 87 different codes including some very general categories (D for drama, F for Farce, P for Pantomime etc.) as well as many more nuanced classifications such as “Military Drama”, “Operatic Drama”, “Poetic Drama” “Romantic Comedy”, “Romantic Comedy Drama”, and “Romantic Drama”. Nicoll further remarks “Where possible, the designation employed in the original bills has here been followed” – so we should take these characterisations as indicative of the language in which Victorian Theatre chose to describe itself, not as a formally organized taxonomy.

I did some counting up of the actual usage of these codes, and found a further 23 codes not specified in either list. However, most of these are used very infrequently : less than 10 times for all except two of them. The exceptions are the emdash which is used for 15 entries that describe non performance items such as published collections of plays and the code “Bsq. O.” which is used for 22 entries, all of them presumably “burlesque operas”.

For what it’s worth, here’s a colourful camembert to show the distributional statistics of these categorisations. Of the 110 different codes used, more than half (64) are used fewer than 10 times. Or, to put it another way, the top six codes between them account for 17,242 out of the total nunber of 25,632 entries – over 67%.; the top eight codes (labelled in the picture) account for more than three-quarters of the whole population.

My next project will be to see if these proportions change significantly over time.

ELTeCTiT : ELTeC Titles in Translation

(I haven’t posted here since last October. I expect it’s lockdown keeping me quiet. But this morning I did manage to dream up quite an interesting research proposal, which I post here for now.)

The ELTeC corpora were designed explicitly and deliberately [ref design criteria] to exclude translated works. [quote] Despite this principled design decision, it seems self-evident that analysis of the mechanisms and results of the cross-cultural dispersal of the novel across Europe – which emphatically is within the scope of the ELTeC project – depended largely if not entirely on the availability of works in translation. It seems probable that the spread of the novel as a popular form was largely determined by the success of particular works, or classes of work, in translation; in particular, we may surmise, those works which responded to common social problems and common cultural trends. Novelists in the traditions from which those works sprang influenced novelists working in entirely different cultural milieux; just as writers raised in other traditions may be presumed to have influenced the development of what we now perceive as a unified European culture by providing easily assimilated versions of the exotic.

Although some multi-lingual expertise was undoubtedly prevalent during this period, the availability of translated versions of novels must have been essential to this diffusion, in both directions. But even basic data about the scale and scope of translations over the period covered by the ELTeC (1840 to 1920) is hard to find [refs needed] being largely diffused across national library catalogues which vary in the extent to which such works are associated with their originals, and rarely give any indication of the exact pedigree of any translation. It seems probable for example that translations into some target languages (say Romanian) would have started not from the version in the original language (say English) but from some other more accessible L2 (say French), but this is hard to determine without substantial research into individual titles and authors. Even harder to find or quantify is any information about the linguistic skills or preferences of a novel’s intended or actual readership. While it is highly probable that the languages of the great imperial powers (English, French, German) would be widely understood in those countries directly under the political or cultural influence of those powers, the extent to which they would be considered appropriate vehicles for reading for pleasure is less clear.

There are many theoretical and formal difficulties associated with any investigation of the relationship between a source and its translation, particularly (perhaps) for works of a literary nature. Translation, like speech itself, is one of the more inexplicable human behaviours. It ought not to be possible, and yet it is done, apparently more or less successfully, every day. [For an entertaining and accessible discussion, written from the perspective of a professional translator, see David Bellos “Is that a fish in your ear?” (2011)] We do not propose to address any such issues in this project, though we may provide some indicative data points to help us others do so. Our goals are more modest. Each completed ELTeC corpus already provides us with a sample of novel production in a given language within a given time frame, hopefully more or less well balanced with respect to date, size, authorship, and impact. We propose to enrich this list of titles with bibliographic data about all translated versions published within a short period (say 15 years) of their first appearance, recording for example the target language, the translated title, date and other details of publication, the translator’s name, and (where this can be determined) the source of the translation. This data will of course be provided in an open format compatible with existing ELTeC deliverables.

Amongst other research questions which availability of this data should address, we identify at least the following:

To assess “impact” or “persistence” of titles, the ELTeC corpora rely on a simple reprint count. Do translation counts complement or contradict this classification?
Are translation counts statistically correlated with any of the ELTeC classification criteria? That is, where a given collection shows an imbalance for a given criterion, is this also reflected in the translation count?
What patterns are discernible in the L1/L2 pairings manifested by our data: for example, which languages are most frequently translated into for each source language?
Is there any correlation between stylistic properties of a given group of sources and the languages into which they are translated? Crudely speaking, are romances more often translated into romance languages?

Where translated texts are available in digital form, it would be easy also to provide an ELTeC encoded version, using existing production pipelines. At this stage in the project, it is impossible to say whether this will be feasible on a sufficiently large scale to constitute true parallel ELTeC corpora: it would in any case require significant investment of time and effort from the existing ELTeC partners, whereas the collection of metadata can be done more simply.

Lou Burnard

February 2021

A tale of precision and recall

Back in the day when “text retrieval” was a thing, I remember learning the difference between precision and recall, and the need for a philosophical attitude to the fact that an optimal search has to maximize both these fairly incompatible factors. I now realise how much this whole ATCL exercise has been about that fact. My earlier efforts to identify ATCL titles in the catalogues of existing digital archives involved comparison on the basis of a manufactured key, algorithmically derived from each resource by the same process, which seemed a good compromise. This method also seemed necessary because of the limited facilities some of those resources offered for querying and manipulating the results of queries. With the availability of the wonderful “opentexts.world” service neither of these constraints applies – but the difficulties of balancing precision and recall have not gone away.

Here are the steps I am jumping through:

1. Generate a list of queries, one for each title in ATCL which doesn’t yet have any digital copy

2. Using CURL, send the queries off to the opentexts.world server and get back an XML representation of the results, including catalogue information and a link to the digital version

3. Process the results to check that this is actually the title we are looking for, and then extract the link to add to my atcl-links database

As of today, my query list has 7791 items. The NLS server doesn’t seem to mind dealing with several thousand CURL requests in rapid succession: it takes about ten minutes to run and dutifully sends me back a fat file containing a fairly straightfoward XML representation of the data.

This is fortunate since I am finding it difficult to decide how exactly to construct my query with maximal precision (to avoid false positives) and maximal recall (to avoid missing any) Most titles contain lots of words, most of which are preserved in most catalogues, so an exact word match for the full title is a good start. There are however still a few problems: punctuation and articles sometimes disappear; some titles appear more than once; some titles are very short, and thus generate many false positives. Quite a few titles have the BTAO problem – that tendency of Victorian publishers to improve the title of a new work by adding to it the formula “By The Author Of [insert previously successful titles by this author]” which results in multiple titles containing the same (irrelevant) string. What’s a good filter to cut down the noise from such things? My first thought was to require that the author’s name should be included; my second was to use the date of publication.

The problem with using the author’s name of course is that that it isn’t necessarily present on the title page, and therefore not necessarily present in the title field of the catalogue record. Many novels are anonymous; many authors published under a pseudonym. The ATCL has done a great job of rounding up and normalising authors, grouping under a single entry all variations of an author’s names. Using this it would be possible to find all the works of “Isabella Harwood” whether published under her name or the more usual pseudonym of “Ross Neil”, by increasing the recall of my “creator” search to allow for either name, but I haven’t yet done that. Instead, for my first experiment, I just use the main ATCL surname of the author, and resign myself to less recall, but more precision.

Running my 7791 queries like this

curl “https://design.opentexts.world/search/export?advanced=true&format=xml&title=Abbot%27s%20Cleve%3A%20or%20Can%20It%20be%20Proved%3F%20A%20Novel&creator=Harwood“

gets me a total of 6503 results saying “nothing doing”, and 1288 for which there is one or more matching record. I anticipate multiple hits for each title, since there are multiple editions, and of course most of these catalogues list works by volume rather than by work. A very large number of hits usually indicates a problem: for example, there is a novel with the title “Arthur” by Christiana Jane Douglas. Searching just for “title: arthur AND creator:douglas” gets many many titles containing the word “arthur”, some of them editions of the Morte D’Arthur, edited by James Douglas, and others being numerous editions of Crimean War memoirs by one Douglas Arthur Reid. But 1288 hits is not too big a list to refine further.

My second experiment searches for the full title as above, but filters by date of publication. This produces slightly different numbers: there are now 6143 “nothing doing” responses, and 1648 with at least one hit. More interesting perhaps is that I can now compare the two result sets and see which titles are not found by either query – by hypothesis these are genuinely not available, because they don’t exist in the OpenTexts database – and which are found by one but not the other. There are 659 records not found by the search-with-author queries but found by the search-with-date option, whereas there are only 299 records not picked up by the search-with-date query but found with the search-with-author option. Looking down that list very quickly, I see that in most cases the disparity in dates is because the digitized copy is of a later edition of the same work, and this starts me wondering how much later an edition has to be before I decide it’s not satisfactory. The ideal might be to include only digitizations of the first edition, but an edition produced a year or two later is probably fine. Some of these texts have a long and complicated publishing history in which distinguishing the edition is quite critical; others were reprinted once or twice and then disappeared forever.

I am now leaning to the view that the way forward is to maximize recall, simply by combining the 299 records missed by the search-with-date strategy with the rest, and then to pass those results through another filter to improve its precision. This filter would check, for example, whether the publication details for each candidate match, or are within an acceptable range. But it’s very pleasing to note that I have now identified at least one digital version for 13,769 of the 19,912 titles in ATCL, i.e. 69%. Now, if I could only persuade the British Library to be a bit less secretive…

Counting the books: yes, there’s more

My efforts to find links to digitized versions of all the titles in ATCL made one huge methodogological leap forward last week, and is now poised on the brink of another.

Going through the titles I had managed to extract from a rather uncooperative Google Books interface last week, I noticed that rather a lot of them were marked as “not available” for some reason: more precisely although my 11,104 searches, each corresponding to an entry in ATCL for which I had not yet found a digitized version, had succeeded in identifying 2186 previously unseen titles, they had also thrown up 3885 titles which Google considered inaccessible, presumably for copyright reasons, and 5033 of which it flatly denied any knowledge. Yet when I looked up a few of these same titles (whether allegedly “inaccessible” or “non-existent”) in SOLO – the Bodleian’s wizard student-friendly query interface to its catalogue– there they were, page images downloadable in PDF, no sweat.

Now, amongst other delights, SOLO allows quite rich facetted searching, so it is easy to formulate a query like “find me all titles classed as fiction published in London or Scotland between 1830 and 1900, which have also been digitized by Google”, which made me think for a few moments that my work was now done. But as with many other classy library interfaces, SOLO stops short of allowing a mere automaton to carry out any searching: you have to sit at a keyboard and type, though it will grudgingly allow you to save and download the results of your query … provided it contains no more than 50 (FIFTY!) hits. Which (as I politely pointed out to the harassed librarian on online-chat duty last week), is almost entirely useless for my purposes.

Then I remembered that Real Librarians Do It With Z39.50 and dusted off my YAZ skills. The Bodleian, like all real libraries, has a perfectly good Z39.50 interface, which is not only entirely unbothered by a succession of several hundred queries but also happy to send back directly as many full catalogue entries for the hits as you can (err) handle. The only catch is that the queries have to be expressed in some antique syntax called PQN (Prefix Query Notation) and the results come back in MARC 11. I cut my programming teeth on Fortran IV, so these ancient tongues scare me a lot less than, say, JSON. I turned my list of queries unsatisfied by Google Books into PQN, fired them at library.ox,ac.uk:210/ALEPH and put the kettle on for a nice cup of tea. PQN is not very discriminating, or not in my hands at any rate, and my queries massively overgenerated. But once my 11297 results had passed through through a couple of utilities (yaz-marcdump to produce marcxml, and my very own `marctotei` to identify and fillet the relevant records) I had a set of 780 CPF format records to add to the ATCL database list, and the tea wasn’t even cold (774 once I’d weeded out some duplicates and mismatches).

A natural question is: can we do the same trick with the British Library? Or any other library offering a Z39.50 interface? In principle, yes. But of course the Bodleian’s use of MARC fields may not be entirely the same as everyone else’s, and so the script I wrote to fillet the results of a query may need fine tuning. For example, the BL does not seem to use Marc code 856 (which I rely on) at all: its digital texts are stored in something called the Digital Store, and their identifiers there don’t seem to map directly to anything like a URL. And while I was thinking about that, something unexpected happened.

A tweet arrived, alerting me to the existence online of the “OpenTexts.world” search engine: a search interface to a much more ambitious and much more comprehensive view of the world’s digital resources, namely the Global Digitised Dataset Network (GDD Network), originally a research project into the feasibility of creating a global catalogue of digitised texts. At the end of this project’s first funding year it has made available not only a nice search interface but also (applause) the underlying complete dataset. The latter looks a bit like the HT snapshot dumps I have processed before, though it is missing quite a few useful fields, such as type of text, place of publication, etc. And the nice search interface so far has only limited functionality: nice if you are exploring the data, and really quite annoying if you know exactly what you want to find. On the bright side, it allows you to download the results of the query as a CSV file and even has a sort of API, apparently supporting Lucene-style queries to be passed in via a URL to a SOLR-indexed version of the data. This could well be the answer…

Counting the Books contd.

A couple of days ago I reported here on some imbalance in the representation of male and female novelists in current digital archives, written while I was still trying to persuade the Google Books server to do tricks for me. I can now report further progress. After 21 iterations, I did finally manage to confect a complete list of all the ATCL titles freely available from Google Books. Putting this together with data already stored in ATCL for Google, I have now identified 2823 ATCL titles in Google Books, which brings the total number of known digitizations up to 11,510: 58% of all available titles. This seemed good enough pretext to revisit the summary table I produced last time, so here it is again in a new and hopefully slightly more comprehensible form:

As might have been predicted, with more data the situation becomes more nuanced. Note first that the percentages of available titles which get digitized (column “%dig”) apparently decreases as the actual number of texts available for digitization (column “All”) increases, suggesting that the more titles you have available the less likely you are to deal with any one of them. Only tentative conclusions are warranted, since we are lacking so much data for the later part of the century. That said, comparing the columns M-dig and F-dig suggests that throughout the century, digitizers are consistently and disproportionately more likely to go for a male-authored text. Even in titles from the 1850s, where there are substantially more female authored titles available than male (778 as opposed to 595), the proportion of them which get digitized is still lower than the proportion of male authored titles (79% as opposed to 84%). In the 1880s, 55% of titles are explicitly female-authored, as opposed to 41% male (the remainder being unspecified); yet the male authors are still sampled for digitization at a far higher rate (59% as opposed to 37%).

My previous accusations of sexism amongst the digitizers en masse thus vindicated I next considered the practice of individual archives. The following table shows the numbers of ATCL titles I found in each of five major archives, and the proportions attributed to male and female authors in each.

	A-dig	M-dig	F-dig	U-dig	%Male	%Fem
All	11510	6207	5050	253	54%	44%
Hathi Trust	5655	3568	2022	65	63%	36%
InternetArchive	1665	889	748	28	53%	45%
Google Books	2823	1138	1580	105	40%	56%
British Lib	5104	2742	2252	110	54%	44%
Gutenberg	2275	1682	590	3	74%	26%

Digitization choices by archive

Overall, the balance is comparable with that shown in the previous table: a small preference for male as opposed to female authored titles (54% to 44%). But this is perhaps concealing a marked variation in practice amongst the archives. At one extreme, Project Gutenberg has nearly three times as many male authored titles as female, while at the other Google Books actually has significantly more female authors than male (56% as opposed to 40%). In between is the British Library Microsoft collection, which matches exactly the proportions for all the archives combined.

Irrespective of gender, how much variation is there in the holdings of these archives? Here’s a frequency distribution showing how many ATCL titles are available from 1, 2, or more archives (note that I cannot distinguish how many of these are actually copies of the same digital version).

1	7247	63%
2	2780	24%
3	1201	10%
4	267	2.3%
5	16	0.7%

Archive overlap: how many titles are available from how many archives?

Encouragingly, this suggests that there is little overlap amongst the holdings of the main digital archives: 63% of all 11,511 digitized titles listed occur in only one archive, 87% in one or two.

Which titles get digitized most frequently? This is hard to tell, for several reasons. Some archives list multi-volume titles as multiple copies; some archives list items simply copied from other archives. For my Google Books listing I excluded titles which were already listed by another archive. But for what it’s worth, here, in no particular order, are the fifteen titles listed as available from all five archives I looked at:

Caine, Hall (1853-1931). The Deemster: A Romance (1887)
Caine, Hall (1853-1931). A Son of Hagar: A Romance of Our Time (1887)
Hamerton, Philip Gilbert (1834-1894). Wenderholme: A Story of Lancashire and YorkshireEdinburgh: Blackwood 1869
Collins, Wilkie (1824-1889). Antonina: or, The Fall of Rome. A Romance of the Fifth Century London: Bentley 1850
Collins, Wilkie (1824-1889). The Woman in White London: Sampson Low1860
Eliot, George (pseud.) (1819-1880). The Mill on the Floss. Edinburgh: Blackwood 1860
Dickens, Charles (1812-1870). Oliver Twist: or, The Parish Boy’s Progress. London: Bentley 1838
Eliot, George (pseud.) (1819-1880). Middlemarch: A Study of Provincial Life. Edinburgh: Blackwood 1872
Gaskell, Elizabeth Cleghorn (1810-1865). Mary Barton: A Tale of Manchester Life. London: Chapman and Hall 1848
Grant, James (1822-1887). The Romance of War: or, The Highlanders in Spain London: Henry Colburn 1847
Dickens, Charles (1812-1870). Barnaby Rudge: A Tale of the Riots of ‘EightyLondon: Chapman and Hall 1841
Dickens, Charles (1812-1870). Bleak HouseLondon: Bradbury and Evans 1853
Wood, Mrs. (-). It May be True: A Novel. London: T. C. Newby 1865
Oliphant, Margaret (1828-1897). Harry Jocelyn. London: Hurst and Blackett 1881
Ouida, (pseud.) (1839-1908). Folle-Farine. London: Chapman and Hall 1871

No, it makes no sense to me either. I expected to see Charles Dickens and George Eliot and Mrs Gaskell on the list, but Hall Caine and Philip Hamerton? Clearly one needs to be very careful in interpreting this data.

See previous bloggage for details of how the numbers were obtained. Supporting data and scripts have been updated in my github repo.