Building the ELTeC : stage 0

Problem: if the ELTeC is supposed to represent in some sense the full range of novel production in a given language (EN in my case) for a given time slot (1850 to  1920, it says here) how do you find out what the population actually is before starting to sample it?

Enter at this point a wonderful database: Bassett, Troy J. At the Circulating Library: A Database of Victorian Fiction, 1837-1901. Victorian Research Web. [accessed 2018-02-09] (http://www.victorianresearch.org/atcl). I say “wonderful” advisedly: I have found nothing comparably complete and usable in a week of scratching around the internet.

According to Bassett’s technical notes, “This website was written on a Macintosh using a MySQL database and PHP.” Consequently its contents are pretty consistently organized and tagged, and consequently screen scraping and munging into a different format are both pretty easy: like this

 #!/bin/bash
 for number in {1850..1901}
 do
 echo "$number "
 wget "http://www.victorianresearch.org/atcl/show_year.php?year=$number" -O $number.html ; tidy -asxml -n --new-empty-tags image $number.html | saxon - bassetConv.xsl > $number.xml
 done
 exit 0

The `bassetConv.xsl` stylesheet generates a TEI format bibl entry for each of the 15,000 plus titles, like this:


<bibl xml:id="11169" n="acounselofperfection|Malet">
<author n="576">Lucas Malet.</author>
1 vol.  London: Kegan Paul.</bibl>
The numbers are the identifiers used by the mySQL database, so they should be unique. The @n attribute on the bibl element is a key I generate for matching purposes (see later).

I would rather like to know how many of these titles are available in
digital form, and from where. The current database has links to
Google Books (500 or so, it claims: I haven’t worked out how to find
them without downloading the entire database) but nothing else so far
as I can tell.

How to accomplish this?

My best plan so far is to extract  lists of keys or fingerprints, derived from the cataloguing information supplied by each online repository  and then start looking for overlaps with Bassett. My assumption is that for any relevant title not to exist in Bassett would be rather remarkable. Some initial experiments suggest that the first few words of the title combined with the surname of the author is a reasonable approximation to a fingerprint for each title, though obviously not
perfect.

So far I have looked at the following four online repositories:

1. Victorian Women Writers Project

This has 75 titles identified as fiction, and all in good TEI format; the following grabs the catalogue info for them

http://webapp1.dlib.indiana.edu/vwwp/search?docsPerPage=100&browseText=fiction&text1=fiction&field1=browse-genre&style=&smode=simple&brand=general

Life is too short to process all the resulting HTML: in any case I am sure if I wanted a proper XML catalogue my friends at Indiana would give me one. So I manually separate out the useful chunk of HTML: and mangle it through a simple XSLT stylesheet (`vwwpConv.xsl`), to produce a file of entries like this:

<bibl xml:id="VAB7046" n="daphne|Ward">
<title>Daphne, or Marriage a la Mode</title>
<author>Ward, Humphry, Mrs., 1851–1920</author>
<publisher>London; New York: Cassell, 1909. 315 p.</publisher>
</bibl>

Note the cunningly constructed @n attribute supplying a key which, all things being equal, should match an entry in the Bassett database, if there is one.

2. The ebooks@adelaide project

This is a quirky site hosted by the University of Adelaide library which makes many texts available in a well structured XHTML format which is easy to munge into TEI. For info about this apparently little known but rather splendid project see https://ebooks.adelaide.edu.au/about/
Selecting relevant titles for our purposes is not so easy, so I grabbed a readable version of their entire catalogue in HTML from the website, and then grepped through it for potentially useful entries, resulting in a file full of lines like this:

<li><a href="/a/abbott/edwin/flatland/">Flatland: a romance of many dimensions / Edwin A. Abbott [1884]</a></li>

along with a lot of less useful lines like this

<li><a href="/a/aristotle/meteorology/">Meteorology / Aristotle; translated by E. W. Webster</a></li>

A perl script was the quickest way of munging this into minimal TEI entries like this:

<bibl><title>Flatland: a romance of many dimensions </title><author>Edwin A. Abbott </author> <date>1884</date><ref>/a/abbott/edwin/flatland/</ref></bibl>

I am not sure that there is much wheat of this kind in this chaff however: I will come back to it later.

3. And then there’s Gutenberg

Yes, the Gutenberg Project also has a catalogue: a huge one, which you
can download as an incomprehensible RDF file or a plain text
monster. I took the latter option, and hacked out of it 49639 lines
like this:

<title>100 Desert Wildflowers in Natural Color</title><author>Natt Noyes Dodge</author><idno>54631</idno>

I won’t rehearse here the manifold  inconveniences of using Gutenberg texts. But it would be useful to come back to this and see how many of the Bassett titles are available here. As a first approximation, I first reduced the title list to just the first part of the title and the author’s surname, with no spaces or punctuation, and then used the invaluable unix  comm command to identify common lines in the two files. Obviously, this procedure is vulnerable to typos and inconsistent editorial practices, which are not uncommon, but in my first experiment I found 984 items with identical titles and authors in both the Gutenberg title list and the Bassett title list.

4 Internet Archive

This wonderful collection has a good search interface, spoiled somewhat by the unreliability of the data. I used it to grab all the entries for a collection called “19thcennov” which looked promising. Sorted by descending date, the very first item had a date of “1983” which, on inspection, turned out to be a typo for “1883”, so a good thing I didn’t use “date” to limit my search. OTOH, the second item in the list was something called “Mathematics in urban science, V : Catastrophe theory” published by the “Monticello, Ill. : Council of Planning Librarians”. No matter: at least the output is in XML, and the text identifiers used by the IA bear a striking resemblance to those I thought I had invented. My search gives me 7829 items which look like this:


<doc>
<str name="creator">North, William, d. 1854</str>
<str name="date">1847-01-01T00:00:00Z</str>
<str name="identifier">impostororbornwi03nort</str>
<str name="language">eng</str>
<str name="publisher">London : T.C. Newby</str>
<str name="title">The impostor, or, Born without a conscience</str>
<str name="volume">3</str>
</doc>


which I then munge into

<bibl xml:id="impostororbornwi01nort" n="theimpostor|North">
<author>North, William, d. 1854</author>
<title>The impostor, or, Born without a conscience</title>
London : T.C. Newby</bibl>


The IA gives each volume a separate catalogue entry, so these 7829 entries boil down to only 2739 unique keys which I can compare with the Bassett keys. On a first pass through I identify 1235 matches, i.e. nearly half. I also identify lots of ways of improving the matching procedure, but for the moment this seems like it might be useful. Though I am not sure that a success rate of around 10% in identifying matches is altogether worth shouting from the rooftops about.

---

---