Building the ELTeC : the quest continues

My quest for a reasonably reliably complete list of 19th c. English novels complete with indication of available electronic versions thereof has made a large leap forward as a result of the generosity of others.

Firstly, Professor Troy Bassett of Purdue University Fort Wayne, whom God Preserve, tells me that he will now start distributing regular extracts from his database at http://www.victorianresearch.org/atcl/index.php in CSV form for my (and others’) mungeing pleasure. I spent some time testing this out, using Libre Office to convert the CSV into an XML form which I could then reprocess with yet another xslt stylesheet. This worked reasonably well (eventually) and I was able to create a new version of this data in TEI, all 15,682 records of it. To merge in identifiers for VWWP, Internet Archive, Gutenberg, and Google texts I used a simple minded stylesheet which tries to match on the magic keys I created earlier, as discussed in previous blog postings. This worked, though not exactly with blistering speed, and I produced a second version of the Bassett TEI file with added references. Of those 15,682 titles, only 3,430 have at least one digital version and there are a total of 3,813 digital versions in all.

Secondly, and with some trepidation, I approached the sacred grove of Big Data and sent a plea for help to the nice people at Hathi Trust, that extraordinary repository of 16 million or so books in digital form. You can download a full list of their holdings in a nice simple processable form from https://www.hathitrust.org/hathifiles : the catch is that their monthly snapshots contain 16 million (and counting) records derived, I think automatically, from contributing libraries’ catalogue systems, not all of them entirely consistent in their metadata usage, and certainly none of them clearly identifying which titles might be considered to be novels or have female authors. I did some initial poking around with a perl script and ascertained that I could whittle the 16,208,265 entries in the May 4th version of the file down to a mere 1,253,950 by selecting only those which are publically acccessible, have a publication date between 1849 and 1820, are books in English, and are not US government documents. These counts are inflated by the fact that we are counting volumes here, rather than works.

Meanwhile, a helpful person on the Hathitrust help desk had got in touch pointing me to the existence of various selections created by other researchers before me, notably Ted Underwood’s file filtered_fiction_metadata.csv [available on Github at https://github.com/tedunderwood/character/blob/master/metadata/filtered_fiction_metadata.csv]
which lists 93,708 volumes of fiction, published between 1800 and 2007. “It focuses on fiction, but will include works in translation. It’s also not restricted to ‘the novel’; short stories and even folk tales are included. But we have tried to exclude fiction aimed explicitly at a juvenile audience across the whole timeline.” Underwood points out that “There is no authoritative master list of all English-language fiction from 1800 to the present. So scholars who want to do research at scale have to construct their own list.” which is true enough, though I wonder if he’s aware of Bassett’s work. Anyway, I wrote more scripts to process his CSV file into TEI, adding my magic keys to enable me to link his data with the titles in the Bassett database. All of which meant that I could provide Bassett with a list of Hathtrust links as well as those already provided for Gutenberg et al.

Bassett’s data includes some titles published only in the US, which I would like to exclude, as well as some “aimed explicitly at a juvenile audience” (the two sets of course have nothing in common, except that I would like to exclude them both). More annoyingly, it stops in 1901. I therefore went back to thinking about how to extract comparable data from the Hathifiles for the period 1901-1920. The most recent Hathi file (dated 1 June) has entries for 16,370,821 volumes. I tweaked my perl script to extract records for books in English published between 1900 and 1921, as before, tweaking it also to suppress duplicate titles, which gave me a more manageable but still daunting total of 348,232 titles.

The question is, as ever, how to pick out the novels from the chaff? The obvious answer is to apply some basic text analytic tools. I have a list of 15k titles which the obliging Bassett has already identified as novels. I have 348,232 titles extracted from the Hathitrust file. If I had to scan through them choosing the titles which are most likely to be novels, which words would catch my eye? Which words in fact appear significantly more frequently in the first list than their frequency distribution in the second would lead you to suppose? This sort of question is surely what computer linguists deal with ten times a day before breakfast.

I install Lawrence Anthony’s excellent AntConc program from http://www.laurenceanthony.net/software.html and battle through its interface a bit. Here are the top twenty “keywords” for the Bassett title list, when I use the aforesaid Hathitrust titles list as the “base corpus” for comparison. The underlying statistic (need you ask) is log likelihood (4 term); the threshold being p < 0.05 (+Bonferroni), i.e. the defaults.

#Keyword Types: 611
#Keyword Tokens: 47261
#Search Hits: 0
1 2693 + 15387.79 0.0556 novel
2 8559 + 10677.59 0.0763 a
3 1969 + 6571.89 0.0388 or
4 1262 + 6402.77 0.0266 tale
5 1426 + 4104.1 0.0283 story
6 721 + 2599.46 0.0152 romance
7 2195 + 1957.72 0.0327 s
8 471 + 1072.51 0.0098 stories
9 364 + 1058.08 0.0077 love
10 837 + 1019.33 0.016 life
11 355 + 1001.66 0.0075 tales
12 9268 + 875.37 0.0384 the
13 272 + 773.82 0.0058 adventures
14 163 + 594.6 0.0035 daughter
15 224 + 591.09 0.0048 lady
16 217 + 546.42 0.0046 woman
17 150 + 484.95 0.0032 wife
18 570 + 478.07 0.011 other
19 221 + 422.18 0.0047 little
20 131 + 395.37 0.0028 miss

This seems like pretty good evidence that simply looking for titles which contain the words “novel”, “tale”, “story”, “romance” or – this one a surprise – “or” might do pretty well. It also suggests that the Bassett novel is quite concerned about women, but that’s another story.

Reassured, I re-tweak my Perl script to extract bibliographic records as before, but this time only if their title contains one of the words “novel”, “tale”, “story”, or “romance”. A glance at the results shows that this approach cuts down the number of titles to a more manageable 16,908 but still has many false positives (“My life and the story of the Gospel hymns and of sacred songs and solos” for example) . A more sophisticated approach is needed, clearly. Time to make a cup of tea.

---

---