NON-TRADITIONAL AUTHORSHIP ATTRIBUTION STUDIES IN EIGHTEENTH CENTURY LITERATURE. STYLISTICS STATISTICS AND THE COMPUTER

Abstract

Non-traditional authorship attribution studies are those attribution studies that make use of the computer, statistics, and stylistics. The hypothesis behind these studies is that an author has a unique and identifiable style. The computer has now become ubiquitous in eighteenth century literary studies and is the main reason why non-traditional authorship studies have advanced to where they are. David Holmes gives a good overview of the field in, The Analysis of Literary Style – A Review.[1] This article surveys a representative sample of authorship studies of eighteenth century literature and gives an exemplum of an ongoing study.

Survey

Because my research is on the canon of Daniel Defoe, the following representative survey of non-traditional authorship studies in eighteenth century literature focuses on literature written in English. However, many non-traditional authorship studies in other languages are important to researchers in English literature for reasons of theory and technique. Among others, Richard Frautschi has done stylistic and authorship work on many eighteenth century French authors: Diderot, d'Alembert, Jaucourt, Rousseau, Voltaire, and Perrault.[2] Estelle Irizarry has done work on Rodríguez Juliá imitating the hyperbolic style of the Spanish intellectuals of the eighteenth century.[3]

In addition to the non-English works, a few seminal studies that use stylistics and statistics but not the computer are mentioned for the same reasons of style and technique.

The following list is not exhaustive but is very representative. There are many authorship attribution problems that exist in the long eighteenth century. Griffin, in Anonymity and Authorship, gives a good synopsis of the why of the problem.[4]

As an aside, it is interesting to note that the English courts allowed language style as evidence for authorship as early as 1728 in the trial of William Hales.[5]

This list does not include stylistic studies on groups of authors such as the one that Burrows did on a group of twenty eighteenth century authors,[6] or the one that Sigelman et al. did on fifteen eighteenth century pamphleteers,[7] or the study that Kroeber did on five eighteenth century novelists.[8]

Two questions to keep in mind as you look at the following studies are: (1) Am I aware of these studies? (2) Have the results of the study become generally accepted by the profession?

• Swift

Louis Milic's study of the quantitative style of Swift, A Quantitative Approach to the Style of Jonathan Swift, is must reading for anyone working on non-traditional attribution studies.[9]

Along with Milic, Cynthia and William Matlack in their, A Statistical Approach to Problems of Attribution: »A Letter of Advice to a Young Poet« deal with A Letter of Advice to a Young Poet.[10]

Corbett identifies several quantifiable style-markers and uses them to analyze A Modest Proposal.[11]

• Smollett

Barbara Laning Fitzpatrick did preliminary work for a non-traditional study of some essays that appeared in the British Magazine.[12] However, she decided (correctly, in my opinion) that important elements for a valid non-traditional study were missing and therefore ceased the study.[13]

• Samuel Johnson and/or Charlotte Lennox

Deborah McLeod studied the attribution of the penultimate chapter of The Female Quixote – Was it Johnson, Lennox, or was it a collaboration? »The results of the statistical analysis [...] are maddeningly contradictory.«[14]

Isobel Grundy discusses this same question and goes on to show the importance of the study of attribution in general and that of attributions to women in particular.[15]

• Sir Josiah Child

O'Brien and Darnell did a study to determine if Sir Josiah Child was Philopatris – the author of the Treatise[...] defending the East India Company.[16]

• Goldsmith

Mannion and Dixon have done non-traditional authorship studies on many of the over one hundred essays that have been attributed to Goldsmith since his death.[17]

• Patrick Henry

Stephen Olson did an authorship study on Patrick Henry's Liberty or Death speech. But, do historians or the general public believe that St. George rather than Patrick Henry wrote the speech?[18]

Olson also did a study, Computerized Thematic Analysis of Selected American Revolution Pamphlets. Unfortunately, all of the digital records, print copies, and notes were lost in a fire in 1995.[19]

• Charles Brockden Brown

Fritz Fleischmann is working on Brown's short fiction and essays that were published pseudonymously or anonymously. After spending years on a traditional study, he is set to begin the non-traditional part of his study.[20]

Larry Stewart also is doing quantitative analysis on Brockden Brown's work. He delivered a paper at the ALLC/ACH 2002 conference in Tübingen, Charles Brockden Brown: Quantitative Analysis and Narrative Voice.[21]

• The Junius Letters

Various practitioners have tried to answer the question: Did Sir Philip Francis, Johnson, or (Pick-a-name) write the Junius Letters?[22]

• The Federalist Papers

The study that arguably is the most famous and the most successful is the Mosteller and Wallace work on the twelve disputed Federalist Papers. Were they written by Hamilton or Madison?[23] It is interesting to note that Mosteller and Wallace used Bayes' Theorem in their study. Bayes was an eighteenth century minister and mathematician.[24] Gavin Budge delivered a paper at the 2002 American Society for Eighteenth Century Studies meeting titled, Bayesian Probability and the Crisis of Representation in Eighteenth Century Mathematics.[25] The following list of studies show the influence that Mosteller and Wallace have over non-traditional authorship studies. They all either test Mosteller and Wallace's results or use Mosteller and Wallace's methods and techniques in their own studies:

– Merriam's An Experiment with the »Federalist Papers«.[26]

– Särndal's On Deciding Cases of Disputed Authorship.[27]

– Tankard's Literary Detective.[28]

– Martindale and McKenzie's On the Utility of Content Analysis in Author Attributions »The Federalist«[29]

– Tweedie, Singh, and Holmes' Neural Network Applications in Stylometry: »The Federalist Papers«.[30]

– Kjell's Discrimination of Authorship Using Letter Pair Frequency Features with Neural Network Classifiers.[31]

– Kjell et al.'s Discrimination of Authorship Using Visualization.[32]

– Wachal's dissertation Linguistic Evidence, Statistical Inference, and Disputed Authorship.[33]

– Rokeach et al.'s A Value Analysis of the Disputed Federalist Papers.[34]

– McColly and Weier's Literary Attribution and Likelihood-Ratio Tests: The Case of the Middle English »Pearl«-Poems.[35]

– Bosch and Smith's Separating Hyperplanes and the Authorship of the Disputed »Federalist Papers«.[36]

– Khmelev and Tweedie's Using Markov Chains for Identification of Writers.[37]

– Fung and Mangasarian's The Disputed Federalist Papers: SVM Feature Selection via Concave Minimization.[38]

– Forsyth's dissertation Stylistic Structures: A Computational Approach to Text Classification.[39]

– Francis' An Exposition of a Statistical Approach to the »Federalist« Dispute.[40]

– Hilton and Holmes' An Assessment of Cumulative Sum Charts for Authorship Attribution.[41]

– Farringdon and Morton's Fielding and the »Federalist«.[42]

– Holmes and Forsyth's The »Federalist« Revisited.[43]

In addition to the above, almost every non-traditional authorship study (done in or out of the eighteenth century) cites Mosteller and Wallace for one reason or another.

• Henry Fielding

Michael and Jill Farringdon did a study of Fielding's translation of the Military History of Charles XII.[44]

Michael Farringdon also contributed a non-traditional authorship attribution section to Martin Battestin's work on Fielding's contributions to the Craftsman.[45]

Michael Farringdon and Andrew Morton worked on Fielding in their, Fielding and the Federalist.[46]

Hugh Amory discussed word usage in Fielding.[47]

Martin Battestin, in New Essays by Henry Fielding: His Contributions to the »Craftsman« (1734-1739) and other Early Journalism with a Stylometric Analysis by Michael Farringdon, discussed the CUSUM method used in authorship attribution in general and attribution in Fielding in particular.[48]

• Sarah Fielding

John Burrows did a study that showed Henry Fielding wrote the beginning of the history of Anna Boleyn and that he allowed Sarah to continue it for him. Burrows also showed that Henry either revised her ending, or added an ending in which he sought to imitate her style.[49]

An editorial review in The Scriblerian discussed Burrows' article and made the statement that »Computers instill confidence«. But the reviewer goes on to quote Burrows concession that, »statistical analysis never yields conclusive answers«.[50]

Sheridan Baker wrote a probing article, Did Fielding Write ›A Vision‹?, casting doubt on Burrows' contentions.[51]

M.W.A. Smith, in Attribution by Statistics: A Critique of Four Recent Studies, looks at Burrows' contributions to the study of Anna Boleyn and criticizes him for neither giving the origin of the methodology nor the origin of its underlying theory. Smith agrees that Burrows has showed Henry's and Sarah's authorship can be differentiated. He calls Burrows »results«[52] impressive.

Burrows wrote a few articles to answer his critics and to explain how statistical evidence should be viewed.[53]

• Jane Austen

Karl Kroeber talked about the perils of quantification and used Emma as a case study.[54]

Although John Burrows' book, Computation into Criticism, is not an attribution study, many of his techniques have been referenced and used in authorship studies.[55]

Michael Hilton and David Holmes included a short study of five of Austen's novels in their An Assessment of Cumulative Sum Charts for Authorship Attribution.[56]

• Aphra Behn

John Burrows and Harold Love did a fine study – traditional and non-traditional – on Caesar's Ghost.[57]

• Shadwell

Burrows and Love also did a study on some works attributed to Shadwell – confirming some of the attributions but not all.[58]

• Defoe

As early as 1966, Edward McAdam used the computer to try to find out if it was Defoe who wrote about one hundred anonymous political tracts.[59] The results of McAdam's study are lost – not as the result of a fire as in Olson's case, but seemingly because no one published or preserved them when McAdam died.

Newsome did a study on the 1745 continuation of Roxana – I have never been able to find this in print. Fortunately, I have a 1987 e-mail pre-print from the author.[60] It is unfortunate that Furbank and Owens did not know of this study when they wrote their The ›Lost‹ Continuation of Defoe's »Roxana«.[61]

Steig Hargevik's monumental work, The Disputed Assignment of »Memoirs of an English Officer« to Daniel Defoe, is a good starting point for anyone doing non-traditional attribution work on Defoe. Hargevik did not use the computer in his 1974 study. However, he did use stylistics and statistics. He hand counted various stylistic traits in a text sample of over two million words.[62]

Irving Rothman looks at Hargevik's study, makes some points for expanding it to other Defoe works, and takes Furbank and Owens to task for ignoring important aspects of etylometrics.[63]

Maximillian Novak has used the computer to generate some concordances as an aid in his Defoe attribution studies – and, therefore,in the compilation of his bibliographies of Defoe's canon.[64] Novak, in a recent essay, re-attributed a de-attributed work by using a mixture of Rothman and historical scholarship.[65] Novak also takes Furbank and Owens to task for promising the use of new technology – stylometrics – and then, although finding the methodology inconclusive, abandoning any real analysis of language. Novak also finds fault with their not using the Eighteenth Century Short Title Catalogue (ECSTC).[66]

Furbank and Owens answered Novak's article but were silent on his charge about stylometrics, language analysis, and the ECSTC.[67]

Paula Backscheider, one of the world's preeminent Defoe scholars, made a serious effort to understand and employ non-traditional authorship techniques while working on her Daniel Defoe: His Life.[68] The results were not convincing enough to use as support evidence for her attributions based on traditional external evidence. Professor Backscheider continues to review the pertinent literature and believes that the non-traditional technology will eventually become valid and valuable.[69]

There are at least three dissertations on Defoe's style that are valuable aids in Defoe attribution studies:

– Horten – concentrated on Defoe's spelling, vowels, consonants, and punctuation[70]

– Dill – concentrated on the quantifiable elements of Defoe's style: phrases, vocabulary, sentence structure, and prose rhythm.[71]

– Lannerd – concentrated on Defoe's use of the indefinite article, indefinite pronoun, and periphrastic tenses.[72]

I began working on attribution studies in Defoe's canon in the late 1970's. It soon became clear that there were monumental problems. Correspondence with Curtis, Furbank and Owens, and other Defoe scholars convinced me that a non-traditional approach offered the best hope for some kind of resolution. A large majority of my research time since the mid-eighties has been on the theory and techniques of non-traditional authorship attribu-
tion.[73]

The bibliography of this paper (embedded in the footnotes) contains the references to many non-traditional authorship attribution studies in the eighteenth century.

Exemplum

The following is an exemplum to give a little better idea of what non-traditional studies are about. It is a part of an ongoing project and, in a broad sense, will point out many of the techniques and potential dangers of using the computer in a non-traditional authorship attribution study.

The questioned anonomous work in this example is A Letter from Scotland to a Friend in London (Letter) – a 1705 political pamphlet that had significant impact on the union of England and Scotland. It is about intrigue, piracy, and revenge. The Letter was first attributed to Defoe by Moore[74] – who later wrote that he was, »[...] much less sure of his authorship than I was in 1939.«[75] I have found no other Defoe scholar who believes the Letter is by Defoe.

Now let me give a major caveat – before any non-traditional study is undertaken, a rigorous and complete traditional study must be done – non-traditional methods are tools to be employed by the traditional scholars – and surely not the most important tools.

The following, for obvious reasons, is necessarily short and incomplete. Two of my published articles that explain and expand on some of the problems and solutions of non-traditional authorship problems are: The State of Authorship Attribution Studies: Some Problems and Solutions.[76] and Non-Traditional Authorship Attribution Studies: Ignis Fatuus or Rosetta Stone.[77]

After the traditional study and all of the preliminary analysis is finished (e.g. are all of the Defoe and control texts valid, is the Letter a valid text) the process of the study begins:

• Enter the text of the Letter into the computer and add the Text Encoding Initiative (TEI) coding.[78]

– I typed the Letter into the computer. The error rate of the optical character reader is too high – although it is down to about 60%. But, even with careful proofing of my typing, an error rate for typos must be calculated and folded into the final experimental error.

– The coding allows automated analysis of style-markers such as parts of speech ratios and phrasal type percentages.

• Enter the sample of Defoe texts into the computer and add the TEI coding.

– This sample should consist of all political tracts written by Defoe and published within (+ or -) five years of the Letter, i.e. 1700-1710.

– Do not include any dubitanda – a certain and stylistically pure Defoe sample must be established – all decisions must err on the side of exclusion. If there can be no certain Defoe touchstone, there can be no non-traditional authorship attribution studies on his canon, and no wide ranging stylistic studies

– Get texts from anywhere possible, then edit, and type in the rest.

– Keep out a random sample of this set as one type of control.

There is a danger in downloading texts on the internet. You must compare the electronic text with the printed copy you have chosen to ensure its integrity.

• Enter a substantial random sample of non-Defoe texts into the computer and add the TEI coding.

– Same genre and time constraints as the Defoe texts.

– Download texts, edit them, and type in the rest.

– This is a random sample of all of the other authors within the constraints. The larger the sample the lower the statistical error on the result.

– This is one of the experimental controls.

• Analyze the texts stylistically

Textual problems must be dealt with before the analysis can begin. For example: quotes (of others, of Defoe's earlier works, of fictional authors), plagiarism, translations, mixed genre, editorial corruption, and accidentals (such as orthography).[79]

There is a danger in using canned computer packages such as TACT or TUSTEP.[80] The user must completely understand the assumptions and techniques that the authors of these programs employed. The user must really understand everything – even something as basic as what the system designers consider a ›word‹ or ›sentence‹.[81]

– Identify all of the style-markers (from the hundreds of thousands available) that Defoe uses consistently (e.g. type/token ratio, word length correlations, function word frequencies).[82]

– Compare Defoe's usage of these style-markers to the writers in the random sample.

– Compare Defoe's usage of these style-markers to the Letter.

• Analyze the results statistically

Again, users of canned packages must understand the assumptions behind them and the methodology that was used in creating them. Users must not let these packages dictate their research plans.

– Look at the results of the stylistic analysis the way you would look at a DNA autoradiogram.[83]

– Cull out the zeitgeist style-markers.

• Answer the question – Did Defoe write the Letter?

– Probabalistic.
No non-traditional authorship attribution study can say with 100% surety that an author wrote a given work. These studies can approach certainty the way that a DNA study can – giving odds such as one in a million that an author did or did not write the questionable work.

Conclusion

The results of nearly all of the studies mentioned in the survey if accepted at all are accepted with a grain of salt.

Why are these studies looked at with such skepticism? Do these studies show non-traditional authorship attribution to be simply ›aspiration‹ and not a science, as Furbank and Owens claim?[84] Has stylistics, with the help of Milic and Dilligan, withstood the onslaught of Fish?[85]

Are most non-traditional authorship studies of eighteenth century literature valid? My answer is no. Much theoretical and experimental work must be done before this answer can change and these studies can take their place in mainstream bibliography.

Joseph Rudman

Jospeph Rudman
Department of English
Carnegie Mellon University
Pittsburgh, PA 15213
USA
jr20@andrew.cmu.edu


(16. Oktober 2002)
[1] David I. Holmes: The Analysis of Literary Style – A Review. In: Philippe Thoiron/Dominique Labbe/Daniel Serant (Eds.): Vocabulary Structure and Lexical Richness. Paris: Champion-Slatkine 1988, pp. 67-76.
[2] Richard L. Frautschi: Lexical and Focal Preferences in Rousseau's Profession de foi du Vicaire Savoyard (Book IV of Emile). In: Computers and the Humanities (CHum) 23 (1989), pp. 347-355. Richard L. Frautschi: Focal Style and the Problem of Attribution in Eight Prose Tales by Perrault. In: Literary and Linguistic Computing (LLC) 2.4 (1987): pp. 213-220. Richard L. Frautschi: A Project for Computer-Assisted Analysis of French Prose Fiction, 1751-1800. In: A.J. Aitken et al. (Eds.): The Computer and Literary Studies. Edinburgh: Edinburgh University Press 1973, pp. 183-195. Richard L. Frautschi: The Authorship of Certain Unsigned Articles in the Encyclopedic. A First Report. In: Computer Studies in the Humanities and Verbal Behavior 3.2 (1970), pp. 66-76.
[3] Estelle Irizarry: One Writer, Two Authors. Resolving the Polemic of Latin America's First Published Novel. In: LLC 6.3 (1991), pp. 175-179. Estelle Irizarry: Exploring Conscious Imitation of Style with Ready-made Software. In: CHum 23.3 (1989), pp. 227-233.
[4] Robert J. Griffin: Anonymity and Authorship. In: New Literary History 30 (1999), pp. 877-895. An example of authors not in the survey is Richard Steele. See: Louis Milic: A Prepositional Analysis of Steele's Guardian No. 12. In: Hakan Ringbom et. al. (Eds.): Style and Text. Studies Presented to Nils Enkvist. Trelleborg, Sweden: Tryckeri AB Allehanda 1975, pp. 118-133.
[5] Gerald R. McManamin: Forensic Stylistics. New York: Elsevier 1993, p. 90.
[6] John F. Burrows: Tiptoeing into the Infinite. Testing for Evidence of Natural Differences in the Language of English Narrative. In: Susan Hockey/Nancy Ide (Eds.): Research in Humanities Computing. Oxford: Clarendon Press 1996, pp. 1-33.
[7] Lee Sigelman/Colin Martindale/Dean McKenzie: The Common Style of Common Sense. In: CHum 30.5 (1996/1997), pp. 373-379.
[8] Karl Kroeber: Styles in Fictional Structure. Princeton, New Jersey: Princeton University Press 1971. This list is not exhaustive. There are other studies such as: Douglas Biber/Edward Finigan: Multi-Dimensional Analysis of Authors' Styles. Some Case Studies from the Eighteenth Century. In: Don Ross/Dan Brink (Eds.): Research in Humanities Computing 4. Oxford: Clarendon Press 1994, pp. 3-17.
[9] Louis Milic: A Quantitative Approach to the Style of Jonathan Swift. The Hague: Mouton 1967. See also: Louis Milic: Unconscious Ordering in the Prose of Swift. In: Jacob Leed (Ed.): The Computer and Literary Style. Kent, Ohio: Kent University Press 1966, pp.79-106. Patricia Köster: Computer Stylistics. Swift and Some Contemporaries. In: R. A. Wisbey (Ed.): The Computer in Literary and Linguistic Research. Cambridge: Cambridge University Press 1971, 129-147. Julius Laffal: A Concept Analysis of Jonathan Swift's A Tale of the Tub and Gullivers Travels. In: Chum 29.5 (1995), pp. 339-361.
[10] Cynthia Matlack/William F. Matlack: A Statistical Approach to Problems of Attribution. A Letter of Advice to a Young Poet. In; College English 29 (1968), 627-632.
[11] Edward P.J. Corbett: A Method of Analyzing Prose Style with a Demonstration Analysis of Swift's A Modest Proposal. In: Glen A. Love/Michael Payne (Eds.): Contemporary Essays on Style. Rhetoric, Linguistics, and Criticism. Glenview, Illinois: Scott, Foresman and Company 1969, pp. 81-98.
[12] Barbara Laning Fitzpatrick: Some Pieces in the British Magazine and A Small Part of the Translation of Voltaire's Works. Smollett Attributions. In: Eighteenth Century Fiction (ECF) 9.1 (1996), pp. 97-100.
[13] Barbara Laning Fitzpatrick: RE: Stylistics and Smollett. Private e-mail to Joseph Rudman, 22 March 2001.
[14] Deborah McLeod: The Female Quixote and Samuel Johnson. A Computer Assisted Analysis. MA Thesis, University of Alberta 1991. The quote is from page 96.
[15] Isobel Grundy: Attribution to Women. In: ECF 8.4 (1996), pp. 522-525.
[16] D.P. O'Brien/A.C. Darnell: Authorship Puzzles in the History of Economics. A Statistical Approach. London: The Macmillan Press Ltd. 1982. See chapter three, »Sir Josiah Child's Economic Writings«.
[17] Peter Dixon/David Mannion: Goldsmith and the British Magazine. In: LLC 13.1 (1998), 37-49. Peter Dixon/David Mannion: Goldsmith's Periodical Essays. A Statistical Analysis of Eleven Doubtful Cases. In: LLC 8.1 (1993), pp. 1-19. David Mannion/Peter Dixon: Authorship Attribution. The Case of Oliver Goldsmith. In: The Statistician 46.1 (1997), pp. 1-18.
[18] Stephen T. Olson: Patrick Henry's Liberty or Death Speech. A Study in Disputed Authorship. In: Thomas W. Benson (Ed.): American Rhetoric. Content and Criticism. Carbondale: Southern Illinois University Press 1989, pp. 14-65.
[19] Stephen T. Olson: RE: Computerized Thematic Analysis of Selected American Revolution Pamphlets. Private e-mail to Joseph Rudman, 20 February 2001.
[20] Fritz Fleischmann: Private e-mail to Joseph Rudman, 2 January 2001.
[21] Larry Stewart: Charles Brockden Browne. Quantitative Analysis and Narrative Voice. Paper delivered at the ALLC/ACH 2002 conference in Tübingen, Germany. 25 July 2002.
[22] A. Ellegård: A Statistical Method for Determining Authorship. The Junius Letters, 1769-1772. In: Gothenburg Studies in English 13 (1962), pp. 1-115. A. Ellegård: Who was Junius? Stockholm: Almquist and Wiksell 19962. Carl-Erik Särndal: On Deciding Cases of Disputed Authorship. In: Applied Statistics 16.3 (1967), 251-268. Linda Katritzky: Johnson and The Letters of Junius. New York: Peter Lang 1966. See also: A review: Anna Boleyn and the Authenticity of Fielding's Feminine Narratives. In: Eighteenth Century Studies (ECS) 21 (1988), pp. 427-453. In: The Scriblerian and the Kit-Cats 22 (1989), p. 14. George Willis Zimmer: The Attribution of Authority. A Computerized Method Evaluated and Compared with Other Methods Past and Future. Dissertation, Michigan State University 1968. See chapter two for Junius Letters. George Willis Zimmer: A review of Who was Junius? and A Statistical Method for Determining Authorship. Alvar Ellegard. In: Journal of English and Germanic Philology LXII (1963), pp. 688-689.
[23] Frederick Mosteller/David L. Wallace: Applied Bayesian and Classical Inference. The Case of The Federalist Papers. New York: Springer-Verlag 1984 (IId. ed.).
[24] Thomas Bayes: An Essay Towards Solving a Problem in the Doctrine of Chances (Communicated by Mr. Price in a letter to John Canton, 23 December 1763.). In: Biometrika 45 (1958), pp. 296-315.
[25] Gavin Budge: Bayesian Probability and the Crisis of Representation in Eighteenth Century Mathematics. Paper delivered at the ASECS 2001 meeting in New Orleans, Louisiana. 19 April 2001.
[26] Thomas Merriam: An Experiment with the Federalist Papers. In: CHum 23.3 (1989), pp. 251-254.
[27] Särndal: On Deciding Cases of Disputed Authorship (footnote 22).
[28] Jim Tankard: The Literary Detective. In: BYTE 11.2 (1986), pp. 231-238.
[29] Colin Martindale/Dean McKenzie: On the Utility of Content Analysis in Author Attribution. The Federalist. In: CHum 29.4 (1995), pp. 259-270.
[30] F.J. Tweedie/S. Singh/D.I. Holmes: Neural Network Applications in Stylometry. The Federalist Papers. In: CHum 30.1 (1996), pp. 1-10.
[31] Bradley Kjell: Authorship Determination Using Letter Pair Frequency Features with Neural Network Classifiers. In: LLC 9.2 (1994), pp. 119-124.
[32] Bradley Kjell/W. Addison Woods/Ophir Frieder: Discrimination of Authorship Using Visualization. In: Information Processing & Management 30.1 (1994), pp. 141-150.
[33] Robert Stanley Wachal: Linguistic Evidence, Statistical Inference, and Disputed Authorship. Dissertation, University of Wisconsin 1966.
[34] Milton Rokeach et al.: A Value Analysis of the Disputed Federalist Papers. In: Journal of Personality and Social Psychology 16.2 (1970), pp. 245-250.
[35] William McColly/Dennis Weier: Literary Attribution and Likelihood-Ratio Tests. The Case of the Middle English Pearl-Poems. In: CHum 17.2 (1983), pp. 65-75.
[36] Robert A. Bosch/Jason Smith: Separating Hyperplanes and the Authorship of the Disputed Federalist Papers. In: The American Mathematical Monthly 105.7 (1998), pp. 601-608.
[37] Dmitri V. Khmelev/Fiona Tweedie: Using Markov Chains for Identification of Writers. In: LLC 16.3 (2001), pp. 299-307.
[38] Glenn Fung/Olvi L. Mangasarian: The Disputed Federalist Papers. SVM Feature Selection via Concave Minimization. Paper delivered at the CSNA 2002 meeting in Madison, Wisconsin. 15 June 2002.
[39] Richard S. Forsyth: Stylistic Structures. A Computational Approach to Text Classification. Dissertation, University of Nottingham 1995.
[40] Ivor Francis: An Exposition of a Statistical Approach to the Federalist Dispute. In: Jacob Leed (Ed.): The Computer and Literary Style. Kent, Ohio: Kent University Press 1966, pp. 38-78.
[41] Michael L. Hilton/David I. Holmes: An Assessment of Cumulative Sum Charts for Authorship Attribution. In: LLC 8.2 (1993), pp. 73-80.
[42] Michael G. Farringdon/Andrew Q. Morton: Fielding and the Federalist. Unpublished manuscript 1990.
[43] David I. Holmes/Richard S. Forsyth: The Federalist Revisited. New Directions in Authorship Attribution. In: LLC 10.2 (1995), 111-127.
[44] Michael Farringdon/Jill Farringdon: A Computer Aided Study of the Prose Style of Henry Fielding and Its Support for His Translation of The Military History of Charles XII. In: D. E. Ager/F. E. Knowles/Joan Smith (Eds.): Advances in Computer-Aided Literary and Linguistic Research. Birmingham: John Goodman & Sons 1979, pp. 95-105.
[45] Martin C. Battestin: New Essays by Henry Fielding. His Contributions to the Craftsman (1734-1739) and Other Early Journalism with a Stylometric Analysis by Michael Farringdon. Charlottesville: University Press of Virginia 1989.
[46] Farringdon/Morton: Fielding and the Federalist (footnote 42).
[47] Hugh Amory: It is Very Probable I am Lord B—ke. Reflections on Fielding's Canon. In: ECS 8.4 (1996), pp. 529-533.
[48] Martin C. Battestin: The Cusum Method. Escaping the Bog of Subjectivism. In: ECF 8.4 (1996), pp.533-538.
[49] John Burrows/A. J. Hassall: Anna Boleyn and the Authenticity of Fielding's Feminine Narratives. In: ECS 21 (1988), pp. 427-453.
[50] Review: Anna Boleyn and the Authenticity of Fielding's Feminine Narratives, (footnote 22)
[51] Sheridan Baker: Did Fielding write ›A Vision‹? In: ECS 22.(3+4) (1989), pp. 548-551.
[52] M. W. A. Smith: Attribution by Statistics. A Critique of Four Recent Studies. In: Revue Informatique et Statistique Dans Les Sciences Humaines 26 (1990), pp. 233-251.
[53] John F. Burrows: »I Lisp'd in Numbers«. Fielding, Richardson, and the Appraisal of Statistical Evidence In: The Scriblerian and the Kit-Cats 33.2 (1991), pp. 234-241. John F. Burrows: »An Ocean Where Each Kind [...]«. Statistical Analysis and Some Major Determinants of Literary Style. In: CHum 23 (1989), pp. 309-321. John F. Burrows: A Vision as a Revision? ECS 22 (1989), pp. 551-565.
[54] Karl Kroeber: Perils of Quantification. The Exemplary Case of Jane Austen's EMMA. In: Lubomír Doleel/Richard W. Bailey (Eds.): Statistics and Style. New York: American Elsevier 1969, pp. 197-213.
[55] John F. Burrows: Computation into Criticism. Oxford: Clarendon Press 1987.
[56] Hilton/Holmes: An Assessment of Cumulative Sum Charts for Authorship Attribution (footnote 41).
[57] John F. Burrows/Harold Love: Did Aphra Behn Write Caesar's Ghost? In: The Culture of the Book. Essays from Two Hemispheres in Honour of Wallace Kirsop. Melbourne: Bibliographical Society of Australia and New Zealand 1999, pp. 148-172.
[58] John F. Burrows/Harold Love: The Role of Stylistics in Attribution. Thomas Shadwell and the Giants' War. In: Studies in the Eighteenth Century 10 22.1 (1998), pp. 18-30.
[59] Thomas O'Toole: Erudite I.B.M. 360 Helping Scholars. In: New York Times, 20 February 1966, p. 26.
[60] Richard Newsome: An Investigation into the Authorship of the 1745 Continuation of Defoe's Roxana. Pre-print from the author. July 1987.
[61] P. N. Furbank/W. R. Owens: The ›Lost'‹ Continuation of Defoe's Roxana. In: ECF 9.3 (1998), pp. 299-308.
[62] Stieg Hargevik: The Disputed Assignment of Memoirs of an English Officer to Daniel Defoe. Part I. Stockholm: Almqvist & Wiksell 1974. Stieg Hargevik: The Disputed Assignment of Memoirs of an English Officer to Daniel Defoe. Part II Stockholm: Almqvist & Wiksell 1974.
[63] Irving N. Rothman: Defoe De-Attributions Scrutinized Under Hargevik Criteria. Applying Stylometrics to the Canon. In: Papers of the Bibliographic Society of America 94.3 (2000), pp. 375-398. See also: Irving N. Rothman: A Stylometric Study of the Variant Styles of Daniel Defoe. Paper delivered at the SCSECS conference in Shreveport, Louisiana. 25 February 1999.
[64] Maximillian E. Novak: The Defoe Canon: Attribution and De-Attribution. In: Huntington Library Quarterly 59.1 (1997), pp. 83-104.
[65] Maximillian E. Novak: Private e-mail to Joseph Rudman, 29 December 2001.
[66] Maximillian E. Novak: Whither the Defoe Canon? In: ECF 9.1 (1996), pp. 89-91.
[67] P. N. Furbank/W. R. Owens: Whence the Defoe Canon? In: ECF 9.2 (1997), pp. 223-225. See also: P. N. Furbank/W. R. Owens: The Defoe That Never Was. A Tale of De-Attribution. In: The American Scholar 66 (1997), pp. 276-284. P. N. Furbank/W. R. Owens: Defoe De-Attributions. London: The Hambledon Press 1994. P. N. Furbank/W. R. Owens: The Canonisation of Daniel Defoe. New Haven: Yale University Press 1988.
[68] Paula R. Backscheider: Daniel Defoe. His Life. Baltimore: The Johns Hopkins University Press 1989.
[69] Paula R. Backscheider: Private e-mail to Joseph Rudman, 12 August 2002. Paula R. Backscheider: Private e-mail to Joseph Rudman, 3 July 2002.
[70] Franz Horton: Studien Über die Sprache Defoe's. Dissertation, Rheinischen Friedrich Wilhelms Universität zu Bonn 1909. (Bonn: Verlag Peter Hanstein 1909).
[71] Stephen H. Dill: An Analysis of Some Aspects of Daniel Defoe's Prose Style. Dissertation, University of Arkansas 1965.
[72] Gustaf Lannerd: An Investigation into the Language of Robinson Crusoe as Compared with that of Other 18th Century Works. Dissertation, University of Lund 1910. (Uppsala: Almqvist & Wiksells 1910).
[73] Joseph Rudman: DNA and Non-Traditional Authorship Attribution. An Inclusive Model. Paper delivered at the ALLC/ACH 2002 conference in Tübingen, Germany. 25 July 2002. Joseph Rudman: Unediting and De-Editing Defoe for Non-Traditional Authorship Attribution Studies. Paper delivered at the SCSECS 2002 conference in South Padre Island, Texas. 21 February 2002. Joseph Rudman: The Style-Marker Mapping Project. A Rational and Progress Report. Paper delivered at the ALLC/ACH 2000 conference in Glasgow, Scotland. July 2000. Joseph Rudman: Non-Traditional Authorship Attribution Studies. Ignis Fatuus or Rosetta Stone? In: Bibliographical Society of Australia and New Zealand Bulletin 24.3 (2000), pp. 163-176. Joseph Rudman: Problems with Attributions to the Canon of Daniel Defoe. Traditional and Non-Traditional Methodologies. Paper delivered at the SCSECS 1999 conference in Shreveport, Louisiana. 25 February 1999. Joseph Rudman: The State of Authorship Attribution Studies. Some Problems and Solutions. In: Chum 31 (1998), pp. 35-365. Joseph Rudman: The Computer and Daniel Defoe. Paper delivered at the MLA convention in San Francisco,California. 29 December 1998. Joseph Rudman: The Prose Style of Daniel Defoe. A Computer Assisted Analysis. Paper delivered at the Eighth International Conference on Computers and the Humanities in Columbia, South Carolina. 11 April 1997. Joseph Rudman: Does ANON = DEFOE? Paper delivered at the Seventh International Conference on Computers and the Humanities in Provo, Utah. 28 June 1985. Joseph Rudman/David Banks: Questionable Attribution in the Canon of Daniel Defoe. A Study of Techniques. Paper delivered at the ALLC/ACH 1992 conference at Oxford, England. 7 April 1992. Laura A. Curtis: Private letter to Joseph Rudman, 13 May 1987. P. N. Furbank/W. R. Owens: Private letter to Joseph Rudman, 29 April 1985.
[74] John Robert Moore: Defoe in the Pillory and Other Studies. Bloomington, Indiana: Indiana University Publications 1939, p. 151.
[75] John Robert Moore: A Checklist of the Writings of Daniel Defoe. Archon Books 1971 (IId ed.), p. 41.
[76] Rudman: The State of Authorship Attribution Studies. Some Problems and Solutions (footnote 73).
[77] Rudman: Non-Traditional Authorship Attribution Studies. Ignis Fatuus or Rosetta Stone? (footnote 73).
[78] Text Encoding Initiative: <http://www.tei-c.org> (14.10.2002).
[79] Rudman: Unediting and De-Editing Defoe for Non-Traditional Authorship Attribution Studies (footnote73).
[80] Ian Lancashire et al.: Using TACT with Electronic Texts. New York: Modern Language Association of America 1996. TUSTEP (Tübingen System of Text Processing Programs) <http://www.uni-tuebingen.de/zdv/tustep/> (14.10.2002).
[81] Gregory Grefenstette/Pasi Tapanainen: What is a Word? What is a Sentence? Problems of Tokenization. In: Proceedings of the 3rd International Conference on Computational Lexicography. Budapest: Research Institute for Linguistics, Hungarian Academy of Sciences 1994, pp. 79-87.
[82] Rudman: The Style-Marker Mapping Project. A Rational and Progress Report (footnote 73).
[83] Rudman: DNA and Non-Traditional Authorship Attribution. An Inclusive Model (footnote 73).
[84] P. N. Furbank/W. R. Owens: Dangerous Relations. In: The Scriblerian and the Kit-Cats 33.2 (1991), pp. 242-244.
[85] Stanley Fish: Is There a Text in This Class? The Authority of Interpretive Communities. Cambridge: Harvard University Press 1980. Louis Milic: Contra Fish. The Arrogance of Misreading. In: Style 19.3 (1985), pp. 385-394. R. J. Dilligan: Effective Stylistics. A Response to Affective Stylistics. In: Revue Internationale De Lexicographic 30.1 (1977), pp. 75-92.