Counting Collections

As will be clear to regular readers of this blog, we are concerned here to encourage the creation of the best possible records of small languages. Since much of this work is done by researchers (linguists, musicologists, anthropologists etc.) within academia, there needs to be a system for recognising collections of such records in themselves as academic output. This question is being discussed more widely in academia and in high-level policy documents as can be seen by the list of references given below.

The increasing importance of language documentation as a paradigm in linguistic research means that many linguists now spend substantial amounts of time preparing corpora of language data for archiving. Scholars would of course like to see appropriate recognition of such effort in various institutional contexts. Preliminary discussions between the Australian Linguistic Society (ALS) and the Australian Research Council (ARC) in 2011 made it clear that, although the ARC accepted that curated corpora could legitimately be seen as research output, it would be the responsibility of the ALS (or the scholarly community more generally) to establish conventions to accord scholarly credibility to such products. Here, we report on some of the activities of the authors in exploring this issue on behalf of the ALS and discuss issues in two areas: (a) what sort of process is appropriate in according some form of validation to corpora as research products, and (b) what are the appropriate criteria against which such validation should be judged?

“Scholars who use these collections are generally appreciative of the effort required to create these online resources and reluctant to criticize, but one senses that these resources will not achieve wider acceptance until they are more rigorously and systematically reviewed.” (Willett, 2004)

We propose that the process, which would parallel quite closely the peer review process applied to traditional publications, should involve the following steps:

1.     Submission of a corpus by its creators to the ALS with a request for review.

2.     Refereeing by an anonymous panel representing the ALS.

3.     Report of the panel provided to the creators, possibly with suggestions for improvements.

4.     Response by the creators, including possibly including revisions to the collection.

5.     Publication of a summary of the panel’s report and the creators’ response under the auspices of the ALS (for example in the ALS newsletter, AJL, or the ALS website and constituting recognition of the corpus as a research product.

The question of what criteria to use in evaluating a corpus is more problematic, but we believe that some aspects should not be controversial. The accessibility of a corpus is clearly fundamental. As only accessible data can be considered to be “published” only such data can be included in the review.  Beyond the data being in principle available to corpus users, accessibility can also be defined (measured?) in terms of the technical resources necessary to use it (including metadata) and in terms of the background information provided in order to make the data comprehensible. On the other hand, we are less certain that consensus will be easily achieved on questions about quantities of data and the weight to be given to different data types. For example, we would expect a useful corpus to include some fully glossed material (in most cases anyway), but is there a minimum amount (or minimum proportion) of such material which should be taken as a threshold for approval? Another difficult question is whether it is useful to attempt to establish any equivalence between corpora as output and traditional publications, and if so, how any such equivalence might be measured.

As researchers are being asked to account for research quality, it is timely to consider the importance of collections of research data as an indicator of research quality. Such collections can, like any research endeavour, vary considerably in size and in the amount of curation that has gone into developing them. Further, primary data, while critically important to preserve for future reuse (particularly by the source community) and for verification of analyses, is best created within a framework that allows for its description and annotation. For example, using international standard metadata systems (such as Dublin Core or Open Archives Initiative) for description of research material means that it can then be located using any targeted search tool, resulting in more precision in searches than is possible using generic tools like google.

How can we ensure that collections of primary language data can count towards academic advancement?

This blog post summarises a presentation given at the Australian Linguistic Society conference in Perth (December 2012), titled: Assessing curated corpora as research output: issues of process and evaluation. The authors are Anna Margetts (Monash) , Stephen Morey (La Trobe), Simon Musgrave (Monash), Adam Schembri (La Trobe), Nick Thieberger (Uni Melbourne).

Suggested process for assessing a collection

1.  Submission of corpus by creators to ALS with request for review

2. Review by ALS committee

3. Report of committee to the creators

– possibly with suggestions for improvements

4. Response by the creators

– including possibly revisions to the collection

5. Publication of report summary & creators’ response
= recognition of the corpus as a research product

– E.g. in ALS newsletter, AJL, or ALS website

Criteria suggested for assessing collections

1. Collection is housed in a repository

– which has a commitment to long-term curation and access

– which provides a citation form for items within the collection.

2. Accessibility

–  are the files in the corpus in a format that is freely accessible?

–  do they depend on certain software?

•  this can render them difficult or impossible to use

–  does the software require additional meta-files?

•  e.g. Toolbox files need .typ and .lng files to be read

3. Contextual information

– background to how the collection came into being and overview of what it contains

4. Metadata

– sufficiently described to allow corpus to be located

– what it contains e.g. texts, media, vocab, speaker info, perhaps content keywords

5. Annotations

– transcription

– text-media linkage

– translation into language(s) of wider communication

– interlinear glosses

6. Content

– good range of speakers of different ages and genders

• relative to opportunity

– good range of text types (narratives, procedural, hortatory texts, written, songs, etc.)

• as per Himmelmann (2006:21) — intro to Gippert et al.

• relative to opportunity

7. Supporting documentation

– list of abbreviations used

– information on orthography and its relation to sound system

– grammatical information sufficient to allow further analysis

– lexicon (quality rated from short wordlist to detailed dictionary)

– ethnographic information

‘Quantity’ criteria

• Hours/units of primary data?

• Proportion of data which is annotated etc. could be handled as a quality issue, not a quantity issue

• Perhaps the quality rating may end up being independent of quantity?

 

Background documents

A. May 2002 ALS Newsletter: Open letter from linguistic postgraduate students:
Given
1) the recognition by linguists of the need for urgent work to record small (‘endangered’) languages and;
2) the availability of funding for that work, and;
3) rapid advances in technological aids to doing that work, then there is a clear need for the members of the Australian linguistic community to consider the following:
– the current PhD in linguistics makes no provision for language documentation beyond an academic grammar (in fact it positively discourages it).
– documentation of a language with few speakers or with little prospect of being spoken in the next generation should be considered a suitable PhD topic in linguistics.
– this language documentation would produce information in the language in a form that makes it accessible to speakers  and their descendants, and to the linguistic community. Typically this would include a grammar sketch, dictionary and texts.
– the form of the documentation would include as much information as possible, but would minimally provide audio and video recording of performance in the language. It could also include ethnobiological information such as pictures of plants and animals, their uses and names and so on.
– the document would include grammatical information, but not of the detail currently expected of a  PhD in linguistics.
– the document would be produced using current standard tools (e.g. digital recording, text/audio linkage).
– the document would be presented in archive quality and placed in an appropriate repository to ensure its accessibility and usability into the future
– This all entails training students in documentary techniques and linguistic data management.
In the long run it is the documentation that will prove more valuable for linguistic analysis than the traditional PhD. At present we have to rely on the writer of a PhD nearly 100% for some languages – and certainly 100% if the language is now gone.
While the current system values language analysis, it places no value on linguistic data management, nor on safely archiving recorded materials.

Claire Bowern – Harvard
Nicolette Bramley – UC/ANU
Anthony Jukes  – Melbourne Uni
Doug Marmion – ANU
Stephen Morey  – Monash Uni
Adam Paliwala  – Sydney Uni
Carol Priestley – ANU
Adam Saulwick – Melbourne Uni
St John Skilton – Sydney Uni
Nick Thieberger – Melbourne Uni
Myfany Turpin – Sydney Uni