Creating archival data in linguistic fieldwork
Nicholas Thieberger
Pacific And Regional Archive for Digital Sources in Endangered Cultures (PARADISEC)
Department of Linguistics & Applied Linguistics
The University of Melbourne Vic 3010
Australia
thien@unimelb.edu.au

Abstract
Many indigenous or endangered languages encounter new technologies through the presence of a linguist whose work usually results in the only permanent record of that language.  How can we, as linguists, modify our fieldwork methods to ensure that the data we create will have use not only in our research projects, but will also be reusable and citable by others in the future. In this paper I discuss methods developed as part of my research on South Efate, an Oceanic language of Vanuatu, and also in the construction of the archive of cultural materials, the Pacific And Regional Archive for Digital Sources in Endangered Cultures (PARADISEC).

Introduction
Linguistic fieldwork typically involves recording information by use of dynamic media (audio or video), writing notes, or taking photos. The task after fieldwork is to analyse this material and to make generalisations based on the data. As we know if we have tried to find these records in archives, the better their quality, the better the information we can derive from them. Furthermore, the context of the recording, and the relationships between these items all need to be recorded in addition to the items (tapes, books, films and so on) themselves. While it has always been possible to document the links between these items, it has become an easier task with the recent advent of digital recording media and standards for the encoding and exchange of digital data. The result of engaging with new technologies and using standard open formats is that our data becomes more accessible, both for ourselves and our research, but also for longterm archiving and for repatriation to the people we record and to their descendants. To this end, then, we need to build the production of well-formed data into our normal operating procedures. This is all the more important for the many small languages (those with few speakers) for which our work may be the only documentary effort. These languages are often put under pressure from central authorities determined to impose national languages, and in some cases the speakers of these languages are targets of military or company-based campaigns that are virtually genocidal. While fieldwork becomes more problematic in these circumstances, the records produced take on even more importance and need to be well cared for. In Australia, we have seen most of the indigenous languages wiped out, mainly the result of successive generations of colonial government policies and public attitudes that unfortunately continue up to the present.

I have been working with speakers of South Efate, an Oceanic language from central Vanuatu, to document their language since the mid-1990s. From a fieldwork perspective, this has involved visiting and living in a village in South Efate and recording various aspects of local knowledge, including stories, events (like court hearings), procedural information (like how to make a canoe), plant names and uses and so on, in addition to elicitation to test my grammatical analysis. I want to be able to make all of this information accessible for myself and for the speakers of the language now, as well as preparing it for future accessibility. My PhD dissertation (Thieberger 2004) included documentary aspects, in particular it gave the reader the ability to play audio or video of example sentences and texts. The need to have a longterm location for collections like my own has also prompted a group of Australian researchers to establish an archive called the Pacific And Regional Archive for Digital Sources in Endangered Cultures (PARADISEC).

Ability to locate material
Traditionally collections of tapes and notes made in the course of fieldwork are kept by the researcher until they retire or later. It has required some effort on the part of anyone interested in reading these notes or in hearing these tapes, first, to know that they exist at all, and, second, to locate them and be able to use them. A solution is to provide descriptive tools for small collections that can both announce the existence of a collection and provide a simple but sufficient description of what is in the collection. By using standard ways of describing these collections and by publishing these descriptions on the web they become a resource available to the research community. We have adopted the standards proposed by the Open Language Archives Community or OLAC, which include a set of basic descriptive information or metadata as outlined in Table 1.

Table 1. Key metadata fields used by PARADISEC
1.	Title	Item name	
2.	Identifier	Unique resource identifier	
3.	Contributor	Name of the person primarily responsible for creating the resource	
4.	Item	Identification as given by the depositor	
5.	Role*	annotator; artist; author; complier; composer; consultant; data_inputter; depositor; developer; editor; illustrator; interviewer; participant; performer; photographer; recorder; researcher; respondent; speaker; signer; singer; sponsor; transcriber; translator	
6.	Coverage	ISO country code (soon to be supplemented by lat/long polygons)	
7.	Language as in source	Freeform language name as given by the depositor	
8.	Subject language and code*	Ethnologue names and codes	
9.	Content language and code*	"	
10.	Date	Date the item was created	
11.	Data type*	lexicon; primary text; language description; song; instrumental music	
12.	Rights	Access conditions	

Assessing legacy material and making it useful today
At PARADISEC we assess collections of field recordings or notes – currently held in disparate physical locations, such as filing cabinets or cupboards – that it usually requires local knowledge and memory to locate. The process of assessment includes deciding whether tapes can be used at all, and if so, what their content is and how important or unique it may be. Tapes, for example, of language-learning sessions in major languages are less likely to enter our queue than are tapes of narratives in languages no longer spoken. A further consideration is the degree of current interest expressed to us by people who want to work with the language materials. 

If the researcher is still alive then we can ask them to provide descriptions of the items in the collection, but if they are no longer able to do this, then we simply do not have the staff or time to work through notes in order to enrich the descriptive metadata. Most of this material will eventually be digitised, but in the first instance we enter brief cataloguing information into our metadata catalogue at a collection-level in order to make the existing analogue collection locatable by interested researchers. They, in turn, can assist in determining the priority of parts of the collection and so the order in which the collection should be digitised. For example, we have been working with a large number of tapes left by the late Professor Stephen Wurm. One part of his collection relates to the Solomon Islands, and the Reef Islands in particular. We have taken images of fieldnotes or tape transcripts and placed them on a website, using software that maintains relationships between the notes and links them via a database. As we digitise tapes in this and other collections we give priority to those tapes requested by current researchers and then make password access available to them available on the internet. We have asked colleagues in Norway and the Netherlands who are currently working in the Reefs and Santa Cruz islands to look at this material on the web. Their advice on current language names and dialect relationships of the items in the collection in turn allows us to improve the finding aids we can provide. 

Locating the speakers or performers or their descendants.
A major motivation for preserving field recordings is to make them available to the speakers recorded and to their descendants. While it may appear that the use of the latest technology for encoding these recordings puts them out of the reach of the speaker in the village, we take it as our responsibility to use the best possible means to secure the data in a way that allows it to be reused in future. Reusability means that the archival copy can be reproduced in more accessible forms, and so we make CD audio copies, or Mp3 copies on CD for distribution. As it is often impossible for us, based in Australia, to know where these CDs should go to for repatriation, we rely on local cultural centres to take the copies and then make them available to the appropriate people. We have returned digital versions of tapes to a speaker of a New Caledonian language who recorded them in 1980. We have also provided CDs to the Vanuatu Cultural Centre and the Institute of Papua New Guinea Studies. We are hoping to expand this network throughout the region.

Citing data in our analysis
In the presentation of scientific hypotheses and arguments it is unusual, if not unheard of, to present conclusions without presenting the set of data on which they are based. It has, on the contrary, long been the accepted practice among linguists to give an example as proof that a certain morpheme, word or construction occurs, without citation of the source of that example. If an example has no citable source it is impossible for the reader to verify that it actually occurred, or what kind of context it may have occurred in. While, in the past, it was difficult to cite points where an example occurred within analogue data, it has become much easier in the past decade with increased use of digital media. Nevertheless, it is still rare to find such citation forms in theses, grammars and academic papers. Part of the problem has been the lack of suitable software for building media-based corpora, and that has been addressed by tools like Transcriber or Elan, especially for creating time-aligned transcripts of media files. Another part of the problem was that there were no suitable repositories which would provide persistent identification for archival data. That too has changed as the international network of digital language archives expands and we are seeing regional archives conforming to the kinds of standards discussed in this paper. When we have citable data sources with persistent identification then we can build on them to create new and more complex objects. For example, dictionaries can point at images, or at points within archival media files, using the names of data files and their locations, rather than having to handcraft the links between these objects. We can create links between fieldnotes and media they refer to, or provide a textual index of manuscript material, linked to images of the material itself.

Avoiding inappropriate use of the material
Having established safe archival digital copies of the data the task of managing this collection includes questions of access. The rights in this kind of data can be difficult to determine, especially if the recorder has passed on and we have no way of knowing how to contact those recorded. If there was provision in their will for the material to be looked after then it is likely that they also provided some indication of how it could be used. Ideally there would be a document outlining the form of consent provided by the speakers to the recorder that outlined who can look at the data, or make copies, and what can they use it for. We encourage the use of consent forms by fieldworkers and ask them to ensure that the people they work with can tell them how the recordings may be used.

When tapes or other data are deposited in an archive a deposit form specifies what conditions are to be placed on their access. Access to any of the material in PARADISEC’s collection is subject to receiving a signed access form. For authorised users the means of accessing the collection is via a password and username and we have the means to assign these to a user for a limited period of time and for access at the level of a single item, or at the level of a collection of items. 

Ensuring its longevity
A major reason for digitising analogue tapes is that it is the best archival form for this data. Digital data can be copied with no loss of resolution, unlike analogue tapes for which each copy lose some of the signal. Copies are the same as originals which means that multiple copies can be made and stored in diverse locations, so increasing the likelihood that copies will survive. Digital data can also be migrated to new platforms and media automatically and globally, that is, a whole collection can be copied without having to handle small media objects like cassettes, tapes or records as we used to do. As technology will keep changing, the key to ensuring longevity of data storage is to keep migrating data and to store it in open formats (those that do not require proprietary software to be read).

Providing a secure base is an aspect of ensuring data longevity that is less easily planned for. There are few institutions that can commit themselves to hosting an archive in the longterm. In Australia, our project was established precisely because there was no adequate repository of this kind. The Australian National Library or National Film and Sound Archive are not funded to look after material recorded outside of Australia. Our collection is housed in four separate locations in two cities, Sydney and Canberra, and we are planning to have mirrored copies on other continents in future as part of the Digital Endangered Languages and Musics Network (DELAMAN). A digital collection can be copied and mirrored without loss of quality, the major requirements are proper authentication procedures to safeguard access to the data, large hard disks and speedy networks. We are yet to place our collection on a secure funding footing, but have the commitment of the Australian Partnership for Advanced Computing to migrate the current collection over time. 

Research possibilities
In addition to the importance of preparing data for repatriation and for use in future, the methods described in this paper allow us to create citable forms of our research data in order to do better research now. As was pointed out earlier, this means we can provide data together with an analysis and so allow our claims to be tested. But it also allows us to create new views of the data that can assist in our analysis. A simple example is the ability to access points within large media recordings via a textual index or transcript which itself is indexed by a concordance, allowing us to view all examples of a given word or morpheme in context and then to discover the rules governing its use. 

By creating the data using good naming conventions and persistent locations we can automatically establish links between objects. Dictionary entries that reference external objects like filenames or points within a media file allow us to create rich linguistic and cultural representations. For example, an entry for a tree name in a lexical database could include a field which is the name of the picture of the tree, and include a time-coded reference to a video clip in which the tree and its uses are discussed. These links can be followed by the user because they have targets that can be resolved, due to good naming practices and persistent locations of the objects involved. 

It is a fundamental part of scholarship that we cite our sources, yet it has not been easy to do so for the numerous recordings we make in fieldwork. Digital data stored in a suitable repository provides us with persistent citable sources that can then be consulted over time.

Alternative to taking this path
There is a growing awareness among linguists that language documentation involves activities that are distinct from those involved in language description (cf Himmelmann 1998, Woodbury 2003). Linguists are now at a point where we can either engage with language documentation and its possibilities, or we can continue to work as we have for some decades now. What if we decide not to pursue the possibilities offered by digitising and engaging with the approach advocated by the emerging network of language archives in the spirit of language documentation? The invaluable recordings we make will become unplayable as the machines we used breakdown and can no longer be repaired. In the worst case, our files will be held in proprietary software that we can no longer read and they too become unusable. We certainly will face problems in using data for new purposes, as, for example, in changing the form of dictionary we want to present from a Microsoft Word document to a structured lexicon for example. If we simply type the dictionary using font changes to indicate parts of the dictionary entry then we are bound to introduce errors and inconsistencies into the work, and any change we want to make to that work has to be similarly made by hand to each individual item in the dictionary. If on the other hand, we structure our dictionary so that each part is marked for the kind of entity that it is (headword, part of speech, definition and so on) then we can allow the output form of the dictionary to be determined by its structure. This also allows us to convert the structured data into a styled MS Word document, a web-based dictionary, or a multimedia tool. The crucial point is that the underlying data is in a well-structured form that can be archived, as it is a simple text file.

This loss of access to the data is a critical issue for our relationship to the people we record. We are entrusted by them and by the funding agencies that support our work to create not only a grammatical analysis but also a permanent record of the language that we work with.

Implications for current fieldwork
At PARADISEC we have been active in providing training in new techniques for linguists, postgraduate students, and for language workers in Aboriginal language centres. We have participated in the establishment of an advisory group called the Resource Network for Linguistic Diversity (RNLD) which maintains a mailing list and web-based advice pages in order to network between anyone involved in language work. New tools and methods for conducting linguistic research are appearing all the time and by sharing ideas and experience we hope that we can strengthen language documentation efforts. Based on the kinds of considerations outlined so far in this paper, there are a number of implications for fieldwork practice. The first is that we have to discuss the expectations of the people recorded about how the data can be used so that we are in a position to safely archive it and to specify access conditions for it. We advocate the use of a 'consent form' that lays out what can and cannot be done with the data. When we record someone we need to clarify with them how the recording can be used, and this is best done by use of a signed consent form, or an acceptable alternative is to record the speaker's intentions on the fieldtape. It is not always an easy task to get across what we want to use the recordings for, and there may be a conflict between the notion of free information for academic purposes and the local conception of particular items of knowledge being owned by individuals. Further, there are very real intellectual property issues involved in exposing local knowledge in a broader venue like a published work. All of this becomes even more complicated when we consider lodging recordings in an archive and then plan for longterm access conditions. Nevertheless, these are issues that need to be recognised, discussed and addressed as part of the broader project of language documentation.

When making a recording it is crucial to note on the recording medium (if it is an analogue recording) and in an appropriate database the metadata or cataloguing information that will make this recording locatable in future. As outlined earlier, this is the basic descriptive information that will be made available via search engines. Recordings themselves should be high quality, which means using good microphones with flashram recorders with standard format wav files and avoiding any compression algorithms like Mp3 or ATRAC (as in minidisks). Older recordings should be digitised using appropriate equipment to ensure their ongoing legibility, but also to provide a means for time-aligned transcription and archiving. Connecting a tape-recorder output to a normal computer input is not a suitable way to digitise archival audio data as internal machine noise can create clicks and dropout in the signal. External soundcards are a preferable means of digitising.

Transcription of these digital files should include time-alignment, using, for example, the tools mentioned earlier, Transcriber or Elan, and resulting in plain text transcripts with stand-off markup that looks like this:
<Sync time="36.972"/>
Ore. Wel kumurin, patli , patli. 
<Sync time="41.188"/>

These transcripts allow us to work with our data at a finegrained level (the sentence, or, if we want, the word) which then allows us to automate retrieval of this information at that level. and to check and improve our transcripts. It will become an integral part of linguistic field analysis to be able to cite parts of a corpus in this way and to use these citations in our descriptive work.

As we select parts of our transcriptions for more detailed analysis, beginning with interlinearisation, we can maintain the timecode references and so produce richly annotated texts that are playable. Subsequent textual work, such as linking to a lexicon can then build on this complex data. 

To produce archival lexical files we can take advantage of specific tools like Shoebox or Toolbox which are designed to both create dictionaries and provide glosses for interlinear files, harvesting words from the texts to add new items to the dictionary.

Ultimately we will be able to create richly interlinked documents that use the predictable structure we have encoded to create automatic links. For example, if we tag all utterances for type (exclamation, narrative, procedural, etc) and the speaker's name, and we have a table listing speakers and their characteristics, it should be possible to hear all exclamations made by a male speaker under thirty. Similarly, a grammatical description can be created in the same way, and so allow automatic interlinking between the information in the grammar and the corpus of texts that it draws on. 

Conclusion
By adapting our fieldwork methods we can ensure that our recordings, our annotations of them, the derived texts and dictionaries, as well as the analysis all persist, are locatable and are usable into the future. There are huge advantages to us in our linguistic analysis, but here are even greater advantages in the legacy we provide for posterity.

References
Himmelmann, Nikolaus P. 1998. Documentary and descriptive linguistics. Linguistics(36). 161-195. 
Woodbury, Anthony C. 2003. Defining documentary linguistics. In Peter K. Austin (ed.), Language documentation and description, Volume 1, 35-51. London: SOAS.
Thieberger, Nicholas. 2004. Topics in the grammar and documentation of South Efate, an Oceanic language of Central Vanuatu. PhD Dissertation, Department of Linguistics and Applied Linguistics, University of Melbourne.