<h2 id="id1">Language Data Commons of Australia</h2>
<h2 id="id2">Australian Text Analytics Platform</h2>
<h3 id="id3">Day of Activities before ALS2022 (Tuesday 29/11)</h3>
<table>
<tr><td>Time</td><td>Activity</td><td>People</td></tr>
<tr><td>09:00 - 10:00</td><td>Introduction to notebooks What are Jupyter notebooks and why are they useful?
</td><td>Sara King Simon Musgrave</td></tr>
<tr bgcolor = 'LightGray'><td>10:00 - 10:30</td><td>Break</td><td></td></tr>
<tr><td>10:30 - 11:00</td><td>General progress report An overview of the projects and of progress to date
</td><td>Michael Haugh Robert McLellan Simon Musgrave Jenny Fewster (ARDC)</td></tr>
<tr><td>11:00 - 12:00</td><td>Introducing the LDaCA Portal How will I find data in LDaCA? Your chance to tell us whether we are meeting your needs!</td><td>Peter Sefton Kathrin Kaiser</td></tr>
<tr bgcolor = 'LightGray'><td>12:00 - 01:00</td><td>Lunch</td><td></td></tr>
<tr><td>01:00 - 02:00</td><td>Data and access How is access to data managed in LDaCA and how can I access data?
</td><td>Peter Sefton Kathrin Kaiser Catherine Travis</td></tr>
<tr><td>02:00 - 02:30</td><td>Auslan project update Report on work by the team at Monash University
</td><td>Louisa Willoughby Trevor Johnston</td></tr>
<tr bgcolor = 'LightGray'><td>02:30 - 03:00</td><td>Break</td><td></td></tr>
<tr><td>03:00 - 03:30</td><td>Text analytics for corpus linguistics Selected ATAP tools showcase
</td><td>Sydney Informatics Hub with Sydney Corpus Lab (Sony Jufri, Kelvin Lee)</td></tr>
<tr><td>03:30 - 05:00</td><td>Discursis workshop How to get started with this tool for tracking topics in conversations </td><td>Ben Foley Sam Hames</td></tr>
</table>
The <a href="https://www.atap.edu.au/">Australian Text Analytics Platform</a> and the <a href="https://www.ldaca.edu.au/">Language Data Commons of Australia</a> are projects supported by the <a href="https://www.ardc.edu.au">Australian Research Data Commons</a> to develop infrastructure for Australian researchers who work with language data. This day of activities organised by the projects will give ALS conference delegates (and anyone else who is interested) the opportunity to learn more about this work. The day will include:
<ul>
<li>an overview of the projects and the work to date</li>
<li>reports on specific sub-projects</li>
<li>introductory workshops on the tools and resources being developed</li>
<li>a workshop on using Discursis, a tool for tracking topics in interactive use of language</li>
<li>the opportunity to influence future work by exploring and providing feedback on the data interface which we are building.</li>
</ul>
The activities will be of interest to anyone who conducts research which includes language data, especially those who use or would like to use computational tools in their research. Participants should bring their own computer; no software is required beyond a web browser.
Catering is provided and therefore a registration fee is being charged for this event to partially cover the cost ($25 full rate and $15 discount).
<a href="https://als.asn.au/Conference/Conference2022/Workshops">Registration</a>
The Australian Text Analytics Platform and the Language Data Commons of Australia received investment from the Australian Research Data Commons (ARDC). The ARDC is funded by the National Collaborative Research Infrastructure Strategy (NCRIS).

# Language Data - Background Information
### [Principles](/principles)
Information about the principles on which the work of LDaCA is based.

### [Technologies](/technologies)
Information about the technologies being used in LDaCA.

### [Metadata](/metadata)
Information about the approach to 
[metadata](https://en.wikipedia.org/wiki/Metadata) being taken by LDaCA.

### [Sample Collections](/collections)
Information about the first datasets which have been added to LDaCA.

### [Case Studies](/case-studies)
Accounts by collectors of various kinds of language data illustrating the 
different solutions they have adopted for the problems encountered in the 
process.

<hr class="dots" />

Background

## Fieldwork in Papua New Guinea
 

Harriet Sheppard's [account](../fieldwork-png) of collecting data on a remote island in Papua New Guinea.

 

## Fieldwork in Sydney
 
 

## Data in a Language Centre
 
 

## Archival data
 
 

Back to [Background](../background/)

Case-studies

LDaCA has begun ingesting data sets, including:

- Corpus of Oz Early English [CoOEE](https://varieng.helsinki.fi/CoRD/corpora/COOEE/basic.html): a collection of texts written in Australia between 1788 and 1900. The corpus is divided into four time periods (1788–1825, 1826–1850, 1851–1875 and 1876–1900) each holding about 500,000 words. Four registers were defined for CoOEE: the Speech-based Register (SB), the Private Written Register (PrW), the Public Written Register (PcW) and the register of Government English (GE). In every Period 1-4 there is a similar number of words in the different registers.
- [Sydney Speaks](https://www.dynamicsoflanguage.edu.au/sydney-speaks/): This project seeks to document and explore Australian English, as spoken in Australia’s largest and most ethnically and linguistically diverse city – Sydney. 
- [From Farms to Freeways](http://omeka.uws.edu.au/farmstofreeways/): This research project sought to analyse the experiences of women who had lived in the Blacktown and Penrith areas since the early 1950s, including their responses to social changes brought about by rapid suburbanisation in the Western Sydney region in the post-war period. Two-hour taped discussions were held with 34 women, aged sixty and over, who were in their early twenties during the Western Sydney region's population growth.

We have also ingested a collection of government documents in various languages. This a very small dataset assembled to check that our technology can handle different languages and different scripts; more information about this work is available in [this presentation](https://ptsefton.com/2022/01/27/DAMTA_Slides_v1/index.html).

Back to [Background](../background/)


Collections

### Contact us

You can contact the Language Data Commons of Australia by [email](mailto:info@ldaca.edu.au)

Our logo was designed by Otis Carmichael. [Read more](/designer) about Otis and the ideas behind his design.

We share a [Twitter account](https://twitter.com/LDaCA_Program) with the [Australian Text Analytics Platform](https://www.atap.edu.au):

Contact us

# Language Data
### [Principles](/principles)
Information about the principles on which the work of LDaCA is based.

### [Technologies](/technologies)
Information about the technologies being used in LDaCA.

### [Metadata](/metadata)
Information about the approach to 
[metadata](https://en.wikipedia.org/wiki/Metadata) being taken by LDaCA.

### [Sample Collections](/collections)
Information about the first datasets which have been added to LDaCA.

### [Case Studies](/case-studies)
Accounts by collectors of various kinds of language data illustrating the 
different solutions they have adopted for the problems encountered in the 
process.

<hr class="dots" />

Language Data


![LDaCA logo](/logo-dark.png)

### Otis Carmichael

The design of the LDaCA logo came from my own experience researching the Waanyi language, the language of [my mob](https://en.wikipedia.org/wiki/Waanyi). The design flows from the beginning to the end, where it flows back into itself at the beginning, representing the cyclical nature of life and the Land. The strands of language interweave with themselves, influenced by everything that touches them until they become something entirely their own. The complexity of these languages is a thing of beauty and their preservation, study and proliferation lays claim to this unceded land. The inspiration of the design is as follows:

**L** - Language is shown as a flow of words that can be merged, split and rearranged to create different meanings, just as these lines can be.

**D** - Data is depicted by these specks of raw information, undecipherable before processing but heavy with power.

**a** - This data is not just numbers and words but signals from the Universe, such as animal tracks.

**C** - Whilst research often appears niche in these commons, there is no limit to the range of it’s influence, the ripples of which disrupt everything it touches.

**A** - The language data of this common has been collected from many individual nations that make up ‘Australia’, often with a foundation on extraction based research techniques. This project aims to make up for the mistakes of the past by celebrating and giving back to these nations, which are so incredibly distinct whilst sharing beautiful connections with each other.

The colours utilised in this design draw from those that are seen in my peoples land of Boodjamulla, in the Gulf of Carpentaria. They reflect the connections between the Sky, the People and the Land in the hope that these connections continue to strengthen indefinitely.

**My Bio**
My name is Otis Carmichael and I am a proud Waanyi man living in Meanjin. My designs reflect my appreciation of the past and my hope for the future.


Otis Carmichael


# Fieldwork in Papua New Guinea

by Harriet Sheppard
![Harriet on an outrigger boat off Vanatɨna](/HS-sailing-outrigger-small.jpg)
Harriet on an outrigger boat off Vanatɨna

My fieldwork experience, in many ways, conforms to stereotypes of language 
fieldwork – a community outsider working with a smaller language community 
located far from an urban centre. For my postgraduate research, I documented 
and described grammatical topics of 
[Sudest](https://glottolog.org/resource/languoid/id/sude1239) (known as Vanga 
Vanatɨna by its speakers), an Oceanic language spoken in Papua New Guinea (PNG
). The language is spoken on the islands of Vanatinai (also known as Sudest 
and Tagula) and Yeina in the Louisiade Archipelago, approximately 350 
kilometres southeast of the PNG mainland. In the following, I discuss 
decisions I made and processes I followed related to the collection of 
language data and archiving the resulting corpus of texts, including ethical 
considerations, ownership of the data, data formats, metadata, and data 
security and reuse.  This is by no means meant as a guide (of best practice) 
for projects where language data are being collected with speakers of un(der)
described languages. Rather, it is an account of some of the types of 
considerations and procedures that need to be considered when collecting 
language data and how I handled them in this instance.


![Map of Louisiade Archipelago](/map-louisiade.jpg)

Before recording with a speaker for the first time, I would explain the 
project to the speaker and their rights as contributors to the project. I 
would explain how I would use the language recordings, how they would be 
stored and accessed and how speakers of the language or other researchers 
might access and use the recordings in the future. This included discussing 
how they, the person(s) making the recording, would retain ownership over the 
recording (i.e., intellectual and cultural property rights). They would also 
be the ones to choose access rights over the text and have the option to 
withdraw any or all recordings they made from the corpus at any point (more 
on these below). After these discussions, if the speaker was still interested 
in the project, I would then record an oral consent form with them. The 
consent form was in English, the language of wider communication in the 
province. In cases where the speaker didn’t speak or spoke limited English, 
we would have a translator present for this process. I chose to use an oral 
consent form as it is accessible to all participants no matter their literacy 
levels.  It also has the advantage of being a more durable format compared 
with a paper consent form, particularly when working in a hot and humid 
climate and it can also be easily backed up.

![Transcription session with Abel Sam](/transcription-session-small.jpg)
Each time I made a recording with a speaker or speakers, I played the 
recording back to them to check that they were happy with the recording and 
so that they could decide on what sort of access conditions they wanted the 
archived recording to have. If the speaker was not satisfied with the 
recording, sometimes they would choose to make another recording and delete 
the original or simply delete the recording and maybe try again later. If 
they were happy with the recording, they would decide on the access 
conditions they wanted for the archived recording. The possibilities ranged 
from the most restricted, being that only I, as the original researcher, 
could access the recordings and use them for my research, to the opposite end 
of the access continuum where anyone interested could download and listen to 
the recordings, read any transcriptions, and potentially use them for 
educational or research purposes. In between these two ends of the continuum, 
they could opt for other access conditions such as only allowing access for 
registered users of the archive, or only for researchers and speakers, or 
just for speakers. They could also choose to have a text embargoed for a 
certain period of time.


![Transcription session with Abel Sam](/elicitation-session-notes-small.jpg)
When I was preparing my ethics application for the university, I applied and 
received ethics approval to work with minors. I didn’t plan to collect data 
from children (although I did want the option of recording older teens if any 
showed an interest in recording), but included minors in my ethics 
application because of the potential for children to be inadvertently filmed, 
for example if they walk into frame or come up to a parent while the parent 
is being recorded. The ethics approval meant that if such instances arose 
during a recording, the recording could still be kept and archived (
assuming approval from the speaker making the recording). While such 
situations didn’t end up arising during my own data collection, this would 
not have been an ideal fix for this issue. More widely, as a discipline, we 
don’t have good model(s) regarding how we deal with such scenarios in regards 
to access and consent. One suggestion I heard was to only make such 
recordings openly available after obtaining informed consent from any 
children involved after they turned 18 but, for most circumstances, such an 
approach is unlikely to be realistic.

As the researcher who collected the data and deposited it in the archive, I 
am the person the archive contacts when someone puts in a request to access 
restricted data. But granting any access to data, whether restricted or not, 
is based on the wishes of the owner(s) of a specific text, that is, the 
speaker(s) who recorded it. As mentioned, the speaker(s) retains ownership of 
the text and intellectual and cultural property rights over the recording. 
They also retain the right to change access restrictions on a recording or 
remove it completely from the archived corpus at any point and request that 
it not be used in any future research from that point forwards (given it 
would be difficult or impossible to remove excerpts from recordings in 
previously submitted or published work on the language, this is not something 
that is generally done). Away from the field, this can get a bit complicated 
due to limited telephone and internet coverage in the islands although it is 
increasingly more feasible as more and more people have access to mobile 
telephones and apps like WhatsApp and Facebook. Any future users of the 
corpus who download data from the archive must agree to only use the data for 
educational or research purposes and not for (direct) financial gain. Having 
said that, it needs to be acknowledged that researchers including myself can 
and do benefit indirectly from such data collections in the form of 
scholarships, grants, qualifications, and other potential employment 
opportunities related to our work with the language.  Although I, like many (
hopefully most) fellow linguists, subscribe to the idea that the source of 
the data, that is the speaker(s), should have control over their own 
recordings, the current systems we have set up make that hard or impossible 
to implement. This is something that is increasingly being acknowledged and 
we as a community of researchers don’t currently have answers for. Each 
speaker community is different and so one-size-fits-all models for 
communities are unlikely to work. 

When collecting spoken or signed language data, the best practice is to make 
audio and video recordings in formats that are ‘lossless’ rather than ‘lossy
’. Lossless compression allows for perfect reconstruction of the original data
. Lossy formats reduce file size but also discard some information that 
cannot be reconstructed which is obviously not an ideal outcome for precious 
language data! For audio recordings, I use a recorder that records Waveform 
Audio File Format or WAV files (.wav) and for video recordings I record 
Advanced Video Coding High Definition or AVCHD files (.mts) which are both 
lossless formats and accepted by many archives. WAV files are also the audio 
format needed for many linguistics software programs. AVCHD files are, however
, not a file type accepted by the archive where the Sudest corpus is housed 
which only accepts MPEG video files (.mpg) which are lossy. This situation is 
likely due to the fact that the archive was set up some two decades ago when 
storage capacity was a big issue and the archive has not yet caught up in 
this regard. This is not an ideal situation and something that should be 
considered if you are able to choose the archive where your data will be 
deposited. For derived data including text-audio aligned transcriptions and 
translations as well as any other annotations, I use plain text (.txt) and 
ELAN Annotation Format files (.eaf). The plain text files are the file type 
created when working in the Field Linguist’s Toolbox software program and can 
be opened, read, and edited in most text editors.  It is also a good format 
for storing information if you want it to persist. The EAF files are more 
specialist although they are a transferable format. They are only created 
and used with ELAN which is an annotation tool for audio and video widely 
used by linguists. Both text and EAF files are some of the standard file 
formats for transcriptions and annotations that are generally accepted by 
language archives today. 

![Setting up for a recording session](/recording-setup-small.jpg)
When collecting metadata, I adopted the Component MetaData Infrastructure (
CMDI) schema used by the archive where I was planning to deposit my data. The 
logistics of fieldwork can be overwhelming at times and following the archive’
s metadata schema meant I would minimally have the metadata required to 
deposit the corpus. The schema was also practical in that there were some 
fields that were obligatory (e.g., text title, creation data, country of 
recording) but many fields are optional (e.g., speaker date of birth/age, 
education level, address where recording took place). Having optional fields 
that could be filled in or not worked well for me because collecting more 
detailed information, particularly biographical information about speaker(s), 
wasn’t always possible or appropriate. Not all speakers, particularly older 
speakers, necessarily knew their date or year of birth and they may only know 
their approximate age. Some speakers I only met when they came to make a 
recording and getting the detailed biographic information one might like for 
metadata records is not necessarily culturally appropriate, particularly 
given the power imbalance between myself, an educated, white researcher from 
Australia, and speakers I don’t have an established relationship with. Aside 
from asking about where the speaker grew up, currently lived, and their 
approximate age, if I didn’t already have an established relationship with 
the individual, I tended to let information emerge incidentally as 
volunteered rather than questioning them in depth. One topic I did discuss 
with speakers regarding metadata was the question of authorship and 
attribution of the recording and whether they wanted their name to be listed 
or would like to be given a pseudonym. With small communities, especially 
when you are recording audio and video of the speaker, complete anonymisation 
is not really possible if the speaker also wants the recording to be 
accessible to other speakers and this could be a potential issue. However, in 
my experiences, (all) speakers chose to have their names listed as the author(
s) of the text without anonymisation and were generally excited or proud to 
be making a record of their language and knowledge.    

While the amount of personal metadata collected for individual speakers varied
, there was also situational data that I wish I had collected and didn’t. I 
collected information including the date and location of the recording (e.g., 
the village and house where the recording took place), who was present at the 
recording, the genre of recording (e.g., a historical narrative, conversation
, instructions on how to complete a specific task, etc.). One piece of 
information that I didn’t include, for example, is the exact location of the 
speaker and their orientation in space. Were they facing towards the north? 
North-west? Since my original fieldwork, I have begun to research the use of 
co-speech gestures in the language, that is, gestures that occur at the same 
time as someone is speaking. Frequently, speakers point to real world 
locations while they are talking but this can be hard to identify if you don’
t know exactly how the speaker was positioned in relation to where they are 
gesturing. Luckily, by watching the videos I can still identify the direction 
a speaker was facing and add this to the recording’s metadata but this would 
be a very difficult if not impossible information for a future user of the 
data to ascertain if they wanted to study gesture. 

Data security is something I had to consider both in the field and at home. 
In order to keep data secure on computers and storage devices, I made sure 
that they were all password protected. In the field, I also had to consider 
specific factors in the environment that could endanger the data - the major 
issue is the humidity. The average humidity level on the island hovers around 
80 percent year-round and I had paper notebooks, a computer, hard drives and 
SD cards that I need to keep in working order in a house that is relatively 
open to the outside environment with no glass windows and no electricity (I 
do remember a field manual suggesting a fridge makes a good home for 
equipment in humid environments if that is an option). To protect equipment,
I stored everything in waterproof pouches with silica sachets. I would 
distribute backup USBs and SD cards across different pouches so if one failed
, I would hopefully have backups. I also tried to double bag equipment and 
notes when travelling, particularly by boat. Boat transport in the islands is 
either by traditional wooden sailing outrigger canoes or fibreglass dinghy ‘
banana’ boat, both of which are open to the elements. 

To secure data, ideally, I would also make backup copies of all new 
recordings and transcription files each day in the field. However, during my 
first fieldtrip, I didn’t have access to a power source and therefore couldn’
t charge my laptop meaning I couldn’t backup recordings. As this also meant I 
was transcribing all texts using pencil and paper, I did photograph all new 
transcriptions each night to preserve a backup copy. Towards the end of my 
trip, I became quite worried about having no backups for recordings, 
particularly when thinking about the boat trip back to the main island which 
can take up to eight hours (or more in bad conditions). To avoid making this 
trip with only the original copies, I tagged along on a walk to the local 
primary school two and a half hours away because it was rumoured that one of 
the teachers had a solar panel connected to a car battery and I would be able 
to charge my laptop and therefore backup the recordings. All the community’s 
primary school students make this walk twice a week, living at school during 
the week. At one point in the journey, you have to cross a river in a small 
outrigger canoe. It was quite alarming when the teenager paddling me across 
nonchalantly pointed out a nearby crocodile while we were mid-river but at 
least I managed to back up all the files! For subsequent fieldtrips, I 
invested in a battery and solar panel for charging equipment and backing up 
data although I still had to cross rivers with crocodiles from time to time. 


![Crossing the Veora river](/crossing-river-small.jpg) 
In order to secure the corpus of texts into the future and make sure that it 
doesn’t get lost on an old hard drive, I have uploaded it to a language 
archive. The archive is a digital repository tasked with safeguarding 
language corpora of smaller languages for which there is limited documented 
data. As well as keeping copies of the uploaded data, the archive will also 
ensure that the data continues to be accessible, for example, by converting 
data files to newer formats if the format a file was uploaded in becomes 
obsolete. Having the corpus saved in an online archive also means that it is 
discoverable for other researchers, speakers, and other interested parties. 
It can be found through the search portal of the archive by searching by 
language name(s), the contributor’s name, or by browsing by country. The 
corpus can also be found by searching through the Open Language Archive 
Community (OLAC) portal which is an online virtual library of information and 
links to language resources available in a specific language (e.g., grammars, 
dictionaries, archived texts). The information on OLAC is automatically 
collected or ‘harvested’ daily from participating archives meaning that when 
you upload a file to an archive, it will automatically be listed in OLAC the 
next time it is updated. Because there is limited information online about 
Sudest, keyword searches on most online search engines such as Google also 
displays links to the corpus quite high up in the results.  

It's often suggested that the researcher should make a will and assign a 
literary executor for the data they collect. This raises the question about 
who to nominate? As a graduate student, I nominated one of my supervisors as 
the person to have direct access and editing powers over the archived corpus. 
This makes sense in that, after myself, my supervisor is one of the people 
that knows the most about the project and data collection. Supervisors are 
also often the chief investigator for ethics approval for graduate data 
collection as well. But supervisors tend to be older than graduate students 
and don’t necessarily have any personal connection to the speaker community. 
Ideally, any succession plan would bring control of the data back to the 
community but what would that mean in practice? There may be issues both 
political and practical if just one individual from the community becomes the 
executor. In some cases, there may be an obvious option if the community has 
a language and culture centre or registered Indigenous corporation. In such 
cases, access and stewardship plans may have been discussed and incorporated 
into the project since the beginning. For many communities, especially in the 
Pacific, this is not the case and therefore there is no clear governing group 
which might take such a task on. 

In future research projects I might work in, I can build on my past 
experiences and hopefully improve on them. This would likely involve 
expanding the list of situational metadata I automatically collect for each 
recording as well as considering the file types accepted by an archive when 
deciding where to deposit any recordings. Although I’ve only touched briefly 
on issues relating to speaker-community control and intellectual and cultural 
property rights, I would also aim to build in more funding and space into the 
timeline of a project for more equitable community consultation and 
collaboration. Such discussion would then likely have a flow-on effect for 
how to secure the data into the future. Ethical and practical questions 
regarding data collection, access, and stewardship are complex. It is 
impossible to take all contingencies into account. Best practice and 
standards change with much of the change led by Indigenous and First Nations 
researchers. Best practices from a decade ago are not those of today and this 
is a good thing!


![Vuwo village](/Vuwo.JPG)

Back to [Language Data](../data/)


Fieldwork PNG

## The Language Data Commons of Australia (LDaCA) will make nationally significant language data available for academic and non-academic use and provide a model for ensuring continued access with appropriate community control.
Australia is a massively multilingual country in one of the world’s most 
linguistically diverse regions. Significant collections of this intangible 
cultural heritage have been amassed, including collections of Australian 
Indigenous languages, regional languages of the Pacific, and of Australian 
English, as well as collections important for cyber-security and for 
emergency communication. LDaCA will integrate this existing work into a 
national research infrastructure while also securing collections which remain 
under-utilised or at risk. LDaCA will thus ensure long-lasting access for 
analysis and reuse of these invaluable data, and will manage the data in a 
culturally, ethically and legally appropriate manner guided by FAIR and CARE 
principles.

To accomplish these goals, LDaCA will:
- Develop a comprehensive language data access policy framework,
- Develop shared technical infrastructure and standards across institutions,
- Build a sustainable long-term repository for ingesting and curating existing
  language data collections of national significance,
- and build a portal for discovery and access of language data.

The result will be an integrated national technical infrastructure to analyse 
language collections at scale which will open up the social and economic 
possibilities of Australia's rich linguistic heritage. The project will build 
connections to other HASS RDC projects and assist in laying the foundation 
for the establishment of a broader HASS Research Data Commons as well as 
positioning Australia internationally as a leading contributor of language 
collections and digital infrastructure.

![Acknowledgement](/AcknowledgeARDC.png)

The Language Data Commons of Australia (LDaCA) project received investment 
([1](https://doi.org/10.47486/DP768) and [2](https://doi.org/10.47486/HIR001))
from the Australian Research Data Commons (ARDC). The ARDC is funded by the 
National Collaborative Research Infrastructure Strategy (NCRIS).

*LDaCA acknowledges and pays respects to the Elders and Traditional Owners of 
the lands on which we live and work.*




Metadata is often defined as 'data about data'. High quality metadata is important in making data FAIR:
- Findable: metadata is the starting point for searching data collections. For example, if we want to find data in a particular language, this will only be possible for data which has a language recorded in its metadata. (Tracking languages is in itself problematic, see [below](#identifying-codes-for-languages).)
- Accessible: access conditions which apply to data should be part of the associated metadata.
- Interoperable: information about the format of data and whether it requires specific software to be usable should be part of the associated metadata.
- Reusable: all of aspects of metadata mentioned above contribute to making data reusable. The more we know about some data, the easier it is to know whether it will be useful to us or not.

RO-Crates in general have basic metadata requirements, but it is possible to specify a **profile** for crates for specific purposes. LDaCA is developing such a profile for our data; we are basing this largely on previous work in the area. An important aspect of the RO-Crate approach is that it uses the principles of [Linked Open Data](https://en.wikipedia.org/wiki/Linked_data#Linked_open_data). This means that terms used in our metadata will (whenever possible) link to an openly available definition. In developing the profile, we are drawing on two existing attempts to provide vocabularies for describing language data.

## [OLAC](http://www.language-archives.org/)

The Open Language Archives Community is an international partnership of institutions and individuals; one of their activities is developing consensus on best current practice for the digital archiving of language resources and this includes making recommendations for metadata. The OLAC metadata scheme is based on [Dublin Core](https://www.dublincore.org/) (DC), a widely used general metadata schema. OLAC have suggested refinements and extensions of the DC base which make it more useful for describing language resources.


## [CMDI](https://www.clarin.eu/content/component-metadata)

The Component Metadata Infrastructure was developed within the [CLARIN](https://www.clarin.eu) project. It draws on the earlier ISLE Metadata Initiative ([IMDI](https://en.wikipedia.org/wiki/IMDI)), but where IMDI attempted to specify a comprehensive scheme for (multimodal) language data, CMDI adopts a more flexible approach where components are assembled into reusable profiles. This is very similar to the RO-Crate approach described above but with an important difference: the components of a CMDI profile are all drawn from a central registry, whereas components of an RO-Crate profile come from any linkable location.


## Identifying Codes for Languages

One very important piece of metadata for language data is a description of the language or languages which the data represent. This is not a simple problem because the relationship between languages and names for them is not one to one. Some languages have more than one name: for example *Farsi* and *Persian* can both be used to refer to the same language. Some names refer to more than one language: for example there are languages called *Buru* used in Nigeria and in Indonesia. To avoid the confusion which can arise from such situations, various systems have been developed to assign unique identifiers to languages. None of these systems gives a comprehensive list of languages and all such systems struggle with another problem, the distinction between separate languages and dialects of one language, as can be seen in the case study below. LDaCA includes identifiers from each of the three systems below where they are available and relevant.

##### [ISO-639](https://iso639-3.sil.org/)
This system is recognised as a standard by the International Standards Organisation. An earlier version of this system used two-letter codes to identify languages; more recent versions use three-letter codes (referred to as ISO 639-3). These codes are used by *[Ethnologue](https://www.ethnologue.com/)*, which is a catalogue of the languages of the world, and in many other contexts. The ISO 639-3 code for French is **fra**, and that for Warlpiri is **wbp**

##### [Glottolog](https://glottolog.org/)
Glottolog is an alternative catalogue of the world's languages, language families and dialects - Glottolog uses the term *languoid* to cover all of these. Each languoid is assigned a unique identifier consisting of four alphanumeric characters and four digits. For example, (standard) French has the code **stan1290**, amd Warlpiri is **warl1254**.

##### [Austlang](https://collection.aiatsis.gov.au/austlang/about)

AustLang provides a controlled vocabulary of persistent identifiers, a thesaurus of languages and peoples, and information about Aboriginal and Torres Strait Islander languages which has been assembled from referenced sources. Alphanumeric codes are used as persistent identifiers, while associated text strings are changeable and can reflect community preferences (including alternative names and spellings). In Austlang, Warlpiri has two codes: **C15** for the language in general, and **C15.1** for the variety named as Wakirti Warlpiri. (French is not covered by Austlang.)

#### Case study - Kala Lagaw Ya

Kala Lagaw Ya is a language spoken in the Torres Strait Islands. The language has several dialects or varieties and the table below shows how the different code schemes deal with this.

<table>
<tr><td>Name</td><td>ISO 639</td><td>Glottolog</td><td>Austlang</td><td>Notes</td></tr>
<tr><td>Kala Lagaw Ya</td><td>mwp</td><td>kala1377</td><td>Y1</td><td>Austlang: Marked with symbol ^ which indicates that the name is used to refer to a language and a dialect of the language.</td></tr>
<tr><td>Kalaw Kawaw Ya</td><td></td><td>kala1378</td><td>Y2</td><td>Ethnologue: Kalaw Kawaw is a dialect</td></tr>
<tr><td>Kawrareg</td><td></td><td>kawr1234</td><td></td><td></td></tr>
<tr><td>Kulkalgau Ya</td><td></td><td>kulk1234</td><td>Y4</td><td></td></tr>
<tr><td>Mabuyag</td><td></td><td>mabu1234</td><td></td><td>Ethnologue: Mabuiag is an alternate name</td></tr>
<tr><td>Kawalgaw Ya</td><td></td><td></td><td>Y5</td><td>Austlang: Kaurareg is an alternative name (probably the same as Glottolog kawr1234)</td></tr>
</table>

 

Back to [Background](../background/)

Metadata

# Organisation

LDaCA is one strand of the partnership between the Australian Research Data Commons ([ARDC](https://ardc.edu.au/)) and the [School of Languages and Cultures](https://languages-cultures.uq.edu.au/) at the [University of Queensland](https://www.uq.edu.au/). This partnership includes a number of projects that explore language-related technologies, data collection infrastructure and Indigenous capability programs. These projects are being led out of the Language Technology and Data Analytics Lab ([LADAL](https://slcladal.github.io/index.html)), which is overseen by [Professor Michael Haugh](https://languages-cultures.uq.edu.au/profile/1498/michael-haugh) and [Dr Martin Schweinberger](https://languages-cultures.uq.edu.au/profile/4295/martin-schweinberger).

The LDaCA project received investment from ARDC through two of its programs:

1. Australian Data Partnerships Program: Developing policy and technology foundations of a nationally integrated research infrastructure for language data collections of high strategic importance for the Australian research community ([DP](https://ardc.edu.au/project/language-data-commons-of-australia-ldaca/)).
2. HASS Research Data Commons and Indigenous Research Capability Program: Capitalising on existing infrastructure, securing vulnerable and dispersed collections and linking with improved analysis environments for new research outcomes ([HASS-RDC](https://ardc.edu.au/news/announcing-3-successful-projects-ardc-hass-rdc/)).

 
<hr class="dots" />
 

## Partner Institutions:

- **University of Queensland** (DP, HASS-RDC): Professor Michael Haugh, Professor Clint Bracknell
- **Australian National University** (DP, HASS-RDC): Professor Catherine Travis
- **Monash University** (DP, HASS-RDC): Associate Professor Louisa Willoughby
- **University of Melbourne** (DP, HASS-RDC): Associate Professor Nick Thieberger
- **University of Sydney** (HASS-RDC): Professor Monika Bednarek (Sydney Corpus Lab)
- **AARNet** (HASS-RDC)
- **First Languages Australia** (HASS-RDC): Beau Williams

### Advisory / Consultative Partners

- PARADISEC
- ARC Centre of Excellence for the Dynamics of Language
- Australian Digital Observatory
- CLARIN

Organisation

## Language Data Commons of Australia

## Australian Text Analytics Platform

### Day of Activities before ALS2022 (Tuesday 29/11)

<table>
<tr><td>Time</td><td>Activity</td><td>People</td></tr>
<tr><td>09:00 - 10:00</td><td>Introduction to notebooks What are Jupyter notebooks and why are they useful?
</td><td>Sara King Simon Musgrave</td></tr>
<tr bgcolor = 'LightGray'><td>10:00 - 10:30</td><td>Break</td><td></td></tr>
<tr><td>10:30 - 11:00</td><td>General progress report An overview of the projects and of progress to date
</td><td>Michael Haugh Robert McLellan Simon Musgrave Jenny Fewster (ARDC)</td></tr>
<tr><td>11:00 - 12:00</td><td>Introducing the LDaCA Portal How will I find data in LDaCA? Your chance to tell us whether we are meeting your needs!</td><td>Peter Sefton Kathrin Kaiser</td></tr>
<tr bgcolor = 'LightGray'><td>12:00 - 01:00</td><td>Lunch</td><td></td></tr>
<tr><td>01:00 - 02:00</td><td>Data and access How is access to data managed in LDaCA and how can I access data?
</td><td>Peter Sefton Kathrin Kaiser Catherine Travis</td></tr>
<tr><td>02:00 - 02:30</td><td>Auslan project update Report on work by the team at Monash University
</td><td>Louisa Willoughby Trevor Johnston</td></tr>
<tr bgcolor = 'LightGray'><td>02:30 - 03:00</td><td>Break</td><td></td></tr>
<tr><td>03:00 - 03:30</td><td>Text analytics for corpus linguistics Selected ATAP tools showcase
</td><td>Sydney Informatics Hub with Sydney Corpus Lab (Sony Jufri, Kelvin Lee)</td></tr>
<tr><td>03:30 - 05:00</td><td>Discursis workshop How to get started with this tool for tracking topics in conversations </td><td>Ben Foley Sam Hames</td></tr>
</table>

The [Australian Text Analytics Platform](https://www.atap.edu.au/) and the [Language Data Commons of Australia](https://www.ldaca.edu.au/) are projects supported by the [Australian Research Data Commons](https://www.ardc.edu.au) to develop infrastructure for Australian researchers who work with language data. This day of activities organised by the projects will give ALS conference delegates (and anyone else who is interested) the opportunity to learn more about this work. The day will include:

- an overview of the projects and the work to date
- reports on specific sub-projects
- introductory workshops on the tools and resources being developed
- a workshop on using Discursis, a tool for tracking topics in interactive use of language
- the opportunity to influence future work by exploring and providing feedback on the data interface which we are building.

The activities will be of interest to anyone who conducts research which includes language data, especially those who use or would like to use computational tools in their research. Participants should bring their own computer; no software is required beyond a web browser.

Catering is provided and therefore a registration fee is being charged for this event to partially cover the cost ($25 full rate and $15 discount).

[Registration](https://als.asn.au/Conference/Conference2022/Workshops)

The Australian Text Analytics Platform and the Language Data Commons of Australia received investment from the Australian Research Data Commons (ARDC). The ARDC is funded by the National Collaborative Research Infrastructure Strategy (NCRIS).

Pre-ALS Conference Activity Day

The Language Data Commons of Australia aims to ensure the long-term access to language data collections for analysis and reuse. Sustainable management of and access to these significant collections of intangible cultural heritage are underpinned by two sets of complementary guiding principles of data management and stewardship, namely the FAIR and CARE principles. 
 
## FAIR Principles
 

The FAIR principles were [first published](https://www.nature.com/articles/sdata201618) by a group of stakeholders representing academia, industry, funding agencies, and scholarly publishers. The principles aim to address issues surrounding data management and stewardship, focusing on four areas (which provide the FAIR acronym).
 
### Findable
Metadata and data should be easy to find for both humans and computers. Making the data findable includes:
- (Meta)data are assigned a globally unique and persistent identifier
- Data are described with rich metadata 
- Metadata clearly and explicitly include the identifier of the data they
describe.
- (Meta)data are registered or indexed in a searchable resource.
 

### Accessible
Once the user finds the required data, she/he/they need to know how they can be accessed, possibly including authentication and authorisation.
- (Meta)data are retrievable by their identifier using a standardised communications protocol
 + The protocol is open, free, and universally implementable
 + The protocol allows for an authentication and authorisation procedure, where necessary
- Metadata are accessible, even when the data are no longer available
 

### Interoperable
The data usually need to be integrated with other data. In addition, the data need to interoperate with applications or workflows for analysis, storage, and processing.
- (Meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation
- (Meta)data use vocabularies that follow FAIR principles
- (Meta)data include qualified references to other (meta)data
 

### Reusable
The ultimate goal of FAIR is to optimise the reuse of data. To achieve this, metadata and data should be well-described so that they can be replicated and/or combined in different settings.
- (Meta)data are richly described with a plurality of accurate and relevant attributes
 + (Meta)data are released with a clear and accessible data usage license
 + (Meta)data are associated with detailed provenance
 + (Meta)data meet domain-relevant community standards
 
 

The Australian Research Data Commons ([ARDC](https://ardc.edu.au/)) supports FAIR data practices and initiatives that make data and related research outputs FAIR. At the same time, the ARDC acknowledges that the implementation of the principles will look different across disciplines and will need discipline-specific approaches and standards. 

## CARE Principles
The [CARE Principles for Indigenous Data Governance](https://www.gida-global.org/care) were developed by the Global Indigenous Alliance (GIDA); they are a response to the FAIR principles and aim to complement them. GIDA highlights how the FAIR principles and the open data movement focus on increasing data sharing among researchers and entities but do not take into account power differentials and historical contexts or fully engage with Indigenous Peoples’ rights and interests. These include the rights to generate value from Indigenous data in ways that are grounded in Indigenous worldviews and to advance Indigenous innovation and self-determination. The CARE Principles also focus on four areas.
 

### Collective Benefit
Data ecosystems shall be designed and function in ways that enable Indigenous Peoples to derive benefit from the data.
- For inclusive development and innovation
- For improved government and citizen engagement
- For equitable outcomes 
 

### Authority to control
Indigenous Peoples’ rights and interests in Indigenous data must be recognised and their authority to control such data be empowered.
- Recognizing rights and interests
- Data for governance
- Governance of data.
 

### Responsibility
Those working with Indigenous data have a responsibility to share how those data are used to support Indigenous Peoples’ self-determination and collective benefit.
- For positive relationships
- For expanding capability and capacity
- For Indigenous languages and worldviews.
 

### Ethics
Indigenous Peoples’ rights and wellbeing should be the primary concern at all stages of the data life cycle and across the data ecosystem.
- For minimizing harm and maximizing benefit
- For justice
- For future use.


Back to [Background](../background/)

Principles

# Resources

- [Indigenous Knowledge](#indigenous-knowledge)
- [Ethics](#ethics)
- [Languages of the World](#languages-of-the-world)
- [Language Archives](#language-archives)
- [Australian Organisations](#australian-organisations)
- [International Organisations](#international-organisations)
- [Publications](#publications)
- [Tools](#tools)

<hr class="dots" />

### Indigenous knowledge {#indigenous-knowledge}

- [Protocols](https://australiacouncil.gov.au/wp-content/uploads/2021/07/protocols-for-using-first-nati-5f72716d09f01.pdf)
 for using First Nations Cultural and Intellectual Property in the Arts (Australia Council)
- Citing Indigenous elders and knowledge keepers
 - [Templates For Citing Indigenous Elders and Knowledge Keepers](https://kula.uvic.ca/index.php/kula/article/view/135)
 (Lorisia Macleod)
 - University of Alberta Library's [Guide](https://guides.library.ualberta.ca/c.php?g=715568&p=5112574)
 - University of Alberta Library's [blog](https://news.library.ualberta.ca/blog/2022/01/27/citing-indigenous-elders-and-knowledge-keepers/)
- [TK Labels](https://localcontexts.org/labels/traditional-knowledge-labels/): Traditional Knowledge labels are an initiative for Indigenous communities and local organizations. Developed through sustained partnership and testing within Indigenous communities across multiple countries, the Labels allow communities to express local and specific conditions for sharing and engaging in future research and relationships in ways that are consistent with already existing community rules, governance and protocols for using, sharing and circulating knowledge and data.
- Decolonising linguistics: [Spinning a better yarn](https://www.dynamicsoflanguage.edu.au/ialr/decolonising-linguistics-spinning-a-better-yarn/) 
- Rawlings, V, Flexner, J L, Riley, L (eds.) (2021) *[Community-Led Research: Walking New Pathways Together](https://open.sydneyuniversitypress.com.au/files/9781743327630.pdf)*. Sydney University Press

#### Indigenous Languages of Australia

- [First Languages Australia](https://www.firstlanguages.org.au/): First 
 Languages Australia encourages communication between communities, the 
 government and key partners whose work can impact Aboriginal and Torres 
 Strait Islander languages.
- [Living Languages](https://www.livinglanguages.org.au/): Living Languages 
 provides grassroots training to people, communities and Language Centres in 
 Australia doing language work on the ground in remote, regional and urban 
 areas.
- Language Centres: The Indigenous Languages and Arts (ILA) program 
 currently supports a number of organisations, including a network of 20 
 language centres. A list of these centres is available from 
 [this page](https://www.arts.gov.au/documents/indigenous-languages-and-arts-program-language-centres) 
 (pdf or docx format). 
- [Austlang](https://collection.aiatsis.gov.au/austlang/about): AustLang 
 provides a controlled vocabulary of persistent identifiers, a thesaurus of 
 languages and peoples, and information about Aboriginal and Torres Strait 
 Islander languages which has been assembled from referenced sources.
- [Nyingarn](https://nyingarn.net/): a platform for primary sources in 
 Australian Indigenous languages 
- [Digital Daisy Bates](https://bates.org.au/): a collection of over 23,000 
 pages of wordlists of Australian languages, originally recorded by Daisy 
 Bates in the early 1900s, made up of the original questionnaires and around 
 4,000 pages of typescripts.
- [50 Words Project](https://50words.online/): The 50 Words Project showcases 
 words from 64 languages across Australia, with further languages being 
 added regularly as more communities around Australia become involved. For 
 each language you can hear the words spoken via a map that shows the 
 general location of the language.
- [Gambay](https://www.abc.net.au/indigenous/gambay-languages-map): a map of 
 Australia's first languages.
- [Australian Indigenous language collections](https://www.nla.gov.au/research-guides/indigenous-language-resources#): 
 a guide to materials held in the National Library of Australia, with links 
 to similar resources at State Libraries. 
- Living Archive of Australian Languages: see [below](#language-archives)
- Contemporary and Historical Reconstruction in the Indigenous Languages of 
 Australia ([CHIRILA](http://www.pamanyungan.net/chirila/)): CHIRILA is a 
 lexical database (a database with words from different languages). 
 Currently there are about 780,000 words, from all over Australia, of which 
 about 20% is publicly available.

### Ethics {#ethics}

- Australian Institute of Aboriginal and Torres Strait Islander Studies [Code of ethics](https://aiatsis.gov.au/research/ethical-research/code-ethics)
- Australian Linguistic Society [policies](https://als.asn.au/AboutALS/Policies): includes Policy on Linguistic rights of Aboriginal and Islander communities and Statement of Ethics.
- [Australian Code for the Responsible Conduct of Research, 2018](https://www.nhmrc.gov.au/about-us/publications/australian-code-responsible-conduct-research-2018)
- Linguistic Society of America: 
 - [Ethics Statement 2019](https://www.linguisticsociety.org/content/lsa-revised-ethics-statement-approved-july-2019)
 - [Further resources](https://www.linguisticsociety.org/resource/ethics-further-resources)
- [Tromsø recommendations for citation of research data in linguistics](https://www.rd-alliance.org/group/linguistics-data-ig/outcomes/troms%C3%B8-recommendations-citation-research-data-linguistics)

Language and linguistics datasets are often not cited, or cited imprecisely, 
because of confusion surrounding the proper methods for citing them. For the 
use of researchers and scholars in the field working with datasets, the 
Tromsø recommendations propose components of data citation for referencing 
language data, both in the bibliography and in the text of linguistics 
publications. 

### Languages of the world {#languages-of-the-world}

- [Glottolog](https://glottolog.org/): Comprehensive reference information for the world's languages, especially the lesser known languages. LDaCA uses Glottolog [language codes](../metadata/#glottologhttpsglottologorg) in our metadata. 

- Open Language Archives Community ([OLAC](http://www.language-archives.org/)) is an international partnership of institutions and individuals who are creating a worldwide virtual library of language resources by: (i) developing consensus on best current practice for the digital archiving of language resources, and (ii) developing a network of interoperating repositories and services for housing and accessing such resources. OLAC harvests metadata and their web site has a search facility to find resources for languages. OLAC [metadata recommendations](../metadata/#olachttpwwwlanguage-archivesorg) are the basis for some of LDaCA's metadata.

- [Ethnologue](https://www.ethnologue.com/): *Ethnologue* provides information about the languages of the world, but operates on a subscription model. The information which is available without a subscription is very limited: the first three lines of individual language entries which include the ISO 639-3 code, the classification of the language into a language family, and a link to the language’s OLAC page.

- The World Atlas of Linguistic Structures Online ([WALS](https://wals.info/)) is a large database of structural (phonological, grammatical, lexical) properties of languages gathered from descriptive materials (such as reference grammars) by a team of 55 authors.

### Language Archives {#language-archives}

- [LAAL](https://livingarchive.cdu.edu.au/): 
The Living Archive of Aboriginal Languages is a digital archive of endangered literature in Australian Indigenous languages of the Northern Territory. It contains nearly 4000 books in 50 languages from 40 communities available to read online or download freely. This is a living archive, with connections to the people and communities where the books were created. This will allow for collaborative research work with the Indigenous authorities and communities.
- [PARADISEC](https://www.paradisec.org.au/): The Pacific and Regional Archive for Digital Sources in Endangered Cultures holds 14,500 hours of audio recordings and 2,000 hours of video recordings that might otherwise have been lost. These recordings are of performance, narrative, singing, and other oral tradition. This amounts to 150 terabytes, and represents 1,315 languages, mainly from the Pacific region.
- [ELAR](https://www.elararchive.org/): The Endangered Languages Archive is a digital repository for preserving multimedia collections of endangered languages from all over the world, making them available for future generations.
- [The Language Archive](https://archive.mpi.nl/tla/): The Language Archive (TLA) is an integral part of the Max Planck Institute for Psycholinguistics in Nijmegen. It contains various types of materials, including: audio and video language corpus data from languages around the world; photographs, notes, experimental data, and other relevant information required to document and describe languages and how people use them; records of speech in everyday interactions in families and communities; naturalistic data from adult conversations from endangered and under-studied languages, and linguistic phenomena.
- [Kaipuleohone Language Archive](http://ling.hawaii.edu/kaipuleohone-language-archive/): Kaipuleohone is the digital language archive of the University of Hawaiʻi. Founded in 2008, the archive houses texts, images, audio, and video collected from around the world by linguists, anthropologists, ethnomusicologists, and more. Our collection includes a wealth of photographs, notes, dictionaries, transcriptions, and other materials related to small and endangered languages.
- [DELAMAN](https://www.delaman.org/): The Digital Endangered Languages and Musics Archive is an international network of archives of data on linguistic and cultural diversity, in particular on small languages and cultures under pressure.

### Australian Organisations {#australian-organisations}

- The Australian Linguistic Society ([ALS](https://als.asn.au/Home)) is the national organisation for linguists and linguistics in Australia. Its primary goal is to further interest in and support for linguistics research and teaching in Australia.

- The Applied Linguistics Association of Australia ([ALAA](https://alaa.net.au/)) is the national professional organisation for applied linguistics in Australia. We welcome academics, teachers, researchers, students and members of the wider community to join us and become part of an active community interested in questions, issues and problems that can be understood and addressed through a focus on language in our world.

- The Australasian Langauge Technology Association ([ALTA](https://www.alta.asn.au/index.html)) has the purpose of promoting language technology research and development in Australia and New Zealand.

- The Australasian Speech Science and Technology Association ([ASSTA](https://assta.org/)) is a scientific association which aims to advance the understanding of speech science and its application to speech technology in a way that is appropriate for Australia and New Zealand.

- The Australian Institute of Aboriginal and Torres Strait Islander Studies ([AIATSIS](https://aiatsis.gov.au/)) is an Indigenous-led, national institute that celebrates, educates and inspires people from all walks of life to connect with the knowledge, heritage and cultures of Australia’s First Peoples.

### International Organisations {#international-organisations}

- Common Language Resources and Technology Infrastructure ([CLARIN](https://www.clarin.eu/)) is a research infrastructure that was initiated from the vision that all digital language resources and tools from all over Europe and beyond are accessible through an online environment for the support of researchers in the humanities and social sciences.

- Endangered Languages Documentation Program ([ELDP](https://www.eldp.net/)) supports the documentation and preservation of endangered languages through granting, training and outreach activities. The collections compiled through our funding are freely accessible at the [Endangered Languages Archive](#language-archives).

- The Linguistic Data Consortium ([LDC](https://www.ldc.upenn.edu/)) is an open consortium of universities, libraries, corporations and government research laboratories. LDC was formed in 1992 to address the critical data shortage then facing language technology research and development. Initially, LDC's primary role was as a repository and distribution point for language resources. but with the help of its members, LDC has grown into an organization that creates and distributes a wide array of language resources. 

### Publications {#publications}

- [The Open Handbook of Linguistic Data Management](https://direct.mit.edu/books/book/5244/The-Open-Handbook-of-Linguistic-Data-Management) 
Edited by Andrea L. Berez-Kroeker, Bradley McDonnell, Eve Koller, Lauren B. Collister. MIT Press 2022.

 "A guide to principles and methods for the management, archiving, sharing, and citing of linguistic research data, especially digital data. “Doing language science” depends on collecting, transcribing, annotating, analyzing, storing, and sharing linguistic research data. This volume offers a guide to linguistic data management, engaging with current trends toward the transformation of linguistics into a more data-driven and reproducible scientific endeavor. It offers both principles and methods, presenting the conceptual foundations of linguistic data management and a series of case studies, each of which demonstrates a concrete application of abstract principles in a current practice."

 This material is Open Access.

- [Language Documentation and Conservation](https://nflrc.hawaii.edu/ldc/)

 “LD&C publishes papers on all topics related to language documentation and conservation, including, but not limited to, the goals of language documentation, data management, fieldwork methods, ethical issues, orthography design, reference grammar design, lexicography, methods of assessing ethnolinguistic vitality, archiving matters, language planning, areal survey reports, short field reports on endangered or underdocumented languages, reports on language maintenance, preservation, and revitalization efforts, plus software, hardware, and book reviews.”

 LD&C is an Open Access journal.

- [Living languages/Lenguas vivas/Línguas vivas journal](https://scholarworks.umass.edu/livinglanguages/)

 “Living Languages is an international, multilingual journal dedicated to topics in language revitalization and sustainability. The goal of the journal is to promote scholarly work and experience-sharing in the field. The primary focus is on bringing together language revitalization practitioners from a diversity of backgrounds, whether academic or not, within a peer-reviewed publication venue that is not limited to academic contributions and is inclusive of a diversity of perspectives and forms of expression.”

 *Living Languages* is an Open Access journal.

- [Pacific Linguistics](http://sealang.net/archives/pl/): a digital archive of many Pacific Linguistics publications up to 2012.

### Tools {#tools}

- [Audacity](https://www.audacityteam.org/): Audacity is a free, easy-to-use, multi-track audio editor and recorder for Windows, macOS, GNU/Linux and other operating systems.
- [ELAN](https://archive.mpi.nl/tla/elan): a tool for making time-aligned annotations on audio and video recordings.
- [Praat](https://www.fon.hum.uva.nl/praat/): Free software for doing phonetics by computer.
- [FieldWorks](https://software.sil.org/fieldworks/): Software tools for managing linguistic and cultural data. FieldWorks supports tasks ranging from the initial entry of collected data through to the preparation of data for publication, including dictionary development, interlinearization of texts, morphological analysis, and other publications.
- [Toolbox](https://software.sil.org/toolbox/): Toolbox is a data management and analysis tool for field linguists. It is especially useful for maintaining lexical data, and for parsing and interlinearizing text, but it can be used to manage virtually any kind of data.
- [AntConc](https://www.laurenceanthony.net/software/antconc/): A freeware corpus analysis toolkit for concordancing and text analysis.

Resources

LDaCA is basing its data storage on the two technologies described below. The overall approach is informed by the [Arkisto](https://arkisto-platform.github.io/) platform, taking the view that research data has interest and value that extends beyond funding cycles and its long term preservation and accessibility must continue to be managed. [This presentation](https://ptsefton.com/2022/02/18/hass_rdc_tech_advisory/index.html) gives further details of the technical architecture.

## Research Object Crates ([RO-Crate](https://www.researchobject.org/ro-crate/))

A Research Object ([RO](https://www.researchobject.org/)) is a structured archive of all the items that contributed to the research outcome, including their identifiers, provenance, relations and annotations. [RO-Crate](https://www.researchobject.org/ro-crate/) is a lightweight approach to packaging research data with their metadata. It is based on [schema.org](https://schema.org/) annotations in [JSON-LD](https://json-ld.org/), and aims to make best-practice in formal metadata description accessible and practical for use in a wide variety of situations. While RO-Crates can be considered general-purpose containers of arbitrary data and open-ended metadata, in practical use within a particular domain, application or framework, it is beneficial to further constrain RO-Crate to a specific profile: a set of conventions, types and properties that one minimally can require and expect to be present in that subset of RO-Crates. LDaCA is developing such a profile to be used for language data.


## Oxford Common File Layout ([OCFL](https://ocfl.io/))

OCFL is a specification for laying out digital collections on file or object storage. It is designed with long-term preservation principles in mind and does not rely on specialised software. Amongst the benefits of using OCFL with RO-Crate objects are:
- completeness: a repository can be re-indexed from the files it stores
- versioning: repositories can make changes to objects and still allow their history to persist
 
 

 
 

Back to [Background](../background/)

Technologies

## Language data in Australia - Mapping a conceptual landscape

### Simon Musgrave

The data which is being made accessible through the Language Data Commons of Australia will contribute to the task of documenting language use and language behaviour in Australia. But what does this include? We are aware of many data sources and, in the short term, the important questions are about priorities:

- What will be immediately useful?
- What data is stored precariously?
- What relationships can be leveraged?

But in thinking for the long term, there is value in asking the question from a more conceptual point of view:

- What were the possibilities for making records of language use over our history and what has resulted (and not resulted) from those possibilities?

I will try to at least start answering that question by providing a conceptual map of a metaphorical landscape and this will be structured around two themes: demography and technology. Demography is important because what languages were being used at any particular time depends on who was living in Australia at that time. On this dimension, the major point of articulation is the arrival of non-Indigenous people. The places of origin of those non-Indigenous people have changed over time, and that has had linguistic consequences, but not on the same scale as the initial change. Technology is important because the kinds of records which might exist of language use depend on the means which were available to make such records at a given time. And on this dimension, I think that there are three points of articulation that are important: the possibility of making written records, the possibility of recording sound and vision, and the possibility of digital records.

Before European contact, Australia had a very diverse linguistic ecology. Credible estimates of the number of distinct languages range from around [250](https://aiatsis.gov.au/explore/living-languages) up to as many as [490](https://twitter.com/anggarrgoon/status/1381642291455086594?lang=en), with many more dialects. Contact between language groups was common and therefore multilingualism and multidialectalism were also common. We know that there was contact between Indigenous Australians and fishermen from the Indonesian archipelago (especially Makassarese people) from the evidence of loan words. But the only record which existed of this period in the linguistic history of the continent was what was passed from one generation to another by oral transmission.

The presence of Europeans on the Australian continent brought a huge change to both dimensions of the map I am developing. Europeans landed on parts of Australia from some time in the 17th century, but I will take two dates in the 18th century to be crucial. Although earlier visitors may have recorded a few words which they heard from Indigenous Australians, written records of the languages only start properly (and even then to a limited extent) with Cook's expedition in 1770, reflecting perhaps the scientific orientation of Joseph Banks. This is the first point of technological articulation in the language data landscape which introduced the possibility of making language records independent of human memory. And then from 1788, there has been a continuous non-Indigenous presence in Australia representing a huge demographic articulation point.

During the 19th century (and into the 20th century), written records of Australian languages were produced by a variety of people such as explorers, missionaries, administrators (in fact, pretty much anyone who could be bothered). These records are scattered and new material continues to be found (for example, Des Crump's ongoing work in the [State Library of Queensland](https://www.slq.qld.gov.au/get-involved/fellowships-awards-residencies/queensland-memory-awards/state-library-queensland-medal) and the Queensland State Archives). If you would like to get a flavour of some records of this kind, the [Nyingarn project](https://nyingarn.net) is making many of them available online. (Nyingarn builds on earlier work which presented the Indigenous language materials collected by Daisy Bates as an [online resource](http://www.bates.org.au/).) However the efforts of these early recorders were sporadic, uncoordinated and poorly focused. In 1945, Sydney Baker wrote: "Records of their languages are extremely deficient for instance, no exhaustive grammar of an aboriginal language has been published. There is no comprehensive or even partially comprehensive dictionary of reference to aboriginal dialects" (p218). And this record did not reflect the richness of the oral tradition or of language use. For example, it is very difficult to make an accurate record of conversation when contemporaneous writing is your only technological resource.

The non-Indigenous group who arrived in 1788 were the First Fleet, the first group of convicts transported from the United Kingdom with their jailers - English has had a permanent presence in Australia since that date. Written records are all that we have until the end of the 19th century, but they are extensive and a sample of them is available in the COrpus of Oz Early English ([COOEE](https://data.atap.edu.au/collection?id=arcp://name,cooee-corpus/corpus/root&_crateId=arcp://name,cooee-corpus/corpus/root), Fritz 2007). This collection, and indeed the overall record, has the problems we expect to be associated with written sources. The authors are not a representative group, and the material is biased towards non-vernacular styles. Material such as Corbyn's more-or-less verbatim accounts of court scenes in mid-century Sydney is uncommon (Corbyn 1854), and it would be of great value to those researching the development of Australian English if more informal material (personal letters, diaries) could be made accessible.

From 1788 on, speakers of non-Indigenous languages other than English have been present in Australia. Within the convict population (and then also amongst free settlers), a significant minority of the European population in Australia knew Irish. Speakers of other languages were occasionally present in the first part of the 19th century, and then, after the discovery of gold in the middle of the century, speakers of many languages, including major European languages and Sinitic languages, were present in Australia. The written records of these languages, represented by periodical publications, are diverse. But before turning to those records, a few words about the relative absence of Irish in the written record.

As mentioned, a large proportion, probably around one third, of those who arrived in Australia from the United Kingdom were Irish. Many had some knowledge of the language, even up to half of the Irish immigrants to Victoria according to Noone (2012). But some Irish transportees were political prisoners and use of the language was viewed with suspicion; speaking Irish could even be construed as a subversive act. There is evidence that the language continued to be spoken: Irish-speaking priests were needed to hear confessions, and interpreters were used in court occasionally. The written record, however, is limited. As O'Farrell (1988) points out, Irish was not even used on tombstones, one place where Irish could be used without consequences. A bilingual magazine was published in Melbourne in the 1920s, but, rather than continuing a tradition, this is a manifestation of the [Gaelic revival](https://en.wikipedia.org/wiki/Gaelic_revival).

Other non-Indigenous languages were also present from quite early in the post-invasion period, but increasingly so after the gold rushes of the mid 19th century. The crucial evidence here is the record of newspapers published in such languages, and here I rely on the [research of Tim Sherratt](https://glam-workbench.net/trove-newspapers/#finding-non-english-newspapers-in-trove) using the resources of the National Library of Australia. German and Chinese were both well represented by published material from at least the middle of the 19th century and French and Italian were also present. Most of us probably think of Greek migration to Australia as a post-WW2 phenomenon, but publication in Greek started in 1931. Greek and Italian had important newspaper publications in the second half of the 20th century, as did Chinese but rather later.

I have suggested that technology is an important factor in mapping out this data landscape and in considering this it is important to track not just the availability of technologies but also who controlled them and to what ends. The interplay of these considerations is evident in the informal and unsystematic approach to making records of Indigenous languages described previously, and it is also evident in the almost complete lack of data on contact varieties which have existed at various times in Australia. Early contact between Europeans and Indigenous people led to the use of pidgins, but there are only minimal accounts of these in early records. Aboriginal Englishes are a range of contact varieties which are in use and still developing today, while Kriol is a contact language spoken in northern Australia. Good records of any of these varieties only began to be made in the 1980s. At least two contact varieties developed in specific social settings. Queensland Kanaka English originated with the presence of approximately 60,000 Melanesian agricultural workers in Queensland between 1860 and 1906, and Broome Pidgin developed when pearl divers from Asia, for whom Malay was a _lingua franca_, worked in Broome between about 1900 and 1930. In both cases, we know almost nothing except that these varieties existed and were used. As in the case of Indigenous languages, these varieties were of little interest to those who controlled the technology used to make records.

The second point of technological articulation is the possibility of making records of sounds and images. These technologies make it possible to document the sounds of speech, and then movement including gesture. Audio recording technologies were developed in the latter part of the 19th century and the earliest sound recording in the National Film and Sound Archive (NFSA) catalogue is from 1888. The first recordings of speech ('comic monologue') are catalogued as c1896. The first NFSA catalogue entry for an Australian language is for 1899; the material is a selection from the three cylinder recordings of 1899 of Fanny Cochrane Smith, who claimed to be 'the last of the [Aboriginal] Tasmanians'. The earliest catalogue entry at the Australian Institute for Aboriginal and Torres Strait Islander Studies (AIATSIS) is for 1898; these are recordings from the Cambridge Anthropological Expedition to Torres Straits (1898). There is a continuing audio record for Australian languages from this point on. Much of this material is curated by AIATSIS, but there is also material elsewhere.

Although audio recordings of English in Australia begin in the 19th century, the first systematic [set of materials](https://speech.library.sydney.edu.au/) recorded with the intention of documenting Australian English was not made until 1959-60 when A.G. Mitchell and Arthur Delbridge carried out a survey of the speech habits of young Australians. Of course, recordings of Australians speaking exist from earlier, but it is not obvious where they might be. Again, the questions about who controls technology are relevant: who would have an interest in a) making recordings and b) preserving them? Two answers to these questions seem interesting, and they are also relevant in the case of other non-Indigenous languages. Firstly, media organisations make recordings and may preserve them, and secondly, preservation of recorded material is an important aim for oral historians. For English, the ABC has a substantial archive including radio recordings starting in 1932 and television material starting in 1956. The ABC treats most of this material as a commercial resource and therefore access and use conditions are not simple - but there is an enormous amount of material there. And since at least the 1980s, considerable amounts of material have been collected by oral historians, an example which shows the benefit of looking for language data beyond what linguists have collected. For other non-Indigenous languages, SBS Radio has existed since 1975 and broadcasts in 68 languages today. The extent of their archive and how it might be accessed are questions we are exploring. Oral history is also likely to be an important source of data for these languages. A range of material from informal recordings to oral history interviews exists, mostly held by individuals but community associations may be a gateway.

Technology to record moving images developed a little later than audio recording. The NFSA catalogue does not specify whether material is silent or has sound, and I have yet to establish the earliest video record of Indigenous language stored by that institution. At AIATSIS, the earliest materials are recordings made at Ernabella by Norman Tindale in 1933, consisting of silent film with an accompanying wax cylinder audio recording. The first clear instance of film of Indigenous language use which I have traced to date is a film called _Aborigines of the sea coast_ produced by the Australian Commonwealth Film Unit in 1948. [This film](https://shop.nfsa.gov.au/aborigines-of-the-sea-coast) is a record of a 1948 expedition to Arnhem Land led by anthropologist Charles Mountford. It depicts the ancestral fishing, hunting, building and boatmaking techniques used by the communities of the region. For English, the NFSA has a substantial collection of filmed material created in Australia and again the ABC archives are potentially a valuable source, especially for non-scripted material (such as interviews and the like).

The digital revolution of the last few decades is the third point of technological articulation and it has fundamentally altered our relationship to language data. This is true both for how we acquire and handle data and for what data is available. Pre-digital audio and video content required expensive equipment, the recorded media had to be stored very carefully and using the recordings caused them to deteriorate. Today, cheap (and very portable) equipment can produce excellent results for multimodal data, the resulting data can be replicated, edited and disseminated easily and without degradation, and any problems relating to storage of data are (largely) general ones. How we collect and view written data has also changed in various ways. Optical Character Recognition (OCR) can transform the printed record into machine readable text (and increasingly can access handwritten material), and new genres of writing have come into existence. These developments mean that the amount of data available is enormous and continues to expand. These large bodies of data would be intractable using traditional methods; fortunately, new tools have also been developed which allow us to analyse large collections of data.

As a result of these changes, we face new and different problems in how we approach language data. Finding data is simple, but choosing what part of the data and/or how much of the data we should acquire may not be simple. Control of data has become complex when the definition of Public Domain has to accommodate new modes of dissemination and large collections of data are considered commercial assets.

Even the question of what should be considered 'Australian language data' does not have an obvious answer in this digital world. Does the language of an Australian resident from a South Asian background contributing to social media in Sri Lanka come under the term? What about the tweets of someone who was born in Australia and grew up here but has lived in Europe for a number of years? The landscape of this latest phase in language data is still emerging and finding good answers to these questions (and many others) will be needed before we can map the new landscape clearly.

_This post is based on presentations given to the LaTrobe University Linguistics Program (27 May 2021) and to the Monash University Linguistics Program (10 May 2022). I am grateful for helpful comments from both those audiences, and for comments on the draft of this post from Leah Gustafson, Sara King and Harriet Sheppard._

**References**: 
Baker, Sidney (1945) _The Australian Language_. Sydney: Angus and Robertson. 
Corbyn, Charles Adam (1854) _Sydney Revels of Bacchus, Cupid, and Momus_. Sydney: Hawksley & Williamson. 
Fritz, Clemens WA. (2007) _From English in Australia to Australian English: 1788-1900_. Peter Lang Frankfurt. 
Noone, V. (2012). _Hidden Ireland in Victoria_. Ballarat: Ballarat Heritage Services. 
O’Farrell, Patrick (1988) _The Irish in Australia_. Kensington, N.S.W.: New South Wales University Press.

Language data in Australia - Mapping a conceptual landscape

# FAIR and CARE
Data is becoming increasingly important in today’s world, so corpus linguists might feel that the rest of the world is finally catching up. But the rest of the world are bringing with them new approaches to how data is handled. This means that fields such as corpus linguistics may need to reassess their practices. Such reassessment includes addressing concerns about how data is stored and who can access it (data stewardship) – concerns that are a part of the [Open Science](https://en.wikipedia.org/wiki/Open_science) movement, ultimately grounded on principles of equity and accountability. 

The most influential approach to data stewardship today is the [FAIR](https://www.go-fair.org/) principles.
According to these principles, data should be:
- *Findable* 
 
&emsp; Metadata and data should be easy to find for both humans and computers. 
- *Accessible*
 
&emsp; Once the user finds the required data, she/he/they need to know how can they be accessed, possibly including authentication and authorisation.
- *Interoperable*
 
&emsp; The data usually need to be integrated with other data. In addition, the data need to interoperate with applications or workflows for analysis, storage, and processing.
- *Reusable*
 
&emsp; The ultimate goal of FAIR is to optimise the reuse of data. To achieve this, metadata and data should be well-described so that they can be replicated and/or combined in different settings.
 

In general, corpus linguists do well on the interoperability criterion. Corpus data is usually stored in non-proprietary formats; even when some structure is imposed on the data, this is almost always in a form which is saved as a simple text file (e.g. csv files or xml annotations). Data stored in such formats is easy to move between applications. But what about the other three criteria? 

Some corpus data is easy to discover; it is findable. For example CLARIN, the [portal](https://www.clarin.eu/content/data) to the European Union language resource infrastructure, provides access to many large data collections, as does the [Linguistic Data Consortium](https://www.ldc.upenn.edu/) in the USA. However, some data is never made part of a large collection and often remains under the control of individual researchers or research teams. Such data may be almost impossible to find. Even if we can find such data, it is unlikely to be accompanied by good descriptions of the data and metadata, making reusability problematic. Of course, big corpora such as the [British National Corpus](http://www.natcorp.ox.ac.uk/) will be both findable and accompanied by comprehensive corpus manuals. However, it is worth considering how to make other corpora more findable, including the provision of corpus manuals or corpus descriptions. Corpus resource databases such as [CoRD](https://varieng.helsinki.fi/CoRD/) do aim to work towards this principle.

Accessibility may also be an issue for some data. Copyright law may allow use of material for individual research but prohibit any further distribution of the material. The FAIR approach to such cases is that metadata should be available so that interested parties can know that a data holding exists (F), and the metadata will include information about the conditions under which the data may or may not be shared or reused (A and R). 

![FAIR and CARE principles](/fair-care.png)

Image from Global Indigenous Data Alliance (https://www.gida-global.org/)
 
For linguists, there is another very important set of principles concerning data, the CARE principles developed by the Global Indigenous Data Alliance:
 
- *Collective Benefit*
 
&emsp; Data ecosystems shall be designed and function in ways that enable Indigenous Peoples to derive benefit from the data.
 
- *Authority to control*
 
&emsp;Indigenous Peoples’ rights and interests in Indigenous data must be recognised and their authority to control such data be empowered.
 
- *Responsibility*
 
&emsp;Those working with Indigenous data have a responsibility to share how those data are used to support Indigenous Peoples’ self-determination and collective benefit.
 
- *Ethics*
 
&emsp;Indigenous Peoples’ rights and wellbeing should be the primary concern at all stages of the data life cycle and across the data ecosystem.

 
These principles are presented as applying particularly to Indigenous data, but we believe that researchers should adopt this approach in all cases where the people who participate in our research can be seen to have some moral rights in the information they have contributed. Respecting those moral rights should be demonstrated by recognising the participants’ authority to control how data is used, by seeking to ensure that participants derive benefit from use of the data, and by acting ethically and transparently in our relations with the participants. Deborah Cameron and her colleagues (Cameron et al 1993) raised similar issues almost 20 years ago, arguing that the imbalance of power in the relation between researchers and participants needed to be reduced. The CARE principles continue along this path, but go even further in explicitly returning power to the sources of information. 

Corpus data is often written language. We have already mentioned that copyright law is relevant to some such material, and that body of law protects at least some rights for the creators of the material. But corpus linguists also work with other kinds of data such as spoken language (spontaneous or produced as a response to some prompt) or written material produced by research participants according to some protocol. In such cases, ethical research practice should include addressing the issues raised by the CARE principles. Some aspects of this practice will fall under institutional ethics requirements (for example, thinking carefully about what permissions we request on consent forms), but other questions must be part of the relationship between the researcher and the research participants. Corpus linguists working with spoken, computer-mediated, or otherwise particularly sensitive data have been aware of at least some of these issues, but the CARE principles offer an opportunity to go further.

Acquiring data for linguistic research takes effort and often that means money. It is therefore a good use of resources if any data we collect can be used by others. The FAIR principles provide a framework to make sharing and reusing data easier, and applying the CARE principles where relevant helps to ensure that our research has a sound ethical basis.

 
<hr />
 

Note: This post is based on the presentation ‘Advance Australia FAIR’, given by Simon Musgrave and Michael Haugh to the 4th Forum on Englishes in Australia (LaTrobe University, August 27, 2021). 

 

Thanks to Leah Gustafson and Monika Bednarek for helpful comments on drafts.

 

**Reference:**
Cameron, Deborah, Elizabeth Frazer, Penelope Harvey, Ben Rampton & Kay Richardson. 1993. Ethics, advocacy and empowerment: Issues of method in researching language. Language & Communication 13(2). 81–94. [https://doi.org/10.1016/0271-5309(93)90001-4](https://doi.org/10.1016/0271-5309(93)90001-4)

Time	Activity	People
09:00 - 10:00	Introduction to notebooks What are Jupyter notebooks and why are they useful?	Sara King Simon Musgrave
10:00 - 10:30	Break
10:30 - 11:00	General progress report An overview of the projects and of progress to date	Michael Haugh Robert McLellan Simon Musgrave Jenny Fewster (ARDC)
11:00 - 12:00	Introducing the LDaCA Portal How will I find data in LDaCA? Your chance to tell us whether we are meeting your needs!	Peter Sefton Kathrin Kaiser
12:00 - 01:00	Lunch
01:00 - 02:00	Data and access How is access to data managed in LDaCA and how can I access data?	Peter Sefton Kathrin Kaiser Catherine Travis
02:00 - 02:30	Auslan project update Report on work by the team at Monash University	Louisa Willoughby Trevor Johnston
02:30 - 03:00	Break
03:00 - 03:30	Text analytics for corpus linguistics Selected ATAP tools showcase	Sydney Informatics Hub with Sydney Corpus Lab (Sony Jufri, Kelvin Lee)
03:30 - 05:00	Discursis workshop How to get started with this tool for tracking topics in conversations	Ben Foley Sam Hames