Language Models for Code-switch Detection of te reo Māori and English in a Low-resource Setting

By Jesin James, Vithya Yogarajan, Isabella Shields, Catherine Inez Watson, Peter J. Keegan, Keoni Mahelona, Peter-Lucas Jones. 2022

11 citations recorded by the source corpus.

Te reo M¯aori, New Zealand’s only indige- 001 nous language, is code-switched with English. 002 Most M¯aori speakers are bilingual, and the 003 use of M¯aori is increasing in New Zealand 004 English. Unfortunately, due to the minimal 005 availability of resources, including digital data, 006 M¯aori is under-represented in technological ad- 007 vances. Cloud-based systems such as Google 008 and Azure support M ¯ aori language detection. 009 However, we provide experimental evidence 010 to show that the accuracy of such systems is 011 low when detecting M ¯ aori. Hence, with the 012 support of M ¯ aori community, we collect M ¯ aori 013 and bilingual data to use natural language pro- 014 cessing (NLP) to improve M ¯ aori language de- 015 tection. We train bilingual sub-word embed- 016 dings and provide evidence to show that our 017 bilingual embeddings improve overall accuracy 018 compared to the publicly-available monolin- 019 gual embeddings. This improvement has been 020 verified for various NLP tasks using three bilin- 021 gual databases containing formal transcripts 022 and informal social media data. We also show 023 that BiLSTM with bilingual sub-word embed- 024 dings outperforms large-scale contextual lan- 025 guage models such as BERT on down stream- 026 ing tasks of detecting M ¯ aori language. The 027 best accuracy of 87% was obtained using BiL- 028 STM with bilingual embeddings for detecting 029 code-switch points of bilingual sentences.

Subfields: Artificial Intelligence, Language and Linguistics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Language and Linguistics

In Natural Language Processing

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “While some research strives to improve performance for underrepresented Indigenous languages [116], the same Indigenous groups resist AI approaches to use of their language [158].”