Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser

By Long Duong, Trevor Cohn, Steven Bird, Paul Cook. 2015

390 citations recorded by the source corpus.

Training a high-accuracy dependency parser requires a large treebank. However, these are costly and time-consuming to build. We propose a learning method that needs less data, based on the observation that there are underlying shared structures across languages. We exploit cues from a different source language in order to guide the learning process. Our model saves at least half of the annotation effort to reach the same accuracy compared with using the purely supervised method.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Multilingual Language Processing From Bytes (2015, background): “Work on multilingual parsing using Neural Networks that share some subset of the parameters across languages (Duong et al., 2015) seems to beneﬁt the low-resource languages; however, we are sharing all the parameters among all languages.”
Latent Multi-Task Architecture Learning (2017, background): “In soft parameter sharing (Duong et al. 2015), each task has separate parameters and separate hidden layers, as in our architecture, but the loss at the outer layer is regularized by the current distance between the models.”
Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts (2018, methodology): “In fact, many DNN-based multi-task learning models are sensitive to factors such as the data distribution di erences and relationships among tasks [15, 34].”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “In [10], for example, the loss is regularized by the L 2 distance between (selective parts of) the main and auxiliary models.”
Which Tasks Should Be Learned Together in Multi-task Learning? (2019, methodology): “The canonical example of soft parameter sharing can be seen in [11].”
An Edge Traffic Flow Detection Scheme Based on Deep Learning in an Intelligent Transportation System (2021, methodology): “To do so, various structure compression and algorithm acceleration models were proposed, e.g., parameter pruning [28], parameter sharing [29], low-rank decomposition [30], compact convolution kernel [31] and knowledge distillation [32].”
Empirical Analysis of Multi-Task Learning for Reducing Model Bias in Toxic Comment Detection (2019, background): “In deep learning models, multi-task learning is usually implemented by either sharing hidden layer model parameters (Long et al. 2017) or regularizing parameters among related tasks to be similar (Duong et al. 2015).”
Visual Reasoning by Progressive Module Networks (2018, background): “Soft sharing refers to each task having independent weights that are constrained to be similar (e.g. via `2 regularization (Duong et al., 2015), trace norm (Yang & Hospedales, 2017)) while hard sharing typically means that all tasks share the base network but have independent layers on the top…”
Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer (2020, background): “The model is trained on a labeled data rich language and adopted to another language where no or a small portion of labeled data is available (Duong et al., 2015; Guo et al., 2016).”
Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates (2020, background): “Open problems: NLP systems have been shown to be inaccurate for low-resource languages (Duong et al., 2015), and exhibit racial and gender disparity (Blodgett and O’Connor, 2017; Zhao et al., 2017).”
Exploiting Multi-typed Treebanks for Parsing with Deep Multi-task Learning (2016, methodology): “As shown in Table 4, the performance of SMTL decreases sig-niﬁcantly (-0 To verify the second issue, we consider a low resource setup following Duong et al. (2015b), where the target language has a small treebank (3K tokens).”
Learning what to share between loosely related tasks (2017, methodology): “In (Duong et al., 2015), for example, the loss is regularized by the L 2 distance between (selective parts of) the main and auxiliary models.”