Crowdsourcing Translation: Professional Quality from Non-Professionals

By Omar Zaidan, Chris Callison-Burch. Annual Meeting of the Association for Computational Linguistics, 2011

395 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/cb3796eee6752fc0824cfcb633f4bb03e8161554

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, methodology): “…manual checks required, we relied on automatic methods that ensured the translations followed these principles: ( i ) translations should be ﬂuent (Zaidan and Callison-Burch, 2011), ( ii ) they should be sufﬁciently different from the source text, ( iii ) translations should be similar to each…”
The freelance translation machine: Algorithmic culture and the invisible industry (2013, background): “They concluded that ‘it is possible to obtain high-quality translations from non-professional translators, and that the cost is an order of magnitude cheaper than professional translation’ (Zaidan and Callison-Burch, 2011: 1228).”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, background): “The guiding principles underlying our choice of automatic filters are: (i) translations should be fluent (Zaidan and Callison-Burch, 2011), (ii) they should be sufficiently different from the source text, (iii) translations should be similar to each other, yet not equal; and (iv) translations should”
Learning Whom to Trust with MACE (2013, background): “While such models have been implemented in other fields (e.g., vision), we are not aware of their availability for NLP tasks (see also Section 6).”
Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines (2014, methodology): “A technique that is more speciﬁc to translation is to display the source sentence as an image rather than text, thus preventing contributors from simply copying the text into an automatic translator, e.g. (Zaidan and Callison-Burch, 2011).”
Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problem (2015, background): “, 2010), and natural language processing (Zaidan and Callison-Burch, 2011), which involve explicit or implicit feedback provided by humans.”
The Language Demographics of Amazon Mechanical Turk (2014, background): “as images (Zaidan and Callison-Burch, 2011), this strategy does not prevent workers from typing the words into an MT system if they are able to type in the language’s script.”
Crowd-sourcing NLG Data: Pictures Elicit Better Data. (2016, methodology): “Currently deployed technologies for NLG utilise domain-dependent methods including hand-written grammars or domain-specific language templates for surface realisation, both of which are costly to develop and maintain.”
Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets (2019, methodology): “The prevalent method for creating new datasets is through crowdsourcing, where examples are generated by workers (Zaidan and Callison-Burch, 2011; Richardson et al., 2013; Bowman et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2017).”
Constructing Parallel Corpora for Six Indian Languages via Crowdsourcing (2012, methodology): “The techniques described here are similar to those described in Zaidan and Callison-Burch (2011), who showed that crowdsourcing with appropriate quality controls could be used to produce professional-level translations for Urdu-English translation.”
Intrinsic Evaluation of Summarization Datasets (2020, background): “struction (Zaidan and Callison-Burch, 2011; Yan et al., 2014; Callison-Burch et al., 2015), such as using additional annotators to validate annotations (c.”