On internal language representations in deep learning: an analysis of machine translation and speech recognition

By Yonatan Belinkov. 2018

30 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Machine Translation

In Speech & Audio

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “Somewhat anecdotally, a few studies observed better performance with more complex probes, but reported similar relative trends (Conneau et al. 2018; Belinkov 2018).”
Analysis Methods in Neural Language Processing: A Survey (2018, neutral): “A few studies compared different classifiers and found that deeper classifiers lead to overall better results, but do not alter the respective trends when comparing different models or components (Qian et al., 2016b; Belinkov, 2018).”
BERT Rediscovers the Classical NLP Pipeline (2019, background): “tion (e.g. Shi et al., 2016; Belinkov, 2018; Tenney et al., 2019), and that more complex structures are represented hierarchically in the higher layers of the model (Peters et al.”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “…approach has been used previously to show that the representations learned by neural machine translation systems encode token-level properties like part-of-speech, semantic tags, and morphology (Shi et al., 2016; Belinkov et al., 2017a;b), as well as pairwise dependency relations (Belinkov, 2018).”
Linguistic Knowledge and Transferability of Contextual Representations (2019, methodology): “In these tasks, given a word pair w1, w2, we input [w1, w2, w1 w2] into the probing model; it is trained to predict information about the relation between the tokens (Belinkov, 2018).”
Analyzing the Structure of Attention in a Transformer Language Model (2019, methodology): “Recurrent models were found to encode dependencies in lower layers for language models (Liu et al., 2019) and in deeper layers for translation models (Belinkov, 2018).”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Somewhat anecdotally, a few studies observed better performance with more complex probes, but reported similar relative trends (Conneau et al. 2018; Belinkov 2018).”
Inducing Syntactic Trees from BERT Representations (2019, result): “A significant difference between our work and most previous works (Hewitt and Manning, 2019; Belinkov, 2018) is that we estimate the reducibilities and dependency trees directly from the models, without any training on syntactically annotated data.”