Natural Language Multitasking: Analyzing and Improving Syntactic Saliency of Hidden Representations

By Gino Brunner, Yuyi Wang, Roger Wattenhofer, Michael Weigelt. 2018

12 citations recorded by the source corpus.

We train multi-task autoencoders on linguistic tasks and analyze the learned hidden sentence representations. The representations change significantly when translation and part-of-speech decoders are added. The more decoders a model employs, the better it clusters sentences according to their syntactic similarity, as the representation space becomes less entangled. We explore the structure of the representation space by interpolating between sentences, which yields interesting pseudo-English sentences, many of which have recognizable syntactic structure. Lastly, we point out an interesting property of our models: The difference-vector between two sentences can be added to change a third sentence with similar features in a meaningful way.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, neutral): “Recent examples include clustering of sentence embeddings in an RNN encoder trained in a multitask learning scenario (Brunner et al., 2017), and phoneme clusters in a joint audio-visual RNN model (Alishahi et al.”
Identifying and Controlling Important Neurons in Neural Machine Translation (2018, background): “, 2016b), sentence embeddings (Adi et al., 2016; Ganesh et al., 2017; Brunner et al., 2018), and NMT representations at different linguistic levels: morphological (Belinkov et al.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, result): “Similar conclusions were drawn by Brunner et al. [157]. who created several LSTM-based encoder–decoder networks, and analyzed the embedding vectors produced.”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “eludes word embeddings [138, 188, 279], RNN hidden states or gate activations [102, 278, 306, 335, 345], and sentence embeddings [2, 3, 47, 81, 106, 117].”