Word Ordering Without Syntax

By Allen Schmaltz, Alexander M. Rush, Stuart M. Shieber. 2016

43 citations recorded by the source corpus.

Recent work on word ordering has argued that syntactic structure is important, or even required, for effectively recovering the order of a sentence. We find that, in fact, an n-gram language model with a simple heuristic gives strong results on this task. Furthermore, we show that a long short-term memory (LSTM) language model is even more effective at recovering order, with our basic model outperforming a state-of-the-art syntactic model by 11.5 BLEU points. Additional data and larger beams yield further gains, at the expense of training and search time.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, methodology): “The task of linearization is to recover the original order of a shufﬂed sentence [10].”
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks (2016, methodology): “Commonly used approaches is to either compare the performance of the sentence embeddings on down-stream tasks (Hill et al., 2016), or to analyze models, speciﬁcally trained for predeﬁned task (Schmaltz et al., 2016; Sutskever et al., 2011).”
Learning to Write with Cooperative Discriminators (2018, background): “, 2016), future cost estimation (Schmaltz et al., 2016), or a diversityboosting objective function (Shao et al.”
Style Transfer from Non-Parallel Text by Cross-Alignment (2017, background): “Word order recovery Our final experiments focus on the word ordering task, also known as bag translation (Brown et al., 1990; Schmaltz et al., 2016).”
Information Leakage in Embedding Models (2020, background): “by considering attacks that recover a set of words without recovering the word ordering. We leave open the problem of recovering exact sequences and one promising direction involves language modeling [64]. A naïve approach for inversion would be enumerating all possible sequences from the vocabulary”
Neural Sentence Ordering (2016, background): “Comparing with word ordering (Tillmann and Ney, 2000; Zhang et al., 2012; Zhang and Clark, 2015; Schmaltz et al., 2016), sentence ordering is still less studied.”
A Comparison of Neural Models for Word Ordering (2017, methodology): “However, standard beam search suffers from search errors when applied to word ordering and Schmaltz et al. (2016) reported that gains often do not saturate even with a large beam of 512.”
Non-Monotonic Sequential Text Generation (2019, methodology): “We ﬁrst evaluate the proposed models for conditional generation on the Word Reordering task, also known as Bag Translation (Brown et al., 1990) or Linearization (Schmaltz et al., 2016).”
Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2020, background): “Language models (Mikolov et al., 2010) with long short-term memory (LSTM) units (Hochreiter and Schmidhuber, 1997) were able to recover the original word order of a sentence from randomly-shufﬂed words even without any explicit syntactic information (Schmaltz et al., 2016).”
Obj2Text: Generating Visually Descriptive Language from Object Layouts (2017, methodology): “We also take in-spiration from recent work by Schmaltz et al. (2016) where the goal was to reconstruct a sentence from a bag-of-words (BOW) representation using a simple surface-level language model based on an encoder-decoder sequence-to-sequence architecture.”