Segmental Recurrent Neural Networks

By Lingpeng Kong, Chris Dyer, Noah A. Smith. 2015

126 citations recorded by the source corpus.

We introduce segmental recurrent neural networks (SRNNs) which define, given an input sequence, a joint probability distribution over segmentations of the input and labelings of the segments. Representations of the input segments (i.e., contiguous subsequences of the input) are computed by encoding their constituent tokens using bidirectional recurrent neural nets, and these "segment embeddings" are used to define compatibility scores with output labels. These local compatibility scores are integrated using a global semi-Markov conditional random field. Both fully supervised training -- in which segment boundaries and labels are observed -- as well as partially supervised training -- in which segment boundaries are latent -- are straightforward. Experiments on handwriting recognition and joint Chinese word segmentation/POS tagging show that, compared to models that do not explicitly represent segments such as BIO tagging schemes and connectionist temporal classification (CTC), SRNNs obtain substantially higher accuracies.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “Hierarchical RNN architectures have also been used to discover the segmentation structure in sequences (Fernández et al., 2007; Kong et al., 2015).”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “More closely to our model, (Yu et al., 2016b) similarly add “shift” and “emit” operations to an RNN. Finally, the Segmental RNN (Kong et al., 2015) treats a segmentation of the input sequence as a latent random variable.”
The Neural Noisy Channel (2016, background): “, 2006) and the more recent segmental recurrent neural networks (SRNN) (Kong et al., 2016).”
Monotonic Chunkwise Attention (2017, neutral): “A decoder RNN then autoregressively produces the output sequence.”
Learning Neural Templates for Text Generation (2018, background): “Finally, there has also been much recent interest in segmental RNN models for non-generative tasks in NLP (Tang et al., 2016; Kong et al., 2016; Lu et al., 2016).”
Online Segment to Segment Neural Transduction (2016, background): “…is general and could be incorporated into any RNN-based encoder-decoder architecture, such as Neural Turing Machines (Graves et al., 2014), memory networks (Weston et al., 2015; Kumar et al., 2016) or stack-based networks (Grefenstette et al., 2015), enabling such models to process data online.”
Syntactic Scaffolds for Semantic Structures (2018, methodology): “Our models use the strongest available neural network architectures for these tasks, integrating deep representation learning (He et al., 2017) and structured prediction at the level of spans (Kong et al., 2016).”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “Since we marginalize over unobserved trees during learning, our work is also related to the line of work on marginalizing out latent variables/structures for sequence transduction tasks [46, 33, 11, 66, 128, 90, 57, 69, 108, 111, inter alia ].”
Learning Structured Natural Language Representations for Semantic Parsing (2017, background): “Within our current framework, these two types of structural mismatches can be handled with semi-Markov assumptions (Sarawagi and Cohen, 2005; Kong et al., 2016) in the parsing (i.e., predicate selection) and the grounding steps, respectively.”
Natural Language Processing with Small Feed-Forward Networks (2017, background): “Recently, neural networks have signiﬁcantly improved segmentation accuracy (Zhang et al., 2016; Cai and Zhao, 2016; Liu et al., 2016; Yang et al., 2017; Kong et al., 2015).”
TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning (2019, background): “Host-language integration is more than just syntactic sugar—it greatly simpliﬁes the implementation of data-dependent models like segmental recurrent neural networks and recursive neural networks (Kong et al., 2015; Socher et al., 2011).”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “The more popular solution of segmental neural language models was pioneered by Kong et al. (2016), who cast the problem of segmentation as a mono-tonic 13 seq2seq task, going from characters to a covering sequence of substrings, i.e., a segmentation.”
The Unstoppable Rise of Computational Linguistics in Deep Learning (2020, unknown): “Also, Semi-Markov CRF models (Sarawagi and Cohen, 2005) can learn segmentations of an input string, which have been used in the output layers of neural models (e.g. (Kong et al., 2015)).”
DyNet: The Dynamic Neural Network Toolkit (2017, methodology): “In some cases, it is desirable to use sophisticated inference algorithms to compute or approximate a diﬀerentiable quantity such as the marginal likelihood or Bayes risk during learning [26, 15, 25, 38].”
Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation (2019, methodology): “While they were originally developed for information extraction (Sarawagi and Co-hen, 2005), it is most commonly applied to speech recognition within the acoustic model to improve segmentation between different words (He and Fosler-Lussier, 2015; Lu et al., 2016; Kong et al., 2015).”
Neural Speed Reading via Skim-RNN (2017, methodology): “Our model is relevant to several recent works that incorporate hard decisions within recurrent neural networks (Kong et al., 2016).”
Segmental Recurrent Neural Networks for End-to-End Speech Recognition (2016, methodology): “Attention-based RNNs have been demonstrated to be a powerful alternative sequence-to-sequence transducer, e.g., in machine translation [10], and speech recognition [11, 12, 13].”
Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling (2018, methodology): “Our span-graph formulation overcomes a key limitation of semi-markov and BIO-based models (Kong et al., 2016; Zhou and Xu, 2015; Yang and Mitchell, 2017; He et al., 2017; Tan et al., 2018): it can model overlapping spans across different predicates in the same output structure (see Figure 1).”