Bridging CNNs, RNNs, and Weighted Finite-State Machines

By Roy Schwartz, Sam Thomson, Noah A. Smith. 2018

24 citations recorded by the source corpus.

Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances. In this paper we present SoPa, a new model that aims to bridge these two approaches. SoPa combines neural representation learning with weighted finite-state automata (WFSAs) to learn a soft version of traditional surface patterns. We show that SoPa is an extension of a one-layer CNN, and that such CNNs are equivalent to a restricted version of SoPa, and accordingly, to a restricted form of WFSA. Empirically, on three text classification tasks, SoPa is comparable or better than both a BiLSTM (RNN) baseline and a CNN baseline, and is particularly useful in small data settings.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, background): “One way towards faithfulness is to introduce architectural modiﬁcations or constraints that produce rationales with desirable properties (Andreas et al., 2016; Schwartz et al., 2018; Jiang et al., 2019, inter alia ).”
RNN Architecture Learning with Sparse Regularization (2019, methodology): “A path starts at q 0 ; at least four tokens must be consumed to reach q 4 , and in this sense it captures 4-gram “soft” patterns (Peng et al., 2018; Schwartz et al., 2018).”