Morphological Inflection Generation with Hard Monotonic Attention

By Roee Aharoni, Yoav Goldberg. 2016

133 citations recorded by the source corpus.

We present a neural model for morphological inflection generation which employs a hard attention mechanism, inspired by the nearly-monotonic alignment commonly found between the characters in a word and the characters in its inflection. We evaluate the model on three previously studied morphological inflection generation datasets and show that it provides state of the art results in various setups compared to previous neural and non-neural approaches. Finally we present an analysis of the continuous representations learned by both the hard and soft (Bahdanau, 2014) attention models for the task, shedding some light on the features such models extract.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Natural Language Generation

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “…have produced visualizations ranging from tasks like NLI (Rocktäschel et al., 2016; Yin et al., 2016), summarization (Rush et al., 2015), MT post-editing (Jauregi Unanue et al., 2018), and morphological inﬂection (Aharoni and Goldberg, 2017), to matching users on social media (Tay et al., 2018).”
Continual Learning with Knowledge Transfer for Sentiment Classification (2021, methodology): “To generate a binary matrix at so that it can work as a mask in both the MCL (and KB) and AC training phases, we borrow the idea of hard attention in [24, 31, 1] where the values in the attention matrix are binary instead of a probability distribution as in soft attention.”
Marian: Fast Neural Machine Translation in C++ (2018, background): “Furthermore, we investigate the effect of hard-attention models or neural transductors (Aharoni and Goldberg, 2016) which seem to be wellsuited for monolingual tasks, as well as combina- tions of both ideas.”
Monotonic Chunkwise Attention (2017, methodology): “In future work, we are interested in applying MoChA to additional problems with (approximately) monotonic alignments, such as speech synthesis (Wang et al., 2017) and morphological inflection (Aharoni & Goldberg, 2016).”
Ensemble Self-Training for Low-Resource Languages: Grapheme-to-Phoneme Conversion and Morphological Inflection (2020, unknown): “The second one is a hard monotonic attention model ( mono ), similar to Aharoni and Goldberg (2017), where the decoder uses a pointer to select the input vector to make a prediction: either produc-1 ing a phoneme, or moving the pointer to the next position.”
Sparse Sequence-to-Sequence Models (2019, background): “, 2015; Aharoni and Goldberg, 2017; Wu et al., 2018) selects exactly one source token. Its discrete, non-differentiable nature requires imitation learning or Monte Carlo policy gradient approximations, which drastically complicate training. In contrast, entmax is a differentiable, easy to use, drop-”
Neural Machine Translation: A Review (2019, background): “Similar ideas have been explored for speech recognition [385], morphological inflection [386], text summarization [387, 388], and image caption generation [82].”
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection (2020, methodology): “IMS and Flexica re-used the hard mono-tonic attention model from (Aharoni and Goldberg, 2017).”
Gender-Aware Reinflection using Linguistically Enhanced Neural Models (2020, unknown): “Sequence-to-sequence models have achieved signiﬁcant results in grammatical error correction (GEC) (Chollampatt and Ng, 2018; Junczys-Dowmunt et al., 2018; Grundkiewicz et al., 2019) and morphological reinﬂection tasks (Faruqui et al., 2016; Kann and Schütze, 2016; Aharoni and Goldberg, 2017).”
Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS (2019, methodology): “Among all the attention variants, monotonic attention has demonstrated to be particularly effective to strictly preserve monotonicity and locality, and has been applied to multiple tasks including TTS [15, 16].”
An Exploration of Neural Sequence-to-Sequence Architectures for Automatic Post-Editing (2017, methodology): “Aharoni and Goldberg (2016) introduce a simple model for monolingual morphological re-inﬂection with hard monotonic attention.”
Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling (2018, methodology): “Aharoni et al. [ 2017 ] present a neural model for morphological inﬂec-tion generation, where a hard attention mechanism is used to align the characters in a word with its inﬂection because the inﬂection mapping is nearly-monotonic.”
Parallel Iterative Edit Models for Local Sequence Transduction (2019, background): “(2018)’s local transduction model (described in Section 4), and LSTM based ED models with hard monotonic attention (Aharoni and Goldberg, 2017) and soft-attention (Bahdanau et al.”
One-Size-Fits-All Multilingual Models (2020, background): “While hard attention has previously performed well for character transduction (Aharoni and Goldberg, 2017; Makarov et al., 2017; Wu et al., 2018; Wu and Cotterell, 2019), it usually requires an elaborate and slow training procedure.”