Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models

By Avery Hiebert, Cole Peterson, Alona Fyshe, Nishant Mehta. 2018

6 citations recorded by the source corpus.

While Long Short-Term Memory networks (LSTMs) and other forms of recurrent neural network have been successfully applied to language modeling on a character level, the hidden state dynamics of these models can be difficult to interpret. We investigate the hidden states of such a model by using the HDBSCAN clustering algorithm to identify points in the text at which the hidden state is similar. Focusing on whitespace characters prior to the beginning of a word reveals interpretable clusters that offer insight into how the LSTM may combine contextual and character-level information to identify parts of speech. We also introduce a method for deriving word vectors from the hidden state representation in order to investigate the word-level knowledge of the model. These word vectors encode meaningful semantic information even for words that appear only once in the training text.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “These include word similarity tasks (Hiebert et al., 2018), syntactic and semantic tagging tasks (Durrani et al.”