Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation

By J. Clark, Dan Garrette, Iulia Turc, John Wieting. 2021

314 citations recorded by the source corpus.

Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model’s ability to adapt. In this paper, we present Canine, a neural encoder that operates directly on character sequences—without explicit tokenization or vocabulary—and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, Canine combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. Canine outperforms a comparable mBert model by 5.7 F1 on TyDi QA, a challenging multilingual benchmark, despite having fewer model parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “We believe techniques such as hash embeddings, local attention and down-sampling (Clark et al., 2021), as well as sparse computation (Fedus et al.”
Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, methodology): “For a given FLOPs budget, this allows us to train a tokenizer-free language model that matches the performance of a baseline model trained with a SentencePiece tokenizer, hence removing the need for hand-crafted and potentially harmful tokenization schemes [6, 14].”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “Notably, the CANINE model uses local attention (Parmar et al., 2018), which could also be swapped with another efﬁcient Transformer variant.”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, neutral): “Several other architectures (Dai et al., 2020; Nawrot et al., 2021; Clark et al., 2021) reduce the processing requirements of Transformers by sequentially compressing the input with attention or convolution, but rely on the use of multiple large, quadratic-complexity attention layers or exploit…”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, background): “Another promising future research direction is completely disposing of any (language-specific or multilingual) tokenizers during pretraining (Clark et al., 2021).”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, background): “Instead of only providing examples as lists of tokens, XTREME-R always provides the full text of an input sentence, thus ensuring that the entire benchmark now supports research on models that operate directly from the raw input string (Clark et al., 2021).”
Text Data Augmentation for Deep Learning (2021, methodology): “Notably, researchers are exploring tokenizer-free models such as byT5 [123] and CANINE [124].”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Moreover,unlike CANINE, the gradient-based sub-word tokenizer output is interpretable.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “One possible solution is used in Clark et al. (2021)’s CANINE, where instead of giving each character its own embedding, they hash all possible codepoints with multiple hash functions into smaller sets to share parameters across codepoints (similar to Svenstrup et al.”
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (2023, methodology): “…been used in image encoder models such as ViT (Dosovitskiy et al., 2020), and down-and up-sampling operations have been used for text encoders (Clark et al., 2022), but such methods cannot be naively applied to decoder-only models without leaking information to future bytes in the same patch.”