A new algorithm for data compression

By Philip Gage. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Algorithms & Complexity

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “One such method (Sennrich et al., 2016b) is based on the byte pair encoding (BPE) compression al-gorithm (Gage, 1994).”
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “BPE was originally introduced in the data compression literature (Gage, 1994).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “Byte-level BPE The tokenizer is a learned subword tokenizer trained using the Byte Pair Encoding (BPE) algorithm introduced by Gage (1994).”
Extracting Training Data from Large Language Models (2020, methodology): “GPT-2 uses a word-pieces [61] vocabulary with a byte pair encoder [22].”
CogView: Mastering Text-to-Image Generation via Transformers (2021, background): “Tokenization for text is already well-studied, e.g. BPE [16] and SentencePiece [28].”
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation (2016, methodology): “The segmentation algorithm used here is very similar (with small differences) to Byte-Pair-Encoding (BPE) which was described in Gage (1994) and was also used in Sennrich et al.”
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants (2022, methodology): “These are used via ‘byte pair encoding’ [24] to allow the LLMs to ingest more text into their fixed-size input windows.”
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search (2019, methodology): “First, we preprocess the input sequences according to their semantics: identifiers appearing in code tokens are split into subtokens ( i.e. a variable camelCase yields two subtokens camel and case ), and natural language tokens are split using byte-pair encoding (BPE) [10, 21].”
Edinburgh Neural Machine Translation Systems for WMT 16 (2016, neutral): “BPE, originally devised as a compression algorithm (Gage, 1994), is adapted to word segmentation as follows: First, each word in the training vocabulary is represented as a sequence of characters, plus an end-of-word symbol.”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “In order to reduce the effective vocabulary size, we use byte pair encoding (BPE) (Gage, 1994; Sennrich et al., 2015), which splits each word into sub-words or characters.”
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data (2024, methodology): “We apply additional loss functions to promote speaker disentanglement, and compress the resulting speechcodes using Byte-Pair Encoding (BPE) [41] to reduce sequence length, enabling us to model longer audio with Transformers.”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “There is one possible reason: BERT uses a character-level BPE (Gage, 1994) vocabulary, while RoBERTa considers byte-level BPE vocabulary.”
Language Models are Multilingual Chain-of-Thought Reasoners (2022, methodology): “Given the unbalanced representation of languages in the training corpora, the byte-pair encoding (BPE; Gage, 1994) algorithm tokenizes sentences in underrepresented languages, especially those in a different alphabet from English, into more tokens.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, background): “manageable size; advanced approaches such as adaptations of the Byte-Pair Encoding (BPE) algorithm [34, 79] are needed to reach this goal and deal with the OOV problem.”
Multi-Domain Neural Machine Translation through Unsupervised Adaptation (2017, methodology): “…the main bottlenecks for the existing NMT systems, state-of-the-art approaches are trained on corpora in which the less frequent words are segmented into their sub-word units (Sennrich et al., 2016) by applying a modi-ﬁed version of the byte pair encoding (BPE) compression algorithm (Gage, 1994).”
Temporal Attention Model for Neural Machine Translation (2016, methodology): “Considering the morphological richness of German, we follow the approach by Sennrich et al. (2015) and segment the German and English sides into subwords by using byte-pair encoding (Gage, 1994 distinct vocabularies.”
Jamba: A Hybrid Transformer-Mamba Language Model (2024, methodology): “The tokenizer is trained with BPE [15, 29, 39] and each digit is a separate token [6].”
Examining Zero-Shot Vulnerability Repair with Large Language Models (2021, methodology): “For tokenization we trained a BPE tokenizer on the source dataset.”