An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks

By Kyubyong Park, Joohong Lee, Seongbo Jang, Dawoon Jung. 2020

83 citations recorded by the source corpus.

Typically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual information of text, how to define a token plays a decisive role in the performance of a model. Even though Byte Pair Encoding (BPE) has been considered the de facto standard tokenization method due to its simplicity and universality, it still remains unclear whether BPE works best across all languages and tasks. In this paper, we test several tokenization strategies in order to answer our primary research question, that is, “What is the best tokenization strategy for Korean NLP tasks?” Experimental results demonstrate that a hybrid approach of morphological segmentation followed by BPE works best in Korean to/from English machine translation and natural language understanding tasks such as KorNLI, KorSTS, NSMC, and PAWS-X. As an exception, for KorQuAD, the Korean extension of SQuAD, BPE segmentation turns out to be the most effective. Our code and pre-trained models are publicly available at https://github.com/kakaobrain/kortok.

Subfields: Artificial Intelligence, Natural Language Processing, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Computational Linguistics & Semantics

Citations in the corpus

BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model (2022, background): “…the ﬁnal trained multilingual model that was the focus of the effort would to the best of our knowledge depend on a range of Modeling choices, including for example tokenization (Park et al., 2020) or architecture (Scao et al., 2022); all of which were explored in speciﬁc modeling working groups.”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, methodology): “A similarity score is calculated by the cosine similarity score using a feature of the in-house BERT.”
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP (2021, methodology): “Park et al. (2020) shows that a hybrid approach of morphological segmentation followed by BPE (§6.4) works best for most Korean datasets.”