What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

By Boseop Kim, Hyoungseok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, D. Jeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, D. Seo and 25 more. 2021

134 citations recorded by the source corpus.

GPT-3 shows remarkable in-context learning ability of large-scale language models (LMs) trained on hundreds of billion scale data. Here we address some remaining issues less reported by the GPT-3 paper, such as a non-English LM, the performances of different sized models, and the effect of recently introduced prompt optimization on in-context learning. To achieve this, we introduce HyperCLOVA, a Korean variant of 82B GPT-3 trained on a Korean-centric corpus of 560B tokens. Enhanced by our Korean-specific tokenization, HyperCLOVA with our training configuration shows state-of-the-art in-context zero-shot and few-shot learning performances on various downstream tasks in Korean. Also, we show the performance benefits of prompt-based learning and demonstrate how it can be integrated into the prompt engineering pipeline. Then we discuss the possibility of materializing the No Code AI paradigm by providing AI prototyping capabilities to non-experts of ML by introducing HyperCLOVA studio, an interactive prompt engineering interface. Lastly, we demonstrate the potential of our methods with three successful in-house applications.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding (2022, background): “There is large room for future work, including but not limited to: Extension to few-shot learning settings, exploring larger generator models [25, 68], better fine-tuning techniques to leverage generated data and better strategies for selecting quality training data.”
Predictability and Surprise in Large Generative Models (2022, background): “0 Titan [81], FLAN [82], Gopher [62], GPT-3 [11], HyperClova [46], Jurassic-1-Jumbo [51], Megatron Turing NLG [72], LaMDA [78], Pan Gu [89], Yuan 1 .”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, methodology): “…makes up the backbone of the GPT series of models [Radford et al., 2018, 2019, Brown et al., 2020] as well as many other recent record-breaking LLMs [Zeng et al., 2021, Kim et al., 2021, Smith et al., 2022, Thoppilan et al., 2022, Rae et al., 2021, Hoffmann et al., 2022, Chowdhery et al., 2022].”
What Language Model to Train if You Have One Million GPU Hours? (2022, methodology): “We use the Adam optimizer (Kingma and Ba, 2014), with β 1 = 0 .”
A Comprehensive Overview of Large Language Models (2023, unknown): “1.8 HyperCLOVA [99]: A Korean language model with GPT-3 architecture.”
Prompt Injection: Parameterization of Fixed Inputs (2022, unknown): “Zero-shot Learning with Task Instructions Recent works [23, 32] have addressed zero-shot generalization to new tasks [3, 14] by multi-task prompted training.”
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022, background): “Prompt tuning has been proved to be comparable to fine-tuning on 10- billion-parameter models on simple classification tasks (Lester et al., 2021; Kim et al., 2021; Liu et al., 2021).”
Regulating ChatGPT and other Large Generative AI Models (2023, background): “LGAIMs are usually trained with several billion, if not hundreds of billions, parameters [43, 44], requiring large amounts of training data and computing power [45].”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “Surprising qualitative evaluation results (e.g., human-like writing) of extreme-scale LMs also fueled the competition in model size [8], [9].”
On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022, methodology): “Table 1: Descriptions of HyperCLOVA corpus (Kim et al., 2021).”