What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers

Challenge:	GPT-3 has been used to train large-scale language models on hundreds of billion scale data.
Approach:	They propose a Korean variant of GPT-3 that uses Korean tokens to train in-context models.
Outcome:	The proposed method shows state-of-the-art zero-shot and few-shot learning on downstream tasks in Korean.

Similar Papers

Overview

Challenge

Approach

Outcome

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022.naacl-main)

Copied to clipboard

Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung

Semantic Text Processing

Challenge:	Recent studies on large-scale in-context language models have reported successful in-const zero- and few-shot learning ability.
Approach:	They investigate the effects of the pretraining corpus on in-context learning in a Korean-centric model.
Outcome:	The study shows that pretraining corpus size does not determine in-context learning ability . the findings suggest that in-constext learning is not always competitive .

On the Multilingual Capabilities of Very Large-Scale English Language Models (2022.lrec-1)

Copied to clipboard

Jordi Armengol-Estapé, Ona de Gibert Bonet, Maite Melero

Semantic Text Processing

Challenge:	Generative Pre-trained Transformers (GPTs) have been scaled to unprecedented sizes in the history of machine learning.
Approach:	They investigate the potential and limits of Generative Pre-trained Transformers in three tasks . they find it can be almost as useful for many languages as it is for English .
Outcome:	The proposed model can perform tasks in five different languages, and its potential is explored . it can learn from a few examples "via text interaction" and is scalable to many languages .

GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021.findings-emnlp)

Copied to clipboard

Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyoung Park

Language Models

Challenge:	Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability.
Approach:	They propose a data augmentation technique that leverages large-scale language models to generate real text samples from a mixture of real samples.
Outcome:	The proposed method outperforms existing methods on diverse classification tasks.

Lessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedish (2022.lrec-1)

Copied to clipboard

Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, Magnus Sahlgren

Language Models

Challenge:	a prerequisite for building large-scale generative models for other languages is access to large amounts of high-quality text data and powerful computational resources.
Approach:	They present a 3.5 billion parameter autoregressive language model, trained on a 100 GB Swedish corpus.
Outcome:	The proposed model performs well on a 100 GB Swedish corpus and is competent in comparison with existing models of similar size.

The Power of Scale for Parameter-Efficient Prompt Tuning (2021.emnlp-main)

Copied to clipboard

Brian Lester, Rami Al-Rfou, Noah Constant

Responsible & Trustworthy NLP

Challenge:	Unlike discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples.
Approach:	They propose a mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks.
Outcome:	The proposed method outperforms fewshot learning using GPT-3 and matches the quality of model tuning as models exceed billions of parameters.

MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization (2023.findings-emnlp)

Copied to clipboard

Yuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen, Wei Wu, Dayiheng Liu, Zhixu Li, Bang Liu, Yanghua Xiao

Semantic Text Processing

Challenge:	Existing research emphasizes the importance of adapting prompts to specific tasks, rather than specific LLMs.
Approach:	They propose a model-adaptive prompt optimizer method that optimizes original prompts for each LLM in downstream tasks.
Outcome:	The proposed method can optimize prompts for an LLM in downstream tasks.

HyperT5: Towards Compute-Efficient Korean Language Modeling (2023.acl-industry)

Copied to clipboard

Dongju Park, Soonwon Ka, Kang Min Yoo, Gichang Lee, Jaewook Kang

Language Models

Challenge:	Pretraining and fine-tuning language models is a common practice in NLP, but deploying general-purpose language models without the abundant computation or data resources is proving difficult.
Approach:	They propose a sequence-to-sequence language model architecture that can be more practical and compute-efficient than the decoder-oriented approach.
Outcome:	The proposed language model outperforms competing models in Korean benchmarks and is more efficient in low-resource settings.

Few-shot Learning with Multilingual Generative Language Models (2022.emnlp-main)

Copied to clipboard

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

Language Models

Challenge:	Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Approach:	They train multilingual generative language models on a corpus covering a diverse set of languages and study their few- and zero-shot learning capabilities.
Outcome:	The proposed model outperforms GPT-3 on 171 out of 182 directions with 32 training examples and surpasses the official supervised baseline in 45 directions.

A Survey of Post-Training Scaling in Large Language Models (2025.acl-long)

Copied to clipboard

Hanyu Lai, Xiao Liu, Junjie Gao, Jiale Cheng, Zehan Qi, Yifan Xu, Shuntian Yao, Dan Zhang, Jinhua Du, Zhenyu Hou, Xin Lv, Minlie Huang, Yuxiao Dong, Jie Tang

Language Models

Challenge:	Large language models (LLMs) have demonstrated proficiency in understanding and generating human natural languages.
Approach:	They propose a framework for scaling large language models using supervised fine-tuning, RLxF and test-time compute methodologies.
Outcome:	The proposed model can be used to understand and generate human natural languages.

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021.naacl-main)

Copied to clipboard

Timo Schick, Hinrich Schütze

Language Models

Challenge:	Pretraining ever-larger language models on massive corpora requires enormous amounts of compute.
Approach:	They propose to convert textual inputs into cloze questions that contain a task description . they also exploit unlabeled data to improve their performance .
Outcome:	The proposed model outperforms GPT-3 with PET/iPET with cloze questions and unlabeled data.