Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, Vikas Chandra
| Challenge: | Language model pretraining is the cornerstone of universal language models (LMs), creating generalpurpose representations to excel across a variety of downstream tasks. |
| Approach: | They propose to use multi-granular tokens to sample large-scale language models for domain-specific use cases. |
| Outcome: | The proposed model outperforms random sampled samples on eight benchmarks with 1% of the data and performs on par with the full RefinedWeb data. |
Similar Papers
Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to train pre-trained models require domain-specific data and computational resources. |
| Approach: | They propose a domain-aware N-gram Adaptor to incorporate unseen and domain-specific words into a generic pretrained model. |
| Outcome: | The proposed model can improve on eight low-resource tasks using limited data with lower computational costs. |
Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection (2023.findings-eacl)
Copied to clipboard
| Challenge: | AfriBERTa shows that training transformer models from scratch on 1GB of data from many unrelated African languages outperforms massively multilingual models on downstream NLP tasks. |
| Approach: | They propose that training on smaller amounts of data but from related languages could match the performance of models trained on large, unrelated data. |
| Outcome: | The proposed model outperforms models trained on large, unrelated datasets on downstream NLP tasks. |
On the importance of pre-training data volume for compact language models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. |
| Approach: | They investigate the impact of pre-training data volume on compact language models . they use a French question answering task to train models with as little as 100 MB of text . |
| Outcome: | The results show that pre-training data volume can improve models with as little as 100 MB of text . the results suggest that the model performance is poorer with less data than with larger datasets . |
Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to perform named entity recognition (NER) on unlabeled data are difficult to obtain in low-resource domains. |
| Approach: | They propose ways to use unlabeled data for pretraining to improve performance in downstream tasks. |
| Outcome: | The proposed methods outperform models trained on unlabeled data on seven domains. |
How much pretraining data do language models need to learn syntax? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining methods are convenient, but expensive in terms of time and resources. |
| Approach: | They investigate the impact of pretraining data size on the syntactic capabilities of RoBERTa by using syntaktic structural probes to determine whether models pretrained on more data encode a higher amount of syntastic information. |
| Outcome: | The proposed models perform better on part-of-speech tagging, dependency parsing and paraphrase identification. |
Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have suggested that the composition of the pretraining corpus exerts a significant impact upon the performance of LLMs. |
| Approach: | They analyze the impact of 48 datasets from 5 major categories of pretraining data of Large Language Models and measure their impacts on LLMs using benchmarks about nine major categories. |
| Outcome: | The proposed analysis provides insights into the organization of data to support more efficient pretraining of Large Language Models. |
How Much Pretraining Does Structured Data Need? (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly adopted for handling structured data, despite pretraining on unstructured text. |
| Approach: | They propose to re-initialize subsets of layers with random weights before fine-tuning on structured datasets. |
| Outcome: | The proposed models are compared to unstructured datasets and show that they perform well over structured data. |
LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are trained based on single-grained tokenization, making it hard to learn the precise meaning of coarse-grain words and phrases. |
| Approach: | They propose a language model pretraining method that incorporates multi-grained information of input text into pre-trained language models. |
| Outcome: | The proposed method improves performance on CLUE and SuperGLUE in Chinese and English with little extra inference cost. |
Exploiting Language Characteristics for Legal Domain-Specific Language Model Pretraining (2023.findings-eacl)
Copied to clipboard
| Challenge: | Pretraining large language models has resulted in tremendous performance improvement for many natural language processing tasks. |
| Approach: | They propose to incorporate pretraining objectives that explicitly exploit domain specific language characteristics into the model. |
| Outcome: | The proposed objectives target token-level feature representation and incorporate sentence level semantics. |
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models process text as a sequence of characters, ignoring more coarse granularity, e.g., words. |
| Approach: | They propose a new pre-training paradigm for Chinese that incorporates word representations along with characters and can model a sentence in a multi-granular manner. |
| Outcome: | The proposed model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. |