SciPedia: Unlocking the Value of Scientific Data for Pre-training (2026.acl-long)
Copied to clipboard
| Challenge: | High-quality scientific data is critical for advancing LLMs, yet academic literature remains underutilized. |
| Approach: | They construct a large-scale raw scientific corpus but identify a critical Learnability Gap . they develop a multi-stage pipeline featuring content cleaning and pedagogical augmentation . |
| Outcome: | The proposed approach boosts average performance by +2.12 (3B) and +2.95 (7B) on in-domain tasks. |
Similar Papers
YuLan-Mini: Pushing the Limits of Open Data-efficient Language Model (2025.acl-long)
Copied to clipboard
Hu Yiwen, Huatong Song, Jie Chen, Jia Deng, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Yang Lu, Xu Miao, Xin Zhao, Ji-Rong Wen
| Challenge: | prevailing pre-training approaches for large language models involve several complexities. |
| Approach: | They propose a low-cost training recipe and a robust optimization approach to mitigate training instability . they also propose synthesis, curriculum, and data selection pipelines to integrate data . |
| Outcome: | The proposed model achieves top-tier performance among models with similar parameter scale . it is comparable to industry-leading models that require significantly more data . |
Pushing the Frontiers of Scientific Fact-Checking: The SCINLP Dataset (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly being used to understand how scientific research evolves, drawing growing interest from the research community. |
| Approach: | They propose a scientific fact-checking dataset, SCINLP, tailored to the NLP domain that verifies the veracity of scientific research questions across varying rationale contexts. |
| Outcome: | The proposed framework examines scientific claims and research focus from a curated collection of influential and reputable NLP papers published between 2000 and 2024. |
Synthetic Pre-Training Tasks for Neural Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | toxicity and bias can be addressed by pre-training with synthetic resources . BLEU scores are used to compare methods with real-world data . |
| Approach: | They propose several ways to generate obfuscated data from large parallel corpus and concatenating phrase pairs from small word-aligned corpus with synthetic parallel data without real human language corpora. |
| Outcome: | The proposed methods can be used to generate obfuscated data or synthetic parallel data without real human language corpora even with high levels of oblication. |
Data, Data Everywhere: A Guide for Pretraining Dataset Construction (2024.emnlp-main)
Copied to clipboard
Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Bo Liu, Aastha Jhunjhunwala, Zhilin Wang, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
| Challenge: | Recent language models have impressive capabilities on a number of evaluation areas. |
| Approach: | They conduct systematic analysis of pretraining set construction to identify which methods yield the greatest gains in model accuracy. |
| Outcome: | The proposed method can be used to refine and improve a pretraining set. |
Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that powerful pre-trained language models can be fooled by small perturbations or intentional attacks. |
| Approach: | They propose a framework for fine-tuning PLMs using a masked language model and Gaussian noise to augment semantically relevant examples with sufficient diversity. |
| Outcome: | The proposed framework improves the robustness of pre-trained language models and alleviates performance degradation under adversarial attacks. |
Probing Language Models for Pre-training Data Detection (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive capabilities, while raising concerns about the data contamination due to privacy issues and leakage of benchmark datasets in the pre-training phase. |
| Approach: | They propose to utilize the probing technique to examine the model’s internal activations to detect pre-training data contamination by examining the model's internal activates. |
| Outcome: | The proposed method outperforms baselines and achieves state-of-the-art performance on both WikiMIA and ArxivMIA, with additional experiments confirming its efficacy. |
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature (2025.emnlp-main)
Copied to clipboard
David Wadden, Kejian Shi, Jacob Morrison, Alan Li, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, Arman Cohan
| Challenge: | ScIRIFF is the only entirely expert-written instruction-following dataset for scientific literature understanding . it features complex instructions with long input contexts, detailed task descriptions, and structured outputs. |
| Approach: | They present a dataset of 137K instruction-following instances for training and evaluation . they finetuned large language models using a mix of general domain and ScIRIFF instructions . |
| Outcome: | The proposed dataset shows that on nine out-of-distribution held-out tasks, the model performs better than baselines trained on general domain instructions. |
Is a Document Educational or Just Wikipedia-Style? — Pitfalls of Classifier-Based Quality Filtering (2026.acl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are pre-trained on massive data corpora, and the quality of these corporales is one of the main factors in achieving stateof-the-art performance. |
| Approach: | They propose to use Wikipedia-style reformatting to alter a model's quality assessment and enable low-quality content to surpass filtering thresholds. |
| Outcome: | The proposed model would reverse filtering decision for approximately 7% of evaluated documents, thereby admitting content into the pre-training corpus that would otherwise have been excluded. |
ACLSum: A New Dataset for Aspect-based Summarization of Scientific Publications (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing statistical phrasal or hierarchical machine translation systems relies on a large set of translation rules which results in engineering challenges. |
| Approach: | They propose to use factorized grammar from the field of linguistics as more general translation rules from XTAG English Grammar to generate a manually crafted summarization dataset. |
| Outcome: | The proposed method outperforms existing methods on low-resource language translation tasks with less training data. |
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)
Copied to clipboard
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro
| Challenge: | Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work . |
| Approach: | They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data . |
| Outcome: | The proposed models can achieve competitive or better performance than BERT under comparable conditions. |