Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, Anna Rogers
| Challenge: | a 1.6TB multilingual text corpus is currently the largest language model . large language models are ubiquitous in modern NLP, used directly to generate text and as building blocks in downstream applications. |
| Approach: | They propose a search engine for the 1.6TB multilingual ROOTS corpus offering both fuzzy and exact search capabilities. |
| Outcome: | The ROOTS Search Tool is an open-source search engine for the 1.6TB multilingual ROOTs corpus. |
Similar Papers
Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs (2026.acl-long)
Copied to clipboard
Yu Li, Xiaoran Shang, Qizhi Pei, Yun Zhu, Xin Gao, Honglin Lin, Zhanping Zhong, Zhuoshi Pan, Zheng Liu, Xiaoyang Wang, Conghui He, Dahua Lin, Feng Zhao, Lijun Wu
| Challenge: | High-quality post-training data is the primary engine driving LLM capabilities . datasets are often treated as isolated artifacts, overlooking their true developmental context . |
| Approach: | They propose a framework to reconstruct the evolutionary graph of dataset development using data lineage. |
| Outcome: | The proposed framework characterizes domain-specific structural patterns in Math-oriented datasets and general-domain corpora. |
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text (2024.findings-naacl)
Copied to clipboard
| Challenge: | Retrievalaugmented LLMs have been used to ground LLM in external knowledge . a gap exists in the current landscape regarding the effectiveness of grounding LLM on heterogeneous knowledge sources. |
| Approach: | They propose a model that uses symbolic language to generate symbolic queries . they use a dataset that is generated using predefined reasoning chains and human annotation . |
| Outcome: | The proposed model outperforms previous approaches by a significant margin in QA tasks over text. |
The DReaM Corpus: A Multilingual Annotated Corpus of Grammars for the World’s Languages (2020.lrec-1)
Copied to clipboard
| Challenge: | Until recently, language descriptions were available in paper form only, with indexes as the only search aid. |
| Approach: | They propose to digitize a multilingual corpus of language descriptions and annotate it with various meta, word, and text attributes to make searching and analysis easier and more useful. |
| Outcome: | The proposed corpus is searchable through a couple of well-established corpus infrastructures. |
Domain Regeneration: How well do LLMs match syntactic properties of text domains? (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent improvements in large language models have improved their ability to approximate distributions . authors find that LLMs can suffer from model collapse due to domain considerations based on pretraining . |
| Approach: | They use open source LLMs to regenerate permissively licensed English text from Wikipedia and news text. |
| Outcome: | The proposed model can faithfully match the human-generated distributions in a semantically-controlled setting. |
Low-Perplexity LLM-Generated Sequences and Where To Find Them (2025.acl-srw)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly applied across various domains, but the ways they leverage their training data during inference remains only partially understood. |
| Approach: | They propose a systematic approach that analyzes low-perplexity sequences and traces them back to their sources in the training data. |
| Outcome: | The proposed pipeline extracts low-perplexity sequences across diverse topics while avoiding degeneration, then trace them back to their sources in the training data. |
Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts (2025.emnlp-main)
Copied to clipboard
Yuho Lee, Jiaqi Deng, Nicole Hee-Yeon Kim, Hyangsuk Min, Taewon Yun, Minjeong Ban, Kim Yul, Hwanjun Song
| Challenge: | HAMLET is a framework for evaluating the long-context comprehension of large language models. |
| Approach: | They propose a framework for evaluating the long-context comprehension of large language models . HAMLET structures key information of source texts into a three-level hierarchy . |
| Outcome: | HAMLET achieves 90% agreement with expert judgments while reducing evaluation cost by up to 25. |
Large Language Models are Built-in Autoregressive Search Engines (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing dual-encoder dense retrievers obtain representations for questions and documents independently, allowing only shallow interactions between them. |
| Approach: | They propose to use large language models to generate URLs for document retrieval by following human instructions. |
| Outcome: | The proposed method achieves better retrieval performance than existing retrieval approaches on open-domain question answering benchmarks. |
Text Mining for History: first steps on building a large dataset (L18-1)
Copied to clipboard
| Challenge: | a new corpus on the history domain is being created to mine text in the domain . primary motivation for the project is the need to query the material in a non-linear way . |
| Approach: | They propose to use a Brazilian historical-biographical dictionary as a resource for text mining. |
| Outcome: | The proposed corpus is a reference work on the Brazilian history domain . it contains almost 12 millions tokens in about three hundred thousand sentences . the authors argue that the proposed corpu is linguistically motivated . |
LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Models (2025.acl-long)
Copied to clipboard
Zihan Zhou, Chong Li, Xinyi Chen, Shuo Wang, Yu Chao, Zhili Li, Haoyu Wang, Qi Shi, Zhixing Tan, Xu Han, Xiaodong Shi, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing studies have focused on extending the context length of large language models (LLMs) due to their quadratic computational complexity and a lack of high-quality long training examples, most LLMs are trained with a limited window size. |
| Approach: | They propose a training-free framework that enables large language models to effectively process long texts using a divide-and-conquer strategy for comprehensive document understanding. |
| Outcome: | The proposed framework outperforms open-source and commercial long-context LLMs and is compatible with several models. |
Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains (2026.findings-acl)
Copied to clipboard
Md. Faiyaz Abdullah Sayeedi, Subhey Sadi Rahman, Md. Mahbub Alam, Md. Adnanul Islam, Jannatul Ferdous Deepti, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda
| Challenge: | Large Language Models (LLMs) have redefined Machine Translation, enabling context-aware and fluent translations across hundreds of languages and textual domains. |
| Approach: | They propose a framework and dataset to evaluate the translation quality and fairness of open-source LLMs. |
| Outcome: | The proposed framework and dataset evaluates translation quality and fairness of open-source LLMs. |