Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models are trained on text data downsampled from massive text corpora like Common Crawl. |
| Approach: | They propose an efficient and scalable system that can make petabyte-level text corpora searchable by using the FM-index data structure. |
| Outcome: | The proposed system indexes 83TB of Internet text in 99 days with a single 128-core CPU node (or 19 hours if using 137 such nodes). |
Similar Papers
ParCourE: A Parallel Corpus Explorer for a Massively Multilingual Corpus (2021.acl-demo)
Copied to clipboard
| Challenge: | 7000 languages worldwide are spoken, but most research is focused on English . multilinguality is essential for multilingual research, and is a key component of the process. |
| Approach: | They propose a wordaligned parallel corpus that can be browsed using an online tool . they use the word alignment tools SimAlign and BabelNet to find the alignments . |
| Outcome: | The proposed tool can be set up for any parallel corpus and explores its quality and properties. |
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts (2021.acl-tutorials)
Copied to clipboard
| Challenge: | . - (EN) |
| Approach: | . - (EN) |
| Outcome: | . - (EN) |
How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)
Copied to clipboard
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran
| Challenge: | generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones . |
| Approach: | They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora . |
| Outcome: | The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size . |
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track) (2023.acl-industry)
Copied to clipboard
| Challenge: | . - (EN) |
| Approach: | . - (EN) |
| Outcome: | . - (EN) |
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track) (2026.acl-industry)
Copied to clipboard
| Challenge: | 153 papers were selected for the ACL 2026 Industry Track . Topic-wise, large language models were once again front and center of almost all submissions . |
| Approach: | 153 papers selected for the ACL 2026 Industry Track . 61 oral talks and 92 posters will be presented . keynote speaker Roberto Navigli will share his insights . |
| Outcome: | the industry track attracted an unprecedented number of submissions . 153 papers were selected for the ACL 2026 Industry Track . |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
Exploring the Best Practices of Query Expansion with Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are foundational in language technologies, particularly in information retrieval (IR). |
| Approach: | They propose a framework that leverages large language models for query expansion . they use LLMs to generate multiple pseudo-references and integrate them with original queries . |
| Outcome: | The proposed framework enhances sparse and dense retrieval methods without pre-indexing. |
Koala: An Index for Quantifying Overlaps with Pre-training Corpora (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can be influenced by the frequency of overlap between pre-training corpora. |
| Approach: | They propose to search over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
| Outcome: | Koala is a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. |
YuLan-Mini: Pushing the Limits of Open Data-efficient Language Model (2025.acl-long)
Copied to clipboard
Hu Yiwen, Huatong Song, Jie Chen, Jia Deng, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Yang Lu, Xu Miao, Xin Zhao, Ji-Rong Wen
| Challenge: | prevailing pre-training approaches for large language models involve several complexities. |
| Approach: | They propose a low-cost training recipe and a robust optimization approach to mitigate training instability . they also propose synthesis, curriculum, and data selection pipelines to integrate data . |
| Outcome: | The proposed model achieves top-tier performance among models with similar parameter scale . it is comparable to industry-leading models that require significantly more data . |
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 5: Tutorial Abstracts) (2025.acl-tutorials)
Copied to clipboard
| Challenge: | ACL 2025 tutorial sessions are a cornerstone event of the conference . 76 tutorial submissions were received this year, many of which were very engaging . |
| Approach: | 76 tutorial submissions were received this year for the tutorial session at ACL 2025 . the tutorials are designed to equip you with the latest insights, tools, and methodologies . |
| Outcome: | the tutorial sessions at ACL 2025 will be held in london on november 8 . the conference received 76 tutorial submissions this year . |