Papers by Jimmy Lin

The Art of Abstention: Selective Prediction and Error Regularization for Natural Language Processing (2021.acl-long)

Challenge:	Currently, practitioners working on dense retrieval face a bewildering number of choices.
Approach:	They propose a framework for thinking about retrieval in terms of nearest-neighbor search over vector representations where these representations can be dense (typically called embeddings, generated from transformers) or flat (with brute-force search)
Outcome:	The proposed model explicates tradeoffs between HNSW and flat indexes from the perspectives of indexing time, query evaluation performance, and retrieval quality.

Copied to clipboard

Ji Xin, Raphael Tang, Yaoliang Yu, Jimmy Lin

Information Extraction & Text Mining

Challenge:	Pre-trained language models have improved the state-of-the-art results on many NLP applications.
Approach:	They propose a simple error regularization trick that improves confidence estimation without substantially increasing the computation budget.
Outcome:	The proposed regularization improves confidence estimation without increasing computation budget.

Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2 (2021.acl-srw)

Copied to clipboard

He Bai, Peng Shi, Jimmy Lin, Luchen Tan, Kun Xiong, Wen Gao, Jie Liu, Ming Li

Farewell Freebase: Migrating the SimpleQuestions Dataset to DBpedia (C18-1)

Challenge:	Experimental results show that pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage.
Approach:	They conduct experiments on an English essay dataset using Chinese-GPT2 . they find that the model can generate better continuations by learning to generate the in the fine-tuning stage.
Outcome:	The pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage.

Copied to clipboard

Michael Azmy, Peng Shi, Jimmy Lin, Ihab Ilyas

UniRAG: Universal Retrieval Augmentation for Large Vision Language Models (2025.findings-naacl)

Challenge:	Existing datasets for question answering over knowledge graphs lack answer triples from Freebase . a defunct knowledge graph makes it difficult to build "real-world" question answering systems .
Approach:	They propose a benchmark dataset for simple question answering over knowledge graphs that maps SimpleQuestions entities and predicates from Freebase to DBpedia.
Outcome:	The proposed dataset provides simple yet strong baselines with and without neural networks.

Copied to clipboard

Sahel Sharifymoghaddam, Shivani Upadhyay, Wenhu Chen, Jimmy Lin

Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards (2025.emnlp-industry)

Challenge:	Large Vision Language Models (LVLMs) have unlocked many complex use cases that require Multi-Modal (MM) understanding and MM generation.
Approach:	They propose a plug-and-play technique that adds relevant retrieved information to prompts as few-shot examples during inference.
Outcome:	The proposed method significantly improves the output quality of large vision language models when input prompts are augmented with relevant information retrieved by Vision-Language retrievers like UniRAG.

Copied to clipboard

Manveer Singh Tamber, Forrest Sheng Bao, Chenyu Xu, Ge Luo, Suleman Kazi, Minseok Bae, Miaoran Li, Ofer Mendelevitch, Renyi Qu, Jimmy Lin

Reasoning

Challenge:	Large language models (LLMs) excel in various tasks, but often produce hallucinations . retrieved contexts, misrepresent information, or generate outright contradictions .
Approach:	They propose a framework that measures hallucination faithfulness of large language models . they introduce a leaderboard that leverages diverse human-annotated hallucinian examples .
Outcome:	The proposed framework improves hallucination evaluations by leveraging human-annotated examples.

SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale (2022.emnlp-industry)

Copied to clipboard

Raphael Tang, Karun Kumar, Gefei Yang, Akshat Pandey, Yajie Mao, Vladislav Belyaev, Madhuri Emmadi, Craig Murray, Ferhan Ture, Jimmy Lin

Applying BERT to Document Retrieval with Birch (D19-3)

Challenge:	End-to-end automatic speech recognition systems require thousands of hours of manual annotation and heavyweight computation to perform inference.
Approach:	They propose to use a third-party ASR system as a weak supervision source and labeling functions derived from implicit user feedback to reduce human labor.
Outcome:	The proposed system improves word-error rate and speed up 600% over third-party ASR.

Copied to clipboard

Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, Jimmy Lin

Don’t Change Me! User-Controllable Selective Paraphrase Generation (2021.eacl-main)

Challenge:	Birch is an open-source document retrieval system that integrates with the Anserini information retrieval toolkit to demonstrate end-to-end search over large document collections.
Approach:	They propose to integrate Anserini with a BERT-based document ranking model that provides an end-to-end open-source search engine.
Outcome:	The proposed system outperforms existing approaches to document retrieval and question answering on standard newswire and social media test collections.

Copied to clipboard

Mohan Zhang, Luchen Tan, Zihang Fu, Kun Xiong, Jimmy Lin, Ming Li, Zhengkai Tu

Text Generation

Challenge:	a new technique allows paraphrase generation to be user-controlled . a user looking for cheap hotels in New York would not find the other answer helpful .
Approach:	They propose a method that provides a user with explicit tags that can be placed around any arbitrary segment of text to mean "don't change me!" they propose allowing user-controllable paraphrase generation by fine-tuning model that exhibits this behavior .
Outcome:	The proposed technique is language agnostic and tested in English and Chinese.

Bridging the Gap between Relevance Matching and Semantic Matching for Short Text Similarity Modeling (D19-1)

Copied to clipboard

Jinfeng Rao, Linqing Liu, Yi Tay, Wei Yang, Peng Shi, Jimmy Lin

VISA: Retrieval Augmented Generation with Visual Source Attribution (2025.acl-long)

Challenge:	Existing techniques for relevance and semantic matching cannot be easily adapted to the other.
Approach:	They propose a model that incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness.
Outcome:	The proposed model incorporates a hybrid encoder module, a relevance matching module and co-attention mechanisms that capture context-aware semantic relatedness.

Copied to clipboard

Xueguang Ma, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Wenhu Chen, Jimmy Lin

Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models (2024.naacl-long)

Challenge:	Existing approaches to retrieval-augmented generation primarily link generated content to document-level references, making it difficult for users to locate evidence among multiple content-rich retrieved documents.
Approach:	They propose a novel approach that combines answer generation with visual source attribution by leveraging large vision-language models to identify evidence and highlight exact regions that support the generated answers with bounding boxes in the retrieved document screenshots.
Outcome:	The proposed approach identifies evidence and highlights exact regions that support the generated answers with bounding boxes in the retrieved document screenshots.

Copied to clipboard

Raphael Tang, Crystina Zhang, Xueguang Ma, Jimmy Lin, Ferhan Ture

Cross-Lingual Training of Neural Models for Document Ranking (2020.findings-emnlp)

Challenge:	Large language models exhibit positional bias in how they use context, which affects listwise ranking.
Approach:	They propose a method to marginalize out different list orders in the prompt to produce an order-independent ranking with less positional bias.
Outcome:	The proposed method improves on five datasets in sorting and passage reranking by 34-52% . it marginalizes out different list orders in the prompt to produce an order-independent ranking .

Copied to clipboard

Peng Shi, He Bai, Jimmy Lin

Two Birds, One Stone: A Simple, Unified Model for Text Generation from Structured and Unstructured Data (2020.acl-main)

Challenge:	a recent study shows that multi-lingual BERT models can be used for document ranking in non-English languages . a blog post by Google suggests that the company is exploring this approach to improve web search across a number of languages.
Approach:	They propose to leverage relevance judgments in English to train neural document ranking models for mono-lingual retrieval in multiple target languages.
Outcome:	The proposed approach improves search quality in non-English languages while requiring low resources.

Copied to clipboard

Hamidreza Shahidi, Ming Li, Jimmy Lin

Text Generation

Challenge:	Recent studies have shown that simpler, properly tuned models are at least competitive across NLP tasks.
Approach:	They propose to use a table-to-text and neural question generation tasks to generate text from structured and unstructured data.
Outcome:	The proposed task generates biographies based on Wikipedia infoboxes . the proposed model can achieve the state of the art in both tasks .

Cross-lingual Text-to-SQL Semantic Parsing with Representation Mixup (2022.findings-emnlp)

Copied to clipboard

Peng Shi, Linfeng Song, Lifeng Jin, Haitao Mi, He Bai, Jimmy Lin, Dong Yu

Rerank Before You Reason: Analyzing Reranking Tradeoffs through Effective Token Cost in Deep Search Agents (2026.findings-acl)

Challenge:	Experimental results show that Rex can benefit from cross-lingual training and improve the effectiveness of semantic parsers.
Approach:	They propose a Representation Mixup Framework for effectively exploiting translations in the cross-lingual Text-to-SQL task.
Outcome:	The proposed framework can benefit from cross-lingual training and improve the effectiveness of semantic parsers, achieving state-of-the-art performance.

Copied to clipboard

Sahel Sharifymoghaddam, Jimmy Lin

Reasoning

Challenge:	Recent work emphasizes improving efficiency in LLM-based systems, especially for longcontext and multi-step reasoning.
Approach:	They analyze the role of listwise reranking in deep search pipelines and compare their results to a novel ETC metric to determine model scale and reasoning effort.
Outcome:	The proposed model scale, reasoning effort, reranking depth, and total token cost (ETC) metric improve retrieval and end-to-end accuracy and moderate reranked agents achieve comparable accuracy at substantially lower cost.

Natural Language Generation for Effective Knowledge Distillation (D19-61)

Copied to clipboard

Raphael Tang, Yao Lu, Jimmy Lin

Better Quality Pre-training Data and T5 Models for African Languages (2023.emnlp-main)

Challenge:	Knowledge distillation can transfer knowledge from deep language representation models to shallow word embedding-based neural networks.
Approach:	They propose to build an unlabeled transfer dataset to enable effective knowledge transfer . they hypothesize that this principled, general approach outperforms rule-based techniques .
Outcome:	The proposed method outperforms OpenAI GPT on four datasets in sentiment classification, sentence similarity, and linguistic acceptability.

Copied to clipboard

Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, Jimmy Lin

rosaOS: Agentic Operating System for Embodied LLMs (2026.acl-demo)

Challenge:	Existing web crawls have demonstrated quality issues for low-resource languages . Existing pretraining corpora have numerous quality issues .
Approach:	They propose to audit existing pretraining corpora to understand and rectify quality issues . they pretrain a new T5-based model and evaluate its performance on multiple tasks .
Outcome:	The proposed model outperforms existing pretrained models on four NLP tasks.

Copied to clipboard

Yijun Ge, Kushaldeep Mujral, Karthik Nambiar, Jimmy Lin

PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024.emnlp-main)

Challenge:	Existing LLM–robotic systems are tightly intertwined, making it difficult to switch hardware, add extra capabilities, or expand to multiple devices without bespoke integration.
Approach:	They propose an open-source agentic operating system for embodied LLMs . rosaOS integrates agentic tool-calling and ROS for robot interactions .
Outcome:	The proposed system integrates with the Reachy Mini robot and supports a multi-device setup with a quadruped robot, wheeled mobile robot, and smart lamp.

Copied to clipboard

Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, Guido Zuccon

Rethinking Complex Neural Network Architectures for Document Classification (N19-1)

Challenge:	Large language models (LLMs) excel in zero-shot document ranking tasks.
Approach:	They propose a prompt-based re-ranking method that requires no further training but is only feasible for reranking a handful of candidates due to computational costs.
Outcome:	The proposed method can retrieve documents from the entire corpus without training and with a large amount of paired text data.

Copied to clipboard

Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy Lin

Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers (2023.findings-acl)

Challenge:	Neural network models for many NLP tasks have grown increasingly complex in recent years . authors of recent papers question the necessity of such architectures and find them quite effective .
Approach:	They propose to use regularization techniques borrowed from language modeling to improve model accuracy . they find that a simple biLSTM architecture with appropriate regularization yields competitive results .
Outcome:	a simple biLSTM model outperforms the state-of-the-art on four benchmark datasets . authors say that improvements are not real, but are attributed to mundane reasons .

Copied to clipboard

Ji Xin, Raphael Tang, Zhiying Jiang, Yaoliang Yu, Jimmy Lin

Green & Sustainable NLP

Challenge:	Natural language processing tasks rely on complex neural models . transformer-based models are typically slow to execute, making it a non-trivial challenge to apply them in real-world applications.
Approach:	They propose to consider an efficiency method as an operator applied on a model . they find that the commutativity and cumulativeness of efficiency operators are plausible .
Outcome:	The proposed method is commutative and cumulative, and the results are estimated by combining methods.

CELI: Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders. (2024.naacl-short)

Copied to clipboard

Crystina Zhang, Minghan Li, Jimmy Lin

Pretrained Transformers for Text Ranking: BERT and Beyond (2021.naacl-tutorials)

Challenge:	Existing cross-encoders do not capture all information into the [CLS] token . Xiong et al., 2021) find that the out-of-domain approach is less robust.
Approach:	They introduce a cross-encoder with late interaction that incorporates a late interaction layer into existing models.
Outcome:	The proposed method improves BEIR by 5% without compromising in-domain effectiveness or search latency.

Copied to clipboard

Andrew Yates, Rodrigo Nogueira, Jimmy Lin

“Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors (2023.findings-acl)

Challenge:	This tutorial provides an overview of text ranking using neural network architectures known as transformers.
Approach:	This tutorial provides an overview of text ranking with neural network architectures known as transformers.
Outcome:	This tutorial provides an overview of text ranking with neural network architectures known as transformers.

Copied to clipboard

Zhiying Jiang, Matthew Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, Jimmy Lin

Information Extraction & Text Mining

Challenge:	Text classification is one of the most fundamental tasks in natural language processing (NLP), but deep neural networks are data-hungry and expensive to train.
Approach:	They propose a non-parametric alternative to DNNs that uses a compressor like gzip and a k-nearest-neighbor classifier to achieve competitive results.
Outcome:	The proposed method outperforms BERT on all five OOD datasets and outperformed other methods on the few-shot setting.

AfriCLIRMatrix: Enabling Cross-Lingual Information Retrieval for African Languages (2022.emnlp-main)

Copied to clipboard

Odunayo Ogundepo, Xinyu Zhang, Shuo Sun, Kevin Duh, Jimmy Lin

Honkling: In-Browser Personalization for Ubiquitous Keyword Spotting (D19-3)

Challenge:	Existing datasets for cross-lingual information retrieval are limited in many languages, especially those spoken in Africa.
Approach:	They propose to build a test collection for cross-lingual information retrieval in 15 diverse African languages.
Outcome:	AfriCLIRMatrix contains 6 million queries in English and 23 million relevance judgments automatically mined from Wikipedia inter-language links, covering many more African languages than any existing information retrieval test collection.

Copied to clipboard

Jaejun Lee, Raphael Tang, Jimmy Lin

Challenge:	keyword spotting systems are used for simple commands recognition on devices . however, voice-enabled web applications are few and far between . a prominent drawback is that most of these systems perform speech recognition in the cloud .
Approach:	Honkling is a JavaScript-based keyword spotting system that can be deployed on user devices.
Outcome:	Honkling is a JavaScript-based keyword spotting system that can be deployed on user devices.

“Knowing When You Don’t Know”: A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation (2024.findings-emnlp)

Copied to clipboard

Nandan Thakur, Luiz Bonifacio, Crystina Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, Jimmy Lin

Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval (2024.naacl-long)

Challenge:	Prior work on RAG grounds Large Language Models to reduce factual hallucinations lacks a comprehensive evaluation of different language families.
Approach:	They propose a human-annotated dataset for evaluating LLM robustness in RAG . they find that most models struggle to balance the two capacities .
Outcome:	The proposed dataset includes both a non-relevant and a relevant subset.

Copied to clipboard

Nandan Thakur, Jianmo Ni, Gustavo Hernandez Abrego, John Wieting, Jimmy Lin, Daniel Cer

Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval (D19-1)

Challenge:	et al., 2020: performance of dense retrieval models in multilingual retrieval is limited due to uneven and scarce training data available across multiple languages.
Approach:	They propose a synthetic retrieval training dataset containing 33 languages for fine-tuning multilingual retrievers without human supervision.
Outcome:	The proposed model outperforms human-supervised retrieval models on three retrieval benchmarks.

Copied to clipboard

Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, Jimmy Lin

What the DAAM: Interpreting Stable Diffusion Using Cross Attention (2023.acl-long)

Challenge:	Existing test collections provide only document-level relevance judgments, and documents exceed the length that BERT was designed to handle.
Approach:	They propose to aggregate sentence-level evidence to rank news articles using BERT . they also leverage passage-level relevance judgments available in other domains to fine-tune BERT models that capture cross-domain notions of relevance.
Outcome:	The proposed model aggregates sentence-level evidence to rank documents on three standard test collections.

Copied to clipboard

Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, Ferhan Ture

Inserting Information Bottlenecks for Attribution in Transformers (2020.findings-emnlp)

Challenge:	a new text-image attribution analysis model for text-to-image generation is understudied due to ethical constraints . corporators have restricted the general public from using the models and their weights .
Approach:	They perform a text-image attribution analysis on Stable Diffusion, a recently open-sourced model.
Outcome:	The proposed method achieves a competitive 58.8-64.8 mIoU on noun segmentation and fair to good mean opinion scores on all parts of speech rated by humans . it also achieves good attribution quality on all part of speech, rated in humans - and the first to interpret large diffusion models from a visuolinguistic perspective.

Copied to clipboard

Zhiying Jiang, Raphael Tang, Ji Xin, Jimmy Lin

CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrieval (2023.acl-long)

Challenge:	Pretrained transformers are a popular approach for understanding features important for prediction.
Approach:	They apply information bottlenecks to analyze attribution of features for prediction on a black-box model.
Outcome:	The proposed method outperforms two competing methods in degradation tests on four datasets.

Copied to clipboard

Minghan Li, Sheng-Chieh Lin, Barlas Oguz, Asish Ghoshal, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen

Green & Sustainable NLP

Challenge:	Existing multi-vector retrieval methods are slower and require more space to store indices compared to their single-vektor counterparts.
Approach:	They propose a multi-vector retrieval method that uses dynamic lexical routing to route different token vectors to the predicted lexicals.
Outcome:	The proposed method achieves state-of-the-art performance on several benchmark datasets while being nearly 40 times faster than the current state-out-of the-art method.

What Part of the Neural Network Does This? Understanding LSTMs by Measuring and Dissecting Neurons (D19-1)

Copied to clipboard

Ji Xin, Jimmy Lin, Yaoliang Yu

MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages (2023.tacl-1)

Challenge:	Biological neural systems consist of a huge number of neurons, and can react to the environment in complicated ways.
Approach:	They propose a metric to quantify the sensitivity of neurons to each label and conduct experiments to prove it.
Outcome:	The proposed metric is based on a set of experiments that show that dropping an arbitrary neuron significantly degrades the accuracy of the model.

Copied to clipboard

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, Jimmy Lin

Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation (2024.emnlp-main)

Challenge:	MIRACL is a multilingual dataset for ad hoc retrieval across 18 languages that collectively encompass over three billion native speakers around the world.
Approach:	They have gathered over 726k high-quality relevance judgments for 78k queries over Wikipedia in these languages, where all annotations have been performed by native speakers hired by their team.
Outcome:	MIRACL covers languages that are typologically close as well as distant from 10 language families and 13 sub-families, associated with varying amounts of publicly available resources.

Copied to clipboard

Raphael Tang, Crystina Zhang, Lixinyu Xu, Yao Lu, Wenyan Li, Pontus Stenetorp, Jimmy Lin, Ferhan Ture

QuackIR: Retrieval in DuckDB and Other Relational Database Management Systems (2025.emnlp-industry)

Challenge:	Current diffusion models do not cover recent models, thus we curate three test sets for evaluation.
Approach:	They propose a human-calibrated measure of variability in a set of images bootstrapped from existing image-pair perceptual distances.
Outcome:	The proposed model outperforms nine baselines by 18 points in accuracy and matches graded human judgements 78% of the time.

Copied to clipboard

Yijun Ge, Zijian Chen, Jimmy Lin

Incorporating Contextual and Syntactic Structures Improves Semantic Similarity Modeling (D19-1)

Challenge:	Existing vector databases for RAG are needed for large language models, but there are no alternatives.
Approach:	They propose to leverage existing relational databases for retrieval-augmented generation . they use duckDB, SQLite, and PostgreSQL integrations to demonstrate their effectiveness .
Outcome:	The proposed approach is comparable to existing IR tools.

Copied to clipboard

Linqing Liu, Wei Yang, Jinfeng Rao, Raphael Tang, Jimmy Lin

Can’t Hide Behind the API: Stealing Black-Box Commercial Embedding Models (2025.findings-naacl)

Challenge:	Semantic similarity modeling is central to many NLP problems such as question answering.
Approach:	They propose a pairwise word interaction model with syntactic structure priors to explore their effectiveness.
Outcome:	Extensive evaluations on eight benchmark datasets show that incorporating structural information improves over strong baselines.

Copied to clipboard

Manveer Singh Tamber, Jasper Xian, Jimmy Lin

CNNs for NLP in the Browser: Client-Side Deployment and Visualization Opportunities (N18-5)

Challenge:	a new study examines the retrieval effectiveness of commercial embedding models . robert mcgahey: can commercial embeds be "stolen" using distillation techniques? he says stealing models can offer benefits to different actors, including reduced costs and security .
Approach:	They propose to "steal" embedding models by training thief models on text–embedding pairs . they replicate retrieval effectiveness of commercial embeddable models with a cost of under $300 .
Outcome:	The proposed methods replicate retrieval effectiveness of commercial embedding models with under $300 . authors suggest measures to mitigate risk of model theft.

Copied to clipboard

Yiyun Liang, Zhucheng Tu, Laetitia Huang, Jimmy Lin

Visual Data in NLP

Challenge:	a JavaScript implementation of a convolutional neural network performs feedforward inference completely in the browser.
Approach:	They propose a JavaScript implementation that performs feedforward inference completely in the browser.
Outcome:	The proposed model performs feedforward inference completely in the browser without server requests . the proposed model is useful for applications with stringent latency requirements or low connectivity .

Spacerini: Plug-and-play Search Engines with Pyserini and Hugging Face (2023.emnlp-demo)

Copied to clipboard

Christopher Akiki, Odunayo Ogundepo, Aleksandra Piktus, Xinyu Zhang, Akintunde Oladipo, Jimmy Lin, Martin Potthast

Evaluating Embedding APIs for Information Retrieval (2023.acl-industry)

Challenge:	a toolkit for reproducible information retrieval research is available for free.
Approach:	They present a tool that integrates Pyserini and Hugging Face to enable the seamless construction and deployment of interactive search engines.
Outcome:	The proposed tool makes state-of-the-art retrieval models more accessible to non-IR practitioners while minimizing deployment effort.

Copied to clipboard

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-hermelo, Mehdi Rezagholizadeh, Jimmy Lin

BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression (2021.eacl-main)

Challenge:	a growing number of language models are limiting their access to the community . we evaluate existing APIs for domain generalization and multilingual retrieval .
Approach:	They evaluate semantic embedding APIs in retrieval scenarios to assess their capabilities . they use BEIR and MIRACL to re-rank BM25 results using the APIs .
Outcome:	The proposed model is based on semantic embedding APIs that build vector representations of a given text.

Copied to clipboard

Ji Xin, Raphael Tang, Yaoliang Yu, Jimmy Lin

Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval (2023.tacl-1)

Challenge:	Existing methods to make exiting decisions are limited to classification tasks . large-scale pre-trained language models such as BERT have brought performance gain but at the cost of heavy computational burden.
Approach:	They propose a fine-tuning strategy and a learning-to-exit module to accelerate BERT inference . they propose to make trade-offs between model quality and efficiency by early exiting .
Outcome:	The proposed approach improves early exiting for BERT, with better trade-offs . it can be combined with other acceleration methods, and the proposed strategy can be applied to regression tasks.

Copied to clipboard

Sheng-Chieh Lin, Minghan Li, Jimmy Lin

Unifying Multimodal Retrieval via Document Screenshot Embedding (2024.emnlp-main)

Challenge:	Pre-trained language models have been successful in knowledge-intensive tasks, but recent research calls into question the robustness of these singlevector models.
Approach:	They propose to exploit knowledge in a pre-trained language model for dense passage retrieval by aggregating contextualized token embeddings into a dense vector.
Outcome:	The proposed model significantly improves the effectiveness of dense retrieval models on in-domain and zero-shot evaluations without introducing substantial training overhead.

Copied to clipboard

Xueguang Ma, Sheng-Chieh Lin, Minghan Li, Wenhu Chen, Jimmy Lin

Strong Baselines for Simple Question Answering over Knowledge Graphs with and without Neural Networks (N18-2)

Challenge:	Existing document retrieval pipelines require document parsing and content extraction to prepare input for indexing.
Approach:	They propose a retrieval paradigm that regards document screenshots as a unified input format . they leverage a large vision-language model to directly encode document screenshot into dense representations .
Outcome:	The proposed method outperforms existing retrieval pipelines in a text-intensive context.

Copied to clipboard

Salman Mohammed, Peng Shi, Jimmy Lin

Showing Your Work Doesn’t Always Work (2020.acl-main)

Challenge:	Existing work on simple question answering over knowledge graphs involves increasingly complex NN architectures.
Approach:	They propose to decompose the problem into entity detection, entity linking, relation prediction, evidence combination and heuristics.
Outcome:	The proposed approach outperforms existing models and benchmarks on a simple QA task.

Copied to clipboard

Raphael Tang, Jaejun Lee, Ji Xin, Xinyu Liu, Yaoliang Yu, Jimmy Lin

Challenge:	countless experimental papers lack empirical rigor, disregarding necessities such as statistical significance tests and computational environments.
Approach:	They propose to report the expected validation effectiveness of the best-tuned model with respect to the computational budget.
Outcome:	The proposed model favors negative errors and yields poor bootstrapped confidence intervals, the authors argue . they find that the proposed model is biased and uses error-prone assumptions .

Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering (2021.findings-emnlp)

Copied to clipboard

Minghan Li, Ming Li, Kun Xiong, Jimmy Lin

Low-Resource NLP

Challenge:	Existing approaches to multitask dense retrieval are not effective due to corpus inconsistency.
Approach:	They propose to train individual dense passage retrievers for different open-domain question-answering tasks and aggregate their predictions during test time.
Outcome:	The proposed method achieves state-of-the-art performance on 5 benchmark QA datasets, with up to 10% improvement in top-100 accuracy compared to a joint-training multi-task DPR on SQuAD.

Pay-Per-Request Deployment of Neural Network Models Using Serverless Architectures (N18-5)

Copied to clipboard

Zhucheng Tu, Mengping Li, Jimmy Lin

Simple Attention-Based Representation Learning for Ranking Short Social Media Posts (N19-1)

Challenge:	Using Amazon’s Lambda service for feedforward evaluation and DynamoDB for word embeddings, we demonstrate a serverless deployment of neural networks for NLP applications.
Approach:	They propose a pay-per-request pricing model for neural network deployment in NLP applications using Amazon’s Lambda service for feedforward evaluation and DynamoDB for storing word embeddings.
Outcome:	The proposed architecture is scalable and inexpensive.

Copied to clipboard

Peng Shi, Jinfeng Rao, Jimmy Lin

Representation Learning

Challenge:	Existing approaches to ranking short social media posts are complex and require different components to capture a multitude of relevance signals.
Approach:	They propose a word-level Siamese architecture with attention-based mechanisms for capturing semantic "soft" matches between query and post tokens.
Outcome:	The proposed model is faster and simpler than existing models and more efficient than existing approaches.

Scalable Knowledge Graph Construction from Text Collections (D19-66)

Copied to clipboard

Ryan Clancy, Ihab F. Ilyas, Jimmy Lin

Structured Data in NLP

Challenge:	Existing open-source solutions for analyzing unstructured text are lacking in the field of knowledge graph construction.
Approach:	They propose a scalable open-source platform that "distills" a text collection into a knowledge graph . they scale out the Stanford CoreNLP toolkit via Apache Spark integration .
Outcome:	The proposed platform scales out the Stanford CoreNLP toolkit via Apache Spark integration . it extracts mentions and relations from documents and then ingests them into a knowledge graph .

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference (2020.acl-main)

Copied to clipboard

Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, Jimmy Lin

XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing (2022.findings-emnlp)

Challenge:	Large-scale pre-trained language models such as BERT are notorious for being slow in both training and inference.
Approach:	They propose a method to accelerate BERT inference by inserting extra classification layers between each transformer layer of BERT.
Outcome:	The proposed method saves up to 40% inference time with minimal degradation in model quality.

Copied to clipboard

Peng Shi, Rui Zhang, He Bai, Jimmy Lin

How Does Generative Retrieval Scale to Millions of Passages? (2023.emnlp-main)

Challenge:	Existing work focuses on English datasets, and it is unclear whether large language models can serve as competitive semantic parsers for other languages.
Approach:	They propose a framework that learns to retrieve relevant English exemplars for a given query to construct prompts.
Outcome:	The proposed framework learns to retrieve relevant English exemplars for a given query to construct prompts.

Copied to clipboard

Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Tran

Precise Zero-Shot Dense Retrieval without Relevance Labels (2023.acl-long)

Challenge:	generative retrieval is a new paradigm for information retrieval, enabling a sequence-to-sequence model with a single Transformer . generative encoders have been used on small corpora, but only on large ones .
Approach:	They propose to encode an entire document corpus within a single Transformer . they find generative retrieval is competitive with state-of-the-art dual encoders on small corpora .
Outcome:	The proposed approach is competitive with state-of-the-art dual encoders on small corpora, the study finds . the proposed approach only evaluates on document corporales on the order of 100K in size .

Copied to clipboard

Luyu Gao, Xueguang Ma, Jimmy Lin, Jamie Callan

Unsupervised Chunking as Syntactic Structure Induction with a Knowledge-Transfer Approach (2021.findings-emnlp)

Challenge:	Existing dense retrieval systems that use semantic embedding similarities can be effective across tasks and languages.
Approach:	They propose to pivot through Hypothetical Document Embeddings (HyDE) given a query, HyDE first zero-shot prompts an instruction-following language model to generate a hypothetical document.
Outcome:	The proposed method significantly outperforms the state-of-the-art unsupervised dense retriever Contriever and shows strong performance comparable to fine-tuned retrievers across tasks and languages.

Copied to clipboard

Anup Anand Deshmukh, Qianqiu Zhang, Ming Li, Jimmy Lin, Lili Mou

Syntactic Text Processing

Challenge:	Existing methods for predicting linguistic structures require labeled data . unsupervised chunking is useful for understanding linguistic structure of human languages .
Approach:	They propose a knowledge-transfer approach that heuristically induces chunk labels from unsupervised parsing models and a hierarchical recurrent neural network (HRNN) they show that their approach bridges the gap between supervised and unsupervised chunking.
Outcome:	The proposed method bridges the gap between supervised and unsupervised chunking.

Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models (2020.coling-main)

Copied to clipboard

Jheng-Hong Yang, Sheng-Chieh Lin, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, Jimmy Lin

BrowseComp-Plus: A Fair and Disentangled Evaluation Benchmark for Deep Search Agents (2026.acl-long)

Challenge:	Existing approaches to extract implicit knowledge from pretrained models are still unclear.
Approach:	They propose to use a template-based approach to extract implicit knowledge for commonsense reasoning on multiple-choice questions.
Outcome:	The proposed template can be extended to other MC tasks with contexts such as supporting facts in open-book question answering settings.

Copied to clipboard

Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Sahel Sharifymoghaddam, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Hosna Oyarhoseini, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, Jimmy Lin

ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models (2024.emnlp-industry)

Challenge:	Existing benchmarks for deep search agents rely on blackbox web search APIs . dynamic and opaque web APIs hinder reproducibility and fair comparisons - authors .
Approach:	They propose a benchmark that employs a fixed corpus for controlled retrieval for deep search agents.
Outcome:	The new benchmark shows that agents that combine large language models with retrieval tools excel at complex, reasoning-intensive queries.

Copied to clipboard

Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeffrey Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, Yunyao Li

Contextualized Query Embeddings for Conversational Search (2021.emnlp-main)

Challenge:	Knowledge Graphs (KGs) are a powerful tool for capturing structured representations of the world.
Approach:	They propose a scalable method for generating up-to-date and configurable conversational KGQA datasets that adheres to human interaction configurations and operates at a significantly larger scale.
Outcome:	Qualitative psychometric analyses show that ConvKGYarn produces high-quality data comparable to popular conversational KGQA datasets across various metrics.

Copied to clipboard

Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin

Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages (2024.acl-short)

Challenge:	Existing approaches to conversational search use multiple inference pipelines that require long inference times . despite their effectiveness, such a pipeline often includes multiple neural models that require longer inference time.
Approach:	They propose to integrate conversational query reformulation directly into a dense retrieval model . they use a dataset with pseudo-relevance labels to overcome the lack of training data .
Outcome:	The proposed model rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets.

Copied to clipboard

Mofetoluwa Adeyemi, Akintunde Oladipo, Ronak Pradeep, Jimmy Lin

End-to-End Open-Domain Question Answering with BERTserini (N19-4)

Challenge:	Large language models (LLMs) have shown impressive zero-shot capabilities in various passage ranking tasks.
Approach:	They analyze and compare the effectiveness of monolingual reranking using query or document translations and evaluate the effectiveness when leveraging their own generated translations.
Outcome:	The proposed models perform better when using their own translations than when using query or document translations.

Copied to clipboard

Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, Jimmy Lin

Question Answering

Challenge:	a new open-domain question answering system integrates best practices from IR with a BERT-based reader to identify answers from a large corpus of Wikipedia articles.
Approach:	They propose an end-to-end question answering system that integrates BERT with an IR reader.
Outcome:	The proposed system improves on a standard benchmark test collection.

Evaluating Token-Level and Passage-Level Dense Retrieval Models for Math Information Retrieval (2022.findings-emnlp)

Copied to clipboard

Wei Zhong, Jheng-Hong Yang, Yuqing Xie, Jimmy Lin

DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025.acl-long)

Challenge:	a recent study has shown that dense retrieval methods are suboptimal for capturing contextual similarities in complex data.
Approach:	They propose to combine a structure search method and efficient bi-encoder dense retrieval models to capture contextual similarities.
Outcome:	The proposed model improves on token-level and passage-level dense retrieval tasks.

Copied to clipboard

Xueguang Ma, Xi Victoria Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih, Xilun Chen

Improving Precancerous Case Characterization via Transformer-based Ensemble Learning (2022.emnlp-industry)

Challenge:	Large language models (LLMs) have shown strong effectiveness and robustness when fine-tuned as dense retrievers.
Approach:	They propose a training framework that leverages pruned LLMs to train smaller generalizable dense retrievers.
Outcome:	The proposed training framework offers better multilingual and long-context capabilities than traditional encoder-based retrievers and achieves strong performance across multiple tasks and languages.

Copied to clipboard

Yizhen Zhong, Jiajie Xiao, Thomas Vetterli, Mahan Matin, Ellen Loo, Jimmy Lin, Richard Bourgon, Ofer Shapira

Aligning Cross-Lingual Entities with Multi-Aspect Information (D19-1)

Challenge:	Application of natural language processing (NLP) to cancer pathology reports has been focused on detecting cancer cases, ignoring precancerous cases.
Approach:	They developed transformer-based deep neural network NLP models to perform the CRC phenotyping with the goal of extracting precancerous lesion attributes and distinguishing cancer and precancirous cases.
Outcome:	The proposed model achieves 0.914 macro-F1 scores for classifying patients into negative, non-advanced adenoma, advanced adénoma and CRC.

Copied to clipboard

Hsiu-Wei Yang, Yanyan Zou, Peng Shi, Wei Lu, Jimmy Lin, Xu Sun

Document Ranking with a Pretrained Sequence-to-Sequence Model (2020.findings-emnlp)

Challenge:	Existing knowledge graphs that represent entities in different languages are not covered by existing systems.
Approach:	They propose two ways to embed entities from multilingual knowledge graphs into the same vector space, where equivalent entities are close to each other.
Outcome:	The proposed method significantly outperforms existing systems on two benchmark datasets.

Copied to clipboard

Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy Lin

Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval (2021.emnlp-main)

Challenge:	Experimental results on the MS MARCO passage ranking task show that our ranking approach is superior to strong encoder-only models.
Approach:	They propose to use a pretrained sequence-to-sequence model to generate relevance labels as "target tokens" they also show how the underlying logits of these target tokens can be interpreted as relevance probabilities for ranking.
Outcome:	The proposed model outperforms existing models in a data-poor setting and significantly outperformed an encoder-only model on the MS MARCO passage ranking task.

Copied to clipboard

Xueguang Ma, Minghan Li, Kai Sun, Ji Xin, Jimmy Lin

Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs (2025.findings-emnlp)

Challenge:	Dense passage retrieval improves ranking accuracy in open-domain question answering but at the cost of large space and memory requirements.
Approach:	They propose a simple unsupervised pipeline that includes principal component analysis (PCA), product quantization, and hybrid search to improve space efficiency.
Outcome:	The proposed pipeline achieves good accuracy–space trade-offs, for example, 48 compression with less than 3% drop in top-100 retrieval accuracy on average or 96 compression without drop in space requirements.

Copied to clipboard

Nandan Thakur, Crystina Zhang, Xueguang Ma, Jimmy Lin

Robustness in NLP

Challenge:	Using LLMs to identify false negatives improves retrieval and reranker models by 0.7-1.4 points on BEIR and by 1.7-1.8 points on AIR-Bench evaluation.
Approach:	They use a simple, cost-effective approach to identify and relabel false negatives in training datasets.
Outcome:	The proposed approach improves retrieval models by 0.7-1.4 points on BEIR and by 1.7-1.8 points on AIR-Bench evaluation.

GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration (2023.acl-demo)

Copied to clipboard

Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf, Stella Biderman, Martin Potthast, Jimmy Lin

Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features (N19-2)

Challenge:	Using the mature and well-tested methods from the domain of Information Retrieval (IR) we propose to integrate Pyserini with Hugging Face to provide qualitative analysis tools for NLP researchers.
Approach:	They propose to integrate Pyserini with Hugging Face to provide qualitative analysis tools for NLP researchers.
Outcome:	The proposed tools can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts.

Copied to clipboard

Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei, Jimmy Lin

Challenge:	e-commerce companies often have the option of escalating complaints by filing grievances with a government authority . this is detrimental to an ecommerce company, but this problem is challenging to solve by integrating recurrent neural networks with manually-engineered features.
Approach:	They propose a model that integrates recurrent neural networks with manually-engineered features to identify cases where the customer expresses such an intent.
Outcome:	The proposed model outperforms baseline models and provides better recall and triage for specialized agents.

AfroBench: How Good are Large Language Models on African Languages? (2025.findings-acl)

Copied to clipboard

Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, David Ifeoluwa Adelani

EWEK-QA : Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systems (2024.acl-long)

Challenge:	Large-scale multilingual evaluations often include only a handful of African languages due to the scarcity of high-quality data and the limited discoverability of existing datasets.
Approach:	They propose a multi-task benchmark to evaluate the performance of LLMs across 64 African languages, 15 tasks and 22 datasets.
Outcome:	The proposed benchmark compares LLMs across 64 African languages, 15 tasks and 22 datasets.

Copied to clipboard

Mohammad Dehghan, Mohammad Alomrani, Sunyam Bagga, David Alfonso-Hermelo, Khalil Bibi, Abbas Ghaddar, Yingxue Zhang, Xiaoguang Li, Jianye Hao, Qun Liu, Jimmy Lin, Boxing Chen, Prasanna Parthasarathi, Mahdi Biparva, Mehdi Rezagholizadeh

MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems (2025.naacl-long)

Challenge:	citation-based QA systems are suffering from two shortcomings . they usually rely only on web as a source of extracted knowledge and external knowledge sources can hamper the efficiency of the system.
Approach:	They propose to use a web-based knowledge graph retrieval solution to enrich extracted knowledge fed to a citation-based QA system.
Outcome:	The proposed model outperforms open-source state-of-the-art models in 7 quantitative and human evaluation tasks.

Copied to clipboard

Nandan Thakur, Suleman Kazi, Ge Luo, Jimmy Lin, Amin Ahmad

Zero-Shot ATC Coding with Large Language Models for Clinical Assessments (2025.naacl-industry)

Challenge:	Traditional retrieval-augmented generation benchmarks use heuristics as the ground truth for evaluation, but require an expensive large language model (LLM) as a judge for a reliable evaluation.
Approach:	They propose to use large language models as a judge for retrieval-augmented generation benchmarks . they use heuristic metrics as input and a large language model as heuriistic input .
Outcome:	The proposed method couples heuristic features with large language models as judge for evaluation.

Copied to clipboard

Zijian Chen, John-Michael Gamble, Jimmy Lin

Voice Query Auto Completion (2021.emnlp-main)

Challenge:	Manual assignment of ATC codes to prescription records is a significant bottleneck in healthcare operations.
Approach:	They propose a method to automate the assignment of ATC codes to prescription records . they use locally deployable large language models to guide LLMs through the ontology .
Outcome:	The proposed method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B.

Copied to clipboard

Raphael Tang, Karun Kumar, Kendra Chalkley, Ji Xin, Liming Zhang, Wenyan Li, Gefei Yang, Yajie Mao, Junho Shin, Geoffrey Craig Murray, Jimmy Lin

Tomato, Tomahto, Tomate: Do Multilingual Language Models Understand Based on Subword-Level Semantic Concepts? (2025.findings-naacl)

Challenge:	Existing methods fail to complete voice queries from incomplete prefixes because they use orthographic prefix and substrings instead of the true phonetic prefix.
Approach:	They propose to condition QAC approaches on intermediate transcriptions to complete voice queries.
Outcome:	The proposed method obtains an 18% relative improvement over previous methods on a speech-enabled smart television with real-life voice search traffic.

Copied to clipboard

Crystina Zhang, Jing Lu, Vinh Q. Tran, Tal Schuster, Donald Metzler, Jimmy Lin

Certified Error Control of Candidate Set Pruning for Two-Stage Relevance Ranking (2022.emnlp-main)

Challenge:	a recent study shows that human understanding of text depends on general semantic concepts of words that are robust to their superficial forms.
Approach:	They evaluate the accuracy of multilingual multilingual language models based on subword-level semantics . they form "semantic tokens" by merging semantically similar subwords and embeddings based upon the results .
Outcome:	The proposed models are able to make predictions on multilingual tasks with different tokenizers and model sizes.

Copied to clipboard

Minghan Li, Xinyu Zhang, Ji Xin, Hongyang Zhang, Jimmy Lin

How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval (2023.findings-emnlp)

Challenge:	In information retrieval, candidate set pruning is used to speed up two-stage relevance ranking but lacks accurate error control and empirical guarantees.
Approach:	They propose a method that guarantees the test error after pruning is controlled under a user-specified threshold with high probability.
Outcome:	The proposed method reduces the average set size from 1000 to 27, increasing reranking speed by about 37 times while keeping MRR@10 greater than a pre-specified value of 0.38 with about 90% empirical coverage.

Copied to clipboard

Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, Xilun Chen