Papers by Benjamin Wang
arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Literature review tables are essential for summarizing and comparing collections of scientific papers. |
| Approach: | They propose to generate a database of literature review tables from a pool of papers and to model retrieval noise via semantically related but out-of-scope distractor papers verified by human annotators. |
| Outcome: | The proposed method improves over strong baselines while the absolute scores remain modest, underscoring the task’s difficulty. |
QuestEval: Summarization Asks for Fact-based Evaluation (2021.emnlp-main)
Copied to clipboard
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, Patrick Gallinari
| Challenge: | Existing evaluation metrics for summarization evaluation are limited and do not correlate well with human judgments. |
| Approach: | They propose to extend existing evaluation metrics to include question answering models to assess whether a summary contains all relevant information in its source document. |
| Outcome: | The proposed framework significantly improves the correlation with human judgments over four evaluation dimensions. |
Mitigating Backdoor Poisoning Attacks through the Lens of Spurious Correlation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern NLP models are often trained over large untrustworthy datasets, raising the potential for a malicious adversary to compromise model behaviour. |
| Approach: | They propose to mitigate spurious correlations between textual triggers and classification labels by combining them with insertion-based attacks. |
| Outcome: | The proposed defence significantly reduces attack success rates across backdoor attacks and provides a near-perfect defence against insertion-based attacks. |
Evaluating and Modeling Attribution for Cross-Lingual Question Answering (2023.emnlp-main)
Copied to clipboard
Benjamin Muller, John Wieting, Jonathan Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Soares, Roee Aharoni, Jonathan Herzig, Xinyi Wang
| Challenge: | Open-retrieval question answering systems are lacking in attribution for cross-lingual question answering . open-research questions are available in 20 languages, but their raw generation often falls short in factuality . |
| Approach: | They are the first to study attribution for cross-lingual question answering . they collect data in 5 languages to assess the attribution level of a state-of-the-art QA system . |
| Outcome: | The proposed approach improves the attribution level of a state-of-the-art cross-lingual QA system. |
Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations (2022.coling-1)
Copied to clipboard
| Challenge: | Existing metrics to measure the performance of conversational AI assistants are difficult to establish due to their slow nature. |
| Approach: | They propose an automatic dialogue evaluation framework that performs goal segmentation and success prediction by adding multi-task learning heads. |
| Outcome: | The proposed model achieves on-par with human annotation compared to a gold annotation benchmark. |
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in diffusion models have enabled high-quality image generation . generating images with desired details requires proper prompts . |
| Approach: | They analyze syntactic and semantic characteristics of diffusion models and their prompts . they pinpoint specific hyperparameter values and prompt styles that can lead to model errors . |
| Outcome: | The first large-scale text-to-image prompt dataset totals 6.5TB . it contains 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. |
Jailbreak Distillation: Renewable Safety Benchmarking (2025.findings-emnlp)
Copied to clipboard
Jingyu Zhang, Ahmed Elgohary, Xiawei Wang, A S M Iftekhar, Ahmed Magooda, Benjamin Van Durme, Daniel Khashabi, Kyle Jackson
| Challenge: | a new framework for safety benchmarking is being developed for large language models . the framework "distills" jailbreak attacks into high-quality and easily-updatable safety benchmarks . |
| Approach: | They propose a framework that "distills" jailbreak attacks into high-quality safety benchmarks. |
| Outcome: | The proposed framework "distills" jailbreak attacks into high-quality safety benchmarks . it requires minimal human effort to rerun the pipeline and produce updated benchmarks. |
Measuring and Mitigating Name Biases in Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Neural machine translation systems exhibit problematic biases, such as stereotypical gender bias in occupation terms. |
| Approach: | They propose a method to reduce biases in person name translations by randomly switching entities during translation. |
| Outcome: | The proposed method eliminates the problem without any effect on translation quality. |
Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilities (2023.emnlp-industry)
Copied to clipboard
Fengjun Wang, Moran Beladev, Ofri Kleinfeld, Elina Frayerman, Tal Shachar, Eran Fainman, Karen Lastmann Assaraf, Sarai Mizrachi, Benjamin Wang
| Challenge: | In the digital age, large-scale online travel platforms face the challenge of extracting valuable insights from massive volumes of textual data. |
| Approach: | They propose a model that uses a bi-encoder transformer architecture to extract structured information from textual data. |
| Outcome: | The proposed model outperforms state-of-the-art models with 92.9% micro mAP and 75.8% macro mA score compared to baseline models . the proposed model can be used to find hotel facilities and hotel rooms based on positive reviews . |
TurkingBench: A Challenge Benchmark for Web Agents (2025.naacl-long)
Copied to clipboard
Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi
| Challenge: | TurkingBench is a benchmark consisting of tasks presented as web pages with textual instructions and multi-modal contexts. |
| Approach: | They propose to use HTML pages to perform various annotation tasks on crowdsourcing platforms. |
| Outcome: | The proposed model outperforms other models on the TurkingBench benchmark. |
Putting words into the system’s mouth: A targeted attack on neural machine translation using monolingual data poisoning (2021.findings-acl)
Copied to clipboard
Jun Wang, Chang Xu, Francisco Guzmán, Ahmed El-Kishky, Yuqing Tang, Benjamin Rubinstein, Trevor Cohn
| Challenge: | Neural machine translation systems are known to be vulnerable to adversarial test inputs, however, they are also vulnerable to training attacks. |
| Approach: | They propose a poisoning attack in which a malicious adversary inserts a small poisoned sample of monolingual text into a training set of a system trained using back-translation. |
| Outcome: | The proposed attack is based on two methods that can be used to craft poisoned examples. |
SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding (2026.findings-acl)
Copied to clipboard
Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, Wenhu Chen
| Challenge: | Existing benchmarks for agentic repository-level code understanding overlook long tail topics and rely on memorized knowledge. |
| Approach: | They propose a repository-level agentic code understanding benchmark that uses long-tail repositories with executable environments to enforce topical balance. |
| Outcome: | Empirically, a Qwen3-8B model trained with the proposed benchmark outperforms GPT-4o by 2.3 points. |
Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework . |
| Approach: | They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations. |
| Outcome: | The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size. |
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)
Copied to clipboard
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman
| Challenge: | State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text. |
| Approach: | They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling. |
| Outcome: | The proposed model can be used to train sentences on language modeling tasks. |
D2GCLF: Document-to-Graph Classifier for Legal Document Classification (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods learn latent representations for each document by considering the semantics and themes of the documents. |
| Approach: | They propose a document-to-graph classifier which extracts facts as relations between key participants in a law case and represents a legal document with four relation graphs. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on a real-world legal document dataset. |
On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)
Copied to clipboard
Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa
| Challenge: | Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models. |
| Approach: | They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. |
| Outcome: | The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task. |
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (2025.acl-long)
Copied to clipboard
Dongwei Jiang, Guoxuan Wang, Yining Lu, Andrew Wang, Jingyu Zhang, Chuyu Liu, Benjamin Van Durme, Daniel Khashabi
| Challenge: | RATIONALYST is a model for process-supervision of reasoning based on pretraining on rationale annotations extracted from unlabeled data. |
| Approach: | They propose a model for process-supervision of reasoning based on pre-training on rationale annotations extracted from unlabeled data. |
| Outcome: | RATIONALYST improves reasoning accuracy by 3.9% on representative reasoning benchmarks. |
Closing the Spatial Execution Gap in Digital Whiteboards via Verifiable Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models suffer from a fundamental Spatial Execution Gap, failing to translate visual semantics into precise, schema-valid coordinate operations in interactive environments. |
| Approach: | They propose a pipeline that leverages Group Relative Policy Optimization to enforce a strict Identify-Reason-Verify protocol and train on execution-verifiable rewards. |
| Outcome: | The proposed pipeline outperforms a state-of-the-art frontier model by 16.75% in operation accuracy. |
CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers? (2025.findings-emnlp)
Copied to clipboard
Jiefu Ou, William Gantt Walden, Kate Sanders, Zhengping Jiang, Kaiser Sun, Jeffrey Cheng, William Jurayj, Miriam Wanner, Shaobo Liang, Candice Morgan, Seunghoon Han, Weiqi Wang, Chandler May, Hannah Recknor, Daniel Khashabi, Benjamin Van Durme
| Challenge: | CLAIMCHECK is an annotated dataset of NeurIPS 2023 and 2024 submissions and reviews from OpenReview. |
| Approach: | They annotate NeurIPS 2023 and 2024 submissions and reviews for weaknesses and dispute them for fine-grained labels of validity, objectivity, and type of the identified weaknesses. |
| Outcome: | The proposed dataset is richly annotated by ML experts for weaknesses statements in the reviews and the claims that they dispute, as well as fine-grained labels of validity, objectivity, and type of the identified weaknesses. |
Foiling Training-Time Attacks on Neural Machine Translation Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Neural machine translation systems are vulnerable to backdoor attacks . successful backdoors can cause slander, hate speech, phishing, etc. attacks can target very short trigger phrases, which can be challenging to detect even when included verbatim in poisoned instances. |
| Approach: | They propose a method that exploits asymmetry between source and target sentences to detect outlier tokens. |
| Outcome: | The proposed method reduces the success of attacks by up to 89.0% while not affecting predictive accuracy. |
Backdoor Attacks on Multilingual Machine Translation (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that multilingual machine translation systems are vulnerable to backdoor attacks through data poisoning. |
| Approach: | They propose to investigate the security of multilingual machine translation systems by exposing poisoned data into low-resource languages to cause malicious translations. |
| Outcome: | The proposed method achieves an average of 20% success rate in attacking high-resource languages. |
LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error (2024.acl-long)
Copied to clipboard
| Challenge: | Existing work on tool-augmented LLMs focuses on the broad coverage of tools and the flexibility of adding new tools. |
| Approach: | They propose a biologically inspired method for tool-augmented LLMs that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. |
| Outcome: | The proposed method improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and outperforms GPT-4. |
SKGSum: Structured Knowledge-Guided Document Summarization (2024.findings-acl)
Copied to clipboard
Qiqi Wang, Ruofan Wang, Kaiqi Zhao, Robert Amor, Benjamin Liu, Jiamou Liu, Xianda Zheng, Zijian Huang
| Challenge: | Existing summarization methods ignore the importance of summary structure, resulting in summaries that emphasize the most prominent information while omitting essential details from other sections. |
| Approach: | They propose a method that uses automatically extracted summary points to generate summaries. |
| Outcome: | The proposed methods improve quality and BERTScore of summaries and broaden the types of documents that can be effectively summarized. |
Mitigating Data Poisoning in Text Classification with Differential Privacy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data poisoning attacks can plant a backdoor in a model by injecting poisoned examples into training data, causing the model to misclassify test instances which include a specific pattern. |
| Approach: | They propose a generic defence mechanism that makes training robust to poisoning attacks by smoothing the gradient from each training example. |
| Outcome: | The proposed method is highly effective in mitigating, or even eliminating, poisoning attacks on text classification, with only a small cost in predictive accuracy. |
SemStamp: A Semantic Watermark with Paraphrastic Robustness for Text Generation (2024.naacl-long)
Copied to clipboard
Abe Hou, Jingyu Zhang, Tianxing He, Yichen Wang, Yung-Sung Chuang, Hongwei Wang, Lingfeng Shen, Benjamin Van Durme, Daniel Khashabi, Yulia Tsvetkov
| Challenge: | Existing watermarked generation algorithms employ token-level designs and are vulnerable to paraphrase attacks. |
| Approach: | They propose a sentence-level watermarking algorithm that uses locality-sensitive hashing to partition the semantic space of sentences. |
| Outcome: | The proposed algorithm is more robust than the existing state-of-the-art method on paraphrasers and domains, while posing only minor degradations to SemStamp. |
As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Mistranslated numbers can cause financial loss or medical misinformation. |
| Approach: | They propose a method to assess the robustness of neural machine translation systems to numerical text via behavioural testing. |
| Outcome: | The proposed method systematically assesses four fundamental capabilities of neural machine translation systems in translation numbers by virtue of a variety of test cases. |