Papers by Andrew Wang
Open-ended Knowledge Tracing for Computer Science Education (2022.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge tracing (KT) is a method used to estimate student mastery of concepts/skills/knowledge components from their responses to questions and to predict future performance. |
| Approach: | They propose a student knowledge-guided code generation approach that combines program synthesis methods with student knowledge tracing methods to solve the OKT problem. |
| Outcome: | The proposed method is based on a student knowledge-guided code generation approach and validates on coding questions. |
Learning Personalized Alignment for Evaluating Open-ended Text Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Traditional evaluation metrics rely heavily on lexical similarity with human-written references, showing poor correlation with human judgments and failing to account for alignment with the diversity of human preferences. |
| Approach: | They propose an interpretable evaluation framework that evaluates alignment with specific human preferences by providing detailed comments and fine-grained scoring. |
| Outcome: | The proposed framework outperforms GPT-4 in Kendall correlation and accuracy with zero-shot reviewers. |
Mitigating Gender Bias in Natural Language Processing: Literature Review (P19-1)
Copied to clipboard
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang
| Challenge: | NLP models propagate and may even amplify gender bias found in text corpora . methods to mitigate gender bias in NLP are relatively nascent . |
| Approach: | They propose to analyze gender bias based on four forms of representation bias and discuss the advantages and drawbacks of existing gender debiasing methods. |
| Outcome: | The proposed methods are based on four forms of representation bias and have advantages and drawbacks. |
Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP (2021.emnlp-main)
Copied to clipboard
| Challenge: | Meta-learning considers learning as an efficient learning process that can leverage its past experience to accurately solve new tasks. |
| Approach: | They propose to provide task distributions for meta-learning by considering self-supervised tasks automatically proposed from unlabeled text to enable large-scale meta- learning in NLP. |
| Outcome: | The proposed distributions show that human learning models perform better on the few-shot benchmark than previous methods. |
Defending LLMs against Jailbreaking Attacks via Backtranslation (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advancement in large language models (LLMs) has shown their extensive applications and transformative potential to reshape people's lives. |
| Approach: | They propose a method which uses backtranslation to infer an input prompt from an input input prompt and then run it again on the backtranslated prompt. |
| Outcome: | The proposed method outperforms baselines and has little impact on the generation quality for benign input prompts. |
AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies (2024.emnlp-main)
Copied to clipboard
Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Murari Tiyyala, Nicholas Andrews, Daniel Khashabi
| Challenge: | Analogical reasoning is an important part of human communication, says a new study . a benchmark to determine analogical reasoning ability in language models is needed . |
| Approach: | They propose to benchmark analogical reasoning ability in language models by collecting 340 analogies from human writings. |
| Outcome: | The proposed benchmark aims to determine analogical reasoning ability in language models. |
Interpretable Math Word Problem Solution Generation via Step-by-step Planning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to solving math word problems focus on obtaining the correct answer. |
| Approach: | They propose a step-by-step planning approach for intermediate solution generation that strategically plans the generation of the next solution step based on the MWP and the previous solution steps. |
| Outcome: | The proposed approach improves the accuracy and interpretability of the solution on automatic metrics and human evaluation. |
Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Model (2024.eacl-long)
Copied to clipboard
| Challenge: | Motivational Interviewing (MI) is a counselling technique used to guide people towards behaviour change. |
| Approach: | They propose a method for distilling reflections from a foundational language model into smaller models that can be owned and controlled. |
| Outcome: | The proposed method achieves 100% success rate on hold-out test set and 90% on the GPT-2 XL. |
Towards Understanding Gender Bias in Relation Extraction (2020.acl-main)
Copied to clipboard
Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang
| Challenge: | Existing bias mitigation techniques have a negative effect on NRE, a study finds . |
| Approach: | They create a dataset to analyze gender bias in relation extraction systems . they find that existing bias mitigation techniques have a negative effect on NRE . |
| Outcome: | The proposed dataset analyzes gender bias in relation extraction systems using a 10% human annotated test set. |
AlignBench: Benchmarking Chinese Alignment of Large Language Models (2024.acl-long)
Copied to clipboard
Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, Jie Tang
| Challenge: | Effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. |
| Approach: | They propose a multi-dimensional benchmark for evaluating LLMs’ alignment in Chinese with 8 main categories, 683 real-scenario rooted queries and corresponding human verified references. |
| Outcome: | The benchmark uses a human-in-the-loop data curation pipeline, 683 real-scenario rooted queries and human verified references. |
NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval (D18-1)
Copied to clipboard
| Challenge: | Existing neural IR models do not have a mechanism for treating expansion terms differently from the original query terms, making it difficult to combine them with existing PRF approaches. |
| Approach: | They propose an end-to-end neural PRF framework that can be used with existing neural IR models by embedding different neural models as building blocks. |
| Outcome: | Extensive experiments on two standard test collections confirm the effectiveness of the proposed framework in improving the performance of two state-of-the-art neural IR models. |
Can language models learn from explanations in context? (2022.findings-emnlp)
Copied to clipboard
Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, Felix Hill
| Challenge: | Language Models can adapt to a few in-context examples, but without training. |
| Approach: | They examine how explanations of few-shot examples can help Language Models (LMs) explanations can improve performance even without tuning, they find . |
| Outcome: | The proposed explanations outperform hand-tuned explanations on small validation sets. |
Model Surgery: Modulating LLM’s Behavior Via Simple Parameter Editing (2025.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for detoxification or preventing jailbreaking involve fine-tuning billions of parameters through gradient descent with substantial computational cost. |
| Approach: | They propose to use supervised fine-tuning and Reinforcement Learning from human feedback to modify LLMs' behavior by directly editing a small subset of parameters. |
| Outcome: | Experiments show that editing a small subset of parameters can modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreak, with only inference-level computational resources. |
Distributional Inclusion Vector Embedding for Unsupervised Hypernymy Detection (N18-1)
Copied to clipboard
| Challenge: | Existing unsupervised methods for learning hypernyms from unlabeled text are not scaled to large vocabularies or yield unacceptably poor accuracy. |
| Approach: | They propose an unsupervised method of hypernym discovery using word contexts . they use word2vec to embed word context distributions without supervision . |
| Outcome: | The proposed method provides double the precision and highest average performance on 11 datasets. |
The ART of LLM Refinement: Ask, Refine, and Trust (2024.naacl-long)
Copied to clipboard
Kumar Shridhar, Koustuv Sinha, Andrew Cohen, Tianlu Wang, Ping Yu, Ramakanth Pasunuru, Mrinmaya Sachan, Jason Weston, Asli Celikyilmaz
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations and self-improve? |
| Approach: | They propose a reasoning with a refinement strategy called *ART: Ask, Refine, and Trust* that asks necessary questions to decide when an LLM should refine its output and uses it to affirm or deny trust. |
| Outcome: | The proposed reasoning with a refinement strategy achieves a performance gain of +5 points over baselines on two multistep reasoning tasks. |
Entailment Tree Explanations via Iterative Retrieval-Generation Reasoner (2022.findings-naacl)
Copied to clipboard
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xiaokai Wei, Henghui Zhu, Xinchi Chen, Peng Xu, Zhiheng Huang, Andrew Arnold, Dan Roth
| Challenge: | Large language models have achieved high performance on various natural language benchmarks, but the explainability of their output remains elusive. |
| Approach: | They propose an architecture called iterative retrieval-generation reasoner that generates an entailment tree that explains a given hypothesis by using premises from C. |
| Outcome: | The proposed model outperforms existing benchmarks on premise retrieval and entailment tree generation with around 300% gain in overall correctness. |
Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to generate arithmetic math word problems are invalid or have unsatisfactory language quality. |
| Approach: | They propose a method for automatically generating arithmetic math word problems from equations and context. |
| Outcome: | The proposed approach improves language quality and mathematical validity on three real-world MWP datasets. |
FaithLM: Towards Faithful Explanations for Large Language Models (2026.eacl-long)
Copied to clipboard
Yu-Neng Chuang, Guanchu Wang, Chia-Yuan Chang, Ruixiang Tang, Shaochen Zhong, Fan Yang, Andrew Wen, Mengnan Du, Xuanting Cai, Vladimir Braverman, Xia Hu
| Challenge: | Large language models (LLMs) produce natural language explanations, but they lack faithfulness and do not reflect the evidence the model uses to decide. |
| Approach: | They propose a model-agnostic framework that evaluates and improves the faithfulness of LLM explanations without token masking or task-specific heuristics. |
| Outcome: | The proposed framework improves faithfulness of large language models without masking or heuristics. |
Deciphering Stereotypes in Pre-Trained Language Models (2023.emnlp-main)
Copied to clipboard
Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, Soroush Vosoughi
| Challenge: | Current approaches for examining stereotypes in PLMs require intricate human knowledge about these stereotypes and entail careful manual curation of examples. |
| Approach: | They propose a framework for examining stereotype-encoding behavior of PLMs using model probing and textual analyses. |
| Outcome: | The proposed approach can debiase PLMs without compromising their language modeling capabilities or performance. |
ChatHF: Collecting Rich Human Feedback from Real-time Conversations (2024.emnlp-demo)
Copied to clipboard
| Challenge: | We present an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Approach: | They propose an interactive framework for chatbot evaluation that integrates configurable annotation within a chat interface. |
| Outcome: | The proposed framework supports fine-grained error detection and human evaluation at the same time. |
AutoCT: Automating Interpretable Clinical Trial Prediction with LLM Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Clinical trials are expensive and time-consuming, and accurate trial prediction is key to advancing medical treatments. |
| Approach: | They propose a framework that combines reasoning capabilities of large language models with the explainability of classical machine learning to generate, evaluate, and refine tabular features without human input. |
| Outcome: | The proposed framework performs better than SOTA methods on clinical trial prediction tasks within a limited number of iterations. |
kNN-LM Does Not Improve Open-ended Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Interpolation-based retrieval-augmented language models (LMs) are a subtype of retrieval augmented language model that computes the probability of the next token by interpolating between the softmax distribution of the original LM and a token distribution formed by retrieving over an external datastore. |
| Approach: | They propose to interpolate the predicted distribution of the next word with a distribution formed from the most relevant retrievals for a given prefix. |
| Outcome: | The proposed methods do not exhibit improvements in open-ended generation quality, as measured by automatic evaluation metrics and human evaluations. |
Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance (2026.eacl-long)
Copied to clipboard
| Challenge: | LISTEN is a controlled benchmark to disentangle lexical reliance from acoustic sensitivity in emotion understanding. |
| Approach: | They propose a benchmark to disentangle lexical reliance from acoustic sensitivity in emotion understanding. |
| Outcome: | LISTEN shows that current LALMs largely "transcribe" rather than "listen" authors note that models underutilize acoustic cues while relying on lexical semantics . |
Can Authorship Representation Learning Capture Stylistic Features? (2023.tacl-1)
Copied to clipboard
Andrew Wang, Cristina Aggazzotti, Rebecca Kotula, Rafael Rivera Soto, Marcus Bishop, Nicholas Andrews
| Challenge: | Existing methods to disentangle an author's style from the content of their writing are limited by the reliance on human labels and the narrow focus of stylistic distinctions. |
| Approach: | They propose to use a surrogate task to learn authorship representations that are sensitive to writing style and to validate their hypothesis . |
| Outcome: | The proposed representations are sensitive to writing style and may be robust to topic drift over time. |
RATIONALYST: Pre-training Process-Supervision for Improving Reasoning (2025.acl-long)
Copied to clipboard
Dongwei Jiang, Guoxuan Wang, Yining Lu, Andrew Wang, Jingyu Zhang, Chuyu Liu, Benjamin Van Durme, Daniel Khashabi
| Challenge: | RATIONALYST is a model for process-supervision of reasoning based on pretraining on rationale annotations extracted from unlabeled data. |
| Approach: | They propose a model for process-supervision of reasoning based on pre-training on rationale annotations extracted from unlabeled data. |
| Outcome: | RATIONALYST improves reasoning accuracy by 3.9% on representative reasoning benchmarks. |
Boosting LLM Agents with Recursive Contemplation for Effective Deception Handling (2024.findings-acl)
Copied to clipboard
Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, Gao Huang
| Challenge: | Recent advances in large language models (LLMs) have led to significant success in using LLMs as agents. |
| Approach: | They propose a cognitive framework that incorporates first-order and second-order perspective transitions into LLMs to enhance their ability to identify and counteract deceptive information. |
| Outcome: | The proposed framework enhances LLMs’ ability to identify and counteract deceptive information without extra fine-tuning and data. |
DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization (2022.acl-short)
Copied to clipboard
Zheng Li, Zijian Wang, Ming Tan, Ramesh Nallapati, Parminder Bhatia, Andrew Arnold, Bing Xiang, Dan Roth
| Challenge: | Empirical analyses show that pre-trained sequence-to-sequence models can achieve a 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts. |
| Approach: | They propose to distill and quantize pre-trained sequence-to-sequence models to reduce memory and latency requirements. |
| Outcome: | Empirical results show that the proposed model achieves 16.5x model footprint compression ratio with little performance drop relative to full-precision counterparts on multiple summarization and QA datasets. |
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)
Copied to clipboard
Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang
| Challenge: | Existing models for NLP evaluations lack the ability to generate informative critiques in pointwise grading and pairwise comparison especially without references. |
| Approach: | They propose a method which can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings. |
| Outcome: | The proposed method outperforms all open-source models and even GPT-4 in system-level correlations of pointwise grading. |
Hindsight: Structured Agent Memory that Retains, Recalls, and Reflects (2026.acl-demo)
Copied to clipboard
Christopher Latimer, Nicolò Boschi, Andrew Neeser, Chris Bartholomew, Gaurav Srivastava, Xuan Wang, Naren Ramakrishnan
| Challenge: | Hindsight organizes long-term memory into four logical networks and exposes three core operations. |
| Approach: | Hindsight organizes long-term memory into four logical networks and exposes three core operations. |
| Outcome: | Hindsight is a working memory system for AI agents that separates facts from beliefs . the system outperforms existing models on LongMemEval and LoCoMo with 83.6% accuracy . |
PaRaDe: Passage Ranking using Demonstrations with LLMs (2023.findings-emnlp)
Copied to clipboard
Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuanhui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, Kai Hui
| Challenge: | Existing studies show that large language models can be instructed to perform zero-shot passage re-ranking . Existing work like UPR demonstrate promising results for zero- shot ranking using LLMs . |
| Approach: | They propose a demonstration selection strategy based on difficulty rather than semantic similarity . they propose to include only one demonstration in the prompt to improve re-ranking . |
| Outcome: | The proposed method improves LLM-based re-ranking by adding one demonstration to the prompt. |
Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models (2024.lrec-main)
Copied to clipboard
Oana Ignat, Zhijing Jin, Artem Abzaliev, Laura Biester, Santiago Castro, Naihao Deng, Xinyi Gao, Aylin Ece Gunal, Jacky He, Ashkan Kazemi, Muhammad Khalifa, Namho Koh, Andrew Lee, Siyang Liu, Do June Min, Shinka Mori, Joan C. Nwatu, Veronica Perez-Rosas, Siqi Shen, Zekun Wang, Winston Wu, Rada Mihalcea
| Challenge: | Recent advances in large language models have led to misleading public discourse that “it’s all been solved.” |
| Approach: | They identify 14 research areas encompassing 45 research directions that require new research and are not directly solvable by LLMs. |
| Outcome: | The research areas identified are 45 research directions that require new research and are not directly solvable by LLMs. |
You can’t pick your neighbors, or can you? When and How to Rely on Retrieval in the kNN-LM (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that retrieval-enhanced language models can improve perplexity by combining text from large external datastores with a k-nearest neighbors model. |
| Approach: | They propose a retrieval-enhanced language model that interpolates existing LMs with a k-nearest neighbors model and requires no additional training. |
| Outcome: | The proposed model improves on two English language modeling datasets and shows that it is most effective when items have high semantic similarity with the query. |
Exploring and Predicting Transferability across NLP Tasks (2020.emnlp-main)
Copied to clipboard
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, Mohit Iyyer
| Challenge: | Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. |
| Approach: | They conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems. |
| Outcome: | The proposed model can improve performance even with low-data source tasks that differ substantially from the target task. |
BPID: A Benchmark for Personal Identity Deduplication (2024.emnlp-industry)
Copied to clipboard
Runhui Wang, Yefan Tao, Adit Krishnan, Luyang Kong, Xuanqing Liu, Yuqian Deng, Yunzhao Yang, Henrik Johnson, Andrew Borthwick, Shobhit Gupta, Aditi Gundlapalli, Davor Golac
| Challenge: | Data deduplication is a critical task in data management and mining, focused on consolidating duplicate records that refer to the same entity. |
| Approach: | They propose to use a dataset with 1,000,000 unlabeled synthetic PII profiles and a subset of 10,000 pairs curated and labeled as matches or non-matches. |
| Outcome: | The proposed datasets contain synthetic profiles built from publicly available sources that do not represent real individuals. |