Papers by Bill Lin

Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning (2023.emnlp-main)

Copied to clipboard

Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lianhui Qin, Xiang Ren, Sean Welleck, Yejin Choi

Language Models

Challenge:	Extreme-scale language models have shown exceptional performance on a variety of language tasks, but the degree of control offered by these models through pure prompting is limited.
Approach:	They propose an inference-time policy adapter which tailors a large base model without fine-tuning it.
Outcome:	The proposed model outperforms baseline methods on five challenging text generation tasks and even over GPT-4.

Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking (2021.emnlp-main)

Copied to clipboard

Weizhe Lin, Bo-Hsiang Tseng, Bill Byrne

Natural Language Interfaces

Challenge:	Existing models for dialogue state tracking are based on Graph Attention Networks . if the relationship between slots and values is modelled explicitly, this can be improved .
Approach:	They propose a model architecture that augments GPT-2 with Graph Attention Networks to allow sequential prediction of slot values.
Outcome:	The proposed architecture improves performance against a strong GPT-2 baseline and with sparsely supervised training.

Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection (2025.emnlp-main)

Copied to clipboard

Jingbiao Mei, Jinghong Chen, Guangyu Yang, Weizhe Lin, Bill Byrne

Responsible & Trustworthy NLP

Challenge:	Large Multimodal Models (LMMs) have shown promise in hateful meme detection, but they face limitations like sub-optimal performance and limited out-of-domain generalization capabilities.
Approach:	They propose a robust adaptation framework for hateful meme detection that enhances in-domain accuracy and cross-domain generalization while preserving the general vision-language capabilities of LMMs.
Outcome:	The proposed framework outperforms larger agentic systems in detecting hateful memes under adversarial attacks while maintaining the general vision-language capabilities of LMMs.

A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks (2020.acl-main)

Copied to clipboard

Angela Lin, Sudha Rao, Asli Celikyilmaz, Elnaz Nouri, Chris Brockett, Debadeepta Dey, Bill Dolan

Multimodality

Challenge:	a web-based algorithm can be used to align instructions for different tasks . video instructions can be noisy and contain far more information than textual instructions.
Approach:	They propose an algorithm that learns pairwise alignments between different recipes . they then use a graph algorithm to derive a joint alignment between multiple video and text recipes based on the same recipe.
Outcome:	The proposed algorithm learns pairwise alignments between different recipes for the same dish.

NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation (2023.findings-emnlp)

Copied to clipboard

Peter West, Ronan Bras, Taylor Sorensen, Bill Lin, Liwei Jiang, Ximing Lu, Khyathi Chandu, Jack Hessel, Ashutosh Baheti, Chandra Bhagavatula, Yejin Choi

Semantic Text Processing

Challenge:	a new commonsense knowledge model, NovaCOMET, combines knowledge and general task models.
Approach:	They propose an open commonsense knowledge model that combines knowledge and general task models.
Outcome:	The proposed model matches or exceeds existing knowledge models on commonsense reasoning tasks.

More Robust Schema-Guided Dialogue State Tracking via Tree-Based Paraphrase Ranking (2023.findings-eacl)

Copied to clipboard

Alexandru Coca, Bo-Hsiang Tseng, Weizhe Lin, Bill Byrne

Robustness in NLP

Challenge:	Pre-trained language models (PLMs) can track only slots drawn from a database or domain ontology.
Approach:	They propose a framework for generating synthetic schemas which uses tree-based ranking to optimise lexical diversity and semantic faithfulness.
Outcome:	The proposed framework improves the generalisation of strong baselines by augmenting training data with prompts generated by the framework.

Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learning (2024.acl-long)

Copied to clipboard

Jingbiao Mei, Jinghong Chen, Weizhe Lin, Bill Byrne, Marcus Tomalin

Semantic Text Processing

Challenge:	Existing systems for detecting hateful memes lack sensitivity to subtle differences in memes that are vital for correct hatefulness classification.
Approach:	They propose to construct a hatefulness-aware embedding space through retrieval-guided contrastive training to identify hatefulness based on data unseen in training.
Outcome:	The proposed system outperforms existing models on the HatefulMemes dataset with an AUROC of 87.0 and improves contextual understanding across domains.

FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering (2023.findings-eacl)

Copied to clipboard

Weizhe Lin, Zhilin Wang, Bill Byrne

Multimodality

Challenge:	Fact-based Visual Question Answering (FVQA) is a visual question answering task that requires information retrieval using common sense knowledge graphs to answer.
Approach:	They propose a new test question with adversarial variants to address this imbalance by using a KB-VQA dataset that is small and contains only one answer per question.
Outcome:	The proposed version reduces the vulnerability of the original FVQA dataset without human annotations.

Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models (2026.acl-long)

Copied to clipboard

Guangyu Yang, Jinghong Chen, Jingbiao Mei, Weizhe Lin, Bill Byrne

Robustness in NLP

Challenge:	Large Language Models (LLMs) remain vulnerable to jailbreak attacks due to evolving nature and diversity of attack strategies.
Approach:	They propose a framework for jailbreak detection that integrates a database of known attack examples into Retrieval-Augmented Generation to infer the underlying, malicious user query and jailbreak strategy used to attack the system.
Outcome:	The proposed framework reduces the effectiveness of strong jailbreak attacks while maintaining low rejection rates for benign queries.

PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers (2024.acl-long)

Copied to clipboard

Weizhe Lin, Jingbiao Mei, Jinghong Chen, Bill Byrne

Multimodality

Challenge:	Large Multimodal Models excel in natural language and visual understanding but are challenged by challenging tasks such as Knowledge-based Visual Question Answering (KB-VQA).
Approach:	They propose a framework for training Large Multimodal Models (LMMs) to perform KB-VQA tasks.
Outcome:	The proposed framework is used to train and evaluate multi-modal retrievers.

Control-DAG: Constrained Decoding for Non-Autoregressive Directed Acyclic T5 using Weighted Finite State Automata (2024.naacl-short)

Copied to clipboard

Jinghong Chen, Weizhe Lin, Jingbiao Mei, Bill Byrne

Dialogue Systems & Conversational Agents

Challenge:	Existing non-autoregressive (NAR) models fail to generate specified entity names in up to 40% of responses and produce OOV errors.
Approach:	They propose a constrained decoding algorithm for Directed Acyclic T5 model which offers lexical, vocabulary and length control.
Outcome:	The proposed model significantly improves on Schema Guided Dialogue and DART datasets, establishing strong results for Task-Oriented Dialog and Data-to-Text NLG.

An Inner Table Retriever for Robust Table Question Answering (2023.acl-long)

Copied to clipboard

Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria de Gispert, Gonzalo Iglesias

Multimodality

Challenge:	Table Question Answering (TableQA) is a task of answering NL user questions using factoid answers extracted from table content.
Approach:	They propose a method for handling long tables in TableQA that extracts sub-tables to preserve the most relevant information for a question.
Outcome:	The proposed method can improve TableQA's accuracy with up to 1.3-4.8% and achieve state-of-the-art in two benchmarks.

Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding (2024.naacl-short)

Copied to clipboard

Guangyu Yang, Jinghong Chen, Weizhe Lin, Bill Byrne

Multilinguality

Challenge:	Recent work shows that MBR decoding can significantly improve translation performance of Multilingual Large Language Models.
Approach:	They propose a method that uses a monolingual fine-tuning set to fine- tune MLLMs to get the gains of MBR without additional computation in inference.
Outcome:	The proposed method outperforms greedy decoding and beam search on multiple NMT tests.

LI-RAGE: Late Interaction Retrieval Augmented Generation with Explicit Signals for Open-Domain Table Question Answering (2023.acl-short)

Copied to clipboard

Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria de Gispert, Gonzalo Iglesias

Structured Data in NLP

Challenge:	Recent open-domain TableQA pipelines use a combination of retriever and reader . a table can be very large and might contain heterogeneous information across rows/columns .
Approach:	They propose to combine a retriever-reader pipeline with a binary relevance token to train the retriever and reader.
Outcome:	The proposed approaches improve on two open-domain TableQA datasets.

Retrieval Augmented Visual Question Answering with Outside Knowledge (2022.emnlp-main)

Copied to clipboard

Weizhe Lin, Bill Byrne

Multimodality

Challenge:	Recent systems that use Dense Passage Retrieval (DPR) for visual question answering use Denses trained separately from answer generation.
Approach:	They propose a joint training scheme that integrates DPR with answer generation to improve the performance of OK-VQA systems.
Outcome:	The proposed model outperforms existing systems with strong Dense Passage Retrieval (DPR) it yields significant improvements in answer quality and computation required for training.