Papers by Handong Zhao
Advancing Vision-Language Models with Adapter Ensemble Strategies (2024.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. |
| Approach: | They propose to combine a "adapter ensemble" with traditional machine learning techniques to augment large-scale pretrained vision-language models. |
| Outcome: | The proposed model outperforms baselines and derives improvement when the number of ensemble parameters increases. |
Edge: Enriching Knowledge Graph Embeddings with External Text (2021.naacl-main)
Copied to clipboard
| Challenge: | Knowledge graphs suffer from sparsity which degrades the quality of representations generated by various methods. |
| Approach: | They propose a knowledge graph enrichment framework called Edge to enhance knowledge graphs based on "hard" co-occurrence of words in knowledge graph entities and external text. |
| Outcome: | The proposed framework achieves "soft" augmentation by combining external text with knowledge graph entities. |
Context-aware Information-theoretic Causal De-biasing for Interactive Sequence Labeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing deep learning models for sequence labeling are expensive and time-consuming. |
| Approach: | They propose an interactive sequence labeling that allows training directly with the user feedback . they identify context and feedback biases by formulating interactive sequence labels via a Structural Causal Model. |
| Outcome: | The proposed approach can effectively alleviate the biases and can be learnt with the user feedback. |
MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities (2025.acl-long)
Copied to clipboard
| Challenge: | Decoder-only large language models are increasingly being adapted for bidirectional modeling . however, their reliance on causal attention restricts their effectiveness in tasks that require understanding of bidirectional context. |
| Approach: | They propose a method to adapt decoder-only large language models to generate robust representations and infill missing text spans. |
| Outcome: | The proposed method surpasses strong decoders on token-level and sentence-level representation learning tasks and generates contextually appropriate text infills without excessive repetition of words or phrases. |
Augment before You Try: Knowledge-Enhanced Table Question Answering via Table Expansion (2025.findings-emnlp)
Copied to clipboard
Yujian Liu, Jiabao Ji, Tong Yu, Ryan A. Rossi, Sungchul Kim, Handong Zhao, Ritwik Sinha, Yang Zhang, Shiyu Chang
| Challenge: | Existing methods to integrate external information into a given table neglect the structured nature of the table. |
| Approach: | They propose a simple yet effective method to integrate external information into a given table by first building an augmenting table and then generating a SQL query over the two tables to answer the question. |
| Outcome: | The proposed method outperforms strong baselines on three table QA benchmarks. |
Few-Shot Dialogue Summarization via Skeleton-Assisted Prompt Transfer in Prompt Tuning (2024.eacl-long)
Copied to clipboard
Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, Mark Riedl
| Challenge: | Existing prompt transfer techniques lack consideration for dialogue-specific information. |
| Approach: | They propose a method which leverages skeleton generation as extra supervision that functions as a medium connecting the distinct source and target task. |
| Outcome: | The proposed method significantly outperforms baselines on two dialogue summarization benchmarks. |
Personalized Federated Learning for Text Classification with Gradient-Free Prompt Tuning (2024.findings-naacl)
Copied to clipboard
Rui Wang, Tong Yu, Ruiyi Zhang, Sungchul Kim, Ryan Rossi, Handong Zhao, Junda Wu, Subrata Mitra, Lina Yao, Ricardo Henao
| Challenge: | Pretrained language models (PLMs) are used for personalized federated learning . communication costs are high with large PLMs, and local training is expensive . |
| Approach: | They propose a framework for federated learning with pretrained language models . they propose 'discrete local search' and compression mechanism for local training . |
| Outcome: | The proposed framework achieves superior performance compared with baselines. |
Tag-grounded Visual Instruction Tuning with Retrieval Augmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have seen remarkable progress in providing general instruction-following ability, but struggle with critical problems when required to provide a detailed and accurate response to a visual instruction. |
| Approach: | They propose to enhance the mapping process by using retrieval-augmented tag tokens, which contain rich object-aware information such as object names and attributes. |
| Outcome: | The proposed model outperforms baselines that share the same language model and training data on 12 benchmarks and shows zero-shot capability when provided with specific datastores. |
Learning Contextualized Knowledge Structures for Commonsense Reasoning (2021.findings-acl)
Copied to clipboard
Jun Yan, Mrigank Raman, Aaron Chan, Tianyu Zhang, Ryan Rossi, Handong Zhao, Sungchul Kim, Nedim Lipka, Xiang Ren
| Challenge: | Recent knowledge graph (KG) augmented models have achieved notable success on commonsense reasoning tasks. |
| Approach: | They propose a KG-augmented model that contextualizes extracted and generated knowledge by reasoning over both within a single graph structure. |
| Outcome: | The proposed model outperforms existing models on four commonsense reasoning benchmarks and a user study on edge validness and helpfulness. |
Aligning as Debiasing: Causality-Aware Alignment via Reinforcement Learning with Interventional Feedback (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce LLMs' biased outputs rely on reward signals from current model outputs without considering the source of biases. |
| Approach: | They propose to leverage the reward model in RL alignment as an instrumental variable to perform causal intervention on LLMs. |
| Outcome: | The proposed method reduces biases by using human feedback to fine tune LLMs to human values. |
GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work of GUI action grounding fine-tunes data from pre-trained MLLMs, but data is limited to specific GUI environments. |
| Approach: | They propose to use a GUI-based agent to collect environment-specific data and fine-tune GUI grounding models with the collected data. |
| Outcome: | The proposed model can be extended to other GUI environments to improve performance. |
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Contrastive language-image pre-training models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval. |
| Approach: | They propose a fine-tuning approach to enhance the representations of CLIP models for paraphrases by leveraging large language models. |
| Outcome: | The proposed model improves on baseline models across paraphrased retrieval, visual genome relation and attribution, and seven semantic textual similarity tasks. |
Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that pre-trained ML-LMs can achieve zero-shot cross-lingual transfer without explicit cross-linguistic supervision. |
| Approach: | They propose a method to remove language-specific factors from multilingual embedding spaces by using a single value decomposition method with multiple monolingual corpora as input. |
| Outcome: | The proposed method can boost language agnosticism without finetuning . Empirical results show that it consistently leads to improvements over existing models. |
Few-Shot Class-Incremental Learning for Named Entity Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models of Named Entity Recognition (NER) are trained on large datasets with predefined entity classes, but data of new classes arrives constantly. Existing work on NER relies on the assumption that there exists abundance of labeled data for the training of new class. |
| Approach: | They propose a few-shot class-incremental learning problem where NER model is trained with only few labeled samples of the new classes without forgetting knowledge of the old ones. |
| Outcome: | The proposed model improves over existing baselines by reconstructing training data of old classes and real data from the training set. |
Learning Adaptive Axis Attentions in Fine-tuning: Beyond Fixed Sparse Attention Patterns (2022.findings-acl)
Copied to clipboard
Zihan Wang, Jiuxiang Gu, Jason Kuen, Handong Zhao, Vlad Morariu, Ruiyi Zhang, Ani Nenkova, Tong Sun, Jingbo Shang
| Challenge: | Adaptive Axis Attention learns different attention patterns for each task and model layer . sparse attention patterns do not improve the run time of the models but they reduce model memory requirements . |
| Approach: | They propose a method that learns different attention patterns for each Transformer layer . they propose 'adaptive axis attention' method that identifies important tokens . |
| Outcome: | The proposed method does not require pre-training to accommodate sparse attention patterns. |
A Critical Analysis of Document Out-of-Distribution Detection (2023.findings-emnlp)
Copied to clipboard
Jiuxiang Gu, Yifei Ming, Yi Zhou, Jason Kuen, Vlad Morariu, Handong Zhao, Ruiyi Zhang, Nikolaos Barmpalios, Anqi Liu, Yixuan Li, Tong Sun, Ani Nenkova
| Challenge: | Existing document understanding models focus on single-modal inputs such as images or texts. |
| Approach: | They propose to use a spatial-aware adapter to adapt transformer-based language models to document domain to exploit multi-modal information. |
| Outcome: | The proposed model significantly improves the OOD detection performance compared to using a standard language model and to competitive baselines. |
Federated Domain Adaptation for Named Entity Recognition via Distilling with Heterogeneous Tag Sets (2023.findings-acl)
Copied to clipboard
| Challenge: | a federated domain adaptation approach is used to learn with NER datasets from multiple platforms while not violating data privacy. |
| Approach: | They propose to use a distillation approach to facilitate knowledge transfer across platforms. |
| Outcome: | The proposed model performs better in the clinic domain. |