Papers by Hongxia Jin
Instruction-following Evaluation through Verbalizer Manipulation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on common instructions that align well with what the model learned during training, but proficiency in responding to these instructions does not necessarily imply strong ability in instruction following. |
| Approach: | They propose a new instruction-following evaluation protocol called verbalizer manipulation that instructs the model to verbalize the task label with words aligning with model priors to different extents. |
| Outcome: | The proposed protocol can be integrated with any classification benchmark to examine the model’s reliance on priors and its ability to override them to accurately follow the instructions. |
A Neural Transition-based Model for Nested Mention Recognition (D18-1)
Copied to clipboard
| Challenge: | Existing methods to recognize nested mentions are based on Stack-LSTM . nesting mentions can be used for downstream tasks like question answering and relation extraction. |
| Approach: | They propose a scalable transition-based method to model the nested structure of mentions. |
| Outcome: | The proposed method gets the state-of-the-art performance in ACE datasets showing its effectiveness in detecting nested mentions. |
Adaptive Rank Selections for Low-Rank Approximation of Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Singular Value Decomposition (SVD) or its weighted variants has progressed in compressing language models. |
| Approach: | They propose a binary masking mechanism for optimizing the number of ranks in a differentiable framework. |
| Outcome: | The proposed algorithm achieves much better accuracy than previous SVD and its weighted variants. |
Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection (2024.naacl-long)
Copied to clipboard
Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, Hongxia Jin
| Challenge: | Instruction-tuned Large Language Models (LLMs) can modulate responses based on human instructions, but they can be maliciously steered to impact society in subtle but persistent ways. |
| Approach: | They propose a backdoor attack setting that allows an attacker to inject a virtual prompt into an LLM to steer it without any explicit injection at its input. |
| Outcome: | The proposed method is able to poison the model's instruction tuning data and show that it is highly effective in steering the model. |
Explicit over Implict: Explicit Diversity Conditions for Effective Question Answer Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent pretrained and large language model-based QAG methods suffer from redundant generation of QA pairs, affecting downstream QA systems. |
| Approach: | They propose to use explicit diversity conditions to generate diverse question-answer synthetic data by focusing on spatial aspects, question types, and entities. |
| Outcome: | The proposed diversity conditions significantly increase diversity in QA generation over existing diversity techniques. |
Generating Dialogue Responses from a Semantic Latent Space (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for dialogue generation are unable to integrate information from multiple semantically similar valid responses of a given prompt. |
| Approach: | They propose to learn the pair relationship between the prompts and responses as a regression task instead of the end-to-end classification on vocabulary. |
| Outcome: | The proposed model learns the pair relationship between the prompts and responses on a latent space instead of the end-to-end classification on vocabulary. |
A New Concept of Deep Reinforcement Learning based Augmented General Tagging System (C18-1)
Copied to clipboard
| Challenge: | Existing systems for general sequence tagging/labeling are based on neural network architectures. |
| Approach: | They propose a deep neural network based sequence labeling model and a augmented tagger to improve system performance by modeling the data with minority tags. |
| Outcome: | The proposed system outperforms the current state-of-the-art model on ATIS and CoNLL-2003 datasets by 1.9% and 1.4%. |
Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase (2021.eacl-main)
Copied to clipboard
| Challenge: | a data augmentation technique is used to boost performance on spoken language understanding tasks. |
| Approach: | They propose a data augmentation technique based on byte pair encoding and a BERT-like self-attention model to boost performance on spoken language understanding tasks. |
| Outcome: | The proposed method performs well on domain and intent classification tasks for a voice assistant and in a user-study focused on utterance naturalness and semantic similarity. |
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing (2025.naacl-long)
Copied to clipboard
James Seale Smith, Chi-Heng Lin, Shikhar Tuli, Haris Jeelani, Shangqian Gao, Yilin Shen, Hongxia Jin, Yen-Chang Hsu
| Challenge: | Empirical evaluations demonstrate substantial performance gains over existing methods . |
| Approach: | They propose a method to prune LLMs that selectively prunes model blocks based on an importance score and replaces them with a low-parameter replacement strategy. |
| Outcome: | The proposed method achieves state-of-the-art performance on 5/6 and 6/6 benchmarks with a compression rate of 30% and 40%. |
Fast Domain Adaptation of Semantic Parsers via Paraphrase Attention (D19-61)
Copied to clipboard
| Challenge: | Semantic parsers are used to convert user’s natural language commands to executable logical form in intelligent personal agents. Labeled datasets required to train such parser are expensive to collect, and are never comprehensive. |
| Approach: | They propose to use a sequence-to-sequence/tree attention based attention-based sequence-based parsers which support fast near real time retraining. |
| Outcome: | The proposed parsers can maintain high accuracy and fast retraining time while leveraging paraphrases already present in the training dataset. |
Numerical Optimizations for Weighted Low-rank Estimation on Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Singular value decomposition (SVD) is one of the most popular methods for estimating a target matrix with smaller matrices. |
| Approach: | They propose a method that approximates a target matrix with smaller matrices by two smaller . they also propose metric to predict when the SVD may introduce a significant performance drop. |
| Outcome: | The proposed method can perform better than current SOTA methods in compressing Transformer-based language models. |
A Bi-Model Based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling (N18-2)
Copied to clipboard
| Challenge: | Intent detection and slot filling are two main tasks for building a spoken language understanding system. |
| Approach: | They propose to use a sequence to sequence model to generate both intent and slot filling tasks together to perform the two tasks jointly. |
| Outcome: | The proposed model achieves 0.5% intent accuracy improvement and 0.9 % slot filling improvement on the ATIS benchmark data. |
DynaMo: Accelerating Language Model Inference with Dynamic Multi-Token Sampling (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid explosion in model sizes has resulted in high inference times . open-source LLMs are democratizing research in natural language processing . |
| Approach: | They propose a suite of multi-token prediction language models that reduce net inference times by leveraging traditional autoregressive weights. |
| Outcome: | The proposed model achieves same-quality generated text as baseline (Pythia-6.9B) with only 5.87% and 2.67% parameter and training time overheads. |
Explainable Slot Type Attentions to Improve Joint Intent Detection and Slot Filling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods analyze and compute features collectively for all slot types, and have no way to explain slot filling model decisions. |
| Approach: | They propose a method that learns to generate additional slot type specific features to improve accuracy and provides explanations for slot filling decisions for the first time in a joint NLU model. |
| Outcome: | The proposed model improves on two widely used datasets and provides an explanation for slot filling decisions for the first time. |
Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing continual learning approaches suffer from low accuracy and performance fluctuation when the distributions of old and new data are significantly different. |
| Approach: | They propose a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. |
| Outcome: | The proposed model outperforms the best state-of-the-art method by 20% in average accuracy and each component contributes effectively to overall performance. |
CRUISE: Cold-Start New Skill Development via Iterative Utterance Generation (P18-4)
Copied to clipboard
| Challenge: | Existing systems require developers to manually generate and annotate a large number of utterances. |
| Approach: | They propose a system that guides ordinary software developers to build a high quality NLU engine from scratch. |
| Outcome: | The proposed system shows that iterative pruning of incorrect utterances reduces human workload and cognitive load. |
SLiM: Speculative Decoding with Hypothesis Reduction (2024.findings-naacl)
Copied to clipboard
| Challenge: | Speculative decoding has emerged as an alternative to autoregressive decoding for expediting inference in large language models (LLMs). prevailing assumptions focus solely on latency reduction, neglecting the computational expenses. |
| Approach: | They propose a speculative decoding enhancement to reduce the speculation set while validating more effective tokens. |
| Outcome: | The proposed method reduces the speculation set while validating more effective tokens. |
Dynamic Low-rank Estimation for Transformer-based Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | RankDyna is a matrix decomposition method that can be used to compress Transformer-based language models. |
| Approach: | They propose a matrix decomposition method that enables dynamic rank resource allocation . they say it can outperform current SOTA methods under various parameter budget levels . |
| Outcome: | The proposed method outperforms current SOTA methods under various budget levels . the proposed method is more efficient with higher compression rates . |
A Progressive Model to Enable Continual Learning for Semantic Slot Filling (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to slot filling training on large scale data are inefficient and require multiple trainings. |
| Approach: | They propose a slot filling model that transfers previously learned knowledge to a small size expanded component and enables it to be fast trained to learn from new data. |
| Outcome: | The proposed model outperforms existing models on two benchmark datasets by 4.24% and 3.03% on the same dataset. |
A New Concept of Knowledge based Question Answering (KBQA) System for Multi-hop Reasoning (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing knowledge based question answering systems are trained based on labeled reasoning paths, which hinder their performance. |
| Approach: | They propose a KBQA system which leverages multiple reasoning paths’ information and only requires labeled answer as supervision. |
| Outcome: | The proposed system can leverage multiple reasoning paths’ information and only requires labeled answer as supervision. |
Enhancing the generalization for Intent Classification and Out-of-Domain Detection in SLU (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for intent classification are expensive to collect and train . evaluators have shown that the ability to detect out-of-domain utterances is limited . |
| Approach: | They propose to train a model with only IND data while supporting both intent classification and OOD detection. |
| Outcome: | The proposed model improves on existing models and strong baselines on four datasets. |
SkillBot: Towards Automatic Skill Development via User Demonstration (N19-4)
Copied to clipboard
| Challenge: | Existing industrial PA products require software developers to build new skills via IDE tools. |
| Approach: | They propose a software that automatically develops a natural language understanding engine and implements the action without the need of coding. |
| Outcome: | The proposed system performs well on both benchmark and in-house datasets. |