Papers with Vicuna
Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. |
| Approach: | They propose a structured pruning algorithm that derives the importance of different components based on intermediate data dependencies and removes coupled components across different layers simultaneously. |
| Outcome: | The proposed algorithm reduces model size and accelerates inference without specialized operators and libraries, while maintaining its utility as versatile problem solvers. |
Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced capabilities but produce complex structured data. |
| Approach: | They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs. |
| Outcome: | The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats. |
BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized various domains with their remarkable capabilities, but their massive parameter sizes pose significant challenges for fine-tuning and inference. |
| Approach: | They propose a Bayesian Knowledge Distillation framework for compact Large Language Models in resource-constrained fine-tuning scenarios that employs Logits Dual-Scaling, Knowledge Alignment Module, and Bayes Distillations Optimization. |
| Outcome: | The proposed framework outperforms baseline methods on various state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna. |
Investigating Table-to-Text Generation Capabilities of Large Language Models in Real-World Information Seeking Scenarios (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing table-to-text generation techniques that transform complex tabular data into comprehensible narratives are lacking in real-world applications. |
| Approach: | They investigate the table-to-text capabilities of different LLMs using four datasets within two real-world information seeking scenarios. |
| Outcome: | The proposed models can generate table-to-text data in two real-world information seeking scenarios and perform better than existing models. |
Do large language models and humans have similar behaviours in causal inference with script knowledge? (2024.starsem-1)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models have superior language understanding abilities, including zero-shot causal reasoning. |
| Approach: | They used a script-based story to manipulate event B in a story which causally depends on a previous event A. |
| Outcome: | The results show that only recent LLMs, like GPT-3 or Vicuna, correlate with human behavior in the A B condition. |
VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization (2024.naacl-long)
Copied to clipboard
Dongsheng Zhu, Daniel Tang, Weidong Han, Jinghui Lu, Yukun Zhao, Guoliang Xing, Junfeng Wang, Dawei Yin
| Challenge: | Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. |
| Approach: | They propose a novel approach to advancing multi-modal language models in zero-shot learning by evaluating and optimizing instructional texts through In-Context Learning. |
| Outcome: | The proposed approach improves zero-shot performance in multi-modal tasks by evaluating and optimizing instructional texts. |
How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilities (2024.naacl-long)
Copied to clipboard
| Challenge: | Rapid progress in open-source Large Language Models (LLMs) is driving AI development, but lacks sufficient trustworthiness to detect and mitigate adversarial demonstrations. |
| Approach: | They propose an extended Chain of Utterances-based (CoU) prompting strategy to attack open-source LLMs. |
| Outcome: | The proposed attack strategy is based on malicious demonstrations and toxicity tests on open-source models. |
ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Attaching suffixes to harmful instructions can hack the defense of Large language models (LLMs) However, due to the unreadable of adversarial suffix, it can be relatively easily penetrated by common defense methods such as perplexity filters. |
| Approach: | They propose an algorithm to embed adversarial suffixes into coherent and understandable text to attack Large language models (LLMs) using a Advbench dataset. |
| Outcome: | The proposed approach reduces the computation time of adversarial suffixes and achieves a much better attack success rate than existing techniques. |
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study validates the effectiveness of chat language models by fine-tuning instruction data. |
| Approach: | They propose to use a large-scale dataset of instructional conversations to fine-tune a conversational model on instruction data. |
| Outcome: | The proposed model outperforms open-source models in key metrics including scale, average length, diversity, coherence, etc. |
Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retraining (2025.coling-main)
Copied to clipboard
| Challenge: | Existing backdoor defense methods are ineffective for generative large language models . generative LLMs output sequences of high-dimensional token logits instead of low-dimensional classification logits . |
| Approach: | They propose a method that leverages sample-wise gradients to identify backdoor samples without retraining LLMs. |
| Outcome: | The proposed method outperforms baselines significantly in identifying backdoor samples without retraining LLMs. |
Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models (2024.acl-long)
Copied to clipboard
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, Kaidi Xu
| Challenge: | Large Language Models (LLMs) show promising results in language generation but often “hallucinate”, making their outputs less reliable. |
| Approach: | They propose to shift attention to more relevant components at token- and sentence-levels for better UQ. |
| Outcome: | The proposed approach improves the performance of a range of popular “off-the-shelf” LLMs with model sizes extending up to 33B parameters. |
SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding (2024.acl-long)
Copied to clipboard
| Challenge: | Despite advances in large language models, they face substantial challenges in terms of safety. |
| Approach: | They develop a safety-aware decoding strategy for large language models to defend against jailbreak attacks. |
| Outcome: | The proposed strategy outperforms six defense methods against jailbreak attacks on five LLMs. |
Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method (2024.naacl-long)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Recent literature reveals that Large Language Models (LLMs) hallucinate intermittently, which impedes their reliability for further utilization. |
| Approach: | They propose a self-detection method to detect which questions an LLM does not know by combining the two components to identify whether the model generates a non-factual response to the question. |
| Outcome: | The proposed method can detect which questions an LLM does not know across factoid question-answering, arithmetic reasoning, and commonsense reasoning tasks. |
PlatoLM: Teaching LLMs in Multi-Round Dialogue via a User Simulator (2024.acl-long)
Copied to clipboard
| Challenge: | Recent efforts to democratize ChatGPT have focused on leveraging real user and ChatGPP dialogues, but the most direct human needs are often ignored. |
| Approach: | They propose a method to simulate human behavior better by using real human-like questions extracted from real human conversations as a learning goal and a user simulator called ‘Socratic’. |
| Outcome: | The proposed model achieves SoTA performance among LLaMA-based 7B models in MT-Bench. |
A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated capabilities for generating content that could be deemed harmful. |
| Approach: | They conduct a comprehensive analysis of existing studies on jailbreaking LLMs and their defense techniques. |
| Outcome: | The proposed techniques underperform existing white-box attacks and include special tokens significantly affects the likelihood of successful attacks. |
Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inference (2025.naacl-long)
Copied to clipboard
Zeping Li, Xinlong Yang, Ziheng Gao, Ji Liu, Guanchen Li, Zhuang Liu, Dong Li, Jinzhang Peng, Lu Tian, Emad Barsoum
| Challenge: | Existing methods such as Medusa lack adequate information interaction between different drafting heads. |
| Approach: | They propose an enhanced speculative decoding framework that builds upon Medusa and integrates a drafting block capable of parallel inference. |
| Outcome: | The proposed framework outperforms Medusa in terms of head accuracy and latency. |
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are often used for multi-faceted language generation and evaluation tasks that require complex user constraints or taking into account multiple aspects and criteria. |
| Approach: | They propose a Large Language Model program that consists of branch, solve, and merge modules that are parameterized with specific prompts to the base LLM. |
| Outcome: | The proposed program improves the evaluation correctness and consistency for each LLM by up to 26%, reducing length and pairwise position biases by up 50%, and allowing LLaMA-2-chat to match or outperform GPT-4 on most domains. |
Unraveling the Mystery: Defending Against Jailbreak Attacks Via Unearthing Real Intention (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly vulnerable to elusive and implicit intentions, causing security risks and compromising user experience. |
| Approach: | They propose a method to detect and mitigate implicit jailbreak attacks using LLMs by unearthing real intentions and a greedy gradient-based algorithm to remove the least important parts of a sentence. |
| Outcome: | The proposed method reduces attacks success rate and Harmful Score while maintaining overall model performance. |
Using Natural Language Explanations to Improve Robustness of In-context Learning (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that large language models excel in many tasks via in-context learning (ICL). However, ICL struggles to execute complex tasks such as arithmetic, commonsense, and symbolic reasoning. |
| Approach: | They propose to augment ICL with natural language explanations (NLEs) to produce further NLEs on adversarial datasets. |
| Outcome: | The proposed approach yields more accurate results than zero-shot-ICL and using only human-generated NLEs on eight adversarial datasets. |
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)
Copied to clipboard
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
| Challenge: | Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly. |
| Approach: | They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source. |
| Outcome: | The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source. |
Enhancing Alignment using Curriculum Learning & Ranked Preferences (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is an effective technique that leverages pairwise preference data to align LLMs to human preferences. |
| Approach: | They propose to use pairwise preference data to create multiple preference pairs for a given prompt. |
| Outcome: | The proposed method outperforms standard DPO on MTbench, Vicuna bench, and WizardLM with a score of 7.43 on the test sets. |
Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. |
| Approach: | They propose to use visual-question-answering (VQA) datasets to annotate a 5k-sample VQA preference dataset and to investigate the degradation of VQA datasets. |
| Outcome: | The proposed model surpasses the instruction-following capabilities of the language model with DPO and SteerLM. |
From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration (2026.acl-long)
Copied to clipboard
| Challenge: | Existing acceleration strategies suffer from severe "backbone dependency" Existing strategies such as token pruning or layer sparsity suffer from this . |
| Approach: | They propose a framework that decouples visual redundancy into IVR and architecture-dependent secondary saturation redundancies. |
| Outcome: | The proposed framework outperforms existing frameworks on Qwen25-VL and Qwa25-LL. |
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization (2023.emnlp-main)
Copied to clipboard
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, Yejin Choi
| Challenge: | a dataset of 1.5 million conversations distilled from everyday spoken situations is limited in scale due to its associated costs. |
| Approach: | They propose to make SODA a publicly available, million-scale high-quality social dialogue dataset . they contextualize social commonsense knowledge from a knowledge graph to distill broad spectrum of social interactions . |
| Outcome: | The proposed dataset is the first publicly available, million-scale high-quality social dialogue dataset. |
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown strong abilities to understand natural language, but how these factors affect the models’ language perception is unclear. |
| Approach: | They compare the self-attention of several existing large language models in different sizes to assess the effect of scaling and instruction tuning on language perception. |
| Outcome: | The proposed models are closer to non-native speakers than native speakers in attention, suggesting a sub-optimal language perception of all models. |
Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM (2023.findings-emnlp)
Copied to clipboard
Sahal Mullappilly, Abdelrahman Shaker, Omkar Thawakar, Hisham Cholakkal, Rao Anwer, Salman Khan, Fahad Khan
| Challenge: | Recent large language models like ChatGPT and Bard excel in a wide variety of NLP tasks but are not specifically tailored for climate related domain specific information. |
| Approach: | They propose a lightweight Arabic Mini-ClimateGPT that is built on an open-source LLM and specifically fine-tuned on a conversational-style instruction tuning curated Arabic dataset Clima500-Instruct. |
| Outcome: | The proposed model surpasses the baseline LLM in 88.3% of cases during ChatGPT-based evaluation and human expert prefers it over other open-source models. |
USDC: A Dataset of ̲User ̲Stance and ̲Dogmatism in Long ̲Conversations (2025.findings-acl)
Copied to clipboard
| Challenge: | Previously, studies on stance and dogmatism in user conversations have focused on training models using annotated datasets at the post level, treating each post as independent and randomly sampling posts from conversation threads. |
| Approach: | They build a dataset for studying user opinion fluctuations in 764 long multi-user Reddit conversation threads, called USDC. |
| Outcome: | The proposed dataset analyzes user opinion fluctuations in 764 long multi-user Reddit conversation threads. |
Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient (2025.acl-long)
Copied to clipboard
| Challenge: | Recent pruning methods rely on heuristically hand-crafted metrics, leading to suboptimal performance. |
| Approach: | They propose a method that optimizes pruning masks by minimizing back-propagation . they learn an underlying Bernoulli distribution to sample binary pruning mask samples . |
| Outcome: | The proposed method is able to support global and heterogeneous pruning without back-propagation. |