Papers with LLaMA-7B
Struc-Bench: Are Large Language Models Good at Generating Complex Structured Tabular Data? (2024.naacl-short)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have advanced capabilities but produce complex structured data. |
| Approach: | They propose a structure-aware fine-tuning method to bolster LLMs' performance by crafting format-specific instructions from the intended outputs. |
| Outcome: | The proposed method outperforms LLMs on all three formats and spans text tables, HTML, and LaTeX formats. |
DL-QAT: Weight-Decomposed Low-Rank Quantization-Aware Training for Large Language Models (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Quantization-aware Training (QAT) is a popular technique for reducing memory usage and improving computational efficiency in large language models. |
| Approach: | They propose a weight-decomposed low-rank quantization-aware training approach that integrates QAT with a group-specific quantization magnitude adjustment. |
| Outcome: | The proposed method outperforms the state-of-the-art method on LLaMA and LLama2 models. |
Composite Backdoor Attacks Against Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated superior performance on various tasks, but untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks. |
| Approach: | They propose a composite backdoor attack that scatters multiple trigger keys in different prompt components. |
| Outcome: | The proposed attack achieves 100% Attack Success Rate (ASR) with a False Triggered Rate (FTR) below 2.06% and negligible model accuracy degradation. |
CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models that use self-attention and position embedding have anomalous behavior that hinder long context window extrapolation. |
| Approach: | They propose a collinear constraint between Q and K to integrate RoPE and self-attention. |
| Outcome: | The proposed model integrates self-attention and position embedding into LLMs without fine-tuning. |
MDR: Model-Specific Demonstration Retrieval at Inference Time for In-Context Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for retrieval-based in-context learning ignore model biases and fail to retrieve the most appropriate demonstrations for different LLMs. |
| Approach: | They propose a model-specific demonstration retrieval method that considers the biases of different LLMs at inference time. |
| Outcome: | The proposed method improves performance on seen and unseen tasks with multi-scale inference LLMs by up to 41.2%. |
Search Augmented Instruction Learning (2023.findings-emnlp)
Copied to clipboard
Hongyin Luo, Tianhua Zhang, Yung-Sung Chuang, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, James Glass
| Challenge: | Large language models (LLMs) have been significantly improved by instruction fine-tuning, but still lack transparency and the ability to utilize up-to-date knowledge and information. |
| Approach: | They propose a search-augmented instruction learning model which grounds the language generation and instruction following abilities on complex search results generated by in-house and external search engines. |
| Outcome: | The proposed model outperforms plain LLMs on zero-shot language tasks and can generate both natural and programming languages following natural language guidance and requests. |
Teaching Language Models to Self-Improve through Interactive Demonstrations (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been shown to improve performance on downstream tasks by prompting them to analyze and revise their outputs. |
| Approach: | They propose a training algorithm that prompts large language models to analyze and revise their own outputs and uses this feedback to train the small model. |
| Outcome: | The proposed approach improves LLaMA-7B's performance on math and reasoning tasks by up to 7.13%. |
Fast Randomized Low-Rank Adaptation of Pre-trained Language Models with PAC Regularization (2024.findings-acl)
Copied to clipboard
| Challenge: | Low-rank adaptation (LoRA) is an efficient way to fine-tune large language models (LLMs) but its memory overhead restricts it to scale up when the model size increases. |
| Approach: | They propose a low-rank adaptation method which decomposes model weight updates into a pair of low-ranked projection matrices and incorporates a regularizer to improve generalization. |
| Outcome: | The proposed method achieves better performance under few-shot settings and similar performance to the SOTA low-rank adaptation methods. |
LPZero: Language Model Zero-cost Proxy Search from Zero (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing zero-cost (ZC) proxies rely on expert knowledge and incur significant trial-and-error costs. |
| Approach: | They propose a framework that automatically designs zero-cost (ZC) proxies for various tasks and incorporates genetic programming to find the optimal symbolic composition. |
| Outcome: | The proposed framework achieves higher ranking consistency than human-designed proxies on NLP tasks. |
LoRaDA: Low-Rank Direct Attention Adaptation for Efficient LLM Fine-tuning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in parameter-efficient fine-tuning techniques allow for adjustments to only a minor fraction of the parameters of language models. |
| Approach: | They propose a low-rank direct attention adapted method for efficient LLM fine-tuning . they propose LMAM, which can bring negative attention to self-attention modules . |
| Outcome: | The proposed method outperforms the full fine-tuning method by 2.1% on GLUE benchmark. |