Papers with initialization
Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering (2024.naacl-short)
Copied to clipboard
| Challenge: | Prompt tuning is an efficient method for initializing pre-trained models . but initialization of prompts is sensitive when the model size is small . |
| Approach: | They propose a method to measure catastrophic forgetting by analyzing prompts for the first time . they characterize a question answering task based on answer format and prompt initialization . |
| Outcome: | The proposed approach can help deepen understanding of prompt tuning. |
NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is essential for adapting large language models (LLMs). However, LoRA suffers from slow convergence and some recent LoRA variants, such as PiSSA, rely on Singular Value Decomposition (SVD) for initialization. |
| Approach: | They propose to introduce a small intermediate matrix between the low-rank matrices (A) and (B) and propose NyströmLoRA (NLoRA) which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. |
| Outcome: | The proposed approach improves on 5 natural language generation tasks and 8 natural language understanding tasks with minimal parameter overhead. |
An Empirical Study of Multimodal Model Merging (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that model merging can generate a multi-task solution without synchronous training. |
| Approach: | They propose to merge vision, language, and cross-modal transformers of a modality-specific architecture to create a parameter-efficient architecture. |
| Outcome: | The proposed model merging outperforms naive models on various tasks with improvements of 3% on VQA, 7% on COCO retrieval, 25% on NLVR2, 14% on Flickr30k and 3% ADE20k. |
AdaPT: A Set of Guidelines for Hyperbolic Multimodal Multilingual NLP (2024.findings-naacl)
Copied to clipboard
| Challenge: | Euclidean space is used for training neural models and performing arithmetic operations, but many data types have complex geometries and cannot be captured in the Euclidesan space. |
| Approach: | They propose a set of guidelines for initialization, parametrization, and training of neural networks that can be generalized over existing neural network training methodologies. |
| Outcome: | The proposed framework outperforms Euclidean methods on three tasks over 12 languages and modalities on a variety of domains. |
Meta-Information Guided Meta-Learning for Few-Shot Relation Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing meta-learning models rely on implicit instance statistics and are unreliability and weak interpretability. |
| Approach: | They propose a meta-information guided meta-learning framework that uses semantics to guide meta- learning . experimental results demonstrate the effectiveness of the proposed framework . |
| Outcome: | The proposed framework can establish connections between instance-based information and semantic-based data, enabling faster initialization and adaptation. |
ModRWKV: Transformer Multimodality in Linear Time (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, multimodal studies are based on large language models with quadratic-complexity Transformer architectures. |
| Approach: | They propose a decoupled multimodal framework built upon the RWKV7 architecture as its LLM backbone and a lightweight architecture to achieve multi-source information fusion. |
| Outcome: | The proposed framework achieves multi-source information fusion through dynamically adaptable heterogeneous modality encoders. |
CSMCIR: CoT-Enhanced Symmetric Alignment with Memory Bank for Composed Image Retrieval (2026.findings-acl)
Copied to clipboard
Zhipeng Qian, Zihan Liang, Yufei Ma, Ben Chen, Huangyu Dai, Yiwei Ma, Jiayi Ji, Chenyi Lei, Han Li, Xiaoshuai Sun
| Challenge: | Existing approaches to search for images using single-modality are limited by representation space fragmentation. |
| Approach: | They propose a unified representation framework that achieves efficient query-target alignment . they introduce a multi-level Chain-of-Thought prompting strategy that guides MLMs to generate discriminative, semantically compatible captions for target images . |
| Outcome: | The proposed framework achieves efficient query-target alignment through synergistic components. |
Efficient One-shot Compression via Low-Rank Local Feature Distillation (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing structured pruning approaches for large language models require calibration data and costly continued pretraining on billions of tokens to recover lost performance. |
| Approach: | They propose a method that locally distills activations with low-rank weights . they compress Mixtral-8x7B on a single GPU and Phi-2 3B by 40% . |
| Outcome: | The proposed method compresses Mixtral-8x7B on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance. |
Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods underestimate the importance of utilizing the teacher's discriminative classifier and face challenges in establishing proper layer mappings. |
| Approach: | They propose to reuse pre-trained teacher classifiers to improve student performance . they use projectors to match hidden size of the teacher model to student . |
| Outcome: | The proposed method outperforms existing methods on 97.7% of the teacher BERT base without additional trainable parameters. |
A Closer Look on Unsupervised Cross-lingual Word Embeddings Mapping (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for word embeddings are limited to a single, unannotated corpus, which means that word representations with similar meaning in distinct languages can be very different. |
| Approach: | They propose an unsupervised method for cross-lingual word embedding mapping that uses stochastic initialization and isometric initialization to verify the method's robustness. |
| Outcome: | The proposed method is robust on different embedding representations and new language pairs, particularly those involving Slavic languages like Polish or Czech. |
LESA: Learnable LLM Layer Scaling-Up (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for depth scaling-up rely on empirical heuristic rules for layer duplication, resulting in poor initialization and slower convergence during continual pre-training. |
| Approach: | They propose a method for learning latent parameters between layers by concatenating parameters from each layer and applying Singular Value Decomposition. |
| Outcome: | Experiments show that LESA outperforms baseline models with less than half the cost of existing methods. |
TLoRA: Task-aware Low Rank Adaptation of Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing low-rank Adaptation (LoRA) methods address only one factor, often at the cost of increased training complexity or reduced practical efficiency. |
| Approach: | They propose a low-rank Adaptation framework that optimizes initialization and resource allocation at the outset of training. |
| Outcome: | The proposed framework performs excellently across various tasks while reducing the number of trainable parameters. |
Behavioral Consistency Validation for LLM Agents: An Analysis of Trading-Style Switching through Stock-Market Simulation (2026.findings-acl)
Copied to clipboard
Zeping Li, Guancheng Wan, Keyang Chen, Yu Chen, Yiwen Zhao, Philip Torr, Guangnan Ye, Zhenfei Yin, Hongfeng Chai
| Challenge: | Recent studies have applied Large Language Models (LLMs) as agents in financial stock market simulations to test if micro-level behaviors aggregate into macro-level phenomena. |
| Approach: | They propose four alignment metrics and use Mann–Whitney U tests to compare agents’ style-switching behavior with financial theory. |
| Outcome: | The proposed model is only partially consistent with financial theory. |