Papers with MQA
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multi-query attention (MQA) can lead to quality degradation and training instability . it may not be feasible to train separate models optimized for quality and inference. |
| Approach: | They propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original training compute. |
| Outcome: | The proposed model achieves comparable quality to multi-head attention with comparable speed. |
MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding (2025.findings-naacl)
Copied to clipboard
| Challenge: | Multi-Layer Key-Value (MLKV) sharing reduces memory usage by 6x compared to Multi-Query Attention and Grouped-Query Attributes. |
| Approach: | They propose a novel approach that extends KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention and Grouped-Query Attributes. |
| Outcome: | The proposed approach reduces KV cache size by 6x with minimal performance loss and scales linearly with model size, batch size, and sequence length. |
MQA-KEAL: Multi-hop Question Answering under Knowledge Editing for Arabic Language (2025.coling-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated significant capabilities across numerous application domains. |
| Approach: | They propose to use Multi-hop Questioning Answering under Knowledge Editing for Arabic Language to update and/or edit prior knowledge and test it via Multi-Hop Question Answering (MQA). |
| Outcome: | The proposed model outperforms baseline models by a significant margin . it can be used to update and/or edit prior knowledge and then test it with MQA . |
PokeMQA: Programmable knowledge editing for Multi-hop Question Answering (2024.acl-long)
Copied to clipboard
| Challenge: | Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine’s comprehension and reasoning abilities, where large language models (LLMs) have widely achieved the human-comparable performance. |
| Approach: | They propose a framework to edit multi-hop question models to update model with up-to-date facts while avoiding expensive re-training or fine-tuning. |
| Outcome: | The proposed framework outperforms all competitors in multi-hop question answering tasks and consistently produces reliable reasoning process. |
From Query to Logic: Ontology-Driven Multi-Hop Reasoning in LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit limitations in complex multi-hop question answering tasks that necessitate non-linear, structured reasoning. |
| Approach: | They propose an ontology-driven reasoning and chain framework that combines LLMs’ generative capabilities with the structural benefits of knowledge graphs. |
| Outcome: | Extensive experiments across a diverse set of models and standard MQA benchmarks demonstrate that the proposed framework achieves competitive performance while producing more interpretable reasoning chains. |
Few-shot Reranking for Multi-hop QA via Language Model Prompting (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multi-hop QA with open-domain questions require a large number of labeled question-document pairs for retrieval. |
| Approach: | They propose a language-based prompt for multi-hop path reranking that relies on language model prompting to generate a relevance score between a question and the path. |
| Outcome: | The proposed method yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples. |