Papers with FFNs
Infinity-MoE: Generalizing Mixture of Experts to Infinite Experts (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing methods to increase the number of experts are -MoE and . |
| Approach: | They propose a mixture of experts that selects a few feed-forward networks per token to increase the number of experts. |
| Outcome: | The proposed model improves on a GPT-2 Small model with 129M active and 186M total parameters by 2.5% over the current model. |
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. |
| Approach: | They propose to convert a model into its MoE version with the same parameters and build expert routers to decide which experts will be used for each input. |
| Outcome: | The proposed model can use 10% to 30% of FFN parameters while maintaining over 95% original performance. |
MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter (2024.acl-long)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-tuning (PEFT) methods are limited on knowledge-intensive tasks due to the limited number of trainable parameters. |
| Approach: | They propose a mechanism that fine-tunes Large Language Models with larger adapters . they store and update the parameters of larger adapter adapters on the CPU . |
| Outcome: | The proposed method achieves comparable results to those obtained with larger memory capacities over the limited bandwidth of PCI Express (PCIe). |
TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors (2026.acl-long)
Copied to clipboard
| Challenge: | Existing attention-aggregation methods focus on individual attention heads or layers, failing to account for the model’s global behavior. |
| Approach: | They propose a unified attention representation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction tensor. |
| Outcome: | The proposed model encapsulates the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction tensor. |
Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis (2026.acl-long)
Copied to clipboard
Zehua Pei, Hui-Ling Zhen, Lancheng Zou, Xianzhi Yu, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu
| Challenge: | Large language models (LLMs) are fast but require expensive pre-training . a new approach to scale large language models into MoEs reduces inference costs . |
| Approach: | They propose an analytical post-training framework that rapidly restructures FFNs into sparse MoE architectures using only a small calibration dataset. |
| Outcome: | The proposed framework outperforms existing methods on a small calibration dataset. |
Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models’ Memories (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models demonstrate excellent abilities to understand texts in the generic domain while struggling in a specific domain. |
| Approach: | They propose to decouple the feed-forward networks of the Transformer architecture into two parts to maintain old-domain knowledge and a mixture-of-adapters gate to inject domain-specific knowledge in parallel. |
| Outcome: | The proposed method achieves superior performance on in-domain, out-of-domain and knowledge-intensive tasks. |
PartialFormer: Modeling Part Instead of Whole for Machine Translation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing feed-forward neural networks have significant computational and parametric overhead. |
| Approach: | They propose a parameter-efficient Transformer architecture that utilizes multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. |
| Outcome: | The proposed architecture reduces computational and parameter overhead while maintaining essential hidden dimensions. |
Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignment (2023.findings-emnlp)
Copied to clipboard
Boyang Xue, Weichao Wang, Hongru Wang, Fei Mi, Rui Wang, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong
| Challenge: | Experimental results show that pretrained language models generate inconsistent factual knowledge in many conversational tasks. |
| Approach: | They propose a method which explicitly introduces extended feedforward networks (FFNs) in Transformers to enhance factual knowledge expressions given the specific patterns of knowledge-grounded dialogue inputs. |
| Outcome: | The proposed methods improve the factual expression capability of feedforward networks (FFNs) in knowledge-grounded dialogue systems by knowledge enhancement and alignment respectively. |
Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models acquire and store factual knowledge for interpretability, reliability, efficiency . prior work on factual recall focused on localizing knowledge within transformer parameters . |
| Approach: | They analyze the evolution of factual knowledge representation in a large language model by tracking its attention heads and feed forward networks over training. |
| Outcome: | The proposed model acquires and stores factual knowledge over time and is adaptively trained . the proposed model can be pruned, optimized, and transparent . |
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-Training (2024.emnlp-main)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) has gained increasing popularity as a framework for scaling up large language models. |
| Approach: | They investigate how to build Mixture-of-Experts (MoE) models from existing large language models . they use expert construction, Continual pre-training and data sampling strategies . |
| Outcome: | The proposed model outperforms existing models with similar parameters on a wide range of tasks. |
MemTR: Enhancing Tool-Calling Reliability via Uncertainty-Triggered FFN-Space Retracing (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing tool-calling methods rely on costly tool-use training data or only constrain syntax, leaving tool selection and argument value errors largely unsolved. |
| Approach: | They propose a method that decodes tool evidence from the tool library and mixes it into the output at the uncertain layer. |
| Outcome: | The proposed method reduces tool calling failures by 2%–9% with only 1%–2% runtime overhead. |
TPA: Next Token Probability Attribution for Detecting Hallucinations in RAG (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches attribute hallucinations to a binary conflict between internal knowledge stored in FFNs and the retrieved context. |
| Approach: | They propose a framework which mathematically attributes each next-token probability to seven distinct sources and aggregates source attributions by POS tags to quantify contribution of each model component to the generation of specific linguistic categories within a response. |
| Outcome: | Extensive experiments show that the proposed framework achieves state-of-the-art performance. |
Detecting What Queries Seek: Steering LLM Safety with FFN Output Activation Monitoring (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for detecting malicious queries rely on residual stream activations, resulting in limited discriminative power and unreliable interventions. |
| Approach: | They propose to use feed-forward networks (FFNs) to generate more discriminative signals for intervention, since these activations more explicitly reflect the intent of a query. |
| Outcome: | Experiments show that the proposed approach achieves state-of-the-art defense performance against various jailbreak attacks while maintaining the model's original performance on benign tasks. |
CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Efficient inference in Large Vision Language Models is constrained by the high cost of processing thousands of visual tokens. |
| Approach: | They propose a framework that prunes visual tokens using attention contribution at critical functional transitions and reduces computations using efficient linear approximations. |
| Outcome: | The proposed framework achieves competent efficiency–performance trade-offs with improved robustness. |
TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating external knowledge into language models do not prioritize learning embeddings for entity-related tokens. |
| Approach: | They propose a framework for incorporating external knowledge into pre-training models that utilize entity-related tokens. |
| Outcome: | The proposed framework reduces pre-training time by 50% and outperforms other KEPLMs in knowledge probing tasks and multiple knowledge-aware language understanding tasks. |
Spectral Scaling Laws in Language Models: emphHow Effectively Do Feed-Forward Networks Use Their Latent Space? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. |
| Approach: | They propose to reframe model width selection as a spectral utilization optimization problem . they quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families . |
| Outcome: | The proposed model maximizes the capacity of feed-forward networks by recasting the problem as a spectral utilization optimization problem. |