Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Decoder-only large language models are brittle to minor grammatical perturbations, causing reliability problems. |
| Approach: | They propose a checkpoint-compatible gated tree cross-attention branch that reads constituency chunk memory while keeping the backbone architecture unchanged. |
| Outcome: | The proposed framework strengthens syntactic competence beyond continued training benchmarks and transformer backbones without compromising commonsense reasoning. |
Similar Papers
Bitune: Leveraging Bidirectional Attention to Improve Decoder-Only LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Decoder-only large language models rely on masked causal attention, which limits expressiveness by restricting information flow to one direction. |
| Approach: | They propose a method that incorporates bidirectional attention into prompt processing to enhance pretrained decoder-only LLMs. |
| Outcome: | The proposed method shows significant improvements in commonsense reasoning, arithmetic, and language understanding tasks. |
Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors (2026.acl-long)
Copied to clipboard
Rui Yin, Tianxu Han, Naen Xu, Changjiang Li, Ping He, Chunyi Zhou, Jun Wang, Zhihui Fu, Tianyu Du, Jinbao Li, Shouling Ji
| Challenge: | Existing methods to inject safety-aligned large language models rely on token-level mappings, which do not guarantee sustained harmful output. |
| Approach: | They propose a method that directly modifies model weights to map a trigger to an attacker-specified response. |
| Outcome: | The proposed method achieves high triggered attack success while maintaining non-triggered safety and general utility. |
SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with safe-alignment training are vulnerable to jailbreak attacks, causing malicious users to generate harmful outputs. |
| Approach: | They propose a safe-alignment jailbreak method that bypasses the middle-to-late layers of large language models by a residual connection. |
| Outcome: | The proposed method improves by 51% over the best performing baseline GCG on HarmBench test set. |
Evolving Sparsity: Leveraging Token Importance Dynamics for Efficient LLM Decoding with Sparse Attention (2026.acl-long)
Copied to clipboard
| Challenge: | Efficient long-context inference remains a major challenge for large language models (LLMs), as the cost of attention computation during auto-regressive decoding grows linearly with the context length. |
| Approach: | They propose to model token importance as a dynamic process that evolves over decoding steps and propagates through model layers. |
| Outcome: | The proposed method outperforms baseline sparse attention methods and achieves speedups of up to 5.36 for attention latency and 2.33 for end-to-end decoding. |
Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Jailbreak attacks pose persistent threats to large language models . current safety alignment methods have insufficient safety alignment depth and unrobust internal defense mechanisms. |
| Approach: | a new safety alignment framework is developed to overcome jailbreak attacks . the framework forces the model to dynamically rebuild its refusal mechanisms from jailbreak states . |
| Outcome: | a new safety alignment framework reduces attack success rates by approximately 95% on four open-source LLM families and six representative attacks. |
TransLLaMa: LLM-based Simultaneous Translation System (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Decoder-only large language models have limited applications in simultaneous machine translation . naively translating each source word immediately results in compromised target quality . |
| Approach: | a study shows that a pre-trained open-source LLM can control input segmentation directly by generating a special "wait" token. |
| Outcome: | a new open-source model can control input segmentation directly by generating a special "wait" token. |
Gamma-Guard: Lightweight Residual Adapters for Robust Guardrails in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are widely deployed as zero-shot evaluators for answer grading, content moderation, and document ranking. |
| Approach: | They propose a system that trains LLMs with adapters to denoise embeddings and refocus attention. |
| Outcome: | The proposed model lifts adversarial accuracy from 5% to 95% a 90 percentage-point gain while reducing clean-data accuracy by just 8 percentage points. |
Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models learn non-trivial abstractions during pretraining, but it is not well understood when and how these specific linguistic abilities emerge. |
| Approach: | They propose a method to track the evolution of linguistic features during pretraining by using sparse crosscoders to discover and align features across model checkpoints. |
| Outcome: | The proposed approach can detect features emergence, maintenance, and discontinuation during training stages. |
FLASH: Focused Layer Attention Sink Hijacking (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) remain vulnerable to jailbreaking attacks despite advances in safety alignment . |
| Approach: | They propose a new diagnostic auditing framework that dismantles the model's internal safety anchor by precisely scaling attention scores in these vulnerable layers. |
| Outcome: | The proposed framework achieves a state-of-the-art Attack Success Rate of over 77% with an unprecedented efficiency of 1.53 queries on average. |
Deputy: Accelerating Large Language Model Inference with Dynamic Low-Rank Substitution (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing dynamic schemes such as early-exit and layer-drop reduce FLOPs but break batch processing or introduce KV-cache inconsistency. |
| Approach: | They propose a dynamic low-rank substitution framework that employs a lightweight decision module at each layer to dynamically determine the execution branch for different tokens. |
| Outcome: | The proposed model reduces computation by approximately 40% compared to the original dense model while outperforming existing baseline methods. |