Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Speech Language Models (LSLMs) typically operate at high token rates to ensure acoustic fidelity, yet this results in sequence lengths that exceed the underlying semantic content, incurring prohibitive inference costs. |
| Approach: | They propose a token-based token merging mechanism that uses a training-free token pooling mechanism to reduce prefilling FLOPs by 27.48% while maintaining competitive accuracy. |
| Outcome: | The proposed method reduces prefilling FLOPs by 27.48% while maintaining competitive accuracy. |
Similar Papers
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |
Reducing Token Redundancy in LVLMs: A Systematic Review of Token Pruning Methods (2026.acl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) excel at visual understanding but face severe computational bottlenecks when processing high-resolution images and long videos due to massive visual token counts. |
| Approach: | They propose a taxonomy categorizing methods into vision-side, LLM-side and hybrid paradigms and analyze token selection mechanisms and pruning strategy. |
| Outcome: | The proposed method selectively removes less informative tokens while maintaining performance. |
Bridging the Granularity Gap for Acoustic Modeling (2023.findings-acl)
Copied to clipboard
Chen Xu, Yuhao Zhang, Chengbo Jiao, Xiaoqian Liu, Chi Hu, Xin Zeng, Tong Xiao, Anxiang Ma, Huizhen Wang, Jingbo Zhu
| Challenge: | Despite the success of speech recognition, how to encode the speech features effectively remains an open problem. |
| Approach: | They propose a Progressive Down-Sampling technique which compresses acoustic features into coarser-grained units containing more complete semantic information, like text-level representation. |
| Outcome: | The proposed method yields comparable or better results on the speech recognition task and inference speedups ranging from 1.20x to 1.47x. |
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization (2026.acl-long)
Copied to clipboard
Negar Foroutan, Clara Meister, Debjit Paul, Joel Niklaus, Sina Ahmadi, Antoine Bosselut, Rico Sennrich
| Challenge: | Tokenization is the first step of most NLP pipelines. |
| Approach: | They propose a parity-aware byte pair encoder that maximizes the compression gain of the currently worst-compressed language for cross-lingual parity. |
| Outcome: | a new algorithm reduces tokenization inequality by 89% compared to classical BPE . the proposed algorithm is based on a fair-max rule that maximizes the compression gain of the currently worst-compressed language . |
Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | LSLMs have impressive conversational generation abilities, but consistently fall short of traditional pipeline systems on semantic understanding benchmarks. |
| Approach: | They propose to analyze the performance gap between speech and text inputs through a systematic experiment . they find that representation similarity is strongly correlated with the modality gap . |
| Outcome: | The proposed models improve the accuracy of speech inputs and their semantic understanding benchmarks. |
RedundancyLens: Revealing and Exploiting Visual Token Processing Redundancy for Efficient Decoder-Only MLLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Current decoder-only architectures achieve higher performance but lower efficiency . cross-attention-based architectures skip visual token computations . |
| Approach: | They propose a training-free framework for analyzing trained MLLMs to investigate redundancy . they propose 'probe-activated Dynamic FFN and Hollow Attention' algorithms for visual token reductions and a layer ranking algorithm for inference acceleration. |
| Outcome: | The proposed framework achieves comparable performance to or better than state-of-the-art methods while remaining compatible with them. |
Retrofitting Large Language Models with Dynamic Tokenization (2025.acl-long)
Copied to clipboard
| Challenge: | Current language models use a static tokenizer, which results in degraded efficiency and language capabilities. |
| Approach: | They propose a way to dynamically decide on token boundaries based on input text . they merge frequent subword sequences in a batch and apply a hypernetwork to compute token embeddings . |
| Outcome: | The proposed method reduces token sequence lengths by >20% across 14 languages while promoting fairness across languages. |
RepCodec: A Speech Representation Codec for Speech Tokenization (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have led to discrete speech tokenization, but this discretization can be costly and impedes performance. |
| Approach: | They propose a new speech representation codec for semantic speech tokenization that reconstructs speech representations from speech encoders like HuBERT or data2vec. |
| Outcome: | The proposed method outperforms the widely used k-means clustering approach in speech understanding and generation. |
Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal large language models have shown remarkable performance for cross-modal understanding and generation, yet suffer from severe inference costs. |
| Approach: | They propose to prune redundant tokens in MLLMs to reduce computation and storage costs. |
| Outcome: | The proposed method reduces the computational and storage costs of MLLMs by identifying redundant tokens and pruning them. |
Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects (2026.findings-acl)
Copied to clipboard
Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li
| Challenge: | Large Vision-Language Models are hindered by a systemic efficiency barrier known as visual token dominance. |
| Approach: | They propose a systematic taxonomy of efficiency techniques structured around the inference lifecycle . they examine visual encoding, prefilling, and decoding to understand bottlenecks . |
| Outcome: | The proposed techniques reveal how upstream decisions dictate downstream bottlenecks . the proposed techniques include hybrid compression and modality-aware decoding . |