Bridging the Temporal Gap in Multimodal LLMs: Deeply Stacking Temporal Tokens for Audio-Visual Speech Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing audio-visual speech recognition systems suffer from a temporal gap . visual speech patterns captured from lip movements provide complementary information that remains inherently robust to acoustic noise. |
| Approach: | They propose a framework that deeply stacks temporal tokens across both encoding and decoding stages to bridge this temporal gap. |
| Outcome: | The proposed framework outperforms existing supervised, self-supervised, and LLM-based methods by 6.1% on LRS2 and 7.8% on LLS3. |
Similar Papers
SEAM: Bridging the Temporal-Semantic Granularity Gap for LLM-based Speech Recognition (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing duration-based methods generate embeddings at fixed rates, creating distributional mismatch with LLM pre-training. |
| Approach: | They propose an encoder-decoder architecture that generates embeddings at variable rates through cross-attention between speech features and text embeddables. |
| Outcome: | The proposed architecture achieves competitive performance on LibriSpeech (2.6%/5.2% WER) and 4.7% WER on TED-LIUM-v2 with a multi-stage training strategy and First Token Guidance. |
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent Large Language Model (LLM) based AVSR systems incur high computational costs due to high temporal resolution of audio-visual speech. |
| Approach: | They propose an efficient multimodal speech LLM framework that minimizes token length while preserving essential linguistic content. |
| Outcome: | The proposed approach reduces token usage by 86% while using only 3.5 tokens per second. |
Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for audio-visual speech recognition use extra data to increase performance . a recent study shows that the use of unimodal self-supervised learning improves performance on multimodal tasks. |
| Approach: | They propose to use unimodal self-supervised learning to train AVSR models on unlabelled unilateral data. |
| Outcome: | The proposed model improves on lip reading sentences 2 by 30% even without an external language model. |
Temporal Token Matters: Investigating and Interpreting the Consistency of Temporal Ordering in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) exhibit notable deficiencies in temporal reasoning . phrasing changes can lead LLMs to produce inconsistent outputs . |
| Approach: | They investigate the mechanistic interpretability of temporal ordering within event temporal reasoning . they identify a sparse subset of attention heads that are causally responsible for reasoning outcomes . |
| Outcome: | The proposed model outperforms other models in a variety of tasks and is validated by intervention-based experiments. |
TemporalVLM: Video LLMs for Temporal Reasoning in Long Videos (2026.findings-acl)
Copied to clipboard
| Challenge: | Several video understanding applications require the ability of temporal reasoning. |
| Approach: | They propose a video large language model for temporal reasoning and fine-grained understanding in long videos. |
| Outcome: | The proposed model outperforms existing methods in time and motion studies and temporal action segmentation evaluations. |
VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) suffer from significant computational overhead due to the quadratic growth of attention computations with the number of multimodal tokens. |
| Approach: | They propose a training-free pruning framework that prunes multimodal tokens without a trained pruning method. |
| Outcome: | The proposed pruning framework outperforms existing token pruning methods and generalizes across diverse MLLMs. |
Mitigating the Discrepancy Between Video and Text Temporal Sequences: A Time-Perception Enhanced Video Grounding method for LLM (2025.coling-main)
Copied to clipboard
| Challenge: | Existing video LLMs excel at capturing the overall description of a video but lack the ability to demonstrate an understanding of temporal dynamics and localized content within the video. |
| Approach: | They propose a Time-Perception Enhanced Video Grounding via Boundary Perception and Temporal Reasoning to improve LLMs' understanding of video temporality. |
| Outcome: | The proposed method improves on three datasets: ActivityNet, Charades, and DiDeMo (up to 11.2% improvement on R@0.3). |
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models (2025.findings-emnlp)
Copied to clipboard
Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, Lifu Huang
| Challenge: | Video Large Language Models (VLMs) have been praised for their performance in coarse-grained video understanding but still face ineffective temporal grounding and inadequate timestamp representations. |
| Approach: | They propose a novel Video-LLM that senses and reasoned over specific video moments with fine-grained temporal precision. |
| Outcome: | The proposed model surpasses existing models in fine-grained video understanding tasks and exhibits strong potential as a general video understanding assistant. |
Markovian Linguistic-Temporal Bridge: Unlocking the Potential of LLMs for Time Series Forecasting (2026.acl-long)
Copied to clipboard
| Challenge: | Pretrained Large Language Models (LLMs) are based on token-level linguistic-temporal alignment, leading to stacking of logically disjointed tokens as input. |
| Approach: | They propose a framework that distills latent evolutionary patterns of language into a Markovian state transition graph, which is transferred as a structural prior to the time series domain. |
| Outcome: | The proposed framework achieves global structural isomorphism between the linguistic and temporal domains. |
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Audio-visual speech recognition (AVSR) leverages multimodal signals to understand human speech. |
| Approach: | They propose an adversarial network to refine frame-level modality-invariant representations to bridge the distribution gap between modalities. |
| Outcome: | The proposed approach outperforms the state-of-the-art on public benchmarks LRS3 and LRS2 on the modalities of AVSR. |