MSCode: Advancing Human Motion-Language Understanding via Modality-Shared Codebook (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for motion understanding lack precise alignment between motion and modalities . existing methods lack precise semantics and a mismatch between motion, text . |
| Approach: | They propose a modality-shared codebook that enables unified representation learning and precise alignment between motion and linguistic modalities. |
| Outcome: | The proposed model surpasses current state-of-the-art methods in many areas . it enables unified representation learning and precise alignment of motion and modalities . |
Similar Papers
CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal representation alignment is crucial for large language models and robotics. |
| Approach: | They propose a framework that optimizes multimodal representation spaces through a modality-shared-specific codebook design. |
| Outcome: | The proposed framework achieves state-of-the-art performance in multimodal classification and retrieval tasks. |
Deep Generative Model for Joint Alignment and Word Representation (N18-1)
Copied to clipboard
| Challenge: | EmbedAlign model embeds words in their complete observed context and learns by marginalisation of latent lexical alignments. |
| Approach: | They exploit translation as a distributional context and embed words as posterior probability densities, rather than point estimates, which allows them to compare words in context using a measure of overlap between distributions. |
| Outcome: | The proposed model performs on a range of lexical semantics tasks and achieves competitive results on benchmarks including natural language inference, paraphrasing, and text similarity. |
Enhancing Multimodal Unified Representations for Cross Modal Generalization (2025.findings-acl)
Copied to clipboard
Hai Huang, Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Minghui Fang, Jieming Zhu, Zhenhua Dong, Sashuai Zhou, Zhou Zhao
| Challenge: | Existing studies on discrete unified representations overlook important distinctions between different dimensions of features. |
| Approach: | They propose to use a codebook to optimize unified representations from pretraining and fine- and coarse-grained disentangling to optimize the representations. |
| Outcome: | The proposed methods improve the interpretability of multimodal unified representations . they use training-free optimization of codebook and fine and coarse cross-modal disentangling . |
Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that deep vision-only and language-only models project inputs into a partially aligned representational space. |
| Approach: | They investigate whether a model's representational code is semantically shared . they find that alignment peaks in mid-to-late layers of both model types . |
| Outcome: | a forced-choice "Pick-a-Pic" task shows human preferences for image-caption matches are mirrored in embedding spaces across vision-language model pairs. |
Modality Alignment between Deep Representations for Effective Video-and-Language Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing Video-and-Language models do not take into account the different characteristics of video and text representations. |
| Approach: | They propose a method that exploits Centered Kernel Alignment (CKA) to enhance cross-modality attention by combining multiple modalities. |
| Outcome: | The proposed method outperforms conventional multi-modal methods significantly on video QA tasks with +3.57% accuracy increment compared to the baseline in a popular benchmark dataset. |
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to visual-language understanding lack unified tokenization for images and videos . lack of unified visual representations makes it difficult to learn multi-modal interactions from poor projection layers. |
| Approach: | They propose to unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. |
| Outcome: | The proposed model outperforms Video-ChatGPT on image benchmarks and on 9 image benchmark benchmarks. |
Text-centric Alignment for Bridging Test-time Unseen Modality (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a text-centric alignment method is used to handle unseen modalities and dynamic modality combinations at test time. |
| Approach: | They propose a text-centric alignment method that unifies different input modalities into a single semantic text representation by leveraging in-context learning with Large Language Models and uni-modal foundation models. |
| Outcome: | The proposed method unifies input modalities into a single semantic representation . it significantly improves the ability to manage unseen, diverse, and unpredictable modality combinations . |
Can Textual Unlearning Solve Cross-Modality Safety Alignment? (2024.findings-emnlp)
Copied to clipboard
Trishna Chakraborty, Erfan Shayegani, Zikui Cai, Nael Abu-Ghazaleh, M. Salman Asif, Yue Dong, Amit Roy-Chowdhury, Chengyu Song
| Challenge: | integrating new modalities into large language models creates new attack surface . existing safety training techniques like SFT and RLHF are not feasible in multi-modal settings . |
| Approach: | They explore whether unlearning in the textual domain can be effective for cross-modality safety alignment. |
| Outcome: | The proposed approach reduces the Attack Success Rate (ASR) to less than 8% and preserves the utility. |
Cross-Modal Discrete Representation Learning (2022.acl-long)
Copied to clipboard
| Challenge: | a new framework for learning representations from multimodal data is proposed . the proposed framework uses discretized embedding vectors to capture finer levels of granularity . |
| Approach: | They propose a self-supervised representation learning framework that captures finer levels of granularity across different modalities. |
| Outcome: | The proposed representation can capture finer levels of granularity across different modalities . it can be used on cross-modal retrieval tasks without direct supervision . |