Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Model (2025.acl-short)
Copied to clipboard
| Challenge: | Existing studies focus on translating a single SL into a spoken language (one-to-one SLT) however, multilingual SLT remains unexplored due to language conflicts and alignment difficulties across SLs and spoken languages. |
| Approach: | They propose a multilingual gloss-free model that can be used to translate a single SL into a spoken language and generate a token-level SL identification and spoken text. |
| Outcome: | The proposed model supports 10 SLs and handles one-to-one, many-to-1, and many- to-many SLT tasks. |
Similar Papers
Better Sign Language Translation with STMC-Transformer (2020.coling-main)
Copied to clipboard
| Challenge: | Current SLT approaches use a sign language recognition system to extract sign language glosses from videos. |
| Approach: | They propose to use a Sign Language Recognition system to extract sign language glosses from videos and a translation system to generate spoken language translations from the glossed sign language. |
| Outcome: | The proposed system outperforms existing methods on gloss-to-text and video-to text translations on the ASLG-PC12 corpus. |
Improving Multilingual Sign Language Translation with Automatically Clustered Language Family Information (2025.coling-main)
Copied to clipboard
| Challenge: | Recent research has focused on bilingual translation models, but multilingual sign language translation presents unique challenges due to the diversity of sign languages across nations. |
| Approach: | They propose a method that leverages sign language families to improve MSLT performance. |
| Outcome: | The proposed approach can achieve balance between translation accuracy and computational cost by regulating the number of language families. |
Neural Machine Translation Methods for Translating Text to Sign Language Glosses (2023.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art techniques common to low resource Machine Translation (MT) are applied to improve MT of spoken language text to Sign Language glosses. |
| Approach: | They propose to use data augmentation, semi-supervised Neural Machine Translation, transfer learning and multilingual NMT to improve MT of spoken language to Sign Language glosses. |
| Outcome: | The proposed models outperform previous work on two German SL corpora and are confirmed by human evaluation. |
An Efficient Gloss-Free Sign Language Translation Using Spatial Configurations and Motion Dynamics with LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for sign language translation rely on glosses, which are written representations of signs. |
| Approach: | They propose a new LLM-based SLT framework that uses off-the-shelf visual encoders to extract spatial and motion features from sign videos. |
| Outcome: | The proposed framework captures spatial configurations and motion dynamics in sign language without domain-specific tuning. |
Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Previous Sign Language Translation methods have relied on gloss annotations to improve performance, but labeling high-quality glosses is labor-intensive and inefficient. |
| Approach: | They propose to integrate Large Language Model (LLM) into SLT by factorizing learning into two stages to improve the learning curve. |
| Outcome: | The proposed approach improves on three SLT datasets conducted under the gloss-free setting. |
Can Small Vision–Language Models Perform Sign Language Translation? (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) have shown strong generalization across multimodal tasks, but their capacity to handle sign language translation (SLT) remains unclear. |
| Approach: | They propose entity- and semantics-aware metrics tailored for SLT to evaluate their performance. |
| Outcome: | The proposed metrics highlight the limitations of general-purpose VLMs to SLT, unlike their applicability in other tasks. |
Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sign language translation (SLT) assume video segments are directly mappable to spoken-language words. |
| Approach: | They propose a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between video and generated text. |
| Outcome: | The proposed model improves coherence and faithfulness over existing gloss-free methods. |
Sign Language Translation with Sentence Embedding Supervision (2024.acl-short)
Copied to clipboard
| Challenge: | State-of-the-art sign language translation systems facilitate learning through gloss annotations when available at scale. |
| Approach: | They propose to use sentence embeddings of the target sentences at training time that take the role of glosses to supervise the learning process. |
| Outcome: | The proposed method significantly outperforms gloss-free approaches on German and American sign languages and with mono- and multilingual sentence embeddings and translation systems. |
Gloss-Free End-to-End Sign Language Translation (2023.acl-long)
Copied to clipboard
| Challenge: | a study of sign language translation without gloss annotations focuses on the problem of gloss annotation . gloss annotation is hard to acquire, especially in large quantities, and limits the domain coverage of translation datasets . |
| Approach: | They propose a gloss-free end-to-end sign language translation framework to solve this problem . gloss annotations are hard to acquire, especially in large quantities, they argue . |
| Outcome: | The proposed framework improves sign language translation performance on large-scale datasets . gloss annotations are hard to acquire, especially in large quantities . |
Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to sign language translation use gloss annotations as an intermediary . a new approach to use large language models and word embeddings to improve Gloss2Text translation is needed. |
| Approach: | They propose to leverage large language models pre-trained on expansive and diverse corpora to improve Gloss2Text translation stage by using data augmentation and label-smoothing loss function. |
| Outcome: | The proposed approach surpasses state-of-the-art methods on the PHOENIX Weather 2014T dataset . it shows that gloss annotations can be used to guide the translation process . |