Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Current sign language recognition methods use spatial graphs and temporal modules to capture spatial and temporal features, but their spatial graph modules are typically built on fixed graph structures. |
| Approach: | They propose a new spatial architecture that captures input-sensitive joint relationships and a temporal module to model multi-scale temporal information to capture complex human dynamics. |
| Outcome: | The proposed method achieves state-of-the-art accuracy on four large-scale SLR benchmarks. |
Similar Papers
Multi-Channel Spatio-Temporal Transformer for Sign Language Production (2024.lrec-main)
Copied to clipboard
| Challenge: | Sign language production models ignore structural correlations between channels and use multi-channel spatial attention to capture correlations across channels. |
| Approach: | They propose a novel approach to transform sign language into a unified feature representation using multi-channel spatial attention and temporal attention to learn sequential dependencies for each channel over time. |
| Outcome: | The proposed model outperforms state-of-the-art models on two sign language datasets from diverse cultures. |
Improving Handshape Representations for Sign Language Processing: A Graph Neural Network Approach (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for sign language recognition process a signing sequence holistically, leaving handshape information implicit, which limits both recognition accuracy and linguistic analysis. |
| Approach: | They propose a graph neural network that separates temporal dynamics from static handshape configurations in continuous signing sequences. |
| Outcome: | The proposed approach achieves 46% accuracy across 37 handshape classes, compared to 25% for baseline methods. |
A Low-Cost Motion Capture Corpus in French Sign Language for Interpreting Iconicity and Spatial Referencing Mechanisms (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for automatic translation of sign language videos into transcribed texts are limited. |
| Approach: | They propose to use deep learning methods to circumvent the use of models in spatial referencing recognition by a 3D skeleton and a software program to capture and post-process the LSF-SHELVES corpus. |
| Outcome: | The proposed system targets iconicity and spatial referencing in french sign language . it is light-weight and low-cost to collect data from a large panel of signers . |
TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that TESTA reduces the number of visual tokens by 75% and thus accelerates video encoding. |
| Approach: | They propose a method to condense video semantics by aggregating similar frames and patches within each frame. |
| Outcome: | The proposed method reduces visual tokens by 75% and accelerates video encoding. |
Bridging the Temporal Gap in Multimodal LLMs: Deeply Stacking Temporal Tokens for Audio-Visual Speech Recognition (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing audio-visual speech recognition systems suffer from a temporal gap . visual speech patterns captured from lip movements provide complementary information that remains inherently robust to acoustic noise. |
| Approach: | They propose a framework that deeply stacks temporal tokens across both encoding and decoding stages to bridge this temporal gap. |
| Outcome: | The proposed framework outperforms existing supervised, self-supervised, and LLM-based methods by 6.1% on LRS2 and 7.8% on LLS3. |
Bidirectional Skeleton-Based Isolated Sign Recognition using Graph Convolutional Networks (2022.lrec-1)
Copied to clipboard
| Challenge: | a new method for computer-based sign recognition from video is proposed . it involves explicit detection of the start and end frames of isolated signs . |
| Approach: | They propose a skeleton-based method that involves explicit detection of start and end frames of signs . they apply a modified WLASL dataset with corrections to the gloss labeling . |
| Outcome: | The proposed method outperforms state-of-the-art methods on the modified WLASL dataset . it has a success rate of 77.43% and 94.54% for top-5 . |
Spatial and Temporal Language Understanding: Representation, Reasoning, and Grounding (2024.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
| Approach: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
| Outcome: | This tutorial provides an overview of cutting edge research on spatial and temporal language understanding. |
GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extraction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to analyzing large language models are limited by their pre-trained knowledge of Small Language Models(SLMs). |
| Approach: | They propose a global distance-aware modeling approach based on Large Language Models (LLMs) they first present a distance-based graph structure utilizing Graph Attention Network(GAT) |
| Outcome: | The proposed approach significantly improves minority relation class learning ability on two publicly available datasets, TB-Dense and MATRES. |
Synergizing Multimodal Temporal Knowledge Graphs and Large Language Models for Social Relation Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have limited capacity to model complex graph-structured relationships. |
| Approach: | They propose a low-coupling method synergizing multimodal temporal Knowledge Graphs and Large Language Models for social relation reasoning. |
| Outcome: | The proposed method exhibits state-of-the-art performance in social relation recognition . it bridges the gap between KGs and LLMs and will be released after acceptance . |
WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language (2022.acl-short)
Copied to clipboard
| Challenge: | Signed Language Processing (SLP) is a major form of NLP, but has been overlooked by the NLP community. |
| Approach: | They leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. |
| Outcome: | The proposed model outperforms existing approaches on signs unobserved during training. |