Papers by Tengfei Yu
Self-Powered LLM Modality Expansion for Large Speech-Text Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models exhibit remarkable performance across diverse tasks . however, these methods require significant resource demands and tend to overfit specific tasks. |
| Approach: | They propose a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning. |
| Outcome: | The proposed model mitigates speech anchor bias and improves the fusion of speech and text modalities in large language models. |
Curriculum Consistency Learning for Conditional Sentence Generation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Consistency learning (CL) has proven to be a valuable technique for improving the robustness of conditional sentence generation models. |
| Approach: | They propose a strategy that guides models to learn consistency in alignment with their current capacity to differentiate between features. |
| Outcome: | The proposed strategy delivers +2.0 accuracy point improvement compared with vanilla IT and +0.7 COMET scores over traditional CL methods in MT tasks. |
Imagination and Contemplation: A Balanced Framework for Semantic-Augmented Multimodal Machine Translation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal Machine Translation (MMT) is effective in resolving linguistic ambiguities, but visual information often introduces redundancy or noise, potentially impairing translation quality. |
| Approach: | They propose a semantic-augmented framework that integrates "Imagination" and "Contemplation" they first generate synthetic images from source text and align them with authentic images via an optimal transport loss . |
| Outcome: | The proposed framework outperforms baselines on translation datasets with visually ambiguous or weakly correlated content. |
Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for timeline summarization ignore the events’ intra-structures and inter-structure connections. |
| Approach: | They propose to represent news articles as an event-graph, thus compressing the whole graph to its salient sub-graph. |
| Outcome: | The proposed method significantly improves on the state-of-the-art on three real-world datasets, including two public benchmarks and a Timeline100 dataset. |
PromptST: Abstract Prompt Learning for End-to-End Speech Translation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show that PromptST can improve speech-to-text translation by capturing richer linguistic knowledge. |
| Approach: | They propose a plug-in prompt-enhanced S2T model that captures richer linguistic knowledge . they use a 10GB linguistic probing benchmark to investigate the fusion of speech and text features . |
| Outcome: | The proposed model can improve on a strong baseline by capturing richer linguistic knowledge. |
Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation (2024.acl-long)
Copied to clipboard
| Challenge: | End-to-end speech translation (ST) models require simultaneous crossmodal and crosslingual transformations to be effective. |
| Approach: | They propose a homophone-aware contrastive learning approach that integrates a speech-text masking strategy to reduce ambiguity. |
| Outcome: | The proposed approach achieves SOTA results on BLEU scores on different MuST-C and CoVoST ST tasks, underlining its effectiveness in reducing speech sense ambiguity. |