Papers by Jan Niehues

Beyond Transcripts: A Renewed Perspective on Audio Chaptering (2026.acl-long)

Challenge:	Existing studies have shown that text-generation models can be overconfident when there are multiple correct options.
Approach:	They propose a QE approach called BoostedProb which boosts the model’s confidence in cases where there are multiple viable output options.
Outcome:	The proposed approach achieves on average +0.194 improvement in Pearson correlation to ground-truth quality and outperforms more costly approaches like supervised or ensemble-based QE in certain settings.

Copied to clipboard

Fabian Retkowski, Maike Züfle, Thai Binh Nguyen, Jan Niehues, Alexander Waibel

How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models? (2024.eacl-long)

Challenge:	despite its relevance, research on audio chaptering remains limited and predominantly textbased . authors: audio chapterers can't be used linearly because they skim, scrub timelines, jump to relevant moments . acoustic features and learning representations are not used for audio chapterer evaluation .
Approach:	They propose to use audio-only architecture to automatically segment audio into coherent sections . they compare audio-based models with acoustic features and a novel audio-oriented architecture .
Outcome:	The proposed audio-only architecture outperforms text-based approaches on acoustic features and LLMs.

Copied to clipboard

Danni Liu, Jan Niehues

Tutorial: End-to-End Speech Translation (2021.eacl-tutorials)

Challenge:	Pretrained multilingual translation models with massive coverage are becoming of the backbone of many translation systems.
Approach:	They propose to use a gradient-based inference-time controller to control a pretrained multilingual model by using a model with attribute annotations.
Outcome:	The proposed model performs well on pretrained multilingual models and is attribute- rather than language-specific.

Copied to clipboard

Jan Niehues, Elizabeth Salesky, Marco Turchi, Matteo Negri

Summarizing Speech: A Comprehensive Survey (2025.emnlp-main)

Challenge:	Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation.
Approach:	This tutorial introduces the techniques used in cutting-edge research on speech translation.
Outcome:	The proposed models achieve state-of-the-art performance with end-to-end speech translation for both high- and low-resource languages.

Copied to clipboard

Fabian Retkowski, Maike Züfle, Andreas Sudmann, Dinah Pfau, Shinji Watanabe, Jan Niehues, Alexander Waibel

Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks (2025.emnlp-main)

Challenge:	Podcasts and other audiovisual content are becoming more and more a part of everyday communication and the digital age is changing from text to voice.
Approach:	They synthesize the current state of the field and highlight the need for realistic evaluation benchmarks and multilingual datasets.
Outcome:	The proposed frameworks are based on evaluation protocols and datasets and highlight the need for realistic benchmarks and multilingual datasets.

Copied to clipboard

Supriti Sinhamahapatra, Jan Niehues

Early-Exit and Instant Confidence Translation Quality Estimation (2026.eacl-long)

Challenge:	Current automatic speech recognition systems rely on only audio information, ignoring multi-modal context.
Approach:	They propose to integrate visual context into existing automatic speech recognition systems to integrate presentation slides with multi-modal information.
Outcome:	The proposed model reduces word error rate by approximately 34% across all words and 35% for domain-specific terms compared to baseline model.

Copied to clipboard

Vilém Zouhar, Maike Züfle, Beni Egressy, Julius Cheng, Mrinmaya Sachan, Jan Niehues

Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach (2024.emnlp-main)

Challenge:	Quality estimation models are often opaque and computationally expensive, making them impractical to be part of large-scale pipelines.
Approach:	They propose an uncertainty-aware quality estimation model that matches previous approaches at a fraction of their costs.
Outcome:	The proposed method reduces evaluation costs by 50% and improves reranking performance.

Copied to clipboard

Siqi Li, Danni Liu, Jan Niehues

Multimodal In-context Learning for ASR of Low-resource Languages (2026.findings-acl)

Challenge:	Incorrect translation of rare words can severely degrade the accuracy of ST models .
Approach:	They propose a retrieval-and-demonstration approach to enhance rare word translation accuracy in ST models by incorporating retrieved examples into ST models.
Outcome:	The proposed approach outperforms other modalities and exhibits higher robustness to unseen speakers.

Copied to clipboard

Zhaolin Li, Jan Niehues

Improving Zero-Shot Translation by Disentangling Positional Information (2021.acl-long)

Challenge:	In-context learning with large language models addresses this limitation, but prior work focuses on high-resource languages covered during training and text-only settings.
Approach:	They propose to use multimodal ICL to learn unseen languages with multimodal learning to improve ASR in large language models.
Outcome:	The proposed model outperforms existing models on unseen languages with multimodal ICL (MICL) and cross-lingual transfer learning matches or outperformed models without using target-language data.

Copied to clipboard

Danni Liu, Jan Niehues, James Cross, Francisco Guzmán, Xian Li

Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation (2024.lrec-main)

Challenge:	Multilingual neural machine translation has shown the capability of directly translating between language pairs unseen in training, i.e. zero-shot translation.
Approach:	They propose to remove residual connections in an encoder layer to reduce the difficulty of generalizing to new translation directions.
Outcome:	The proposed model outperforms pivot-based translation in terms of quality and ease of integration of new languages.

Copied to clipboard

Matthias Sperber, Ondřej Bojar, Barry Haddow, Dávid Javorský, Xutai Ma, Matteo Negri, Jan Niehues, Peter Polák, Elizabeth Salesky, Katsuhito Sudoh, Marco Turchi

Machine Translation

Challenge:	a meta-analysis of human evaluation for speech translation has not been conducted . noisy data and segmentation mismatches are challenges for automatic metrics .
Approach:	They propose an evaluation strategy based on automatic resegmentation and direct assessment with segment context.
Outcome:	The proposed evaluation strategy is robust and scores well-correlated with other types of human judgements.

Speech Recognition Corpus of the Khinalug Language for Documenting Endangered Languages (2024.lrec-main)

Copied to clipboard

Zhaolin Li, Monika Rind-Pawlowski, Jan Niehues

F-Actor: Controllable Conversational Behavior in Full-Duplex Models (2026.findings-acl)

Challenge:	Existing tools to document endangered languages are limited due to data scarcity and the need for training.
Approach:	They propose to use a speech corpus for Khinalug, an endangered language spoken in northern Azerbaijan, to create a model that can be used in language documentation scenarios.
Outcome:	The proposed model achieves 6.65 CER points and 25.53 WER points in low-resource scenarios.

Copied to clipboard

Maike Züfle, Ondrej Klejch, Nicholas Sanders, Jan Niehues, Alexandra Birch, Tsz Kin Lam

Dialogue Systems & Conversational Agents

Challenge:	Current spoken conversational systems lack customization capabilities, limiting their naturalness and usability.
Approach:	They propose an instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints.
Outcome:	The proposed model requires just 2,000 hours of data to be trained under typical academic resource constraints.

End-to-End Evaluation for Low-Latency Simultaneous Speech Translation (2023.emnlp-demo)

Copied to clipboard

Christian Huber, Tu Anh Dinh, Carlos Mullov, Ngoc-Quan Pham, Thai Binh Nguyen, Fabian Retkowski, Stefan Constantin, Enes Ugan, Danni Liu, Zhaolin Li, Sai Koneru, Jan Niehues, Alexander Waibel

Speech & Audio in NLP

Challenge:	a framework to evaluate low-latency speech translations is currently only limited to specific aspects and is not able to compare different approaches.
Approach:	They propose a framework to perform and evaluate low-latency speech translation in realistic conditions.
Outcome:	The proposed framework evaluates various aspects of low-latency speech translation under realistic conditions.

Analyzing Challenges in Neural Machine Translation for Software Localization (2023.eacl-main)

Copied to clipboard

Sai Koneru, Matthias Huck, Miriam Exel, Jan Niehues

Machine Translation

Challenge:	Neural machine translation (NMT) is a new form of machine translation that reduces the post-editing time of human annotators.
Approach:	They propose to use a novel multilingual UI corpus collection to test NMT for user interfaces.
Outcome:	The proposed test set evaluates state-of-the-art methods on a UI translation task from English to German and identifies its limitations.

Sigmoid Head for Quality Estimation under Language Ambiguity (2026.acl-long)

Copied to clipboard

Tu Anh Dinh, Jan Niehues

Benchmarking Diffusion Models for Machine Translation (2024.eacl-srw)

Challenge:	Language model (LM) probability is not reliable quality estimator, as natural language is ambiguous.
Approach:	They propose to train a language model (LM) probability module on top of pre-trained LMs to address these limitations.
Outcome:	The proposed module is an extra unembedding head with sigmoid activation to tackle the first limitation.

Copied to clipboard

Yunus Demirag, Danni Liu, Jan Niehues

Machine Translation

Challenge:	Diffusion models have shown great potential on many generative tasks, but their application to natural language processing (NLP) is still a less explored direction.
Approach:	They adapt two diffusion-based text generation models, Diffusion-LM and DiffuSeq, to perform machine translation.
Outcome:	The proposed models struggle more on long-range dependencies than other models.

Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs (2025.acl-long)

Copied to clipboard

Danni Liu, Jan Niehues

Semantic Text Processing

Challenge:	Effective cross-lingual transfer is hindered by performance gaps and the scarcity of fine-tuning data in many languages.
Approach:	They propose a middle-layer alignment objective integrated into task-specific training to improve cross-lingual transfer across languages.
Outcome:	The proposed method improves cross-lingual transfer to lower-resource languages and can be merged with existing modules without full re-training.

Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing (2024.naacl-long)

Copied to clipboard

Sai Koneru, Miriam Exel, Matthias Huck, Jan Niehues

BOOM: Beyond Only One Modality KIT’s Multimodal Multilingual Lecture Companion (2026.eacl-demo)

Challenge:	Large language models have demonstrated considerable success in various natural language processing tasks, but their performance in NMT tasks is still underexplored.
Approach:	They propose to use LLMs as automatic post-editors rather than direct translators to improve BLEU and COMET performance.
Outcome:	The proposed approach improves BLEU but COMET performance compared to in-context learning.

Copied to clipboard

Sai Koneru, Fabian Retkowski, Christian Huber, Lukas Hilgert, Seymanur Akti, Enes Yavuz Ugan, Alexander Waibel, Jan Niehues

Can a Large Language Model Keep My Secrets? A Study on LLM-Controlled Agents (2025.acl-srw)

Challenge:	a multimodal multilingual lecture companion is needed to preserve lecture content in its entirety . globalization of education and rapid growth of online learning have made localizing educational content a challenge .
Approach:	They propose a multimodal multilingual lecture companion that translates lecture audio and slides to produce synchronized outputs across three modalities.
Outcome:	The proposed solution preserves the original content in its entirety while preserving translations across three modalities.

Copied to clipboard

Niklas Hemken, Sai Koneru, Florian Jacob, Hannes Hartenstein, Jan Niehues

Responsible & Trustworthy NLP

Challenge:	Using large language models, agents can assist with natural language tasks when given access to confidential data.
Approach:	They created a synthetic dataset consisting of confidentiality-aware planning and deduction tasks in organizational access control.
Outcome:	The proposed model can perform tasks similar to humans when given access to confidential data.

Language-Independent Representations Improve Zero-Shot Summarization (2024.naacl-short)

Copied to clipboard

Vladimir Solovyev, Danni Liu, Jan Niehues

Semantic Text Processing

Challenge:	Pretrained models can be fine tuned on downstream generation tasks, but they can fail in zero-shot conditions.
Approach:	They propose query-key finetuning to decouple task-specific knowledge from pretrained models . they propose a variant that more directly enforces language-agnostic representations .
Outcome:	The proposed model decouples task-specific knowledge from pretrained language generation abilities.

KIT Lecture Translator: Multilingual Speech Translation with One-Shot Learning (C18-2)

Copied to clipboard

Florian Dessloch, Thanh-Le Ha, Markus Müller, Jan Niehues, Thai-Son Nguyen, Ngoc-Quan Pham, Elizabeth Salesky, Matthias Sperber, Sebastian Stüker, Thomas Zenkel, Alexander Waibel

Incremental processing of noisy user utterances in the spoken language understanding task (D19-55)

Challenge:	In today's globalized world, communication is difficult and often the language barrier still prevents communication.
Approach:	They have developed a low-latency translation system that is adapted to lectures and covers several language pairs.
Outcome:	The proposed system improves performance but also covers several European languages.

Copied to clipboard

Stefan Constantin, Jan Niehues, Alex Waibel

Challenge:	triggered actions with high executions times can cause dialog systems to react slowly due to high latency and high latex.
Approach:	They propose a model-agnostic method to achieve high quality in processing incrementally produced partial utterances.
Outcome:	The proposed method improves the metric F1-score by 47.91 percentage points . the proposed method can be used to create low-latency natural language understanding components on ATIS datasets.

KIT-Multi: A Translation-Oriented Multilingual Embedding Corpus (L18-1)

Copied to clipboard

Thanh-Le Ha, Jan Niehues, Matthias Sperber, Ngoc Quan Pham, Alexander Waibel

Automated Evaluation of Out-of-Context Errors (L18-1)

Challenge:	Cross-lingual word embeddings are representations of words across languages in a shared continuous vector space.
Approach:	They propose a multilingual word embedding corpus which is acquired by neural machine translation and is based on monolingual data.
Outcome:	The proposed method is competitive with existing methods but on the cross-lingual document classification task, it obtains the best figures.

Copied to clipboard

Patrick Huber, Jan Niehues, Alex Waibel

Challenge:	Existing methods to modify text understanding systems use only one sentence at a time . however, considering a larger context can improve performance for text understanding tasks.
Approach:	They propose to modify existing text data to insert out-of-context errors . they use a 2016 TEDTalk corpus to evaluate computational models for text understanding .
Outcome:	The proposed method targets real-world problems of transcription and translation systems by inserting authentic out-of-context errors.

How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations (2025.naacl-short)

Copied to clipboard

Hyunji Lee, Danni Liu, Supriti Sinhamahapatra, Jan Niehues

SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading (2024.emnlp-main)

Challenge:	Recent advances in foundation models have sparked growing interest in expanding their text processing capabilities to speech.
Approach:	They analyze the model activations from semantically equivalent sentences across languages in the text and speech modalities and examine how text and spoken are represented in recent multimodal foundation models.
Outcome:	The proposed models exhibit cross-lingual differences, but are not explicitly trained for modality-agnostic representations.

Copied to clipboard

Tu Dinh, Carlos Mullov, Leonard Bärmann, Zhaolin Li, Danni Liu, Simon Reiß, Jueun Lee, Nathan Lerzer, Jianfeng Gao, Fabian Peller-Konrad, Tobias Röddiger, Alexander Waibel, Tamim Asfour, Michael Beigl, Rainer Stiefelhagen, Carsten Dachsbacher, Klemens Böhm, Jan Niehues

LibriS2S: A German-English Speech-to-Speech Translation Corpus (2022.lrec-1)

Challenge:	Large Language Models (LLMs) are rapidly developing and are becoming more and more useful in scientific tasks.
Approach:	They propose to use LLM-as-a-judge to grade LLMs on SciEx to assess their ability on scientific tasks.
Outcome:	The proposed benchmarks show that the LLMs perform decently on free-form exams, achieving 0.948 Pearson correlation with expert grading.

Copied to clipboard

Pedro Jeuris, Jan Niehues

Continuous Learning in Neural Machine Translation using Bilingual Dictionaries (2021.eacl-main)

Challenge:	Recent advances in speech-to-text translation have led to significant improvements, but the availability of appropriate training data is limiting.
Approach:	They propose a new text-to-speech and speech-tospech translation model that directly learns to generate the speech signal based on the pronunciation of the source language.
Outcome:	The proposed model learns to generate speech signal based on pronunciation of source language.

Copied to clipboard

Jan Niehues