An Interpretable and Crosslingual Method for Evaluating Second-Language Dialogues (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies on second language (SL) assessment of conversational fluency and interactivity have focused on written correction or pronunciation from ASR. |
| Approach: | They propose a framework that assesses the relationships between micro-level linguistic features and macro-level interactivity labels for Chinese-as-a-second-language dialogues. |
| Outcome: | The proposed framework is interpretable and can be adapted to other languages for second-language dialogue evaluation. |
Similar Papers
Understanding Cross-Lingual Alignment—A Survey (2024.findings-acl)
Copied to clipboard
| Challenge: | Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. |
| Approach: | They propose a taxonomy of methods to improve cross-lingual alignment . they argue that an effective trade-off between language-neutral and language-specific information is key . |
| Outcome: | The proposed methods can be applied to encoder models and encoder-decoder-only models . they show that language-neutral and language-specific information is key . |
IM^2: an Interpretable and Multi-category Integrated Metric Framework for Automatic Dialogue Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Evaluation metrics for dialogue systems are expensive and time-consuming . current evaluation metrics focus on a single quality or several qualities . |
| Approach: | They propose an interpretable, multi-faceted, and controllable framework to combine dialogue metrics which are good at measuring different qualities. |
| Outcome: | The proposed framework integrates a large number of evaluation metrics to improve the performance of the model. |
End-to-End Evaluation for Low-Latency Simultaneous Speech Translation (2023.emnlp-demo)
Copied to clipboard
Christian Huber, Tu Anh Dinh, Carlos Mullov, Ngoc-Quan Pham, Thai Binh Nguyen, Fabian Retkowski, Stefan Constantin, Enes Ugan, Danni Liu, Zhaolin Li, Sai Koneru, Jan Niehues, Alexander Waibel
| Challenge: | a framework to evaluate low-latency speech translations is currently only limited to specific aspects and is not able to compare different approaches. |
| Approach: | They propose a framework to perform and evaluate low-latency speech translation in realistic conditions. |
| Outcome: | The proposed framework evaluates various aspects of low-latency speech translation under realistic conditions. |
Disentangling Language and Culture for Evaluating Multilingual Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Extensive evaluations of large language models (LLMs) are conducted on a wide range of models, revealing a notable cultural-linguistic synergy phenomenon, where models exhibit better performance when questions are culturally aligned with the language. |
| Approach: | They propose a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of large language models by decomposing evaluation along dimensions of linguistic medium and cultural context. |
| Outcome: | The proposed framework allows for a nuanced analysis of LLMs’ ability to process questions within both native and cross-cultural contexts cross-lingually. |
XNLI: Evaluating Cross-lingual Sentence Representations (D18-1)
Copied to clipboard
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, Veselin Stoyanov
| Challenge: | State-of-the-art natural language processing systems rely on annotated data to learn competent models. |
| Approach: | They extend the development and test sets of the Multi-Genre Natural Language Inference Corpus to 14 languages, including Swahili and Urdu. |
| Outcome: | The proposed evaluation set extends the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 14 languages including low-resource languages such as Swahili and Urdu. |
Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in training multilingual models on large datasets have shown promising results in knowledge transfer across languages. |
| Approach: | They challenge the assumption that high zero-shot performance reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. |
| Outcome: | The proposed model can achieve high performance on multilingual benchmarks and on low-resource languages. |
xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, human evaluation is the most reliable way to holistically judge the quality of the dialogue. |
| Approach: | They propose to use English dialogue evaluation metrics to generalize them to other languages. |
| Outcome: | The proposed metrics outperform OpenAI’s ChatGPT in terms of average Pearson correlations over all datasets and languages. |
Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Multilingual task-oriented dialogue (ToD) datasets suffer from severe limitations, such as being small in scale and lacking naturalness and cultural specificity in the target language. |
| Approach: | They propose a novel outline-based annotation process where domain-specific abstract schemata of dialogue are mapped into natural language outlines. |
| Outcome: | The proposed approach improves understanding, dialogue state tracking, and end-to-end dialogue evaluation in Arabic, Indonesian, Russian, and Kiswahili. |
MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for engagingness evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. |
| Approach: | They propose to use large language models to evaluate engagingness in dialogue . they propose to include prompts and translated prompts in the model . |
| Outcome: | The proposed model outperforms existing methods on evaluation of engagingness in dialogue across languages. |
Natural Language Processing for Multilingual Task-Oriented Dialogue (2022.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial will examine the challenges and gaps in multilingual ToD research . multilingual systems are difficult to build, and are limited to English and other languages . |
| Approach: | This tutorial will discuss the importance of multilingual task-oriented dialogue systems . it will provide an overview of current research gaps, challenges and initiatives related to multilingual ToD systems - with a particular focus on their connections to current research and challenges in multilingual and low-resource NLP. |
| Outcome: | This tutorial will provide an overview of current research gaps, challenges and initiatives related to multilingual ToD systems. |