| Challenge: | Second language learners tend to imprint the prosody of their mother language onto the second language (L2) . this can hamper communication between learners and natives, and can also affect the credibility of learners and how they are evaluated by others. |
| Approach: | They propose a tool that focuses on stress perception for speakers whose L1 is a fixed-stress language, such as French. |
| Outcome: | The tool is particularly useful for speakers whose L1 is a fixed-stress language, such as French. |
Similar Papers
Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias (2021.emnlp-main)
Copied to clipboard
| Challenge: | Lexical disambiguation is a major challenge for machine translation systems . previous work focused on automatic post-hoc analysis of translations, but rules of what makes a disambiguations correct or incorrect tend to be imprecise. |
| Approach: | They propose a black-box method that uses contrastive conditioning to detect disambiguation errors. |
| Outcome: | The proposed method is scalable and reliable for disambiguation evaluations. |
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation (2025.acl-long)
Copied to clipboard
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker
| Challenge: | Reliable multilingual evaluation is difficult and culturally appropriate evaluation is even harder to achieve. |
| Approach: | They propose a multilingual evaluation framework that aims to mitigate these biases by improving translations and annotation practices. |
| Outcome: | The proposed framework improves translation quality and cultural coverage and is culturally sensitive and culturally agnostic. |
CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models (2026.acl-short)
Copied to clipboard
| Challenge: | Membership inference attacks are a canonical way to assess a machine learning model’s privacy properties. |
| Approach: | They propose a framework for principled evaluation of membership inference attacks against large language models by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. |
| Outcome: | The proposed framework can be used to evaluate membership inference attacks against large language models. |
Your Stereotypical Mileage May Vary: Practical Challenges of Evaluating Biases in Multiple Languages and Cultural Contexts (2024.lrec-main)
Copied to clipboard
Karen Fort, Laura Alonso Alemany, Luciana Benotti, Julien Bezançon, Claudia Borg, Marthese Borg, Yongjian Chen, Fanny Ducel, Yoann Dupont, Guido Ivetta, Zhijian Li, Margot Mieskes, Marco Naguib, Yuyan Qian, Matteo Radaelli, Wolfgang S. Schmeisser-Nieto, Emma Raimundo Schulz, Thiziri Saci, Sarah Saidi, Javier Torroba Marchante, Shilin Xie, Sergio E. Zanotto, Aurélie Névéol
| Challenge: | Recent studies have identified a gap in the availability of tools and resources to study bias in languages other than English and social contexts outside the north of America. |
| Approach: | They use stereotypes to build a corpus of sentence pairs that cover biases in seven cultural contexts. |
| Outcome: | The proposed resource covers a wide range of languages and cultural settings . it favors sentences that express stereotypes in most bias categories . |
Extrinsic Evaluation of French Dependency Parsers on a Specialized Corpus: Comparison of Distributional Thesauri (2020.lrec-1)
Copied to clipboard
| Challenge: | Using a frequency-based method, we can identify subsets of the same word contexts without any reference data. |
| Approach: | They compare 11 different French dependency parsers on a specialized corpus to generate distributional thesauri using a frequency-based method. |
| Outcome: | The proposed method can identify relevant subsets without reference data and the similarity is confirmed on a restricted distributional benchmark. |
Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training. |
| Approach: | They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals. |
| Outcome: | The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 . |
DEMETR: Diagnosing Evaluation Metrics for Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | BLEU scores are based on string overlap, but they are opaque in comparison to newer learned metrics. |
| Approach: | They propose a dataset to evaluate MT evaluation metrics based on linguistic perturbations in English . they find learned metrics perform substantially better than string-based metrics . |
| Outcome: | The proposed dataset shows that learned metrics perform better than string-based metrics . the dataset contains 31K English examples that cover 35 different linguistic phenomena . |
VLStereoSet: A Study of Stereotypical Bias in Pre-trained Vision-Language Models (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing studies on pre-trained vision-language models have focused on measuring biases and stereotypes in a single modality. |
| Approach: | They extend a recently released stereotypical bias dataset into a vision-language probing dataset called VLStereoSet to measure stereotypical biased vision-linguistic models. |
| Outcome: | The proposed probing task measures stereotypical bias in vision-language models and its intra-modal and inter-modal biases. |
EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions (2026.acl-long)
Copied to clipboard
| Challenge: | Mainstream multilingual models are generally trained on a much higher proportion of English data . this raises questions about their ability to capture linguistic features specific to non-English languages . |
| Approach: | They propose a benchmark to test multilingual LLMs' ability to capture linguistic features in other languages. |
| Outcome: | The proposed benchmark shows that multilingual models can capture features in non-English languages and cultural norms. |
Detecting Training Data of Large Language Models via Expectation Maximization (2026.eacl-long)
Copied to clipboard
| Challenge: | Membership inference attacks aim to determine whether a specific example was used to train a given language model. |
| Approach: | They propose a membership inference approach that iteratively refines prefix effectiveness and membership scores using an expectation-maximization strategy without requiring labeled non-member examples. |
| Outcome: | The proposed approach outperforms baselines under systematically varied distributional overlap and difficulty. |