Papers by Marine Carpuat
Copied to clipboard
| Challenge: | LLMs can rewrite inputs but in machine translation, they are primarily used to re-write outputs via post-editing. |
| Approach: | They propose to use LLMs to rewrite inputs automatically to improve machine translation (MT) they propose to simplify inputs and use quality estimation to assess translatability. |
| Outcome: | The proposed methods can be improved by using quality estimation to assess translatability. |
Copied to clipboard
| Challenge: | Existing methods to improve machine translation (MT) in low-resource settings are limited in the number of languages spoken in the world. |
| Approach: | They apply cartography techniques to characterize the contribution of training samples in two low-resource MT tasks (Swahili-English and Turkish-English) they argue that data augmentation strategies for low-Resource ML would benefit from model-in-the-loop strategies to maximize improvements. |
| Outcome: | The proposed methods show that training samples contribute to model training in low-resource MT tasks, albeit not uniformly throughout the training process. |
Copied to clipboard
| Challenge: | Prior work on text complexity has focused on simplifying input text in one language, primarily English. |
| Approach: | They propose a method to align news articles written for different levels of target language proficiency. |
| Outcome: | The proposed model outperforms pipeline approaches that translate and simplify text independently. |
Copied to clipboard
| Challenge: | Recent efforts focus on single-LLM, single-turn generation approaches, but it can be challenging for any single model to support all cultures equally well. |
| Approach: | They propose to exploit the complementary strengths of multiple LLMs to promote cultural adaptability. |
| Outcome: | The proposed model improves accuracy and cultural group parity over single-LLM models. |
Copied to clipboard
| Challenge: | Synthetic translations have been used for a wide range of NLP tasks, but it remains unclear how they differ from naturally occurring data. |
| Approach: | They propose to use a semantic equivalence classifier to improve bitext quality without additional bilingual supervision to replace the originals. |
| Outcome: | The proposed samples improve bitext quality without additional bilingual supervision and are validated intrinsically and extrinsically through bilingual induction and MT tasks. |
Copied to clipboard
| Challenge: | Neural sequence generation models produce outputs that are unrelated to the source text, and are potentially harmful, yet it remains unclear in what conditions they arise and how to mitigate their impact. |
| Approach: | They first identify internal model symptoms of hallucinations by analyzing the relative token contributions to the generation in contrastive hallucinous vs. non-hallucinated outputs generated via source perturbations. |
| Outcome: | The proposed detector outperforms both baseline models and strong classifiers on English-Chinese and German-English translation test beds. |
Copied to clipboard
| Challenge: | Despite the growing use of large language models for writing tasks, it remains unclear whether users can effectively reshape LLM-generated text to reflect their personal style. |
| Approach: | They conduct an online study in which participants post-edit LLM-generated drafts for writing tasks where personal style matters to them. |
| Outcome: | The results show that post-editing increases stylistic similarity to unassisted writing and reduces similarity with fully LLM-generated output. |
Copied to clipboard
| Challenge: | a lack of standardized and reliable methods for automatic evaluation hinders ST . prior work has employed as many as nine different automatic systems to rate formality alone . |
| Approach: | They evaluate automatic metrics on the oft-researched task of formality style transfer . they outline best practices for automatic evaluation in (formality) style transfer and identify models that correlate well with human judgments. |
| Outcome: | The proposed models correlate well with human judgments and are robust across languages. |
Copied to clipboard
| Challenge: | Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer while speeding up decoding dramatically compared to constrained beam search. |
| Approach: | They propose an Edit-Based TransfOrmer with Repositioning that integrates lexical preferences into output sequences by iterative editing hypotheses. |
| Outcome: | The proposed model uses soft lexical constraints more effectively than the Levenshtein Transformer while speeding up decoding dramatically compared to constrained beam search. |
Copied to clipboard
| Challenge: | et al. (2017) show that imitation learning algorithms for machine translation introduce mismatches between training and inference that lead to undertraining and poor generalization in editing scenarios. |
| Approach: | They propose a framework for training non-autoregressive sequence-to-sequence models for editing tasks where the original input sequence is iteratively edited to produce the output. |
| Outcome: | The proposed framework significantly improves output quality and controls complexity better on the simplification task. |
Copied to clipboard
| Challenge: | Existing models that generate generic simplified outputs for a given source text have been used to specify output properties. |
| Approach: | They propose a non-autoregressive model that iteratively edits an input sequence and incorporates lexical complexity information into the refinement process to generate simplifications that better match the desired output complexity. |
| Outcome: | The proposed model incorporates lexical complexity information into the refinement process to achieve more complex simplification operations such as content deletion and paraphrasing, as well as sentence splitting. |
Copied to clipboard
| Challenge: | Generating natural language requires conveying content in an appropriate style. |
| Approach: | They propose to solve two related tasks on generating text of varying formality jointly using multi-task learning. |
| Outcome: | The proposed models achieve state-of-the-art performance for formality transfer and formality-sensitive machine translation without training on style-annotated translation examples. |
Copied to clipboard
| Challenge: | supervised systems have not replaced dedicated supervised models for machine translation tasks. |
| Approach: | They propose to guide LLMs to post-edit MT with feedback from MQM annotations . they then fine-tune the LLM to improve its ability to exploit the feedback . |
| Outcome: | The proposed model improves TER, BLEU and COMET scores on Chinese-English, English-German and English-Russian data. |
Copied to clipboard
| Challenge: | Using machine translation tools for everyday tasks is becoming more commonplace, but a lack of evaluation strategies and alternatives can cause users to over-rely on it. |
| Approach: | They propose to use MT evaluation techniques to promote MT quality and MT literacy among its users. |
| Outcome: | The findings highlight the need for evaluation and NLP explanation techniques to promote MT quality and MT literacy among its users. |
Copied to clipboard
| Challenge: | Recent work at the intersection of AI explainability and fairness has focused on how explanations can improve human-plus-AI task performance . |
| Approach: | They propose to characterize what constitutes an explanation that is itself "fair" they use not just accuracy and label time, but psychological impact of explanations on different groups . |
| Outcome: | The proposed method is based on content moderation of potential hate speech and its differential impact on Asian vs. non-Asian proxy moderators across explanation approaches. |
Copied to clipboard
| Challenge: | Neologisms and emerging slang are central to daily conversation, yet challenging for non-native speakers (NNS) to interpret and use appropriately in cross-cultural communication with native speakers (NS). |
| Approach: | They use AI to learn English neologisms and write messages using the learned word to an NS friend. |
| Outcome: | The proposed model shows that AI Explanation yields the largest gains over no support in NS-rated competence, while contextual appropriateness judgments show indifference across support. |
Copied to clipboard
| Challenge: | Existing approaches to machine translation (MT) systems degrade when faced with code-mixed text. |
| Approach: | They propose a system that can augment Vietnamese-English code-mixed text with iterative fine-tuning and targeted filtering. |
| Outcome: | The proposed framework outperforms strong back-translation baselines and improves zero-shot models by up to +11.9 points. |
Copied to clipboard
| Challenge: | Existing MT error detection and quality estimation (QE) techniques do not address this practical scenario. |
| Approach: | They propose a question generation and answering framework that detects critical MT errors and provides actionable feedback to help users decide whether to accept or reject MT outputs even without the knowledge of the target language. |
| Outcome: | The proposed framework has higher Kendall’s Tau correlation and decision accuracy with human ratings compared to other QE metrics. |
Copied to clipboard
| Challenge: | Existing work has addressed this problem by leveraging monolingual or multilingual data. |
| Approach: | They propose to introduce a differentiable reconstruction loss for neural machine translation to exploit the limited amounts of parallel text available in low-resource settings. |
| Outcome: | The proposed approach achieves small but consistent BLEU improvements on four language pairs in both translation directions and outperforms an alternative differentiable reconstruction strategy based on hidden states. |
Copied to clipboard
| Challenge: | Neural machine translation (NMT) performance drops when domains do not match and in-domain training data is scarce. |
| Approach: | They propose a curriculum learning approach to adapt generic neural machine translation models to a specific domain. |
| Outcome: | The proposed approach outperforms unadapted and adapted baselines in two domains and two language pairs. |
Copied to clipboard
| Challenge: | Recent advances in machine translation (MT) have focused on scaling multilingual machine translation models and evaluation data to hundreds of languages, including multiple under-resourced languages. |
| Approach: | They propose to use n-gram matching metrics to measure progress in multilingual machine translation to 13 typologically diverse African languages to create high-quality human evaluation data with simplified MQM guidelines. |
| Outcome: | The proposed metrics have a higher correlation with human judgments than n-gram matching metrics such as BLEU and METEOR. |
Copied to clipboard
| Challenge: | Existing research on machine translation tools has not revealed how users perceive MT errors and how they evolve through interaction. |
| Approach: | They propose a framework where users accept MT output or request professional re-translation to answer questions based on information presented in a foreign language. |
| Outcome: | The proposed framework can predict where the system is likely to be wrong and how it evolves through interaction. |
Copied to clipboard
| Challenge: | Current approaches to incorporating terminology constraints in machine translation (MT) typically assume that the constraint terms are provided in their correct morphological forms. |
| Approach: | They propose a framework for incorporating lemma constraints in machine translation . they use a cross-lingual inflection module that inflects the target lemmo constraints based on the source context. |
| Outcome: | The proposed framework outperforms existing methods with lower training costs and linguistic knowledge in domain adaptation and low-resource MT settings. |
Copied to clipboard
| Challenge: | Many commonsense reasoning questions require a hard selection of a single correct answer . ambiguity and semantic mismatches are common in many MCQs . |
| Approach: | They collect plausibility judgments on 5 000 commonsense reasoning questions . they find that the answer rated most plausible does not match the benchmark gold answers . |
| Outcome: | Experiments with LLMS reveal low accuracy and high variation in performance on the subset . high plausibility rating for the most plausible answer is highlighted in bold . |
Copied to clipboard
| Challenge: | Recent advances in automatic quality estimation for machine translation focus on written language, leaving the speech modality underexplored. |
| Approach: | They propose a new quality estimation system based on cascaded and end-to-end architectures. |
| Outcome: | The proposed system is better suited to estimating the quality of direct speech translation than existing systems designed for text translation. |
Copied to clipboard
| Challenge: | Existing approaches to correct exposure bias in machine translation are inadequate . scheduled sampling assumes that words are aligned at each time step . |
| Approach: | They propose a differentiable sampling algorithm that optimizes the probability that the reference can be aligned with the sampled output. |
| Outcome: | The proposed approach improves BLEU on translation tasks and is simpler to train with no sampling schedule. |
Copied to clipboard
| Challenge: | a major challenge in the practical use of Machine Translation (MT) is that users lack information on translation quality to make informed decisions about how to rely on outputs. |
| Approach: | They evaluate quality estimation feedback in vivo with a human study in a medical setting. |
| Outcome: | The proposed method improves appropriate reliance on MT, but backtranslation helps detect harmful errors. |
Copied to clipboard
| Challenge: | Prior work treats all types of mismatches between source and target as noise . Consequently, it remains unclear how noisy parallel training samples impact NMT training. |
| Approach: | They propose a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences. |
| Outcome: | The proposed framework improves translation quality and model calibration on EN-FR tasks. |
Copied to clipboard
| Challenge: | Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. |
| Approach: | They propose to use large language models to model difficulty of multiple-choice math questions for real-world students. |
| Outcome: | The proposed model predicts difficulty of multiple-choice math questions for students . correlations between model and real-world difficulty are high, the authors show . |
Copied to clipboard
| Challenge: | despite its potential to help users, NLP research on explicitation is limited because of the lack of adequate evaluation methods. |
| Approach: | They propose automatic methods to generate explicitations from a Wikipedia dataset . they use both intrinsic and extrinsic evaluation to evaluate the system's effectiveness . |
| Outcome: | The proposed system bridges the gap between the source speaker and the target audience . it is effective based on intrinsic and extrinsic evaluation, the authors show . |
Copied to clipboard
| Challenge: | Existing approaches to text simplification control output complexity at corpus level disregarding complexity of individual inputs and considering only one level of output complexity. |
| Approach: | They propose a method that predicts edit operations required for a specific grade level . they say this approach improves the quality of the simplified outputs over corpus-level heuristics . |
| Outcome: | The proposed method improves the readability of simplified outputs over corpus-level search-based heuristics. |
Copied to clipboard
| Challenge: | Existing studies on the impact of feedback on human decision-making are limited as people are not equipped to assess the quality of AI predictions. |
| Approach: | They compare the quality of MT inputs and outputs with explicit and implicit feedbacks that directly give users an assessment of translation quality using error highlights and LLM explanations. |
| Outcome: | The proposed model improves decision accuracy and appropriate reliance by using error highlights and explanations, and by using backtranslation and question–answer tables. |
Copied to clipboard
| Challenge: | a common strategy to explain NLP predictions is to highlight salient tokens in their inputs. |
| Approach: | They propose a technique to generate contrastive phrasal highlights that explain the predictions of a semantic divergence model via phrase alignment guided erasure. |
| Outcome: | The proposed techniques match human rationales of cross-lingual semantic differences better than popular post-hoc saliency techniques and help people detect fine-grained meaning differences in human translations and critical machine translation errors. |
Copied to clipboard
| Challenge: | Iterative Back-Translation and Dual Learning use different objectives and heuristic gradient approximation strategies, and have not been extensively compared. |
| Approach: | They propose a dual reconstruction objective that provides a unified view of Iterative Back-Translation and Dual Learning. |
| Outcome: | The proposed method is more effective than Dual Learning on German-English and Turkish-English tasks. |
Copied to clipboard
| Challenge: | Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. |
| Approach: | They propose a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. |
| Outcome: | The proposed model outperforms the previous state-of-the-art by 18.5% WER and 4.7 BLEU on downstream audio-visual speech recognition and translation tasks. |
Copied to clipboard
| Challenge: | Existing NLP systems can only access the retrieved context to determine the answer, resulting in a knowledge gap between the information that is required to answer the question and the information available to assess the model’s correctness. |
| Approach: | They ask whether adding relevant background helps mitigate users’ over-reliance on predictions. |
| Outcome: | The proposed approach reduces over-reliance on model predictions even in the absence of sufficient information to assess their correctness. |
Copied to clipboard
| Challenge: | Query-focused summarization of foreign-language documents can help a user understand whether a document is relevant to a query term. |
| Approach: | They propose to use machine translation and post-editing to improve human relevance judgments . they include a query term in a summary when its translation appears in the source document . |
| Outcome: | The proposed approach improves human relevance judgments by including a query term in a summary when its translation appears in the source document. |
Copied to clipboard
| Challenge: | Authorship obfuscation has been evaluated in narrow settings in the NLP literature . superficial edit operations can lead to unnatural outputs, authors say . |
| Approach: | They propose an automatic text privatization framework that fine-tunes a large language model via reinforcement learning to produce rewrites that balance soundness, sense, and privacy. |
| Outcome: | The proposed method maintains high text quality according to automated metrics and human evaluation, and successfully evades several automated authorship attacks. |
Copied to clipboard
| Challenge: | Parallel sentence pairs are sentences that are translations of each other and convey the same meaning in the source and target languages. |
| Approach: | They propose a model which detects meaning divergences in parallel sentence pairs . parallel sentence pair are translations of each other, therefore often assumed to convey the same meaning . |
| Outcome: | The proposed model detects divergences more accurately than models based on word alignments. |
Copied to clipboard
| Challenge: | a good SimulMT system will allow the downstream QA system to answer correctly as quickly as possible. |
| Approach: | They propose a word-by-word question answering evaluation task to evaluate if models translate salient elements of a question correctly. |
| Outcome: | a new evaluation task aims to show whether models translate salient elements of a question accurately and quickly . evaluators can reveal weaknesses in existing neural systems, hallucinating or omitting facts . human evaluation is too costly and slow to guide system development, authors say . |
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer prompts in many languages despite being pre-trained mostly on English text. |
| Approach: | They propose a Discriminative Alignment Index to quantify instance-level alignment across 24 languages other than English and three distinct NLU tasks. |
| Outcome: | The proposed model can perform natural language understanding tasks in 24 languages other than English and three distinct NLU tasks. |
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual hypernymy detection are sparse and can be trained on related languages with negligible loss of performance. |
| Approach: | They propose a family of unsupervised approaches for cross-lingual hypernymy detection which learns sparse, bilingual word embeddings based on dependency contexts. |
| Outcome: | The proposed approach significantly improves performance on this task, compared to approaches based only on lexical context. |
Copied to clipboard
| Challenge: | Using demographics, we hypothesize that the ability of translation systems to correctly translate female-associated names is significantly lower than male-associated name. |
| Approach: | They propose a translation evaluation procedure based on round-trip translation of names that are demographically aligned and analyze the effect of name demographics on translation quality using generalized linear mixed effects models. |
| Outcome: | The proposed evaluation procedure is based on round-trip translation of names from a dataset of names that are demographically aligned and shows that the ability of translation systems to translate female-associated names is significantly lower than male-associated name. |
Copied to clipboard
| Challenge: | Existing methods for authorship analysis and text detection are limited . authors: human-AI collaborative writing poses a potential challenge for existing methods . |
| Approach: | They investigate the extent to which existing AI detection and authorship analysis models can perform classification on data generated in human-AI collaborative writing sessions. |
| Outcome: | The proposed models outperform existing models on human-AI collaborative writing data . authors say human- AI co-written text will require adapting models in the near future . |
Copied to clipboard
| Challenge: | Detecting fine-grained differences in content conveyed in different languages is expensive and hard to scale. |
| Approach: | They propose a training strategy for multilingual BERT models by learning to rank divergent examples of varying granularity. |
| Outcome: | The proposed model improves the prediction and annotation of fine-grained semantic divergences. |
Copied to clipboard
| Challenge: | Despite progress in MT, a gap persists between how the technology is developed and how it is used in real-world contexts. |
| Approach: | They propose a human-centered approach to machine translation (MT) they argue that MT should be evaluated with diverse goals and contexts of use . |
| Outcome: | The proposed approach emphasizes alignment of evaluation and design with diverse communicative goals and contexts of use. |
Copied to clipboard
| Challenge: | despite growing interest in explainable NLP, it remains unclear how explanation strategies shape user behavior in tasks like authorship identification. |
| Approach: | They propose two explanation types to support their analysis of user behavior . they use example-based style rewrites and feature-based rationales to generate explanations . |
| Outcome: | The proposed explanations support appropriate reliance, whereas explanations increase AI overreliance, the study finds . |
Copied to clipboard
| Challenge: | Prior work suggests that distilled training data is less complex than manual translations. |
| Approach: | They propose to use sequence-level knowledge distillation to match autoregressive models' translation quality. |
| Outcome: | The proposed model can match translation quality of autoregressive models with distilled training data. |
Copied to clipboard
| Challenge: | Existing work characterizes differences in meaning between words across languages using semantic relations . however, because of translation ambiguity, semantic relations are not always preserved by translation. |
| Approach: | They propose a cross-lingual relation classifier trained only with English examples and a bilingual dictionary to account for translation ambiguity when transferring knowledge from English to cross-linguistic settings. |
| Outcome: | The proposed model outperforms baselines that rely on bilingual embeddings or dictionaries for cross-lingual transfer and approaches fully supervised systems on English tasks. |