Papers by Jasmijn Bastings

“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification (2022.emnlp-main)

Copied to clipboard

Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, Katja Filippova

Language Models

Challenge:	Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared.
Approach:	They propose a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking.
Outcome:	The proposed method is based on partially synthetic data and is compared with lexical shortcuts on a range of datasets and LSTM models.

We Need To Talk About Random Splits (2021.eacl-main)

Copied to clipboard

Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, Katja Filippova

Responsible & Trustworthy NLP

Challenge:	argued that random splits, like standard splits lead to overly optimistic performance estimates.
Approach:	They argue that random splits, like standard splits lead to overly optimistic performance estimates.
Outcome:	The proposed method leads to more realistic performance estimates than standard splits.

MiTTenS: A Dataset for Evaluating Gender Mistranslation (2024.emnlp-main)

Copied to clipboard

Kevin Robinson, Sneha Kudugunta, Romina Stella, Sunipa Dev, Jasmijn Bastings

Machine Translation

Challenge:	Existing studies on gender mistranslation in translation systems have highlighted the problem . a dataset of 26 languages is presented to measure the extent of such errors .
Approach:	They propose a dataset that measures the extent of gender mistranslation in translation systems . they use handcrafted passages that target known failure patterns and synthetically generated passages .
Outcome:	The proposed dataset covers 26 languages from a variety of language families and scripts, including several traditionally under-represented in digital resources.

Training Text-to-Text Transformers with Privacy Guarantees (2022.findings-acl)

Copied to clipboard

Natalia Ponomareva, Jasmijn Bastings, Sergei Vassilvitskii

Semantic Text Processing

Challenge:	Recent advances in NLP often stem from large transformer-based pre-trained models.
Approach:	They propose differentially private (DP) training as a potential mitigation for models that can memorize parts of training data.
Outcome:	The proposed model can memorize parts of training data and mitigate memorization concerns.

Amplifying Trans and Nonbinary Voices: A Community-Centred Harm Taxonomy for LLMs (2025.acl-long)

Copied to clipboard

Eddie L. Ungless, Sunipa Dev, Cynthia L. Bennett, Rebecca Gulotta, Jasmijn Bastings, Remi Denton

Speech & Audio in NLP

Challenge:	Existing studies on harms of language technology to transgender and nonbinary people focus on misgendering and stereotyping .
Approach:	They propose a taxonomy of harms for large language models and heuristics for evaluation to help identify harmful behavior in LLMs.
Outcome:	The proposed model-based approach combines surveys and focus groups with community experts to identify harmful behavior in large language models.

Dissecting Recall of Factual Associations in Auto-Regressive Language Models (2023.emnlp-main)

Copied to clipboard

Mor Geva, Jasmijn Bastings, Katja Filippova, Amir Globerson

Semantic Text Processing

Challenge:	Existing studies have focused on identifying where factual knowledge is encoded in the network, but little is known about how it is extracted from the model parameters during inference.
Approach:	They examine how factual associations are stored and retrieved internally in LMs . they use attention edges to identify critical points where information propagates to the prediction .
Outcome:	The proposed model aggregates information about subject and relation to predict the correct attribute . the model “queries” the enriched subject to extract the attribute based on the proposed model .

Low-Rank Adaptation for Multilingual Summarization: An Empirical Study (2024.findings-naacl)

Copied to clipboard

Chenxi Whitehouse, Fantine Huot, Jasmijn Bastings, Mostafa Dehghani, Chu-Cheng Lin, Mirella Lapata

Information Extraction & Text Mining

Challenge:	Pre-trained Large Language Models have significantly advanced NLP, but their ever-increasing size poses significant challenges for conventional fine-tuning.
Approach:	They investigate the potential of Low-Rank Adaptation (LoRA) in multilingual summarization, a task that is challenging and relatively unexplored.
Outcome:	The proposed method outperforms full fine-tuning and cross-lingual transfer strategies in multilingual summarization tasks.

Joey NMT: A Minimalist NMT Toolkit for Novices (D19-3)

Copied to clipboard

Julia Kreutzer, Jasmijn Bastings, Stefan Riezler

Multilinguality

Challenge:	a recent study shows that novices perform better than experts in a code quiz.
Approach:	They present a minimalist neural machine translation toolkit based on PyTorch . they evaluate the accessibility of the toolkit in a user study .
Outcome:	The proposed toolkit performs comparable to more complex toolkits on standard benchmarks.

The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020.emnlp-demos)

Copied to clipboard

Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, Ann Yuan

Responsible & Trustworthy NLP

Challenge:	Existing tools for modeling and understanding models are limited . existing tools can assist practitioners in understanding and evaluating models .
Approach:	They present an open-source platform for visualization and understanding of NLP models.
Outcome:	The language interpretability tool (lit) is an open-source platform for visualization and understanding of NLP models.

Interpretable Neural Predictions with Differentiable Binary Variables (P19-1)

Copied to clipboard

Jasmijn Bastings, Wilker Aziz, Ivan Titov

Explainability & Interpretability in NLP

Challenge:	Neural networks are bringing incredible performance gains on text classification tasks, but they also require interpretability.
Approach:	They propose a latent model that selects a rationale and a classifier that learns from the words in the rationale alone.
Outcome:	The proposed model can predict expected value of penalties without REINFORCE and can be directly optimised towards a pre-specified text selection rate.

Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks (N18-2)

Copied to clipboard

Diego Marcheggiani, Jasmijn Bastings, Ivan Titov

Multilinguality

Challenge:	Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods.
Approach:	They propose to integrate semantic representations into neural machine translation by injecting a semantic bias into sentence encoders and achieving improvements in BLEU scores.
Outcome:	The proposed representations achieve better BLEU scores over the linguistic-agnostic and syntax-aware versions on the English–German language pair.