Papers by Timothy Baldwin

Noisy Label Regularisation for Textual Regression (2022.coling-1)

Challenge:	Existing approaches to assess and improve model fairness have been inconsistent and inconsistent.
Approach:	They propose an open-source python library for assessing and improving model fairness.
Outcome:	The proposed framework can be used for natural language, images, and audio.

Copied to clipboard

Yuxia Wang, Timothy Baldwin, Karin Verspoor

Cost-effective Distillation of Large Language Models (2023.findings-acl)

Challenge:	Existing methods to regularise noisy labels are ineffective in the face of noisy data.
Approach:	They propose a method that regularises noisy labels and prevents error propagation from the input layer.
Outcome:	The proposed method regularises noisy labels and improves generalisation performance over real-world human-disagreement annotations and randomly-corrupted and data-augmented labels.

Copied to clipboard

Sayantan Dasgupta, Trevor Cohn, Timothy Baldwin

Evaluating Debiasing Techniques for Intersectional Biases (2021.emnlp-main)

Challenge:	Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets.
Approach:	They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture.
Outcome:	The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets.

Copied to clipboard

Shivashankar Subramanian, Xudong Han, Timothy Baldwin, Trevor Cohn, Lea Frermann

Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World (2025.findings-emnlp)

Challenge:	Existing methods for debiasing protected attributes have been limited to binary attributes in isolation, however many corpora involve multiple such attributes, possibly with higher cardinality.
Approach:	They propose to evaluate a bias-constrained model which is new to NLP and an extension of the iterative nullspace projection technique which can handle multiple identities.
Outcome:	The proposed model is based on a new iterative nullspace projection technique which can handle multiple identities.

Copied to clipboard

Saeed Almheiri, Rania Elbadry, Mena Attia, Chenxi Wang, Preslav Nakov, Timothy Baldwin, Fajri Koto

Narrative Modeling with Memory Chains and Semantic Supervision (P18-2)

Challenge:	a recent study examined the potential for cross-cultural transfer of commonsense reasoning . merely 12 culture-specific examples from one country can improve performance in others by 10% on average .
Approach:	They evaluate cross-cultural transfer of commonsense reasoning within the arab world . they use in-context learning and demonstration-based reinforcement to evaluate alignment methods .
Outcome:	The proposed model can improve performance in cultures with cultural similarities in the Arab world by 10% on average.

Copied to clipboard

Fei Liu, Trevor Cohn, Timothy Baldwin

Challenge:	Story comprehension requires a deep semantic understanding of the narrative, making it a challenging task.
Approach:	They propose a method that tracks various semantic aspects with external neural memory chains . they propose to encourage each to focus on a particular semantic aspect .
Outcome:	The proposed method outperforms baselines on the task of story ending prediction.

Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks (2026.findings-eacl)

Copied to clipboard

Go Inoue, Bashar Alhafni, Nizar Habash, Timothy Baldwin

Syntactic Text Processing

Challenge:	Diacritics can significantly influence language processing tasks in Arabic . their presence can increase subword fragmentation during tokenization, reducing performance .
Approach:	They analyze the impact of diacritics on tokenization and benchmark task performance across major Large Language Models.
Outcome:	The proposed model is robust to diacritics, but full diacritization leads to token fragmentation and degraded performance.

CULG: Commercial Universal Language Generation (2022.naacl-industry)

Copied to clipboard

Haonan Li, Yameng Huang, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

LipKey: A Large-Scale News Dataset for Absent Keyphrases Generation and Abstractive Summarization (2022.coling-1)

Challenge:	Pre-trained language models have improved performance for many NLP tasks in finance and healthcare.
Approach:	They propose a large-scale commercial universal language generation model which is pre-trained on a corpus drawn from 10 markets across 7 languages.
Outcome:	The proposed model outperforms other models on commercial generation tasks and on other markets, languages, and tasks.

Copied to clipboard

Fajri Koto, Timothy Baldwin, Jey Han Lau

Summarization

Challenge:	Existing work has addressed each element individually, but this study focuses on LipKey, the largest news corpus with human-written abstractive summaries, absent keyphrases, and titles.
Approach:	They propose a novel news dataset that consists of highly absent keyphrases . they combine lips keyphrase and TF-IDF to obtain abstractive summaries .
Outcome:	The proposed dataset is the largest news corpus with human-written abstractive summaries, absent keyphrases, and titles.

COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations (2026.findings-eacl)

Copied to clipboard

Rui Xing, Preslav Nakov, Timothy Baldwin, Jey Han Lau

Qorǵau: Evaluating Safety in Kazakh-Russian Bilingual Contexts (2025.findings-acl)

Challenge:	X, Meta, and TikTok are experimenting with community-based factchecking . community-driven verification is a way to provide explanatory notes that clarify why a post might be misleading .
Approach:	They propose a framework that optimizes the helpfulness of explanatory notes and the reason for this by automatically optimizing the prompt definitions.
Outcome:	The proposed framework improves helpfulness and reason prediction on 104k posts with user-provided notes and helpfulness labels.

Copied to clipboard

Maiya Goloburda, Nurkhan Laiyk, Diana Turmakhan, Yuxia Wang, Mukhammed Togmanov, Jonibek Mansurov, Askhat Sametov, Nurdaulet Mukhituly, Minghan Wang, Daniil Orel, Zain Muhammad Mujahid, Fajri Koto, Timothy Baldwin, Preslav Nakov

A Chinese Dataset for Evaluating the Safeguards in Large Language Models (2024.findings-acl)

Challenge:	Large language models (LLMs) have the potential to generate harmful content, posing risks to users.
Approach:	They propose a dataset specifically designed for safety evaluation in Kazakh and Russian . they use a bilingual context in Kazakhstan where both Kazakh (a low-resource language) and Russian (a high-resourced language)
Outcome:	The proposed dataset is designed for safety evaluation in Kazakh and Russian . it shows that both multilingual and language-specific LLMs perform better than others .

Copied to clipboard

Yuxia Wang, Zenan Zhai, Haonan Li, Xudong Han, Shom Lin, Zhenxuan Zhang, Angela Zhao, Preslav Nakov, Timothy Baldwin

Does Vision Accelerate Hierarchical Generalization in Neural Language Learners? (2025.coling-main)

Challenge:	a recent study has shown that large language models can produce harmful responses, exposing users to unexpected risks.
Approach:	They propose a dataset for the safety evaluation of Chinese LLMs in Mandarin Chinese . they extend the dataset to better identify false negative and false positive examples .
Outcome:	The proposed dataset is for the safety evaluation of Chinese LLMs, and is based on a Chinese dataset.

Copied to clipboard

Tatsuki Kuribayashi, Timothy Baldwin

Multimodality

Challenge:	Neural language models (LMs) are arguably less data-efficient than humans from a language acquisition perspective.
Approach:	They investigate the advantage of grounded language acquisition over visual input to improve syntactic generalization.
Outcome:	The proposed model is less efficient than humans in language acquisition . it shows that visual input helps syntactic generalization, but not vision .

Semi-supervised User Geolocation via Graph Convolutional Networks (P18-1)

Copied to clipboard

Afshin Rahimi, Trevor Cohn, Timothy Baldwin

What does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text (2022.findings-acl)

Challenge:	Social media user geolocation is vital to many applications such as event detection.
Approach:	They propose a multiview geolocation model that uses both text and network context.
Outcome:	The proposed model outperforms baseline models and the state-of-the-art models under minimal supervision.

Copied to clipboard

Biaoyan Fang, Timothy Baldwin, Karin Verspoor

Optimising Equal Opportunity Fairness in Model Training (2022.naacl-main)

Challenge:	Current research on anaphora resolution is mostly based on declarative text, such as chemical patents or instruction manuals.
Approach:	They propose a framework for anaphora annotation for the chemical domain for modeling anamorphic phenomena in recipes and chemical patents.
Outcome:	The proposed framework improves resolution of anaphora in recipes, suggesting transferability of general procedural knowledge.

Copied to clipboard

Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, Lea Frermann

Liputan6: A Large-scale Indonesian Dataset for Text Summarization (2020.aacl-main)

Challenge:	Existing methods to reduce bias have been shown to be effective over real-world datasets.
Approach:	They propose two new training objectives which directly optimise for the widely-used criterion of equal opportunity.
Outcome:	The proposed training objectives directly optimise for the widely-used criterion of equal opportunity while maintaining high performance over two classification tasks.

Copied to clipboard

ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning (2026.acl-demo)

Challenge:	Despite having the fourth largest speaker population in the world, 1 Indonesian is under-represented in NLP.
Approach:	They propose to use a large-scale Indonesian summarization dataset to test extractive and abstractive summarizing methods.
Outcome:	The proposed methods are compared with multilingual and monolingual BERT-based models.

Copied to clipboard

Vladislav Smirnov, Quang-Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Mrinmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, Artem Shelmanov

A Multilingual Social Bias Benchmark Incorporating Thinking Processes (2026.acl-long)

Challenge:	Existing TTC scaling strategies and reasoning scorers are fragmented and evaluated under inconsistent protocols.
Approach:	They propose a framework for seamless test-time compute scaling of large language model reasoning . they use a modular Python library to implement state-of-the-art scaling strategy and scorer families .
Outcome:	The proposed framework evaluates performance and computational efficiency on mathematical and coding tasks.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

M3: Multi-level dataset for Multi-document summarisation of Medical studies (2022.findings-emnlp)

Challenge:	Large Language Models (LLMs) can learn useful knowledge and harmful stereotypes, making bias evaluation essential.
Approach:	They propose a multilingual social bias benchmark that incorporates human-generated reasoning as part of the thinking process.
Outcome:	The proposed method demonstrates superior performance over LLM-generated methods . human-generated thinking yields higher-quality evaluations than template-based approaches .

Copied to clipboard

Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, Antonio Jimeno Yepes, Jey Han Lau

NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages (2023.eacl-main)

Challenge:	Existing summarisation systems are not up to such complex tasks, yet limited tools exist to determine where and why they are failing.
Approach:	They propose to use a dataset to evaluate the quality of summarisation systems in the biomedical domain.
Outcome:	The proposed model can be used to evaluate the quality of summarisation systems in the biomedical domain.

Copied to clipboard

Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, Sebastian Ruder

Arabic Dataset for LLM Safeguard Evaluation (2025.naacl-long)

Challenge:	In Indonesia, many languages are endangered and some are even extinct due to the unavailability of data resources and benchmarks.
Approach:	They propose a high-quality multilingual parallel corpus that covers 10 local languages from Indonesia.
Outcome:	The proposed resource includes sentiment and machine translation datasets, and bilingual lexicons.

Copied to clipboard

Yasser Ashraf, Yuxia Wang, Bin Gu, Preslav Nakov, Timothy Baldwin

Robustness in NLP

Challenge:	Existing studies on large language models have focused on English, but the safety of LLMs in Arabic remains under-explored.
Approach:	They propose to use Arabic-region-specific questions to evaluate LLMs' safety . they use a dual-perspective evaluation framework to examine differences between LLM responses .
Outcome:	The proposed framework assesses the LLM responses from both governmental and opposition viewpoints.

Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models (2025.naacl-long)

Copied to clipboard

Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov

One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia (2022.acl-long)

Challenge:	Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs).
Approach:	They propose to use a well-established method for text generation to extract token embeddings from multiple layers of LLMs and compute MD scores for each token.
Outcome:	The proposed method improves on existing methods and provides accurate and computationally efficient uncertainty scores for sequence-level selective generation and claim-level fact-checking tasks.

Copied to clipboard

Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, Sebastian Ruder

On the Interplay between Human Label Variation and Model Fairness (2026.findings-eacl)

Challenge:	There are more than 700 languages spoken in Indonesia, equal to 10% of the world's languages, second only to Papua New Guinea.
Approach:	They focus on the languages spoken in Indonesia, the world's second most linguistically diverse nation, and the fourth most populous nation of the world.
Outcome:	The proposed model is based on the languages spoken in Indonesia, the world's second-most linguistically diverse nation, with 273 million people spread over 17,508 islands.

Copied to clipboard

Kemal Kurniawan, Meladel Mistica, Timothy Baldwin, Jey Han Lau

BiMediX2 : Bio-Medical EXpert LMM for Diverse Medical Modalities (2025.findings-emnlp)

Challenge:	Existing studies on the impact of human label variation on model fairness have not explored the interaction between HLV and performance.
Approach:	They compare human label variation (HLV) training methods with four other methods . they find that HLV methods improve performance without harming fairness .
Outcome:	The proposed methods improve fairness without explicit debiasing under certain configurations.

Copied to clipboard

Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Sara Pieri, Saeed Yahya Alseiari, Shanavas Cholakkal, Khaled M Aldahmani, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan, Timothy Baldwin, Hisham Cholakkal

Multimodality

Challenge:	BiMediX2 is a bilingual (Arabic-English) large multimodal model that supports text-based and image-based medical interactions.
Approach:	They introduce BiMediX2, a bilingual (Arabic-English) Bio-Medical EXpert Large Multimodal Model that supports text-based and image-based medical interactions.
Outcome:	The model outperforms existing models by over 9% in English and more than 20% in Arabic evaluations.

The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature (2022.acl-long)

Copied to clipboard

Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, Jey Han Lau

Unsupervised Lexical Simplification with Context Augmentation (2023.findings-emnlp)

Challenge:	Existing evaluation approaches to multi-document summarization of biomedical literature lack consistency and transparency.
Approach:	They propose a systematic approach to human evaluation of biomedical summaries and apply it to analyze the summary generated by two current evaluation models.
Outcome:	The proposed evaluation framework is based on two state-of-the-art models and examines the summaries generated by the two models to understand the deficiencies of existing evaluation approaches.

Copied to clipboard

Takashi Wada, Timothy Baldwin, Jey Lau

LM-Polygraph: Uncertainty Estimation for Language Models (2023.emnlp-demo)

Challenge:	Existing unsupervised lexical simplification methods only use monolingual data and pre-trained models.
Approach:	They propose an unsupervised method that generates substitutes based on monolingual data and pre-trained language models.
Outcome:	The proposed method outperforms existing models on the TSAR-2022 task in English, Portuguese, and Spanish.

Copied to clipboard

Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov

Zero-shot Sentiment Analysis in Low-Resource Languages Using a Multilingual Sentiment Lexicon (2024.eacl-long)

Challenge:	Large language models often "hallucinate" i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements.
Approach:	They propose a framework with implementations of state-of-the-art UE methods for LLMs with unified program interfaces in Python.
Outcome:	The proposed framework implements state-of-the-art UE methods for LLMs with unified program interfaces in Python and an extendable benchmark for consistent evaluation by researchers.

Copied to clipboard

Fajri Koto, Tilman Beck, Zeerak Talat, Iryna Gurevych, Timothy Baldwin

Unsupervised Paraphrasing of Multiword Expressions (2023.findings-acl)

Challenge:	Prior work extended multilingual models to other languages due to the unavailability of labeled and unlabeled training data.
Approach:	They use multilingual lexicons to enhance multilingual models capabilities in low-resource languages . they focus on zero-shot sentiment analysis tasks across 34 languages based on a single sentence .
Outcome:	The proposed model improves zero-shot performance across 34 languages without using any sentence-level sentiment data.

Copied to clipboard

Takashi Wada, Yuji Matsumoto, Timothy Baldwin, Jey Han Lau

Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020.acl-main)

Challenge:	Existing methods for paraphrasing multiword expressions in context are unsupervised . multiwords are notoriously difficult to model because the meaning of the whole can diverge substantially from that of the component words.
Approach:	They propose an unsupervised approach to paraphrasing multiword expressions in context using monolingual corpus data and pre-trained language models.
Outcome:	The proposed method outperforms all unsupervised systems and rivals supervised systems on the SemEval 2022 idiomatic text similarity task.

Copied to clipboard

Nitika Mathur, Timothy Baldwin, Trevor Cohn

Uncertainty Quantification for Large Language Models (2025.acl-tutorials)

Challenge:	Existing methods for judging metrics are sensitive to the translations used for evaluation, leading to falsely confident conclusions about a metric’s efficacy.
Approach:	They propose a method for thresholding performance improvement under an automatic metric against human judgements by using a pairwise system ranking method.
Outcome:	The proposed method allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected.

Copied to clipboard

Artem Shelmanov, Maxim Panov, Roman Vashurin, Artem Vazhentsev, Ekaterina Fadeeva, Timothy Baldwin

Loki: An Open-Source Tool for Fact Verification (2025.coling-demos)

Challenge:	Large language models (LLMs) produce hallucinations, which undermine user trust and reliability.
Approach:	This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks.
Outcome:	The proposed framework provides tools for communicating the reliability of a model answer.

Copied to clipboard

Haonan Li, Xudong Han, Hao Wang, Yuxia Wang, Minghan Wang, Rui Xing, Yilin Geng, Zenan Zhai, Preslav Nakov, Timothy Baldwin

Hierarchical Structured Model for Fine-to-Coarse Manifesto Text Analysis (N18-1)

Challenge:	Loki is an open-source fact-checking tool designed to address the growing problem of misinformation.
Approach:	They propose a tool that breaks down the fact-checking task into five steps . they propose LOKI, which offers a semiautomated, human-in-the-loop approach .
Outcome:	a new open-source tool is designed to address the growing problem of misinformation . the tool breaks down the fact-checking task into five steps to assist human judgment .

Copied to clipboard

Shivashankar Subramanian, Trevor Cohn, Timothy Baldwin

Challenge:	Election manifestos document the intentions, motives, and views of political parties.
Approach:	They propose a hierarchical structured deep model to predict fine- and coarse-grained positions and a probabilistic soft logic model to perform post-hoc calibration of coarse- and fine-grain positions.
Outcome:	The proposed model outperforms state-of-the-art approaches at both granularities using manifestos from twelve countries, written in ten different languages.

Recurrent Entity Networks with Delayed Memory Update for Targeted Aspect-Based Sentiment Analysis (N18-2)

Copied to clipboard

Fei Liu, Trevor Cohn, Timothy Baldwin

Sentiment Analysis

Challenge:	Recent work on target-dependent biLSTMs has shown that they are ineffective in aspect-based sentiment analysis.
Approach:	They propose a novel architecture that uses external memory chains with a delayed memory update mechanism to track entities.
Outcome:	The proposed model improves on a TABSA task using external memory chains with a delayed memory update mechanism.

Investigating How Pre-training Data Leakage Affects Models’ Reproduction and Detection Capabilities (2025.emnlp-main)

Copied to clipboard

Masahiro Kaneko, Timothy Baldwin

Balancing out Bias: Achieving Fairness Through Balanced Training (2022.emnlp-main)

Challenge:	Existing studies do not examine how leaked instances in training datasets influence LLMs’ output and detection capabilities.
Approach:	They conduct an experimental survey to examine the relationship between data leakage in training datasets and its effects on the generation and detection by Large Language Models (LLMs).
Outcome:	The results show that enhancing leakage detection through few-shot learning can help mitigate the impact of the leakage rate in the training data on detection performance.

Copied to clipboard

Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation (2022.aacl-main)

Challenge:	Existing approaches to reducing group bias do not account for correlations between author demographics and linguistic variables, limiting their effectiveness.
Approach:	They extend a method for countering group bias using balanced training by balancing each demographic group in training and using protected attributes as input.
Outcome:	The proposed model outperforms all other methods when combined with balanced training.

Copied to clipboard

Thinh Hung Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau, Karin Verspoor

Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation (P19-1)

Challenge:	Negation is an important linguistic phenomenon which denotes non-existence, denial, or contradiction.
Approach:	They propose a natural language inference test suite to test models for negation . they use a linguistic framework to analyze negation types and constructions .
Outcome:	The proposed test suite is more challenging than existing benchmarks on negation . it includes annotation of negation types and constructions grounded in linguistic theory .

Copied to clipboard

Nitika Mathur, Timothy Baldwin, Trevor Cohn

NusaCrowd: Open Source Initiative for Indonesian NLP Resources (2023.findings-acl)

Challenge:	Existing evaluation metrics are limited and can be easily portable to new languages.
Approach:	They propose a simple unsupervised metric and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences.
Outcome:	The proposed model outperforms existing metrics on the WMT 2017 dataset and is more accurate than existing models.

Copied to clipboard

Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan, Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, James Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ichwanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebastian Ruder, Pascale Fung, Herry Sujaini, Sakriani Sakti, Ayu Purwarianti

Challenge:	Existing NLP research in Indonesian languages has been held back by factors such as language diversity, orthographic variation, resource limitation and other societal challenges.
Approach:	They present a collaborative initiative to collect and unify existing resources for Indonesian languages and open access to previously non-public resources.
Outcome:	The results show that the datasets are highly reliable and can be used to generate the first zero-shot benchmarks for natural language understanding and generation in Indonesian and the local languages of Indonesia.

Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (2025.emnlp-main)

Copied to clipboard

Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Gleb Kuzmin, Ivan Lazichny, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov, Artem Shelmanov

A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (2025.emnlp-main)

Challenge:	Uncertainty quantification (UQ) is a promising approach for detecting hallucinations and low-quality outputs of Large Language Models (LLMs).
Approach:	They propose to learn conditional dependency between autoregressive LLM generation steps from attention-based features and a two-staged training procedure to incorporate recurrent features.
Outcome:	The proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.

Copied to clipboard

Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin

What’s in a Domain? Learning Domain-Robust Text Representations using Adversarial Training (N18-2)

Challenge:	Uncertainty quantification (UQ) is a framework for assessing the reliability of model outputs.
Approach:	They introduce pre-trained UQ heads for LLMs that are highly robust and generalized to languages they were not explicitly trained on.
Outcome:	The pre-trained heads significantly improve their ability to capture uncertainty compared to unsupervised methods.

Copied to clipboard

Yitong Li, Timothy Baldwin, Trevor Cohn

Robustness in NLP

Challenge:	a key roadblock is application to new domains, unseen in training.
Approach:	They propose a method to optimise in- and out-of-domain accuracy by combing domain-specific and domain-general components with adversarial training for domain.
Outcome:	The proposed method improves on domain adaptation and domain-adversarial training.

CMMLU: Measuring massive multitask language understanding in Chinese (2024.findings-acl)

Copied to clipboard

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, Timothy Baldwin

SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning (2026.eacl-long)

Challenge:	Existing large language models struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams.
Approach:	They propose to use CMMLU to evaluate Chinese multilingual and Chinese LLMs in a comprehensive benchmark that covers various subjects and settings.
Outcome:	The proposed benchmark covers natural sciences, social sciences, engineering, and the humanities and aims to improve on existing models.

Copied to clipboard

Renxi Wang, Honglin Mu, Liqun Ma, Lizhi Lin, Yunlong Feng, Timothy Baldwin, Xudong Han, Haonan Li

Systematic Evaluation of Predictive Fairness (2022.aacl-main)

Challenge:	Long-context understanding is a critical capability for large language models . evaluating this capability requires extensive human annotation, which is time-consuming and costly.
Approach:	They propose a benchmark to assess citation-grounded long-context reasoning in academic writing.
Outcome:	The proposed benchmark compares state-of-the-art models with human experts on two tasks . human experts achieve 90% accuracy, but most models struggle with the cloze-style task .

Copied to clipboard

Xudong Han, Aili Shen, Trevor Cohn, Timothy Baldwin, Lea Frermann

Decoupling Adversarial Training for Fair NLP (2021.findings-acl)

Challenge:	Several methods have been proposed to mitigate bias in training on biased datasets.
Approach:	They propose to examine the effect of target class imbalance and stereotyping on model performance by analyzing binary classification, profession prediction and regression tasks.
Outcome:	The proposed methods show that data conditions have a strong influence on relative model performance.

Copied to clipboard

On the (In)Effectiveness of Images for Text Classification (2021.eacl-main)

Challenge:	Existing work assumes main task labels and protected attributes are available in the dataset, but protected labels are often unavailable or only available in limited numbers.
Approach:	They propose a method which uses only a small volume of protected labels to train adversarial models using a dataset with a discriminator.
Outcome:	The proposed method can be used to transfer private-labelled instances from one dataset to another without requiring large amounts of protected labels.

Copied to clipboard

Chunpeng Ma, Aili Shen, Hiyori Yoshikawa, Tomoya Iwakura, Daniel Beck, Timothy Baldwin

WikiUMLS: Aligning UMLS to Wikipedia via Cross-lingual Neural Ranking (2020.coling-main)

Challenge:	Existing studies have focused on text classification, but have shown that images do not improve NLP tasks.
Approach:	They focus on text classification, where images complement the text and the Wikipedia page can be in one of a number of different languages.
Outcome:	The proposed model trains without external pre-training, but when combined with BERT models pre-trained on large-scale external data, images contribute nothing.

Copied to clipboard

Afshin Rahimi, Timothy Baldwin, Karin Verspoor

Do-Not-Answer: Evaluating Safeguards in LLMs (2024.findings-eacl)

Challenge:	Using a neural reranking model, we can match a UMLS concept with a Wikipedia page, enabling manual alignment with minimal effort.
Approach:	They propose a cross-lingual neural reranking model to match a UMLS concept with a Wikipedia page, which achieves a recall@1of 72%, a substantial improvement of 20% over word- and char-level BM25.
Outcome:	The proposed model achieves recall@1of 72%, 20% better than word- and char-level BM25, and will facilitate easier access to Wikipedia for health professionals, patients, and NLP systems, including in multilingual settings.

Copied to clipboard

Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, Timothy Baldwin

ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic (2024.findings-acl)

Challenge:	a dataset evaluating harmful capabilities in large language models is available at https://github.com/Libr-AI/do-not-answer.
Approach:	They collect an open-source dataset to evaluate the safeguards in large language models . they find that simple BERT-style classifiers can achieve results comparable to GPT-4 .
Outcome:	The proposed dataset compares the safety of six popular LLMs to GPT-4 on automatic safety evaluation.

Copied to clipboard

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, Timothy Baldwin

Challenge:	evaluating language models in Arabic remains challenging due to limited datasets . focus has shift to reasoning and knowledge-intensive tasks due to lack of relevant datasets.
Approach:	They propose to use ArabicMMLU to evaluate models' understanding of Arabic . they use 40 tasks and 14,575 multiple-choice questions from school exams in different countries .
Outcome:	The ArabicMMLU is the first multi-task language understanding benchmark for the Arabic language . it is based on 40 tasks and 14,575 multiple-choice questions in modern standard Arabic . the models are based in different countries across North Africa, the Levant, and the Gulf regions .

Deep Ordinal Regression for Pledge Specificity Prediction (D19-1)

Copied to clipboard

Shivashankar Subramanian, Trevor Cohn, Timothy Baldwin

Unsupervised Lexical Substitution with Decontextualised Embeddings (2022.coling-1)

Challenge:	Currently, there are no publicly available annotated datasets of pledges . a novel approach to specificity prediction is needed to predict the specificity of pledged issues.
Approach:	They propose deep ordinal regression approaches for specificity prediction using supervised and semi-supervised settings.
Outcome:	The proposed methods demonstrate their utility over several baseline approaches.

Copied to clipboard

Takashi Wada, Timothy Baldwin, Yuji Matsumoto, Jey Han Lau

Content-based Popularity Prediction of Online Petitions Using a Deep Regression Model (P18-2)

Challenge:	Existing methods for lexical substitution using pre-trained language models have some limitations.
Approach:	They propose an unsupervised method for lexical substitution using pre-trained language models.
Outcome:	The proposed method outperforms baseline models and establishes a state-of-the-art without supervision or fine-tuning.

Copied to clipboard

Shivashankar Subramanian, Timothy Baldwin, Trevor Cohn

Challenge:	Existing work on predicting popularity of online petitions based on initial popularity trajectory has focused on estimating the number of signatures a petition gets in the first x hours, and predicting the total number of signed petitions at the end of its lifetime.
Approach:	They propose a CNN-based model to predict the popularity of a petition based on its textual content and use it to model the influence of other petition signers.
Outcome:	The proposed model is based on UK and US government petition datasets and is compared with previous work on predicting popularity over time based upon initial popularity trajectory.

Top-down Discourse Parsing via Sequence Labelling (2021.eacl-main)

Copied to clipboard

Evaluating Document Coherence Modeling (2021.tacl-1)

Challenge:	Discourse analysis is a systematic way to understand how texts are segmented hierarchically into discourse units.
Approach:	They propose a top-down approach to discourse parsing that is conceptually simpler than its predecessors.
Outcome:	The proposed model eliminates the decoder and reduces the search space for splitting points.

Copied to clipboard

Aili Shen, Meladel Mistica, Bahar Salehi, Hang Li, Timothy Baldwin, Jianzhong Qi

Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings (2024.naacl-long)

Challenge:	a new study examines pretrained language models' ability to model discourse and pragmatic phenomena.
Approach:	They propose a sentence intrusion detection task using a dictionary dataset . they show that pretrained LMs perform impressively in in-domain evaluation .
Outcome:	The proposed dataset shows that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalization capacity.

Copied to clipboard

Chen Liu, Fajri Koto, Timothy Baldwin, Iryna Gurevych

Natural Language Interfaces

Challenge:	Large language models (LLMs) are adept at question answering and reasoning tasks, but when reasoning in situational context, human expectations vary depending on the relevant cultural common ground.
Approach:	They construct and evaluate a dataset for proverb understanding with conversational context for six different languages and their usage within the context.
Outcome:	The proposed model is able to reason with proverbs and sayings in conversational contexts.

NAT: Enhancing Agent Tuning with Negative Samples (2025.naacl-long)

Copied to clipboard

Renxi Wang, Xudong Han, Yixuan Zhang, Timothy Baldwin, Haonan Li

Challenge:	Existing methods for fine-tuning and reinforcement learning use only positive examples, limiting their efficiency in low-resource scenarios.
Approach:	They propose a method that leverages both successful and failed trajectories for fine-tuning, maximizing the utility of limited resources.
Outcome:	The proposed method surpasses existing methods, including SFT, DPO, and PPO, across various tasks.

ChEMU-Ref: A Corpus for Modeling Anaphora Resolution in the Chemical Domain (2021.eacl-main)

Copied to clipboard

Biaoyan Fang, Christian Druckenbrodt, Saber A Akhondi, Jiayuan He, Timothy Baldwin, Karin Verspoor

Revisiting subword tokenization: A case study on affixal negation in large language models (2024.naacl-long)

Challenge:	Using a novel annotation scheme, we identify anaphoric references in chemical patents and determine the chemical relation between linked entities.
Approach:	They propose a neural approach to anaphora resolution based on coreference and bridging links in chemical patents.
Outcome:	The proposed framework can be used to identify anaphoric references in chemical patents and determine the chemical relation between linked entities.

Copied to clipboard

Thinh Truong, Yulia Otmakhova, Karin Verspoor, Trevor Cohn, Timothy Baldwin

Syntactic Text Processing

Challenge:	Negation is central to language understanding but is not properly captured by modern NLP methods.
Approach:	They propose to use subword tokenization methods to detect negation in large language models . they find that models can reliably recognize negation, despite mismatches in tokenization accuracy .
Outcome:	The proposed models can detect negation in English using subword tokenization methods despite some mismatches in tokenization accuracy and negation detection performance.

Human Interest Framing across Cultures: A Case Study on Climate Change (2025.coling-main)

Copied to clipboard

Gisela Vallejo, Christine de Kock, Timothy Baldwin, Lea Frermann

Challenge:	Human Interest (HI) framing is a narrative strategy that injects news stories with a relatable, emotional angle and a human face to engage the audience.
Approach:	They perform a systematic analysis of HI stories to understand its role in climate change reporting in English-speaking countries from four continents.
Outcome:	The proposed approach has shown to capture and retain readership and enhance political engagement of the population.

Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)

Copied to clipboard

Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov

Automatic Classification of Neutralization Techniques in the Narrative of Climate Change Scepticism (2021.naacl-main)

Challenge:	Large language models are notorious for producing erroneous claims in their output.
Approach:	They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use.
Outcome:	The proposed method can fact-check the atomic claims in the output of large language models.

Copied to clipboard

Shraey Bhatia, Jey Han Lau, Timothy Baldwin

Text Classification

Challenge:	neutralisation is used to justify lack of action or promote an alternative view of climate change . action on climate change has become an increasingly partisan issue with strong opposition voices discrediting scientists and spreading scepticism and misinformation.
Approach:	They propose to use neutralisation techniques to introduce the problem to the nlp community and to collect manual annotations of neutralised techniques in text relating to climate change.
Outcome:	The proposed models are supervised and semi-supervised by a team of researchers from the nlp and the ccsc.

Collective Human Opinions in Semantic Textual Similarity (2023.tacl-1)

Copied to clipboard

Yuxia Wang, Shimin Tao, Ning Xie, Hao Yang, Timothy Baldwin, Karin Verspoor

Diverse Adversaries for Mitigating Bias in Training (2021.eacl-main)

Challenge:	Existing benchmarks for semantic textual similarity (STS) use averaged human ratings as gold standard.
Approach:	They propose to use a Chinese sentence-to-sentence dataset to study collective human opinions in semantic textual similarity (STS) neither a scalar nor a single Gaussian fits a set of observed judgments adequately, they argue .
Outcome:	The proposed dataset does not capture disagreements on individual instances, but rather the confidence over the aggregate dataset.

Copied to clipboard

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP (2020.coling-main)

Challenge:	Existing adversarial methods only partially mitigate the problem of model bias, added to which their training procedures are unstable.
Approach:	They propose a method where discriminators are encouraged to learn orthogonal hidden representations from one another to reduce model bias.
Outcome:	The proposed method significantly reduces bias and stability of training over standard methods.

Copied to clipboard

Fajri Koto, Afshin Rahimi, Jey Han Lau, Timothy Baldwin

The Gaps between Fine Tuning and In-context Learning in Bias Evaluation and Debiasing (2025.coling-main)

Challenge:	despite being spoken by 200 million people, the Indonesian language is underrepresented in NLP research.
Approach:	They propose a dataset for Indonesian that includes seven NLP tasks . they also propose 'indonesian language evaluation Montage' tasks that are based on previous work .
Outcome:	The proposed dataset shows that IndoBERT outperforms IndoLEM over most of the tasks.

Copied to clipboard

Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin

Target Word Masking for Location Metonymy Resolution (2020.coling-main)

Challenge:	FT-based debiasing methods cause a performance degradation in downstream tasks . FT works by updating some or all parameters, while ICL uses prompts without modifying the model parameters.
Approach:	They propose to use ICL to customize PLMs to downstream tasks without parameter updates.
Outcome:	The proposed method lowers the performance degradation of FT-based debiasing methods compared to FT models . the proposed method improves performance on large datasets while allowing for smaller changes to PLMs .

Copied to clipboard

Haonan Li, Maria Vasardani, Martin Tomko, Timothy Baldwin

Reevaluating Argument Component Extraction in Low Resource Settings (D19-61)

Challenge:	Existing word sense disambiguation and named entity recognition systems have no explicit metonymy detection.
Approach:	They propose an end-to-end word-level classification approach based only on BERT . they show that their approach generalises well to unseen data .
Outcome:	The proposed approach surpasses conventional models and benchmarks on 5 datasets and generalises well to unseen data.

Copied to clipboard

Anirudh Joshi, Timothy Baldwin, Richard Sinnott, Cecile Paris

Deep-speare: A joint neural model of poetic language, meter and rhyme (P18-1)

Challenge:	Argument component extraction is a challenging and complex high-level semantic extraction task.
Approach:	They propose to use character-level, GloVe, ELMo, and BERT encodings to compare arguments extracted using standard BiLSTM-CRF encoders.
Outcome:	The proposed approaches perform better than baselines in higher-level semantic extraction tasks and suggest future improvements.

Copied to clipboard

Jey Han Lau, Trevor Cohn, Timothy Baldwin, Julian Brooke, Adam Hammond

Challenge:	a recent surge of interest in deep learning has led to creative applications for poetry generation . a novel joint architecture captures language, rhyme and meter for sonnet modelling .
Approach:	They propose a joint architecture that captures language, rhyme and meter for sonnet modelling.
Outcome:	The proposed architecture captures language, rhyme and meter for sonnet modelling.

Towards Robust and Privacy-preserving Text Representations (P18-2)

Copied to clipboard

Yitong Li, Timothy Baldwin, Trevor Cohn

Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation (2026.findings-acl)

Challenge:	Written text often provides sufficient clues to identify the author, gender, age, and other important attributes.
Approach:	They propose to obscure important author characteristics at training time so that models are invariant to these attributes.
Outcome:	The proposed approach leads to increased privacy in the learned representations, and robust models to varying evaluation conditions, including out-of-domain corpora.

Copied to clipboard

Ekaterina Fadeeva, Aleksandr Rubashevskii, Dzianis Piatrashyn, Roman Vashurin, Shehzaad Dhuliawala, Artem Shelmanov, Timothy Baldwin, Preslav Nakov, Mrinmaya Sachan, Maxim Panov

KFCNet: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021.findings-emnlp)

Challenge:	Existing approaches to mitigating hallucinations conflate factuality with faithfulness to the retrieved evidence, incorrectly labeling factually correct statements as hallucinos . Existing methods to mitigate hallucinics rely on a lack of training data coverage, input ambiguity, and architectural constraints.
Approach:	They propose a method for hallucination detection in Large Language Models enhanced with knowledge retrieval based on faithfulness to the retrieved context.
Outcome:	The proposed method outperforms unsupervised UQ baselines, RAG-specific methods, and supervised classifiers across multiple tasks and LLMs.

Copied to clipboard

Haonan Li, Yeyun Gong, Jian Jiao, Ruofei Zhang, Timothy Baldwin, Nan Duan

Discourse Probing of Pretrained Language Models (2021.naacl-main)

Challenge:	Pre-trained language models have led to substantial gains over a broad range of NLP tasks, but have limitations for high-quality tasks such as commonsense generation and ad keyword generation.
Approach:	They propose a Knowledge Filtering and Contrastive learning Network which references external knowledge and achieves better generation performance.
Outcome:	The proposed model outperforms the current state of the art on the CommonGen benchmark by a large margin.

Copied to clipboard

Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling (2026.findings-acl)

Challenge:	Existing work on probing of pretrained language models has focused on sentence-level syntactic tasks.
Approach:	They introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document- level relations.
Outcome:	The proposed model performs best in encoder, but only in the encoder layer.

Copied to clipboard

Ivan Rodkin, Daniil Orel, Konstantin Smirnov, Arman Bolatov, Bilal Elbouardi, Besher Hassan, Yuri Kuratov, Aydar Bulatov, Preslav Nakov, Timothy Baldwin, Artem Shelmanov, Mikhail Burtsev

IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization (2021.emnlp-main)

Challenge:	Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear.
Approach:	They evaluate how large language models learn multi-step reasoning without memorization . they find that most neural architectures trained from scratch can learn rule inference .
Outcome:	The proposed framework fails to solve a natural-language proxy task with high accuracy.

Copied to clipboard

Language models are not naysayers: an analysis of language models on negation benchmarks (2023.starsem-1)

Challenge:	In IndoBERTweet, a pretraining model for Indonesian Twitter is extended with domain-specific vocabulary.
Approach:	They propose a pretraining model that extends a monolingual Indonesian BERT model with domain-specific vocabulary.
Outcome:	The proposed model can be initialized with the average BERT subword embedding five times faster than existing methods for vocabulary adaptation.

Copied to clipboard

Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, Trevor Cohn

Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU (2023.emnlp-main)

Challenge:	Negation has been shown to be a major bottleneck for masked language models, such as BERT, but whether this finding still holds for larger-sized auto-regressive language models has not been studied comprehensively.
Approach:	They evaluate the ability of current-generation auto-regressive language models to handle negation using a wide range of benchmarks and models.
Outcome:	The proposed models are compared against a wide range of negation benchmarks and show that they are insensitive to negation, inability to capture the lexical semantics of negations, and failure to reason under negation.

Copied to clipboard

Fajri Koto, Nurul Aisyah, Haonan Li, Timothy Baldwin

Inference-Time Selective Debiasing to Enhance Fairness in Text Classification Models (2025.naacl-short)

Challenge:	Existing studies on large language models based on English datasets do not provide adequate data for evaluating their capabilities beyond English.
Approach:	They propose a multi-task language understanding benchmark for Indonesian culture and languages . it measures language proficiency, reasoning abilities and real-world knowledge .
Outcome:	The proposed model passes the primary school level in Indonesia, while other models perform at lower levels.

Copied to clipboard

Gleb Kuzmin, Neemesh Yadav, Ivan Smirnov, Timothy Baldwin, Artem Shelmanov

Modelling Uncertainty in Collaborative Document Quality Assessment (D19-55)

Challenge:	Several studies have investigated and promoted fairness, and a variety of definitions have been proposed to address this problem.
Approach:	They propose a selective debiasing method that removes bias from model predictions instead of discarding them at inference time.
Outcome:	The proposed method achieves better results than standard uncertainty quantification methods on text classification datasets with encoder-based classification models.

Copied to clipboard

Aili Shen, Daniel Beck, Bahar Salehi, Jianzhong Qi, Timothy Baldwin

Challenge:	Existing work on document quality assessment relies on predicting the quality of a document relative to a putative gold standard, without paying attention to the subjectivity of this task.
Approach:	They propose to use Gaussian processes and random forests to measure the uncertainty in document quality predictions.
Outcome:	The proposed methods can predict the quality of Wikipedia articles while providing an estimate of uncertainty when there is inconsistency in the quality labels from the contributors.

Encoding Sentiment Information into Word Vectors for Sentiment Analysis (C18-1)

Copied to clipboard

Zhe Ye, Fang Li, Timothy Baldwin

Improving negation detection with negation-focused pre-training (2022.naacl-main)

Challenge:	Existing methods for embedding sentiment knowledge into word vectors are generally trained independently of the downstream task.
Approach:	They propose to encode sentiment knowledge into pre-trained word vectors to improve sentiment analysis.
Outcome:	The proposed method improves sentiment analysis on four popular sentiment datasets compared to benchmark methods.

Copied to clipboard

Thinh Truong, Timothy Baldwin, Trevor Cohn, Karin Verspoor

More than Votes? Voting and Language based Partisanship in the US Supreme Court (2023.findings-emnlp)

Challenge:	Negation is a common linguistic feature that is crucial in many language understanding tasks.
Approach:	They propose a new approach to detect negation in language models using data augmentation and negation masking.
Outcome:	The proposed approach improves negation detection performance and generalizability over the strong baseline NegBERT.

Copied to clipboard

Biaoyan Fang, Trevor Cohn, Timothy Baldwin, Lea Frermann

Semi-supervised Stochastic Multi-Domain Learning using Variational Inference (P19-1)

Challenge:	partisanship and ideology have been a key topic in legal studies of the US Supreme Court . most research quantifies partisan behavior based on voting behavior, and oral arguments have not been well studied for this purpose.
Approach:	They propose a framework for analyzing justices' oral arguments for partisan signals and how they align with voting patterns.
Outcome:	The proposed framework shows that the affiliated party of justices can be predicted reliably from their oral contributions.

Copied to clipboard

Yitong Li, Timothy Baldwin, Trevor Cohn

BiMediX: Bilingual Medical Mixture of Experts LLM (2024.findings-emnlp)

Challenge:	Supervised NLP models rely on large collections of text which closely resemble intended testing setting. however, data is often messy, with domain labels not always available, or providing limited information about the style and genre of text.
Approach:	They propose a method to distill the important domain signal as part of a multi-domain learning system using a latent variable model.
Outcome:	The proposed model improves performance over benchmark domain adaptation methods . text corpora are often collated from several different sources, including news, literature, microblogs, and web crawls .

Copied to clipboard

Sara Pieri, Sahal Shaji Mullappilly, Fahad Khan, Rao Anwer, Salman Khan, Timothy Baldwin, Hisham Cholakkal

Evaluating Evidence Attribution in Generated Fact Checking Explanations (2025.naacl-long)

Challenge:	a new bilingual medical mixture of experts LLM is designed for seamless interaction in both English and Arabic.
Approach:	They propose a semi-automated English-to-Arabic translation pipeline with human refinement to ensure high-quality translations.
Outcome:	The proposed model outperforms state-of-the-art medical LLMs in Arabic and Arabic . it outperformed the generic Arabic-English bilingual LLM, Jais-30B by 10% and 15% .

Copied to clipboard

Rui Xing, Timothy Baldwin, Jey Han Lau

MultiSpanQA: A Dataset for Multi-Span Question Answering (2022.naacl-main)

Challenge:	Existing fact-checking systems struggle with attribution quality, as their generated explanations can include hallucinations.
Approach:	They propose a protocol to assess attribution quality in fact-checking explanations using human annotation and automatic annotation.
Outcome:	The proposed protocol can be automated, the authors show . best-performing LLMs still generate explanations that are not always accurate .

Copied to clipboard

Haonan Li, Martin Tomko, Maria Vasardani, Timothy Baldwin

Natural Language Interfaces

Challenge:	Existing reading comprehension datasets focus on single-span answers, but multi-spread questions are less studied.
Approach:	They propose a new reading comprehension dataset that focuses on multi-span questions . they introduce new metrics for the purposes of multi--spontaneous question answering evaluation .
Outcome:	The proposed model beats baselines and achieves state-of-the-art on the existing dataset.

Psychometric Predictive Power of Large Language Models (2024.findings-naacl)

Copied to clipboard

Tatsuki Kuribayashi, Yohei Oseki, Timothy Baldwin

Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models (2026.acl-long)

Challenge:	despite efforts to align large language models with human preferences, instruction tuning does not always make LLMs human-like from a cognitive modeling perspective.
Approach:	They find that instruction tuning does not always make large language models human-like from a cognitive perspective.
Outcome:	The proposed prompts improve predictive power but are still inferior to small base models.

Copied to clipboard

Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan

Does Representational Fairness Imply Empirical Fairness? (2022.findings-aacl)

Challenge:	Existing verification approaches, such as Process Reward Models, are computationally expensive and limited to specific domains.
Approach:	They propose a transformer-based probe that uses internal states of frozen LLMs to estimate credibility of reasoning steps during generation.
Outcome:	The proposed probes match or exceed PRMs that are up to 810 larger.

Copied to clipboard

Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, Lea Frermann

Evaluating the Efficacy of Summarization Evaluation across Languages (2021.findings-acl)

Challenge:	Neural methods have been trained on datasets which embody cultural and societal stereotypes, captured in spurious correlations between target labels and protected attributes.
Approach:	They propose a debiasing method that encourages a latent space that separates instances based on target label, while mixing instances that share protected attributes.
Outcome:	The proposed method shows that representational fairness does not imply empirical fairness across methods.

Copied to clipboard

Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024.acl-srw)

Summarization

Challenge:	Using multilingual summarization evaluation methods is more reliable and interpretable than manual methods.
Approach:	They propose to use multilingual BERT within BERTScore to evaluate summarization evaluation metrics . they use English datasets that are not representative of modern summarizing systems .
Outcome:	The proposed methods perform well across all languages, at a level above that for English.

Copied to clipboard

Renxi Wang, Haonan Li, Minghao Wu, Yuxia Wang, Xudong Han, Chiyu Zhang, Timothy Baldwin

Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models (2025.emnlp-main)

Challenge:	Instruction tuning is effective for aligning large language models with human instructions, but the procedure to optimizing the mixing of instruction datasets is still unclear.
Approach:	They categorize instructions into three primary types: NLP downstream tasks, coding, and general chat.
Outcome:	The proposed method improves performance of large language models (LLMs) but it is difficult to combine different instruction datasets to optimize overall performance.

Copied to clipboard

Masahiro Kaneko, Alham Fikri Aji, Timothy Baldwin

Contextualization of Morphological Inflection (N19-1)

Challenge:	Existing approaches address key factors that influence multilingual ICL, but they do not integrate them into the model.
Approach:	They propose a method that quantifies and optimally balances three factors for improved example selection.
Outcome:	Experiments on mCSQA and TYDI show that the proposed method outperforms existing methods.

Copied to clipboard

Ekaterina Vylomova, Ryan Cotterell, Trevor Cohn, Timothy Baldwin, Jason Eisner

Morphology

Challenge:	In this paper, we isolate the task of predicting a fully inflected sentence from its partially lemmatized version.
Approach:	They propose a task that requires morphological features to be inferred from sentential context . they propose morphology-based models that explicitly reconstruct morphologic features before predicting inflected forms .
Outcome:	The proposed model is able to predict inflected sentences without relying on morphological annotations.

Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks (2023.findings-emnlp)

Copied to clipboard

Yichen Huang, Timothy Baldwin

Robustness in NLP

Challenge:	BERTScore, BLEURT, and COMET are automatic evaluation metrics that are often underperformed on adversarially-synthesized texts.
Approach:	They examine MT evaluation metric performance on adversarially-synthesized texts . they validate that automatic metrics tend to overpenalize adversarial-degraded translations .
Outcome:	The results show that automatic metrics tend to overpenalize adversarially-degraded translations.

Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression (2022.tacl-1)

Copied to clipboard

Yuxia Wang, Daniel Beck, Timothy Baldwin, Karin Verspoor

‘Just What do You Think You’re Doing, Dave?’ A Checklist for Responsible Data Use in NLP (2021.findings-emnlp)

Challenge:	State-of-the-art classification and regression models are often not well calibrated and can be inaccurate.
Approach:	They quantify calibration of pre- trained language models for text regression . they apply uncertainty estimates to augment training data in low-resource domains .
Outcome:	The proposed model calibrations improve performance and generalizability in low-resource settings.

Copied to clipboard

Anna Rogers, Timothy Baldwin, Kobi Leins

Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLP (2023.eacl-main)

Challenge:	a key part of the NLP ethics movement is responsible use of data, but what that means is unclear . a proposed checklist for responsible data (re-)use could standardise peer review of submissions .
Approach:	They propose a checklist for responsible data use that could standardise peer review . they propose implementing a standard for data (re-)use across NLP conferences .
Outcome:	The proposed checklist would standardise peer review of submissions and enable more in-depth view of published research across the community.

Copied to clipboard

Topic Intrusion for Automatic Topic Model Evaluation (D18-1)

Challenge:	Modern NLP systems exhibit a range of biases, which a growing literature on model debiasing attempts to correct.
Approach:	They propose to clarify the current situation and plot a course for meaningful progress in fair learning by making clear inter-relations among the current gamut of methods and their relation to fairness theory.
Outcome:	The proposed approach addresses the practical problem of model selection, which involves a trade-off between fairness and accuracy and has led to systemic issues in fairness research.

Copied to clipboard

Shraey Bhatia, Jey Han Lau, Timothy Baldwin

Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis? (2020.acl-main)

Challenge:	Topic coherence is increasingly being used to evaluate topic models and filter topics for end-user applications.
Approach:	They propose to use topic intrusion to guess an outlier topic given a document and a few topics to automate the task.
Outcome:	The proposed method improves upon the state-of-the-art method and shows it can be used as an alternative to topic perplexity evaluation.

Copied to clipboard

Kobi Leins, Jey Han Lau, Timothy Baldwin