Papers by Paula Buttery
Prompting open-source and commercial language models for grammatical error correction of English learner text (2024.findings-acl)
Copied to clipboard
Christopher Davis, Andrew Caines, O Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, Paula Buttery
| Challenge: | Recent advances in generative AI have enabled us to prompt large language models (LLMs) to produce texts which are fluent and grammatical. |
| Approach: | They evaluate model performance by measuring their performance on established benchmarks. |
| Outcome: | The proposed models outperform supervised English GEC models on fluency correction benchmarks and commercial LLMs on edit benchmarks. |
Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions (2020.acl-main)
Copied to clipboard
| Challenge: | a growing demand for the ability to communicate in English means automated tutoring and assessment systems are becoming more popular. |
| Approach: | They propose to use automatic speech recognition transcripts to grade spontaneous speech based on textual features. |
| Outcome: | The proposed system improves on a transformer encoder with native language identification as an auxiliary task. |
Rubrik’s Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset (2025.acl-long)
Copied to clipboard
Diana Galvan-Sosa, Gabrielle Gaudeau, Pride Kavumba, Yunmeng Li, Hongyi Gu, Zheng Yuan, Keisuke Sakaguchi, Paula Buttery
| Challenge: | Large-Language Models (LLMs) are increasingly being used in explanation generation tasks due to their unreliability. |
| Approach: | They propose a rubric and a dataset of 26k explanations written and quality-annotated using the rubric by humans and six open- and closed-source LLMs to test their proposed rubric. |
| Outcome: | The proposed rubric and CUBE dataset focuses on reasoning and language tasks and provides the necessary diversity to test it. |
Distractor Generation Using Generative and Discriminative Capabilities of Transformer-based Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Multiple Choice Questions (MCQs) are used to test language learners' comprehension and knowledge. |
| Approach: | They propose an automatic distractor generation approach which generates correct and incorrect answer options and then discriminates potential correct options from distractors. |
| Outcome: | The proposed approach outperforms previous models on multiple choice questions and reading comprehension questions. |
Logging Keystrokes in Writing by English Learners (2024.lrec-main)
Copied to clipboard
Georgios Velentzas, Andrew Caines, Rita Borgo, Erin Pacquetet, Clive Hamilton, Taylor Arnold, Diane Nicholls, Paula Buttery, Thomas Gaillat, Nicolas Ballier, Helen Yannakoudakis
| Challenge: | Essay writing is a skill commonly taught and practised in schools. |
| Approach: | They collect and analyse data representing the essay writing process from start to finish by recording every keystroke from multiple writers participating in the study. |
| Outcome: | The data collected from 1,006 writers is compared against a standard dataset of texts, keystroke logs and metadata for public release. |
Tending Towards Stability: Convergence Challenges in Small Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Increasing the number of parameters in language models is a common strategy to enhance performance, but smaller models often underperform compared to their larger counterparts due to their reduced representational capacity. |
| Approach: | They use the Pythia model suite to analyse the training dynamics that underlie this phenomenon. |
| Outcome: | The proposed model suite enables us to examine the training dynamics of small models. |
Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasing (2026.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (LMs) have grown substantially in both societal adoption and training costs. |
| Approach: | They propose to use low-cost proxy models to democratise pre-model debiasing research by using small and mutable corpora. |
| Outcome: | The proposed model can approximate bias acquisition and learning dynamics of larger models despite their reduced size. |
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models rely on frequency information because they maximize the likelihood of tokens during training. |
| Approach: | They propose a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. |
| Outcome: | The proposed method reduces the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. |
REPROLANG 2020: Automatic Proficiency Scoring of Czech, English, German, Italian, and Spanish Learner Essays (2020.lrec-1)
Copied to clipboard
| Challenge: | a new paper aims to reproduce the work described in Vajjala & Rama (2018) . the paper focuses on features-based and neural approaches to essay scoring in Czech, German and Italian . |
| Approach: | They propose to replicate the work described in Vajjala & Rama 2018, ‘Experiments with universal CEFR classification’, as part of REPROLANG 2020. |
| Outcome: | The proposed methods perform better than feature-based models for large text datasets, though neural network modifications do bring performance closer to the best feature-driven models. |
Using LLMs to simulate students’ responses to exam questions (2024.findings-emnlp)
Copied to clipboard
Luca Benedetto, Giovanni Aradelli, Antonia Donvito, Alberto Lucchetti, Andrea Cappelli, Paula Buttery
| Challenge: | Existing studies have used Large Language Models to simulate students answering exam questions . a proposed prompt for GPT-3.5 is not suitable for all LLMs, and there is no correlation between the quality of the rationales obtained with the model and the accuracy of the student simulation task. |
| Approach: | They propose a large language model prompt engineered for GPT-3.5 that can be used to answer exam questions simulating students of different skill levels. |
| Outcome: | The proposed prompt is robust to different educational domains and generalise to data unseen during prompt engineering phase. |
PictureStories: Predicting the Task Adherence of Language Learner Answers to a Picture Story-Based Writing Task (2026.eacl-long)
Copied to clipboard
| Challenge: | a lack of suitable training and evaluation data limits the evaluation of language learning tasks to language proficiency only. |
| Approach: | They develop a marking rubric that covers task adherence with respect to form and content. |
| Outcome: | The proposed model can predict the adherence of learners to written tasks using picture stories. |
Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research (2025.emnlp-demos)
Copied to clipboard
Richard Diehl Martinez, David Demitri Africa, Yuval Weiss, Suchir Salhan, Ryan Daniels, Paula Buttery
| Challenge: | Recent advances in large language models (LLMs) have enabled strong performance across diverse tasks, but small enough to train on modest budgets. |
| Approach: | They propose a lightweight, modular framework that enables systematic, hypothesis-driven research for small and medium-scale language model development. |
| Outcome: | The proposed framework enables systematic, hypothesis-driven research for small and medium-scale language model development. |
Beyond Names: How Grammatical Gender Markers Bias LLM-based Educational Recommendations (2026.eacl-long)
Copied to clipboard
| Challenge: | grammatical gender cues alone trigger substantial distributional shifts in educational recommendations . authors show that up to 76% of the bias exhibited when using prompts with proper names is already present with grammatical gender markers alone. |
| Approach: | They investigate gender biases exhibited by LLM-based virtual assistants in Italian . they show that simply changing noun and adjective endings significantly shifts recommendations . |
| Outcome: | The findings highlight the need for robust bias evaluation and mitigation strategies before deploying LLM-based virtual assistants in student-facing contexts. |
CEPOC: The Cambridge Exams Publishing Open Cloze dataset (2022.lrec-1)
Copied to clipboard
| Challenge: | This paper presents the first dataset of open cloze tests for language learners at different proficiency levels. |
| Approach: | They present the Cambridge Exams Publishing Open Cloze (CEPOC) dataset . they perform a set of experiments on three tasks: gap filling, gap prediction, and CEFR text classification. |
| Outcome: | The results of the study are promising for a number of NLP tasks. |
Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing open cloze tests are laborious to design because they require a large number of variables to predict the distribution of words in a text passage. |
| Approach: | They propose a transformer-based model that exploits generation and discrimination capabilities to improve performance. |
| Outcome: | The proposed model outperforms previous work and baselines in 82% accuracy and can be used as a future benchmark. |
Grammatical error detection in transcriptions of spoken English (2020.coling-main)
Copied to clipboard
| Challenge: | CrowdED corpus of spoken English monologues on business topics was crowdsourced from native speakers of English and learners of English with German as their first language. |
| Approach: | They propose to use the corpus recordings to correct existing speech transcriptions and edit them to make them more fluent. |
| Outcome: | The proposed transcription corrections and annotations can be used for automatic transcription post-editing and grammatical error correction for spoken English. |