Papers by Mark Gales
Is It Possible to Modify Text to a Target Readability Level? An Initial Investigation Using Zero-Shot Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Text simplification and elaboration tasks are limited to only relatively altering the readability of texts to cater to a diverse audience. |
| Approach: | They propose to generate 8 versions of a text at different readability levels using ChatGPT and Llama-2 and introduce a two-step process to generate paraphrases. |
| Outcome: | The proposed task requires the generation of 8 versions at various target readability levels for each input text. |
LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have enabled impressive zero-shot capabilities across various natural language tasks. |
| Approach: | They propose two ways to exploit the emergent abilities of large language models for NLG assessment. |
| Outcome: | The proposed methods improve performance and positional biases in comparisons between candidates. |
WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent work has shown that small, context-dependent shifts in word distributions can be used to apply and detect watermarks, but little work has analyzed the impact of these perturbations on the quality of generated texts. |
| Approach: | They propose a framework that allows for analysis of the impact of watermark settings on the quality of generated texts. |
| Outcome: | The proposed framework provides easy visualization of the quality-detection trade-off of watermark settings. |
Investigating the Emergent Audio Classification Ability of ASR Foundation Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Text and vision foundation models can perform many tasks in a zero-shot setting . however, there has been little work on the zero-shoot abilities of ASR foundation models . |
| Approach: | They investigate the ability of ASR foundation models to perform zero-shot audio classification using text prompts and a decoding probability generator. |
| Outcome: | The proposed model outperforms state-of-the-art models on audio classification datasets without training them on extra data or adding any parameters. |
LLM Task Interference: An Initial Study on the Impact of Task-Switch in Conversational History (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing (NLP) have led to the widespread deployment of large language models (LLMs) across various applications. |
| Approach: | They propose to formalize the study of task-switches in conversational LLMs by analyzing conversational history. |
| Outcome: | The proposed study formalizes and investigates the sensitivity of large language models to taskswitch scenarios in conversational LLMs. |
Efficient LLM Comparative Assessment: A Product of Experts Framework for Pairwise Comparisons (2024.emnlp-main)
Copied to clipboard
| Challenge: | LLM-as-a-judge approaches are effective but cost scales quadratically with number of candidates, which has practical limitations. |
| Approach: | They propose a Product of Expert (PoE) framework for efficient LLM Comparative Assessment where individual comparisons are considered experts that provide information on a pair’s score difference. |
| Outcome: | The proposed framework can generate score predictions that correlate well with human judgements on multiple NLG tasks with as few as 2% of comparisons. |
Efficient Sample-Specific Encoder Perturbations (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing approaches to training encoder-decoder systems often depend on teacher-forcing with the likelihood criteria, e.g. next token prediction of the reference sequence. |
| Approach: | They propose an inference-efficient way to modify the behaviour of an encoder-decoder system according to a specific attribute of interest by using a small proxy network. |
| Outcome: | The proposed framework improves the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition. |
Unlearning vs. Obfuscation: Are We Truly Removing Knowledge? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent methods often rely on obfuscation by injecting incorrect or irrelevant information to suppress knowledge, leaving models vulnerable to probing. |
| Approach: | They propose a method that flattens the model predictive distribution over automatically generated multiple-choice questions, effectively removing knowledge about target individuals. |
| Outcome: | The proposed method achieves unlearning with over 90% refusal rate and a higher uncertainty than obfuscation on probing questions. |
Who Needs Decoders? Efficient Estimation of Sequence-Level Attributes with Proxies (2024.eacl-long)
Copied to clipboard
| Challenge: | Autoregressive decoding is expensive for many sequence-to-sequence tasks, but for some downstream tasks, the actual decoding output is not needed, just attributes of the sequence. |
| Approach: | They propose non-autoregressive proxy models that can efficiently predict scalar-valued sequence-level attributes from the encodings, avoiding the expensive decoding stage. |
| Outcome: | The proposed models outperform ensembles in machine translation (MT) and automatic speech recognition (ASR) while being significantly faster. |
Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities and versatility in NLP tasks, however they often fail to maintain crucial invariances for specific tasks. |
| Approach: | They propose to distill the capabilities of a computationally intensive, debiased, teacher model into a more compact student model. |
| Outcome: | The proposed model outperforms larger, biased teacher models by reducing the invariances of the student model by incorporating encoders. |
Residue-Based Natural Language Adversarial Attack Detection (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to detect adversarial examples for deep learning based systems focus on image embedding feature spaces . however, existing approaches focus on text features, without considering model embeddable spaces. |
| Approach: | They propose a sentence-embedding “residue” detector to identify adversarial examples from embedded feature spaces. |
| Outcome: | The proposed detector outperforms existing model-focused detectors on many tasks. |
Finetuning LLMs for Comparative Assessment Tasks (2025.coling-main)
Copied to clipboard
| Challenge: | Automated assessment in natural language generation is a challenging task. |
| Approach: | They propose a framework for fine-tuning LLMs for comparative assessment to align the model’s output with the target distribution of comparative probabilities. |
| Outcome: | The proposed framework improves state-of-the-art performance while maintaining high performance with an efficient subset of comparisons. |
Grammatical Error Correction Systems for Automated Assessment: Are They Susceptible to Universal Adversarial Attacks? (2022.aacl-main)
Copied to clipboard
| Challenge: | With advances in deep learning, GEC systems are susceptible to adversarial attacks, in which a small change at the input can cause large undesired changes at the output. |
| Approach: | They propose to use a concatenative universal attack to deceive the system into not correcting grammatical errors to create the perception of higher language ability. |
| Outcome: | The proposed attack can deceive the system into not correcting (concealing) grammatical errors to create the perception of higher language ability. |
Cross-Lingual Transfer Learning for Speech Translation (2025.naacl-short)
Copied to clipboard
| Challenge: | Increasing interest in building multilingual foundation models for NLP and speech research has led to limited data collection for training ST systems. |
| Approach: | They propose to use Whisper to explore the behavior of multilingual speech foundation models with restricted data. |
| Outcome: | The proposed model can translate to Chinese with a single language, and it can perform transcriptions in other languages. |
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing fact-checking approaches require access to external databases or external databases . a lack of external databases can undermine trust in large language models. |
| Approach: | They propose a sampling-based approach to fact-check black-box models without external databases. |
| Outcome: | The proposed approach can be used to fact-check black-box models without external databases . it can detect non-factual and factual sentences and rank passages in terms of factuality . |
Analyzing Biases to Spurious Correlations in Text Classification Tasks (2022.aacl-short)
Copied to clipboard
| Challenge: | Often these systems exceed human performance, but there is a caveat: standard benchmarks often assume that training and evaluation data are drawn independently and identically from the same underlying distribution. |
| Approach: | They propose to exploit spurious correlations in training data to exploit these correlations . they show that even when only ‘stop’ words are available, it is possible to predict the class significantly better than random. |
| Outcome: | The proposed model can predict class significantly better when only ‘stop’ words are available at the input stage, but can degrade the ability of the system to generalize well to out-of-domain data. |
Answer Uncertainty and Unanswerability in Multiple-Choice Machine Reading Comprehension (2022.findings-acl)
Copied to clipboard
| Challenge: | Machine reading comprehension (MRC) systems focus on selecting the correct answer to a question given a context paragraph. |
| Approach: | They propose to use machine reading comprehension (MRC) to assess the ability of systems to understand natural language. |
| Outcome: | The proposed system outperforms a system built with an NOA option . the results show that the system is not confident about the NOA choice . |
Muting Whisper: A Universal Acoustic Adversarial Attack on Speech Foundation Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | 'special' tokens in large speech foundation models such as Whisper are used to guide their language generation process, but can be exploited by adversarial attacks to manipulate the model's behavior. |
| Approach: | They propose a method to learn a universal acoustic realization of Whisper's |endoftext|> token, which encourages the model to ignore the speech and only transcribe the special token, effectively muting the model. |
| Outcome: | The proposed method can mute Whisper models for over 97% of speech samples and can be used to bypass speech moderation systems and protect private speech data. |
An Information-Theoretic Approach to Analyze NLP Classification Tasks (2024.acl-long)
Copied to clipboard
| Challenge: | Natural language processing (NLP) tasks take either a single or multiple text elements to predict an output variable. |
| Approach: | They propose an information-theoretic framework to analyse the influence of inputs on the output of text classification tasks. |
| Outcome: | The proposed framework is available at: https://github.com/WangLuran/nlp-element-influence. |
Sparsity and Sentence Structure in Encoder-Decoder Attention of Summarization Systems (2021.emnlp-main)
Copied to clipboard
| Challenge: | Training and inference using large transformer models can be computationally expensive because the self-attention's time and memory grow quadratically with sequence length. |
| Approach: | They propose a modified transformer architecture that constrains the encoder-decoder attention mechanism to a subset of input sentences while maintaining system performance. |
| Outcome: | The proposed architecture can be trained and inferenced using large transformer models with expensive training and induction costs. |
Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations . however, no study has examined the vulnerability of judge-LLM to adversarial manipulation . |
| Approach: | They propose a simple surrogate attack where a surrogated model is attacked and the learned attack phrase transferred to unknown judge-LLMs. |
| Outcome: | The proposed algorithm shows that judge-LLMs can be significantly more susceptible to adversarial attacks when used for absolute scoring, rather than comparative assessment. |
SkillAggregation: Reference-free LLM-Dependent Aggregation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods in NLP assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection. |
| Approach: | They propose a method that learns to combine LLM judgments without additional data or ground truth to exploit the judge estimates during inference. |
| Outcome: | The proposed method outperforms Crowdlayer on all tasks and yields the best performance over all approaches on the majority of tasks. |
Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | acoustic adversarial attacks on speech LLMs may make them more vulnerable to adversarials . flexible speech encoders and large language models have enabled speech Llms to handle a wide range of processing tasks. |
| Approach: | They investigate universal adversarial attacks on speech LLMs by pre-trained speech encoders and large language models. |
| Outcome: | The proposed model can handle a wide range of spoken language processing tasks. |
Long-Span Summarization via Local Attention and Content Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Transformer-based models are state-of-the-art for a wide range of natural language processing tasks, including document summarization. |
| Approach: | They exploit large pre-trained transformer-based models and address long-span dependencies in abstractive summarization using two methods: local self-attention; and explicit content selection. |
| Outcome: | The proposed models achieve state-of-the-art on Spotify Podcast, arXiv, and PubMed datasets. |