BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for evaluating language models are brittle, corpus-level perplexities are vague, and the choice of benchmarks is endless. |
| Approach: | They propose a method that uses contextual embeddings to find fine-grained features of text where one model outperforms another. |
| Outcome: | The proposed method extracts features that demonstrate differences with respect to ease of generation between two language models. |
Similar Papers
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types (2023.findings-acl)
Copied to clipboard
Keerthiram Murugesan, Sarathkrishna Swaminathan, Soham Dan, Subhajit Chaudhury, Chulaka Gunasekara, Maxwell Crouse, Diwakar Mahajan, Ibrahim Abdelaziz, Achille Fokoue, Pavan Kapanipathi, Salim Roukos, Alexander Gray
| Challenge: | Existing evaluation metrics for machine text are inadequate to capture quality of text . a recent study has focused on task-specific evaluation metrics or on properties of machine-generated text based on mismatch errors . |
| Approach: | They propose a new evaluation scheme based on fine-grained mismatch errors . they propose 13 mismatch error types to guide the model for better prediction of human judgments . |
| Outcome: | The proposed evaluation scheme is based on mismatch errors in 7 NLP tasks . the mismatch error types guide the model for better prediction of human judgments . |
Latent Inter-User Difference Modeling for LLM Personalization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly integrated into users’ daily lives, leading to a growing demand for personalized outputs. |
| Approach: | They propose a framework that models inter-user differences in the latent space instead of relying on language-based prompts. |
| Outcome: | The proposed framework outperforms baseline methods on personalized review generation. |
EnCBP: A New Benchmark Dataset for Finer-Grained Cultural Background Prediction in English (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing research on cultural background modeling is coarse-grained and does not examine cultural differences among speakers of the same language. |
| Approach: | They use a news-based cultural background prediction dataset to annotate, validate and benchmark NLP models with cultural background features. |
| Outcome: | The proposed model improves on nine syntactic, semantic, and psycholinguistic tasks while introducing cultural background information does not improve the Go-Emotions task due to text domain conflicts. |
Are All Languages Equally Hard to Language-Model? (N18-2)
Copied to clipboard
| Challenge: | a fair comparison of language models is tricky because of the size of the corpora and the variability of orthographic systems. |
| Approach: | They propose a framework for fair cross-linguistic comparison of language models . they show that in some languages, textual expression is harder to predict with n-gram models compared to LSTM models based on translated text . |
| Outcome: | The proposed framework is based on translated text and language models on 21 languages. |
How to Improve LLMs’ Performance on Specific Languages: A Perspective on LLM-Derived Language Similarity (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit uneven performance across languages. |
| Approach: | They propose to use a framework to quantify the similarity within each language pair through both the lenses of language-specific performance patterns and cross-lingual transferability. |
| Outcome: | The proposed approach outperforms traditional linguistic typology and cross-lingual transferability measures on multilingual LLMs. |
LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests (2025.findings-emnlp)
Copied to clipboard
Jon Saad-Falcon, Rajan Pathe Vivek, William Berrios, Nandita Shankar Naik, Matija Franklin, Bertie Vidgen, Amanpreet Singh, Douwe Kiela, Shikib Mehri
| Challenge: | Using natural language unit tests, language models are costly and noisy, and automated metrics provide only coarse, difficult-to-interpret signals. |
| Approach: | They propose a paradigm that decomposes response quality into explicit, testable criteria and a unified scoring model, LMUnit, which combines multi-objective training across preferences, direct ratings, and natural language rationales. |
| Outcome: | The proposed paradigm significantly improves inter-annotator agreement and enables more effective LLM development workflows. |
LMdiff: A Visual Diff Tool to Compare Language Models (2021.emnlp-demo)
Copied to clipboard
| Challenge: | LMdiff visually compares probability distributions of two different language models . notably absent from the range of available tools are those that aim to compare distributions produced by different models. |
| Approach: | They propose a tool that visually compares probability distributions of two different language models that differ through finetuning, distillation, or simply training with different parameter sizes. |
| Outcome: | The proposed tool allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing interesting text instances from large corpora. |
BECEL: Benchmark for Consistency Evaluation of Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing definitions of behavioural consistency are inconsistent across many studies. |
| Approach: | They propose a behavioural consistency model and propose behavioural taxonomy that classifies consistencies into several sub-categories. |
| Outcome: | The proposed model performs poorly on 19 test cases while exhibiting high inconsistency in many cases. |
Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies focus on data selection but lack a clear, unified framework . variability in experimental settings complicates systematic comparisons . |
| Approach: | They propose a three-stage scheme to standardize data selection for fine-tuning large language models . they propose unified comparison approach that incorporates ratio-based efficiency and ranking-based feasibility metrics to address inconsistencies across experiments. |
| Outcome: | The proposed scheme outperforms existing methods in a dozen key studies and identifies key challenges. |