Language Model Transformers as Evaluators for Open-domain Dialogues (2020.coling-main)
Copied to clipboard
| Challenge: | Computer-based systems for communication with humans are a cornerstone of AI research since the 1950s. |
| Approach: | They propose to use transformer neural networks to predict one or more words based on an already given context to provide an efficient, automatic indication of dialogue quality. |
| Outcome: | The proposed language models show that human evaluators have a positive correlation between the output of the models and scores. |
Similar Papers
An Investigation of Suitability of Pre-Trained Language Models for Dialogue Generation – Avoiding Discrepancies (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have been widely used in open-domain dialogue generation. |
| Approach: | They propose to use decoder-only architecture to achieve excellent performance for dialogue generation. |
| Outcome: | The proposed frameworks are based on transformer-ED, transformer-Dec, transformer MLM and transformer-AR. |
Evaluating Lexical Proficiency in Neural Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Processing have been significantly shaped by the Deep Learning tsunami and the introduction of Transformer-based Language Models. |
| Approach: | They validated a framework to assess the lexical proficiency and linguistic creativity of Transformer-based Language Models (LMs) by analyzing performance of LMs of different sizes across tasks involving the generation, definition, and contextual usage of lexicals, neologisms, and nonce words. |
| Outcome: | The framework evaluates LMs in mono- and multilingual configuration across tasks involving the generation, definition, and contextual usage of lexicalized words, neologisms, and nonce words. |
What Context Features Can Transformer Language Models Use? (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. |
| Approach: | They propose to use lexical and structural information to ablate usable information in transformer language models. |
| Outcome: | The proposed model improves when conditioning on contexts of thousands of previous tokens. |
LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Transformer-based language models (LMs) are opaque and unexplained, causing problems for endusers and developers who wish to debug or fix their behaviour. |
| Approach: | They propose an interactive debugger tool for transformer-based LMs that provides a fine-grained interpretation of the model's internal prediction process and a powerful framework for intervening in LM behavior. |
| Outcome: | The proposed tool provides a fine-grained interpretation of the model's internal prediction construction process, and a powerful framework for intervening in LM behavior. |
Explaining How Transformers Use Context to Build Predictions (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for analyzing input attributions for a model's prediction are unclear how prior words affect the model' s decision throughout the layers. |
| Approach: | They propose a procedure to analyze models for language generation using the Transformer and a comparison of their results with evidence of the linguistic phenomena. |
| Outcome: | The proposed method consistently aligns better than gradient-based and perturbation-based baselines and generates human-like source-target alignments for building predictions. |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
Lightweight Transformers for Conversational AI (2022.naacl-industry)
Copied to clipboard
| Challenge: | Commercial dialogue systems typically require a small footprint and fast execution time, but recent trends are in the other direction, resulting in difficulties in model deployment. |
| Approach: | They build Transformer-based Language Models from scratch on large corpora of conversational data and compare their performance against BERT and other strong baselines on dialogue probing tasks. |
| Outcome: | The proposed model outperforms existing models on dialogue probing tasks and can be fine-tuned on a single consumer GPU card. |
Language Models in Dialogue: Conversational Maxims for Human-AI Interactions (2024.findings-emnlp)
Copied to clipboard
Erik Miehling, Manish Nagireddy, Prasanna Sattigeri, Elizabeth Daly, David Piorkowski, John Richards
| Challenge: | Modern language models exhibit some inherent shortcomings, particularly in conversational settings. |
| Approach: | They propose a set of maxims for describing effective human-AI conversation that include quantity, quality, relevance, manner, benevolence, and transparency. |
| Outcome: | The proposed maxims are applied to human-AI interactions and are based on extensive research from the social science and AI communities. |
Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Backchannels and fillers are important linguistic expressions in dialogue, but often ignored in modern transformer-based language models. |
| Approach: | They use clustering analysis to learn backchannels and fillers in dialogues in English and Japanese and use natural language generation metrics to confirm this. |
| Outcome: | The proposed models can learn representations of backchannels and fillers using three fine-tuning strategies. |
Do dialogue representations align with perception? An empirical study (2023.eacl-main)
Copied to clipboard
| Challenge: | masked language models produce stronger correlations than auto-regressive models, but humans and models make different response selection mistakes. |
| Approach: | They propose to use spoken conversation as a model to measure human comprehension behaviour. |
| Outcome: | The proposed model outperforms the model which produces the strongest correlation with human responses. |