What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability (2023.emnlp-main)
Copied to clipboard
| Challenge: | In Natural Language Generation tasks, multiple communicative goals are plausible and any goal can be put into words, or produced, in multiple ways. |
| Approach: | They characterise the extent to which human production varies lexically, syntactically, and semantically across four NLG tasks, connecting human production variability to aleatoric or data uncertainty. |
| Outcome: | The proposed model can be calibrated to human production variability using multiple samples and, when possible, multiple references. |
Similar Papers
Semantic Accuracy in Natural Language Generation: A Thesis Proposal (2023.acl-srw)
Copied to clipboard
| Challenge: | Using large pre-trained language models, it is essential to research their reliability . if a human does not know the answer to a question, the socially acceptable behavior is to say 'I do not know' failing to fulfill this expectation can lead to distrust, or spread of misinformation. |
| Approach: | They propose a method for evaluating semantic accuracy and a benchmark for NLG metrics. |
| Outcome: | The proposed method evaluates semantic accuracy and provides a benchmark for NLG metrics. |
Uncertainty Quantification for Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | Large language models (LLMs) produce hallucinations, which undermine user trust and reliability. |
| Approach: | This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks. |
| Outcome: | The proposed framework provides tools for communicating the reliability of a model answer. |
Predict the Next Word: <Humans exhibit uncertainty in this task and language models _____> (2024.eacl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are statistical models trained to assign probability to human-generated text. |
| Approach: | They evaluate language models' ability to reproduce variability that humans exhibit in the ‘next word prediction’ task. |
| Outcome: | The language models are trained to assign probability to human-generated text . they exhibit low calibration to human uncertainty, and advise against it . |
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)
Copied to clipboard
Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, Edgar Dobriban
| Challenge: | Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs. |
| Approach: | They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses . |
| Outcome: | The proposed framework assesses uncertainty and confidence measures for LMs. |
On Decoding Strategies for Neural Text Generators (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models. |
| Approach: | They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation. |
| Outcome: | The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say . |
Towards Pragmatic Production Strategies for Natural Language Generation Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using language to communicate successfully requires effort. |
| Approach: | They propose a conceptual framework for the design of natural language generation systems that follow efficient and effective production strategies to achieve complex communicative goals. |
| Outcome: | The proposed framework is applied to visually grounded referential games and abstractive text summarisation tasks with real-world applications. |
Systematic Task Exploration with LLMs: A Study in Citation Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) provide unprecedented flexibility in defining and executing complex, creative natural language generation tasks. |
| Approach: | They propose a framework that consists of input manipulation, reference data, and output measurement to explore citation text generation. |
| Outcome: | The proposed framework explores citation text generation, a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric and has not yet been tackled within the LLM paradigm. |
Automatic and Human-AI Interactive Text Generation (with a focus on Text Simplification and Revision) (2024.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we focus on text-to-text generation, a class of natural language generation tasks, that takes a piece of text as input and then generates a revision that is improved according to some specific criteria. |
| Approach: | This tutorial focuses on text-to-text generation, a class of natural language generation tasks that takes a piece of text as input and generates a revision that is improved according to some specific criteria. |
| Outcome: | This tutorial focuses on text-to-text generation, a class of natural language generation tasks, that takes a piece of text as input and generates a revision that is improved according to some specificcriteria. |
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Natural language generation (NLG) tasks have complex nature and require manual evaluation. |
| Approach: | They propose a unifying perspective based on the nature of information change in NLG tasks . they propose 'information alignment' metrics that can be used to evaluate different aspects of NLG . |
| Outcome: | The proposed metrics achieve stronger or comparable correlations with human judgement compared to state-of-the-art metrics in diverse tasks. |
Generating Text from Language Models (2023.acl-tutorials)
Copied to clipboard
| Challenge: | a growing percentage of natural language processing tasks focus on the generation of text from probabilistic language models. |
| Approach: | They will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |
| Outcome: | This tutorial will provide a centralized discussion of critical considerations when choosing how to generate from a language model. |