| Challenge: | a new technique for layering explicit pragmatic inference on top of models for sequential tasks is proposed . explicit pragmatic reasoning is used to generate and follow natural language instructions . |
| Approach: | They propose a pragmatic speaker that uses the base listener to simulate the interpretation of candidate descriptions and a listener that reasons counterfactually about alternative descriptions. |
| Outcome: | The proposed model improves state-of-the-art models for interpreting human instructions and speaker models in diverse settings. |
Similar Papers
Pragmatically Informative Text Generation (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to pragmatics have been used to improve the informativeness of generated text in grounded language learning problems. |
| Approach: | They propose to use pragmatics to improve the informativeness of conditional text models . they propose to apply pragmatic reasoning to more traditional language generation tasks . |
| Outcome: | The proposed methods improve the performance of strong existing systems for abstractive summarization and generation from structured meaning representations. |
A fine-grained comparison of pragmatic language understanding in humans and language models (2023.acl-long)
Copied to clipboard
| Challenge: | Pragmatics and non-literal language understanding are essential to human communication . a long-standing challenge for artificial language models is to capture pragmatics . |
| Approach: | They compare language models and humans on seven pragmatic phenomena using curated English materials. |
| Outcome: | The proposed model achieves high accuracy and matches human error patterns . the results suggest pragmatic behaviors can emerge in models without explicit representations of mental states . |
Eliciting Instruction-tuned Code Language Models’ Capabilities to Utilize Auxiliary Function for Code Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Using auxiliary functions to implement functions is important for instruction-tuned models because it reduces the implementation difficulty of a target function compared to implementing them from scratch. |
| Approach: | They propose several ways to provide auxiliary functions to the models by adding them to the query or providing a response prefix to incorporate the ability to utilize auxiliary function with the instruction following capability. |
| Outcome: | The proposed models outperform the recent powerful language models, gpt-4o, in the code generation task. |
LLM-driven Instruction Following: Progresses and Concerns (2023.emnlp-tutorial)
Copied to clipboard
| Challenge: | a tutorial on task instruction is aimed at researchers and practitioners interested in NLP generalization . labeled examples are unlikely to be available in large numbers or do not exist . |
| Approach: | This tutorial will examine the progress of natural language processing (NLP) using labeled examples. authors propose that task instructions act as a novel resource for supervision. |
| Outcome: | This tutorial aims to answer questions about instruction-driven NLP . it focuses on the use of task instructions in a low-shot scenario . |
Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches (2023.findings-emnlp)
Copied to clipboard
| Challenge: | People rely heavily on context to enrich meaning beyond what is literally said. |
| Approach: | They analyze how task goals, environmental contexts, and communicative affordances in each work enrich linguistic meaning. |
| Outcome: | The proposed frameworks are based on linguistic goals, environmental contexts, and communicative affordances to enrich linguistic meaning. |
Unnatural language processing: How do language models handle machine-generated prompts? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are outperformed by automatically generated token sequences with no apparent meaning or syntactic structure. |
| Approach: | They propose to use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. |
| Outcome: | The proposed model outperforms human-crafted prompts on a target zero-shot task. |
Fine-Tuning Large Language Models with Sequential Instructions (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing instruction-tuned models struggle to adhere to a query with multiple intentions, which impairs their performance when the completion of several tasks is demanded by a single command. |
| Approach: | They develop an automatic process that turns existing data into diverse and complex task chains and a new benchmark to evaluate a model’s ability to follow all the instructions in a sequence. |
| Outcome: | The proposed model can follow instructions better and deliver higher results in coding, maths, and open-ended generation. |
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models can perform unseen tasks by conditioning on a few input-output demonstrations, but task inference is implicit and the ability of models to explicitly reason about it remains unexplored. |
| Approach: | They propose an instruction induction challenge in which a model is asked to generate a natural language instruction that fits a set of labeled examples. |
| Outcome: | The proposed model achieves 65.7% of human performance while the original model only reaches 9.8% of human performances. |
Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work examines the cognitive capabilities of language models through psychological tests designed for humans. |
| Approach: | They propose to use human-like cognitive capabilities to evaluate language models . they propose to augment language models with better listeners to improve their performance . |
| Outcome: | The proposed method boosts language models with better models of the listener and improves them. |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |