MEEP: Is this Engaging? Prompting Large Language Models for Dialogue Evaluation in Multilingual Settings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for engagingness evaluate the response without the conversation history, are designed for one dataset, or have limited correlation with human annotations. |
| Approach: | They propose to use large language models to evaluate engagingness in dialogue . they propose to include prompts and translated prompts in the model . |
| Outcome: | The proposed model outperforms existing methods on evaluation of engagingness in dialogue across languages. |
Similar Papers
Multilingual Prompting for Improving LLM Generation Diversity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models lack cultural representation and diversity in their generations . lack of demographic diversity can lead to unfair lack of exposure of artists . |
| Approach: | They propose a prompting method which generates several variations of a base prompt with added cultural and linguistic cues from several cultures, generates responses, and then combines the results. |
| Outcome: | The proposed method outperforms existing diversity-enhancing techniques . it can generate multiple variations of a base prompt with cultural cues from multiple cultures . |
What Makes a Good Natural Language Prompt? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies on prompt quality show imbalanced support across models and tasks, and research gaps. |
| Approach: | They propose a property- and human-centric framework for evaluating prompt quality . they propose comparing prompt quality to other factors such as adverbs and apverbs . |
| Outcome: | The proposed framework reveals imbalanced support across models and tasks and substantial research gaps. |
You don’t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments (2024.naacl-long)
Copied to clipboard
Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, David Jurgens
| Challenge: | Large Language Models (LLMs) are popular for research in social sciences . currently, prompting LLMs is insufficient to accurately and reliably capture model perceptions, and we discuss potential alternatives to improve this. |
| Approach: | They construct a dataset that contains 693 questions encompassing 39 different instruments of persona measurement on 115 persona axes and a set of questions containing minor variations. |
| Outcome: | The proposed model can generate answers and negate statements in a consistent and robust manner. |
EnDex: Evaluation of Dialogue Engagingness at Scale (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that measure engagement use expensive human annotas and abstract definitions of the term. |
| Approach: | They propose a human-reaction based model to evaluate dialogue engagingness . they propose combining distant-supervision with a theoretical foundation for engagement . |
| Outcome: | The proposed model is trained on 80k Reddit-based engagement datasets . it uses distant-supervision from human-reaction feedback to evaluate dialogue engagementness . |
What do Large Language Models Need for Machine Translation Evaluation? (2024.emnlp-main)
Copied to clipboard
Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, Fred Blain
| Challenge: | Existing research shows that large language models can perform better in machine translation tasks. |
| Approach: | They propose to use large language models for machine translation evaluations . authors explore what translation information is needed for LLMs to evaluate MT quality . |
| Outcome: | The proposed model performs comparable to fine-tuned multilingual pre-trained models. |
Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that ChatGPT has limitations such as failing to ask clarifying questions to ambiguous queries or refusing problematic user requests. |
| Approach: | They propose a Proactive Chain-of-Thought prompting scheme which augments LLMs with the goal planning capability over descriptive reasoning chains to trigger proactivity. |
| Outcome: | The proposed scheme augments LLMs with the goal planning capability over descriptive reasoning chains to trigger the proactivity of LLM-based proactive dialogue systems. |
DialogBench: Evaluating LLMs as Human-like Dialogue Systems (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate LLMs' abilities for task completion as assistant AI. |
| Approach: | They propose a dialogue evaluation benchmark that contains 12 dialogue tasks to evaluate LLMs' capabilities as human-like dialogue systems. |
| Outcome: | The proposed benchmark contains 12 tasks to evaluate LLMs' capabilities . it shows that instruction tuning improves human likeness, but not as human-like systems . |
Evaluating the Effectiveness of Large Language Models in Establishing Conversational Grounding (2024.emnlp-main)
Copied to clipboard
| Challenge: | despite its importance, there has been limited research on conversational grounding in recent years . pre-trained language models have been costly and time-consuming to evaluate . |
| Approach: | They evaluate the performance of large language models in various aspects of conversational grounding . they propose ways to enhance the capabilities of the models that lag in this aspect . |
| Outcome: | The proposed model performance is based on pre-trained language models and a large pre-training dataset. |
DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models (2024.naacl-demo)
Copied to clipboard
| Challenge: | DIALIGHT is a toolkit for developing and evaluating multilingual Task-Oriented Dialogue systems. |
| Approach: | They propose a toolkit for developing and evaluating multilingual Task-Oriented Dialogue systems which facilitates systematic evaluations and comparisons between ToD systems using pretrained language models and those utilising the zero-shot and in-context learning capabilities of Large Language Models. |
| Outcome: | The toolkit enables systematic evaluations between ToD systems using pretrained language models and those utilising the zero-shot and in-context learning capabilities of Large Language Models (LLMs). |
Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large audio-language models (LALMs) have expanded their impact beyond natural language processing (NLP) to multimodal domains. |
| Approach: | They propose a systematic taxonomy for LALM evaluations, categorizing them into four dimensions based on their objectives: (1) General Auditory Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented Ability, and (4) Fairness, Safety, and Trustworthiness. |
| Outcome: | The proposed taxonomy categorizes LALM evaluations into four dimensions based on their objectives and highlights challenges in this field. |