Distilling ChatGPT for Explainable Automated Student Answer Assessment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing automated student answer assessment models lack explainable and faithful feedback. |
| Approach: | They propose a framework that leverages ChatGPT for student answer scoring and rationale generation. |
| Outcome: | The proposed method improves the overall QWK score by 11% compared to ChatGPT. |
Similar Papers
AERA Chat: An Interactive Platform for Automated Explainable Student Answer Assessment (2025.emnlp-demos)
Copied to clipboard
| Challenge: | Existing systems that use pretrained language models to score student answers are noisy and unreliable. |
| Approach: | They propose a visualization platform for automated student answer assessment that leverages multiple LLMs to generate rationales. |
| Outcome: | The proposed platform enables educators to mark tasks and researchers to evaluate rationale quality from different models. |
ChatGPT Rates Natural Language Explanation Quality like Humans: But on Which Scales? (2024.lrec-main)
Copied to clipboard
| Challenge: | Traditionally, evaluating NLEs through gathering human judgments is a tedious task due to the subjective nature of human evaluations. |
| Approach: | They examine the alignment between ChatGPT and human assessments across multiple scales and compare them using paired comparisons and dynamic prompting. |
| Outcome: | The proposed model aligns better with humans in coarser scales and provides semantically similar examples in the prompt. |
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
Towards Interpretable Mental Health Analysis with Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large language models lack adequate evaluations and prompting strategies for explainability. |
| Approach: | They evaluate the mental health analysis and emotional reasoning ability of large language models (LLMs) using 11 datasets across 5 tasks. |
| Outcome: | The proposed model shows strong in-context learning ability but still has a significant gap with advanced task-specific methods. |
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study validates the effectiveness of chat language models by fine-tuning instruction data. |
| Approach: | They propose to use a large-scale dataset of instructional conversations to fine-tune a conversational model on instruction data. |
| Outcome: | The proposed model outperforms open-source models in key metrics including scale, average length, diversity, coherence, etc. |
ChatGPT Is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | acquiring and representing commonsense in machines has posed a long-standing challenge (Li et al., 2021; Zhang e t al, 2022; Zhou e al. 2023) . |
| Approach: | They use a commonsense-based LLM to evaluate ChatGPT's commonsensing abilities by analyzing 11 datasets and generating knowledge descriptions. |
| Outcome: | The proposed model can achieve good QA accuracies while still struggling with certain domains of datasets. |
Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation protocols for large language models (LLMs) are inadequate for conversational recommender systems. |
| Approach: | They propose an evaluation approach based on LLMs that harnesses LLM-based user simulators to evaluate ChatGPT's performance. |
| Outcome: | The proposed evaluation approach can simulate various system-user interaction scenarios. |
Credible without Credit: Domain Experts Assess Generative Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | ChatGPT has been criticized for its lack of accuracy and coherence . authors argue that language models could replace search engines and make college essays obsolete . |
| Approach: | a team of 10 domain experts conducts an initial assessment of language models using 100 expert-written questions. |
| Outcome: | The results show that language models are mixed in their accuracy. |
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2022.naacl-main)
Copied to clipboard
| Challenge: | Large language models are capable of generating fluent-appearing text with little task-specific supervision. |
| Approach: | They propose a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. |
| Outcome: | The proposed model can generate freetext explanations in a fewshot setting with human-written examples. |
Interview Evaluation: A Novel Approach for Automatic Evaluation of Conversational Question Answering Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation methods for CQA use pre-collected human-human conversations . previous methods use model-predicted dialogue history instead of ground truth . |
| Approach: | They propose an automatic evaluation approach that uses the model's dialogue history to evaluate models. |
| Outcome: | The proposed method improves on existing models and their evaluations on QuAC and CoQA. |