Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants’ Question-Answering in Asynchronous Learning Environments (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing assessments rely on surface-level metrics and lack sufficient grounding in educational theory . a new framework is proposed to evaluate VTAs in asynchronous learning environments . |
| Approach: | They propose a pedagogically-oriented evaluation framework tailored to asynchronous forum discussions . they construct classifiers using expert annotations of VTA responses on a diverse set of forum posts . |
| Outcome: | The proposed evaluation framework is rooted in learning sciences and tailored to asynchronous forum discussions. |
Similar Papers
A Large-Scale Real-World Evaluation of an LLM-Based Virtual Teaching Assistant (2025.acl-industry)
Copied to clipboard
| Challenge: | Empirical studies on their effectiveness and acceptance in real-world classrooms are limited, leaving their practical impact uncertain. |
| Approach: | They develop an LLM-based virtual teaching assistant and deploy it in an introductory AI programming course with 477 graduate students. |
| Outcome: | The proposed system is tested in an introductory AI programming course with 477 graduate students. |
EducationQ: Evaluating LLMs’ Teaching Capabilities Through Multi-Agent Dialogue Framework (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used as educational tools, yet evaluating their teaching capabilities remains challenging due to the resource-intensive nature of teacher-student interactions. |
| Approach: | They propose a multi-agent dialogue framework that efficiently assesses teaching capabilities through simulated dynamic educational scenarios. |
| Outcome: | The proposed framework outperforms open-source models on 1,498 questions across 13 disciplines and 10 difficulty levels on 1,400 questions. |
Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on large language models lacks scalability and assesses pedagogic quality. |
| Approach: | They propose a multi-agent workflow leveraging large language models to simulate interactive teaching-learning conversations. |
| Outcome: | The proposed workflow integrates teacher and learner agents, an interaction manager, and an evaluator to facilitate procedural learning and assess pedagogic quality. |
NoiseQA: Challenge Set Evaluation for User-Centric Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Question-Answering (QA) systems are deployed in the real world . a lack of research attention has been devoted to studying the issues that arise when people use QA systems. |
| Approach: | They show that component components that precede an answering engine can introduce varied and considerable sources of error. |
| Outcome: | The proposed evaluations highlight the need for QA evaluation to expand to consider real-world use. |
Accurate Training of Web-based Question Answering Systems with Feedback from Ranked Users (2023.acl-industry)
Copied to clipboard
| Challenge: | Recent work shows that large-scale annotated datasets are essential for training state-of-the-art Question Answering (QA) models. |
| Approach: | They use large-scale annotated datasets to train question answering models . they use feedback data collected from deployed QA systems to provide cheaper supervision . |
| Outcome: | The proposed model improves on the large scale annotated datasets from QA systems . the proposed model can be easily supervised on large-scale unlabeled web data . |
Generative Interpretation: Toward Human-Like Evaluation for Educational Question-Answer Pair Generation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing evaluation methods often fail to produce objective results and favor high similarity to the ground-truth question-answer pairs. |
| Approach: | They propose an alternative approach to evaluate question-answer generation using Generative Interpretation (GI) GI outperforms existing evaluation methods in terms of human alignment . |
| Outcome: | The proposed approach outperforms existing evaluation methods in human alignment and shows comparable performance with GPT3.5, only with BART-large. |
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)
Copied to clipboard
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen
| Challenge: | Existing methods to explain predictions by highlighting salient features are often unstated. |
| Approach: | They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. |
| Outcome: | The proposed framework allows principled, automatic, model-agnostic evaluation of attributions. |
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | despite the importance of question answering, evaluations of QA systems are typically limited by manual annotations . despite this, little progress has been made in QA evaluations based on a single answer . |
| Approach: | They propose to extend over exact match (EM) with predefined rules or token-level F1 measure . they propose to use a BERT matching measure to approximate QA predictions . |
| Outcome: | The proposed model improves AE approximations and more accurately reflects the performance of systems. |
Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing evaluations of large language models have been limited to subjective protocols and benchmarks. |
| Approach: | They propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles to assess the pedagical value of LLM-powered AI tutor responses grounded in student mistakes or confusions in the mathematical domain. |
| Outcome: | The proposed taxonomy, benchmark, and human-annotated labels will streamline the evaluation process and help track the progress in AI tutors’ development. |
Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvement (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using simulated feedback, our system (called TeachMe) continually improves with time, and without model retraining. |
| Approach: | They propose to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections toerroneous model beliefs that users identify during interaction. |
| Outcome: | The proposed system improves with time and without model retraining, and with real users, by 15% on a hidden test set after teaching. |