Papers by Taha Aksu
Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking (2024.lrec-main)
Copied to clipboard
| Challenge: | Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit three primary limitations: i) erroneously presume a uniform distribution of slots throughout the dialog; ii) neglect to assign partial scores for individual turns; c) frequently overestimate or underestimate performance by repeatedly counting the models’ successful or failed predictions. |
| Approach: | They propose a new metric: Granular Change Accuracy (GCA) which evaluates the predicted changes in dialogue state over the entire dialogue history. |
| Outcome: | The proposed metric reduces biases arising from distribution uniformity and the positioning of errors across turns, resulting in a more precise evaluation. |
CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs (2023.emnlp-main)
Copied to clipboard
Taha Aksu, Devamanyu Hazarika, Shikib Mehri, Seokhwan Kim, Dilek Hakkani-Tur, Yang Liu, Mahdi Namazifar
| Challenge: | Instruction-based multitasking has played a critical role in the success of large language models (LLMs) when exposed to complex instructions with multiple constraints, they lag against state-of-the-art models like ChatGPT. |
| Approach: | They propose a framework that unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without manual effort. |
| Outcome: | The proposed framework unifies a large number of dialog tasks in the same format and allows programmatic induction of complex instructions without manual effort. |