Papers by Dominic Petrak
Towards Automated Error Discovery: A Study in Conversational AI (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that LLMs require information about the nature of an error or hints about its occurrence for accurate detection. |
| Approach: | They propose an encoder-based approach to detect and define errors in conversational AI. |
| Outcome: | The proposed framework outperforms baselines across multiple error-annotated dialogue datasets and shows strong generalization to unknown intent detection. |
Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models (2023.starsem-1)
Copied to clipboard
| Challenge: | Recent work suggests that pretrained language models perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. |
| Approach: | They propose an extended pretraining approach that addresses both in one extended step . they propose a novel extended pre training objective called Inferable Number Prediction Task to improve numeracy. |
| Outcome: | The proposed approach improves reading comprehension and inference-on-tables tasks without architectural changes or pretraining from scratch. |
Learning From Free-Text Human Feedback – Collect New Datasets Or Extend Existing Ones? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for learning from free-text human feedback are scarce. |
| Approach: | They manually annotate a subset of a popular dialogue dataset with error and user response types using an improved version of the Integrated Error Taxonomy and a newly proposed user response type taxonomies. |
| Outcome: | The proposed dataset provides new insights into dataset composition, error types, user response types, and the relations between them. |
Lessons Learned from a Citizen Science Project for Natural Language Processing (2023.eacl-main)
Copied to clipboard
Jan-Christoph Klie, Ji-Ung Lee, Kevin Stowe, Gözde Şahin, Nafise Sadat Moosavi, Luke Bates, Dominic Petrak, Richard Eckart De Castilho, Iryna Gurevych
| Challenge: | Annotations are expensive and difficult to obtain, which is why many NLP systems outsource their work to paid crowdworkers. |
| Approach: | They propose to use Citizen Science to re-annotate parts of a pre-existing crowdsourced dataset to gain high-quality annotations. |
| Outcome: | The proposed approach yields high-quality annotations and motivated volunteers, but requires consideration of scalability, participation over time, and legal and ethical issues. |
Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialogues (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Imlicit user feedback, user emotions and demographic information are promising sources for improving the accuracy and user engagement of dialogue responses, but the impact of such information on task completion and factual consistency is not known. |
| Approach: | They introduce the first English task-oriented and document-grounded dialogue dataset annotated with this information. |
| Outcome: | The proposed dataset shows that the model's responses are more informative and factual consistent. |