Papers with creators
Quality Estimation for Partially Subjective Classification Tasks via Crowdsourcing (2020.lrec-1)
Copied to clipboard
| Challenge: | a common approach to quality estimation is to ask multiple reviewers to evaluate the same artifacts. |
| Approach: | They propose a probabilistic model for subjective classification tasks that incorporates the qualities of artifacts as well as the abilities and biases of creators and reviewers as latent variables to be jointly inferred. |
| Outcome: | The proposed model estimates the quality of speech more effectively than a vote aggregation, measured by correlation with a fine-grained classification by experts. |
Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving LLMs in simulations are limited due to privacy concerns and limited domain knowledge. |
| Approach: | They propose a pipeline that elicits qualitative feedback from a domain-expert and transforms it into a set of principles that govern an LLM-prompted roleplay. |
| Outcome: | The proposed pipeline shows a 30% improvement in response quality and principle following for the downstream task. |