Papers by Michael Tseng
Community-Driven Crowdsourcing: Data Collection with Local Developers (L18-1)
Copied to clipboard
| Challenge: | a community-driven approach to annotation applications and crowdsourcing programs is feasible, says a new study. |
| Approach: | They propose to partner with local developers to create custom annotation applications . they recruit and motivate crowd contributors from their communities to perform an annotation task . |
| Outcome: | The proposed approach combines local developers' knowledge of their social networks to collect labeled data. |
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024.acl-long)
Copied to clipboard
Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva
| Challenge: | Recent literature discusses automatic methods to evaluate reasoning to improve their correctness, but no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods. |
| Approach: | They propose to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings using a dataset that includes comprehensive labels for relevance, attribution to evidence passages, and logical correctness of each reasoning step. |
| Outcome: | The proposed dataset shows that verifiers struggle at verifying reasoning chains, particularly verifying logical correctness and detecting contradictions. |