Papers by Jeffrey Heer
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing counterfactual generation methods rely on manual labor to create very few counterf actuals or only instantiate limited types of perturbations such as paraphrases or word substitutions. |
| Approach: | They propose a general-purpose counterfactual generator that allows for control over perturbation types and locations. |
| Outcome: | The proposed generator produces diverse sets of realistic counterfactuals that are useful in various applications. |
BLADE: Benchmarking Language Model Agents for Data-Driven Science (2024.findings-emnlp)
Copied to clipboard
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai Zhang, Lanyi Zhu, Mike Merrill, Jeffrey Heer, Tim Althoff
| Challenge: | Language model-based agents can be used to conduct and support data-driven science, but evaluating them on open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. |
| Approach: | They propose a benchmark to automatically evaluate agents’ multifaceted approaches to open-ended research questions. |
| Outcome: | BLADE evaluates agents’ multifaceted approaches to open-ended research questions using data from 12 datasets and research questions drawn from existing scientific literature. |
Errudite: Scalable, Reproducible, and Testable Error Analysis (P19-1)
Copied to clipboard
| Challenge: | Errudite is an interactive tool for informative error analysis. |
| Approach: | They propose an interactive tool for model and task agnostic error analysis . they propose a model-agnosed approach to model and tasks error analysis with Errudite . |
| Outcome: | The proposed tool enables reproducible error analyses with less effort and ambiguities in previous tools. |