Papers by David Heineman
Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Traditional human evaluation methods for text simplification often relies on individual, shallow sentence-level ratings, easily affected by the annotator's preference or bias. |
| Approach: | They propose an edit-based human annotation framework that enables holistic and fine-grained text simplification evaluation. |
| Outcome: | The proposed framework is able to predict sentence- and word-level quality simultaneously and report promising results. |
LENS: A Learnable Evaluation Metric for Text Simplification (2023.acl-long)
Copied to clipboard
| Challenge: | Existing metrics for text simplification are based on unitary or outdated models, making them unsuitable for this approach. |
| Approach: | They present a learnable evaluation metric for text simplification using language models . they also introduce a human evaluation framework that rates simplifications from several models a list-wise manner . |
| Outcome: | The proposed model correlates much better with human judgment than existing metrics. |
Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Existing tools for fine-grained human evaluation lack adaptability to different domains or languages, or modify annotation settings according to user needs. |
| Approach: | They propose a unified platform for fine-grained evaluation that is customizable and deployable with a single YAML configuration file. |
| Outcome: | The proposed frameworks are based on a single YAML configuration file and can be easily extended to different domains or languages. |
Improving Minimum Bayes Risk Decoding with Multi-Prompt (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to generate LLMs with a single ‘best’ prompt are unstable and sub-optimal in practice. |
| Approach: | They propose to decode multiple candidate generations from a prompt bank at inference-time and use Minimum Bayes Risk (MBR) to select a final output. |
| Outcome: | The proposed method improves MBR across a set of conditional generation tasks and models. |