Papers by Samuel Carton
Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Dataset (2024.findings-acl)
Copied to clipboard
Satanu Ghosh, Neal Brodnik, Carolina Frey, Collin Holgate, Tresa Pollock, Samantha Daly, Samuel Carton
| Challenge: | Existing methods for ad-hoc schema-based information extraction are brittle and non-transferable, limiting their practicality for this type of one-off extraction task. |
| Approach: | They propose to use GPT-4 to perform ad-hoc schema-based information extraction from scientific literature. |
| Outcome: | The proposed model can replicate two existing material science datasets, one pertaining to multi-principal element alloys and one to silicate diffusion, and draw on their insights to suggest future research directions. |
Human-Centered Evaluation of Explanations (2022.naacl-tutorials)
Copied to clipboard
Jordan Boyd-Graber, Samuel Carton, Shi Feng, Q. Vera Liao, Tania Lombrozo, Alison Smith-Renner, Chenhao Tan
| Challenge: | This tutorial will provide an overview of human-centered evaluations of explanations . |
| Approach: | This tutorial will provide an overview of human-centered evaluations of explanations . it will introduce the psychological foundation of explanation and types of NLP explanations. |
| Outcome: | This tutorial will provide an overview of human-centered evaluations of explanations . it will cover the two categories of evaluation: evaluation based on human-annotated explanations and evaluation with human-subjects studies. |
What to Learn, and How: Toward Effective Learning from Rationales (2022.findings-acl)
Copied to clipboard
| Challenge: | Increasing interest in learning from rationales has led to the use of human-annotated explanations to inject useful inductive biases into models. |
| Approach: | They propose several novel loss functions and learning strategies to exploit human rationales to augment model prediction accuracy. |
| Outcome: | The proposed learning strategies improve on three datasets with human rationales and show that they are more efficient than baselines. |
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (D19-1)
Copied to clipboard
| Challenge: | Existing evaluation methods for natural language generation are inadequate . distinguishing machine-generated text is challenging even for human evaluators . |
| Approach: | They compare human-based evaluators with automated evaluation procedures . they find human evaluers do not correlate well with discriminative evalators . |
| Outcome: | The proposed evaluation methods are compared with a dozen state-of-the-art generators for online product reviews. |
Evaluating and Characterizing Human Rationales (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new study examines how human rationales perform on automatic metrics . human-generated rationale evaluation is difficult because of its ambiguity . |
| Approach: | They propose to use model-dependent baseline performance to evaluate rationale quality . they propose to also use "fidelity curves" to reveal properties such as irrelevance and redundancy . |
| Outcome: | The proposed methods characterize rationale quality based on model retraining and using "fidelity curves" the proposed methods lead to actionable suggestions for evaluating and characterizing rationales . |
Learning to Ignore Adversarial Attacks (2023.eacl-main)
Copied to clipboard
| Challenge: | Despite the strong performance of current NLP models, they can be brittle against adversarial inputs. |
| Approach: | They propose a rationale model that explicitly learns to ignore adversarial tokens . their approach leads to sizable improvements in robustness over baseline models . |
| Outcome: | The proposed model outperforms data augmentation with adversarial examples and closes the gap between model performance and an attacked test set. |
Extractive Adversarial Networks: High-Recall Explanations for Identifying Personal Attacks in Social Media Posts (D18-1)
Copied to clipboard
| Challenge: | Existing work on explaining classifier decisions has not addressed local feature redundancy . a common way to explain why a model classified an example is to extract a sparse subset of features that were responsible for the decision . |
| Approach: | They propose an adversarial method for producing high-recall explanations of text classifier decisions . they use a method which scans the residual of attention for remaining predictive signal . |
| Outcome: | The proposed method produces high-recall explanations of text classifier decisions . it uses a set of human-annotated personal attacks to evaluate the impact . |
Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification (2021.acl-long)
Copied to clipboard
| Challenge: | Text simplification reduces the language complexity of professional content for accessibility purposes. |
| Approach: | They propose that text simplification can be decomposed into a pipeline of tasks . they show that the pipeline can be used to predict whether a text needs to be simplified . |
| Outcome: | The proposed model improves the performance of out-of-sample simplification tests on a blackbox lexical model . the proposed model reduces the complexity of professional text by a large margin . |