Papers by Kellie Webster
Query Refinement Prompts for Closed-Book Long-Form QA (2023.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer questions and produce long-form texts, but the latter is difficult to evaluate since they are subjective in nature. |
| Approach: | They propose query refinement prompts that encourage LLMs to express multifacetedness and generate long-form answers covering multiple facets of the question. |
| Outcome: | The proposed model outperforms fully finetuned models in the closed-book setting and retrieve-then-generate open-book models. |
A Challenge Set and Methods for Noun-Verb Ambiguity (D18-1)
Copied to clipboard
| Challenge: | English part-of-speech taggers make egregious errors related to noun-verb ambiguity, despite having achieved 97%+ accuracy on the WSJ Penn Treebank since 2002. |
| Approach: | They propose to use a WSJ dataset to identify 30,000 examples of noun-verb ambiguity . they find that english part-of-speech taggers make egregious errors related to nouns and verbs . |
| Outcome: | The proposed model improves on the WSJ Penn Treebank by 14% and 52% relative to the previous model. |
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020.acl-main)
Copied to clipboard
| Challenge: | toxicity prediction and sentiment analysis models perpetuate undesirable social biases from the data on which they are trained. |
| Approach: | They propose to use toxicity prediction and sentiment analysis to examine whether NLP models perpetuate undesirable biases towards mentions of disability. |
| Outcome: | The proposed models contain undesirable biases towards mentions of disability in two English language models. |
Automatically Identifying Gender Issues in Machine Translation using Perturbations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | a novel approach to machine translation has addressed outstanding challenges, including the modeling and treatment of gendered language. |
| Approach: | They propose a method to mine examples from real world data to explore challenges for deployed systems. |
| Outcome: | The proposed method exposes where model representations are gendered and the unintended consequences of genderes in downstream applications. |
Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias (2020.emnlp-main)
Copied to clipboard
| Challenge: | English challenge datasets highlight gender-ambiguous occurrences of ‘doctor’ as male doctors, but they are not useful for other languages. |
| Approach: | They propose to build multi-task challenge datasets for detecting gender bias that lead to unambiguously wrong model predictions for languages with type B reflexivization. |
| Outcome: | The proposed dataset can detect gender bias in languages with type B reflexivization and spans four languages and four NLP tasks. |
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing question answering datasets lack diversity in gender, profession, and nationality. |
| Approach: | They focus on how well QA models generalize across demographic subsets . english-language QA datasets mostly ask about US men from a few professions - this is problematic because most English speakers are not from the US or UK . |
| Outcome: | The proposed model accuracy is lower for people based on gender, profession, and nationality, but there is more variation on professions (question topic) and question ambiguity. |