Papers by Zachary Lipton
Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have improved summarization, but they still face a challenge of hallucination. |
| Approach: | They propose a taxonomy of errors to address the problem of hallucination in LLMs . they propose two prompt-based approaches for fine-grained error detection . |
| Outcome: | The proposed model outperforms existing metrics in identifying the novel "Contextual Inference" error type. |
Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Several studies claim that domain-adaptive pretraining improves performance on downstream medical tasks. |
| Approach: | They compare medical LLMs and VLMs against their corresponding base models . they find that medical Lms outperform their base models in 12.1% of cases . |
| Outcome: | The proposed models outperform their base models on medical questions and tasks in 12.1% of cases and reach a tie in 49.8% of cases. |
Model-tuning Via Prompts Makes NLP Models Adversarially Robust (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained models are typically adapted to downstream tasks by appending a randomly initialized multilayer perceptron to their topmost representation layer and fine-tuning the entire model on a downstream task. |
| Approach: | They propose to append a multilayer perceptron to a CLS token and fine-tune the entire model on a downstream task. |
| Outcome: | The proposed model-tuning via prompts outperforms adversarial training-based state-of-art defenses by 3.5% and improves against adversarials by 8% over standard methods. |
Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains (2024.eacl-short)
Copied to clipboard
| Challenge: | Recent work has shown that large language models can generate zero-shot summaries without explicit supervision that are often comparable or even preferred to manually composed reference summary. |
| Approach: | They evaluate large language models (LLMs) that generate zero-shot summaries without explicit supervision that are often comparable to manual reference summary . they acquire annotations from domain experts to identify inconsistencies in summaires and categorize errors. |
| Outcome: | The proposed model outperforms fine-tuned models in biomedical articles and legal bills across specialized domains. |
Goodhart’s Law Applies to NLP’s Explanation Benchmarks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Popular methods for "explaining" the outputs of natural language processing (NLP) models operate by highlighting a subset of input tokens that ought, in some sense, to be salient. |
| Approach: | They propose to inflate a model’s comprehensiveness and sufficiency scores dramatically without altering its predictions or explanations on in-distribution inputs. |
| Outcome: | The proposed metrics exploit the tendency for extracted explanations and complements to be “out-of-support” relative to each other and in-distribution inputs. |
USB: A Unified Summarization Benchmark Across Tasks and Domains (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing summarization benchmarks lack the rich annotations needed to address important problems related to control and reliability. |
| Approach: | They propose a Wikipedia-derived summarization benchmark with crowd-sourced annotations . they find that fine-tuned models outperform larger few-shot prompted language models . |
| Outcome: | The proposed model outperforms many-shot prompted language models on multiple tasks . the proposed model is based on Wikipedia annotations and can be used in other domains . |
Downstream Datasets Make Surprisingly Good Pretraining Corpora (2023.acl-long)
Copied to clipboard
| Challenge: | a dominant practice is to fine tune large pretrained transformer models using smaller downstream datasets . performance gains are not always attributable to the use of external data in massive amounts . |
| Approach: | They propose to use the same (downstream) training data for pretraining and finetuning to compare models. |
| Outcome: | The proposed model outperforms standard pretraining on the BookWiki corpus on 7 and 5 datasets. |