Papers by Dan Iter
PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents (2024.eacl-long)
Copied to clipboard
| Challenge: | Using chain-of-thought prompting, large language models perform better on complex reasoning tasks. |
| Approach: | They propose a prompting framework that decomposes a question into a sequence of actions and executes them over the document to obtain the answer. |
| Outcome: | The proposed framework outperforms zero-shot and chain-of-thought prompting on a QuALITY dataset . it proposes a plan based on actions mined from a training set and executes it step by step . |
The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions (2023.emnlp-main)
Copied to clipboard
Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, Jiawei Han
| Challenge: | Recent advances in large language models (LLMs) have produced models that exhibit remarkable performance across a variety of NLP tasks. |
| Approach: | They analyze a large-scale collection of user-GPT conversations to identify a significant gap between academic research in NLP and the needs of real-world NLP applications. |
| Outcome: | The proposed model outperforms existing models in a large-scale collection of user-GPT conversations and identifies a significant gap between the tasks that users frequently request from LLMs and the tasks commonly studied in academic research. |
In-Context Demonstration Selection with Cross Entropy Difference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can use in-context demonstrations to improve performance on zero-shot tasks. |
| Approach: | They propose a cross-entropy difference method for selecting in-context demonstrations that uses parameter efficient finetuning to train small models on training data. |
| Outcome: | The proposed method outperforms baseline selection methods on a mix-domain dataset and shows that the effectiveness of in-context demonstrations negatively correlates with the perplexity of the test example. |
Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models (2023.findings-emnlp)
Copied to clipboard
Zhihan Zhang, Shuohang Wang, Wenhao Yu, Yichong Xu, Dan Iter, Qingkai Zeng, Yang Liu, Chenguang Zhu, Meng Jiang
| Challenge: | Large language models can perform a wide range of tasks by following natural language instructions without task-specific fine-tuning. |
| Approach: | They propose a method to automatically improve the quality of LLM instructions . they leverage the generative ability of LMS to generate diverse candidate instructions based on a scoring model trained on 575 existing NLP tasks. |
| Outcome: | The proposed method surpasses human-written and LLM-generated instructions on 118 out-of-domain tasks. |
Automatic Prompt Optimization with “Gradient Descent” and Beam Search (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown impressive performance as general purpose agents, but their abilities remain highly dependent on prompts which are hand written with onerous trial-and-error effort. |
| Approach: | They propose an algorithm that uses numerical gradient descent to automatically improve prompts by rewriting vague task descriptions into more precise annotation instructions. |
| Outcome: | The proposed algorithm outperforms previous methods and improves performance on three benchmark NLP tasks and the novel problem of LLM jailbreak detection. |
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | Recent models for unsupervised representation learning of text have put little focus on discourse-level representations. |
| Approach: | They propose an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. |
| Outcome: | The proposed model outperforms the BERT-Large model on the discourse representation benchmark DiscoEval and yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse. |
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in large language models have revolutionized the way summarization is generated. |
| Approach: | They propose a summarization model derived from GPT-3.5 through distillation that is compact and has comparable summarizing capabilities to GPT-3. |
| Outcome: | The proposed model outperforms the established best small models in prefix-tuning and full-data fine-tuned scenarios. |
LMGQS: A Large-scale Dataset for Query-focused Summarization (2023.findings-emnlp)
Copied to clipboard
Ruochen Xu, Song Wang, Yang Liu, Shuohang Wang, Yichong Xu, Dan Iter, Pengcheng He, Chenguang Zhu, Michael Zeng
| Challenge: | Lack of large-scale datasets for query-focused summarization hinders model development . lack of data limits the ability of QFS models to train robust neural models . |
| Approach: | They propose to generate a query for each summary sentence in a generic summarization annotation using a pretrained language model. |
| Outcome: | The proposed model achieves state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks. |
The Trade-offs of Domain Adaptation for Neural Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Neural Language Models (LMs) trained on large generic training sets have been shown to be effective at adapting to smaller, specific target domains for language modeling and other downstream tasks. |
| Approach: | They propose a framework for a Neural Language Models (LM) to be presented in a common framework. |
| Outcome: | The proposed framework highlights similarities and subtle differences between adaptation techniques and the framework. |
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment (2023.emnlp-main)
Copied to clipboard
| Challenge: | Conventional reference-based metrics have low correlation with human judgments, especially for open-ended generation tasks. |
| Approach: | They propose to use large language models as reference-free NLG evaluators to assess the quality of NLG outputs. |
| Outcome: | The proposed framework outperforms all previous methods in two generation tasks, and has a Spearman correlation of 0.514 with human on summarization task, and a large variance in human judgments. |
Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreference (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches simplify by considering coreference only within document clusters, but this fails to handle inter-cluster coreference, common in many applications. |
| Approach: | They propose to model entities/events in a reader’s focus as a neighborhood within a learned latent embedding space which minimizes the distance between mentions and the centroids of their gold coreference clusters. |
| Outcome: | The proposed model achieves state-of-the-art for events and entities on the ECB+, Gun Violence, Football Coreference, and Cross-Domain Cross-DDocument Coreference corpora. |
How Does In-Context Learning Help Prompt Tuning? (2024.findings-eacl)
Copied to clipboard
| Challenge: | a growing number of parameter-efficient adaptation methods are needed to fine-tune large language models. |
| Approach: | They propose a method that combines prompt tuning and in-context learning to improve prompt tuning by concatenating a natural language demonstration with learned prompt embeddings. |
| Outcome: | The proposed method outperforms prompt tuning and prompt tuning on five language generation tasks. |