Papers by Kalpesh Krishna
RankGen: Improving Text Generation with Large Ranking Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Modern language models assign high probabilities to output sequences that are repetitive, incoherent, or irrelevant to the prefix. |
| Approach: | They propose a 1.2B parameter encoder model for English that scores model generations given a prefix. |
| Outcome: | The proposed model outperforms decoding algorithms on automatic metrics and human evaluations with English writers. |
Generating Question-Answer Hierarchies (P19-1)
Copied to clipboard
| Challenge: | a novel text generation task uses reading comprehension datasets to generate a hierarchy of question-answer pairs . users can click on high-level questions to reveal related but more specific questions . |
| Approach: | They propose a text-generating task which converts a document into a hierarchy of question-answer pairs . users can click on high-level questions to reveal related but more specific questions . they then use the specificity-labeled reading comprehension dataset to generate the hierarchy . |
| Outcome: | The proposed system can be used to generate a hierarchy of question-answer pairs based on reading comprehension datasets. |
LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization (2023.eacl-main)
Copied to clipboard
| Challenge: | Human evaluation is labor-intensive, expensive to scale, and difficult to design. |
| Approach: | They propose a set of guidelines for human evaluation of faithfulness in long-form summaries that address the following challenges: (1) How can we achieve high inter-annotator agreement on faithfulness scores? (2) How can our annotator minimize workload while maintaining accurate faithfulness? |
| Outcome: | The proposed framework reduces inter-annotator variance in faithfulness scores while minimizing annotator workload while maintaining accuracy. |
Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature (2022.emnlp-main)
Copied to clipboard
Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, Mohit Iyyer
| Challenge: | Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators . a dataset of non-English language novels is used to study literary MT . |
| Approach: | They use a dataset of non-English language novels aligned to human and automatic English translations to study literary MT. |
| Outcome: | The proposed model prefers human translations over machine translations at a rate of 84% . state-of-the-art MT metrics do not correlate with preferences, the study finds . |
Syntactically Supervised Transformers for Faster Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Standard decoders for neural machine translation generate a single token per timestep, which slows inference . a series of controlled experiments demonstrates that SynST decodes sentences 5x faster than the baseline autoregressive Transformer. |
| Approach: | They propose a syntactically supervised Transformer that generates all target tokens in one shot . synST is a variant of the Transformer architecture that autoregressively predicts a chunked parse tree . |
| Outcome: | The proposed method decodes sentences 5x faster than the baseline method on En-De and En-Fr datasets while achieving higher BLEU scores. |
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution (2023.findings-eacl)
Copied to clipboard
Ankita Gupta, Marzena Karpinska, Wenlong Zhao, Kalpesh Krishna, Jack Merullo, Luke Yeh, Mohit Iyyer, Brendan O’Connor
| Challenge: | Existing datasets vary in definition of coreferences and are curated for linguistic experts. |
| Approach: | They propose to use ezCoref to create a crowdsourcing-friendly coreference annotation methodology that teaches annotators only cases that are treated similarly across existing datasets. |
| Outcome: | The proposed method reannotates 240 passages from seven existing english coreference datasets while teaching annotators only cases that are treated similarly across them. |
Long Document Summarization in a Low Resource Setting using Pretrained Language Models (2021.acl-srw)
Copied to clipboard
Ahsaas Bajaj, Pavitra Dangati, Kalpesh Krishna, Pradhiksha Ashok Kumar, Rheeya Uppaal, Bradford Windsor, Eliot Brenner, Dominic Dotterrer, Rajarshi Das, Andrew McCallum
| Challenge: | Existing abstractive summarization methods only achieve 17.9 ROUGE-L in low-resource settings. |
| Approach: | They propose to use a modern abstractive summarization algorithm to extract salient sentences from long documents to improve their performance. |
| Outcome: | The proposed method beats several competitive salience detection baselines and the identified salient sentences agree with independent human labeling by domain experts. |
Do Long-Range Language Models Actually Use Long-Range Context? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. |
| Approach: | They analyze two long-range Transformer language models that accept 8K token inputs . they find that providing long-term context only improves their predictions on a small set of tokens - not sentence-level ones . |
| Outcome: | The proposed model improves on PG-19 with only 2K tokens and does not help at all for sentence-level prediction tasks. |
SLING: Sino Linguistic Evaluation of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Using pre-trained language models, we find that the accuracy of LMs is far below human performance. |
| Approach: | They propose a benchmark of Sino LINGuistics which consists of 38K sentence pairs in Mandarin Chinese grouped into 9 high-level linguistic phenomena. |
| Outcome: | The proposed model performs better on local phenomena than hierarchical models and has a strong gender and number bias. |
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023.emnlp-main)
Copied to clipboard
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, Hannaneh Hajishirzi
| Challenge: | Evaluating the factuality of long-form text generated by large language models (LMs) is non-trivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate and (2) human evaluation is time-consuming and costly. |
| Approach: | They introduce a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atom facts supported by a reliable knowledge source. |
| Outcome: | The proposed model breaks a generation into atomic facts and computes the percentage of atomic fact supported by a reliable knowledge source. |
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2025.naacl-long)
Copied to clipboard
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui
| Challenge: | Recent advances in Large Language Models (LLMs) have significantly enhanced their capabilities across various cognitive tasks. |
| Approach: | They propose a high-quality evaluation dataset to test LLMs' ability to provide factual responses, assess retrieval capabilities, and evaluate the reasoning required to generate final answers. |
| Outcome: | The proposed framework improves performance in end-to-end RAG scenarios. |
RELiC: Retrieving Evidence for Literary Claims (2022.acl-long)
Copied to clipboard
| Challenge: | Literature scholars often provide evidence for claims they make in the form of quotations from a work . a novel evidence retrieval task requires a deep understanding of complex literary phenomena . |
| Approach: | They use a large-scale dataset to formulate a task of literary evidence retrieval . they implement a RoBERTa-based dense passage retriever that outperforms existing methods . |
| Outcome: | The proposed model outperforms pretrained information retrieval baselines, but there is room for improvement. |
Revisiting the Importance of Encoding Logic Rules in Sentiment Classification (D18-1)
Copied to clipboard
| Challenge: | Neural models that explicitly encode word order, syntax and semantic features are unequipped to deal with complex syntactic structures that affect sentiment, such as contrastive conjunctions. |
| Approach: | They propose to use contextualized ELMo embeddings instead of logic rules to compare models on syntactically complex sentences. |
| Outcome: | The proposed models outperform baseline models even on sentences with ambiguous sentiment labels. |
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for style transfer warp the input’s meaning through attribute transfer, which changes semantic properties such as sentiment. |
| Approach: | They propose a method for fine-tuning pretrained language models on automatically generated paraphrase data to improve the efficiency of style transfer. |
| Outcome: | The proposed method outperforms state-of-the-art style transfer systems on human and automatic evaluations and proposes fixed variants. |
PostMark: A Robust Blackbox Watermark for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect LLM-generated text require access to the underlying LLM’s logits, which LLM providers are loath to share due to fears of model distillation. |
| Approach: | They develop a post-hoc watermarking procedure that inserts an input-dependent set of words into the text after the decoding process has completed. |
| Outcome: | The proposed method is more robust to paraphrasing attacks than existing methods. |
GEE! Grammar Error Explanation with Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing grammatical error correction tools do not provide natural language explanations of errors . a system needs to provide one-sentence explanations for each grammamatical errors in a pair of erroneous and corrected sentences. |
| Approach: | They propose a grammar error explanation task that uses one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences. |
| Outcome: | The proposed pipeline identifies grammar errors in German, Chinese, and English . human evaluation reveals that 93.9% of German errors, 96.4% of Chinese errors, and 92.20% of English errors are correctly detected and explained. |
Hurdles to Progress in Long-form Question Answering (2021.naacl-main)
Copied to clipboard
| Challenge: | Long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. |
| Approach: | They propose a long-form question answering system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. |
| Outcome: | The proposed system tops the public leaderboard on the ELI5 LFQA dataset, but it has several troubling issues. |
Few-shot Controllable Style Transfer for Low-Resource Multilingual Settings (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for few-shot style transfer often copy inputs verbatim . a new method is better at controlling the style transfer magnitude using an input scalar knob. |
| Approach: | They propose a method to model the stylistic difference between paraphrases by rewriting a sentence into a target style while preserving semantics. |
| Outcome: | The proposed method achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages. |
Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | evaluating large language models' output is difficult due to the high cost of human evaluation. |
| Approach: | They propose a family of foundational large autorater models that train on over 100 quality assessment tasks. |
| Outcome: | The proposed model outperforms models on 8 of 12 autorater benchmarks on 53 quality assessment tasks. |