Papers by Navdeep Jaitly
Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are effective at answering clear questions but when faced with ambiguous queries they act unpredictably and produce incorrect outputs. |
| Approach: | They propose to use a surrogate problem to assess an LLMs’s ability to deduce an entity unknown to itself, but revealed to a judge, by asking the judge a series of queries. |
| Outcome: | The proposed model outperforms human players on the entity-deducing task by a large margin. |
Construction of Paired Knowledge Graph - Text Datasets Informed by Cyclic Evaluation (2024.lrec-main)
Copied to clipboard
Ali Mousavi, Xin Zhan, He Bai, Peng Shi, Theodoros Rekatsinas, Benjamin Han, Yunyao Li, Jeffrey Pound, Joshua M. Susskind, Natalie Schluter, Ihab F. Ilyas, Navdeep Jaitly
| Challenge: | Prior studies have shown that sequence-to-sequence models learn to hallucinate when the conditioning data has poor correlation with the sequence being produced. |
| Approach: | They construct a dataset that pairs Knowledge Graphs (KG) and text together and compare their results to a cyclic evaluation model. |
| Outcome: | The proposed model performs better on cyclic generation of KGs than on KG-T, but less well on synchronization of KTs. |
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (2024.acl-long)
Copied to clipboard
| Challenge: | Large language model pre-training is infeasible due to the large compute costs and duration associated with pre- training and the impending scarcity of high-quality data on the web. |
| Approach: | They propose to use an off-the-shelf instruction-tuned model prompted to paraphrase documents on the web in specific styles such as “like Wikipedia” or in “question-answer format” to jointly pre-train LLMs on real and synthetic rephrases. |
| Outcome: | The proposed model speeds up pre-training by 3x on the C4 dataset, and improves perplexity by 50% on average across different subsets of the Pile. |
Divide-or-Conquer? Which Part Should You Distill Your LLM? (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that Large Language Models (LLMs) can solve reasoning tasks better when they are encouraged to solve subtasks of the main task first. |
| Approach: | They propose a strategy that breaks down reasoning tasks into a problem decomposition phase and a solution phase and propose 'smaller' models that can achieve good generalization. |
| Outcome: | The proposed approach outperforms a single stage solution in two tasks and their impact on reasoning outcomes and inference cost. |
Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts. |
| Approach: | They propose to use retrieval-attention-probing to filter and de-noise long contexts during decoding and joint retrieval head training alongside the generation head to improve LCLM performance. |
| Outcome: | The proposed approach outperforms RAG and GPT-4-Turbo on most tasks despite being a much smaller model. |