Papers by Lajanugen Logeswaran
GRACE: Discriminator-Guided Chain-of-Thought Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) can assign a high likelihood to incorrect steps . Existing models (LLMs), however, struggle with complex multi-step reasoning. |
| Approach: | They propose a stepwise decoding approach that steers the decoding process towards producing correct reasoning steps. |
| Outcome: | The proposed approach outperforms existing methods on math and symbolic reasoning tasks. |
Auto-Intent: Automated Intent Discovery and Self-Exploration for Large Language Model Web Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work to improve LLMs' planning and reasoning capabilities has shown that they have strong priors to reason about intermediate subgoals, but their accuracy is limited especially in complex environments such as web navigation. |
| Approach: | They propose a method to adapt a pre-trained large language model (LLM) as an agent for a target domain without direct fine-tuning by extracting intents from target domain demonstrations in a highly compact form. |
| Outcome: | The proposed method significantly improves the performance of GPT-3.5, 4 and Llama-3.1-70B, 405B agents on large-scale real-website navigation benchmarks from Mind2Web and online navigation tasks with its cross-benchmark generalization from Mind2. |
Code Models are Zero-shot Precondition Reasoners (2024.naacl-long)
Copied to clipboard
Lajanugen Logeswaran, Sungryull Sohn, Yiwei Lyu, Anthony Liu, Dong-Ki Kim, Dongsub Shim, Moontae Lee, Honglak Lee
| Challenge: | Existing methods to reason about action preconditions are lacking for agent to complete tasks. |
| Approach: | They propose a method to reason about action preconditions using pre-trained code models. |
| Outcome: | The proposed approach improves few-shot policy learning approaches across task-oriented dialog and embodied textworld benchmarks. |
Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing value probing methods that capture in-context information and predict models’ real-world actions are limited and lack systematic comparisons. |
| Approach: | They compare three widely used value probing methods: token likelihood, sequence perplexity, and text generation. |
| Outcome: | The proposed methods exhibit large variances under non-semantic perturbations in prompts and option formats, with sequence perplexity being the most robust overall. |
Prospector: Improving LLM Agents with Self-Asking and Trajectory Ranking (2024.findings-emnlp)
Copied to clipboard
Byoungjip Kim, Youngsoo Jang, Lajanugen Logeswaran, Geon-Hyeong Kim, Yu Jin Kim, Honglak Lee, Moontae Lee
| Challenge: | Existing LLMs are limited in their ability to incorporate feedback from an environment. |
| Approach: | They propose an LLM agent that consists of an Actor and a Critic. |
| Outcome: | The proposed agent outperforms existing LLMs on benchmark environments and shows that it can generate diverse trajectories and pick the most rewarding trajectory. |
Few-shot Reranking for Multi-hop QA via Language Model Prompting (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multi-hop QA with open-domain questions require a large number of labeled question-document pairs for retrieval. |
| Approach: | They propose a language-based prompt for multi-hop path reranking that relies on language model prompting to generate a relevance score between a question and the path. |
| Outcome: | The proposed method yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples. |
Zero-Shot Entity Linking by Reading Entity Descriptions (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to link entities to unseen entities require in-domain labeled data. |
| Approach: | They propose a zero-shot entity linking task where mentions must be linked to unseen entities without in-domain labeled data. |
| Outcome: | The proposed task can generalize to unseen entities without metadata or alias tables . the proposed system improves over baselines, including BERT, on a new dataset . |
A Picture is Worth a Thousand Words: Language Models Plan from Pixels (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work uses pre-trained language models to reason about plans from text instructions in embodied visual environments. |
| Approach: | They propose to use pre-trained language models to reason about plan sequences from text instructions in embodied visual environments. |
| Outcome: | The proposed approach outperforms previous approaches on the ALFWorld and VirtualHome benchmarks. |
Unsupervised Task Graph Generation from Instructional Video Transcripts (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to task graph generation use instruction-tuned language models to generate accurate task graphs. |
| Approach: | They propose an unsupervised approach that combines the reasoning capabilities of instruction-tuned language models with clustering and ranking components to generate accurate task graphs. |
| Outcome: | The proposed approach generates more accurate task graphs than a supervised learning approach on tasks from the ProceL and CrossTask datasets. |
Few-shot Subgoal Planning with Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown successful progress in many text understanding benchmarks. |
| Approach: | They propose a strategy to re-rank language model predictions based on interaction and feedback from the environment. |
| Outcome: | The proposed approach shows competitive performance on subgoal prediction and task completion in the ALFRED benchmark compared to prior methods that assume more subgoals supervision. |
Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense (2024.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated substantial commonsense understanding through numerous benchmark evaluations. |
| Approach: | They conduct a comprehensive examination of the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks. |
| Outcome: | The language used to query the LLMs can impact their performance on cultural-related tasks. |
You don’t need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments (2024.naacl-long)
Copied to clipboard
Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, David Jurgens
| Challenge: | Large Language Models (LLMs) are popular for research in social sciences . currently, prompting LLMs is insufficient to accurately and reliably capture model perceptions, and we discuss potential alternatives to improve this. |
| Approach: | They construct a dataset that contains 693 questions encompassing 39 different instruments of persona measurement on 115 persona axes and a set of questions containing minor variations. |
| Outcome: | The proposed model can generate answers and negate statements in a consistent and robust manner. |
TOD-Flow: Modeling the Structure of Task-Oriented Dialogues (2023.emnlp-main)
Copied to clipboard
Sungryull Sohn, Yiwei Lyu, Anthony Liu, Lajanugen Logeswaran, Dong-Ki Kim, Dongsub Shim, Honglak Lee
| Challenge: | Recent advances in task-oriented dialogue systems have limitations regarding transparency and controllability. |
| Approach: | They propose to infer the TOD-flow graph from dialog data annotated with dialog acts and integrate it with any dialogue model to improve its prediction performance, transparency, and controllability. |
| Outcome: | The proposed approach improves dialog act classification and response generation performance in the MultiWOZ and SGD benchmarks. |
Merging Generated and Retrieved Knowledge for Open-Domain QA (2023.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain question answering systems often have retrieval modules but retrieving passages from external knowledge sources is known to suffer from insufficient knowledge coverage. |
| Approach: | They propose a Compatibility-Oriented knowledge Merging framework to leverage both sources of information by matching LLM-generated passages with retrieved counterparts into compatible pairs. |
| Outcome: | The proposed framework outperforms baselines on three out of four tested open-domain QA benchmarks. |
When ”A Helpful Assistant” Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Commercial AI systems often define the role of the LLM in system prompts. |
| Approach: | They conduct a systematic evaluation of personas in system prompts by adding 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. |
| Outcome: | The proposed model does not improve performance in the system prompt setting where no persona is added. |
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows that an adversary can extract training data from Pretrained Language Models including Personally Identifiable Information (PII) such as names, phone numbers, and email addresses. |
| Approach: | They propose to use knowledge unlearning to reduce privacy risks for LMs by performing gradient ascent on target token sequences instead of trying to unlearn all the data at once. |
| Outcome: | The proposed method can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. |
Small Language Models Need Strong Verifiers to Self-Correct Reasoning (2024.findings-acl)
Copied to clipboard
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
| Challenge: | Existing studies show that large language models can self-correct their outputs by generating a critique and revising it based on the critique. |
| Approach: | They propose a pipeline that prompts small language models to collect self-correction data that supports the training of self-refinement abilities. |
| Outcome: | The proposed pipeline improves the self-correction abilities of two models on five datasets spanning math and commonsense reasoning. |