Papers by Jiarui Lu
Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Aegis is an advanced LLM-based multi-agent for intelligent functional safety engineering that can perform all phases of a vehicle's lifecycle, including design, development, production, operation, and decommissioning. |
| Approach: | They introduce Aegis: An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering. |
| Outcome: | The proposed solution can perform Hazard Analysis and Risk Assessment (HARA), document Functional Safety Requirements (FSR), and plan test cases for Automatic Emergency Braking (AEB) systems. |
CREAD: Combined Resolution of Ellipses and Anaphora in Dialogues (2021.naacl-main)
Copied to clipboard
Bo-Hsiang Tseng, Shruti Bhargava, Jiarui Lu, Joel Ruben Antony Moniz, Dhivya Piraviperumal, Lin Li, Hong Yu
| Challenge: | Traditionally, anaphora resolution and ellipses resolution are limited in dialogues . despite rapid progress in dialogue systems, several difficulties remain . |
| Approach: | They propose a joint learning framework for modeling coreference resolution and query rewriting for complex, multi-turn dialogues. |
| Outcome: | The proposed model outperforms the state-of-the-art model on a rewritten dialogue dataset. |
Can Large Language Models Understand Context? (2024.findings-eacl)
Copied to clipboard
Yilun Zhu, Joel Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
| Challenge: | Existing evaluation methodologies for Large Language Models (LLMs) have been inadequate to evaluate their ability to understand contextual features. |
| Approach: | They propose a benchmark to assess large language models' ability to understand context by adapting existing datasets to suit their evaluation. |
| Outcome: | The proposed model performs better under the in-context learning pretraining scenario than state-of-the-art models. |
STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants (2023.emnlp-industry)
Copied to clipboard
Leon Zhang, Jiarui Lu, Joel Ruben Antony Moniz, Aditya Kulkarni, Dhivya Piraviperumal, Tien Dung Tran, Nick Tzou, Hong Yu
| Challenge: | Existing training datasets for steering use cases are limited due to the cold-start problem. |
| Approach: | They propose a steering detection model that predicts whether a follow-up turn is a user’s attempt to steer the previous command. |
| Outcome: | The proposed model outperforms existing models on human-graded evaluation sets and shows that it can identify steering intent with over 95% accuracy. |
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains (2025.findings-naacl)
Copied to clipboard
Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe Zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Simon Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang
| Challenge: | Existing benchmarks focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. |
| Approach: | They propose a Massive Multitask Agent Understanding benchmark that evaluates LLMs across five domains and offline tasks. |
| Outcome: | The Massive Multitask Agent Understanding (MMAU) benchmark evaluates models across five domains including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics. |
ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities (2025.findings-naacl)
Copied to clipboard
Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, Ruoming Pang
| Challenge: | Recent advances in large language models have led to a growing interest in tool assisted LLMs . toolSandbox includes stateful tool execution, implicit state dependencies between tools . |
| Approach: | a new tool-based evaluation tool is released to help LLMs evaluate their tool-use capabilities. a tool-driven evaluation tool includes stateful tool execution, implicit state dependencies between tools and a built-in user simulator. |
| Outcome: | the toolSandbox evaluation benchmark shows that open source and proprietary models have a performance gap . the benchmarks show that even the most capable LLMs are challenged by state dependent tasks . |
Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are effective at answering clear questions but when faced with ambiguous queries they act unpredictably and produce incorrect outputs. |
| Approach: | They propose to use a surrogate problem to assess an LLMs’s ability to deduce an entity unknown to itself, but revealed to a judge, by asking the judge a series of queries. |
| Outcome: | The proposed model outperforms human players on the entity-deducing task by a large margin. |