Papers by Jielin Qiu
Embodied Executable Policy Learning with Language-based Scene Summarization (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing Large Language models with text inputs lack the capability to evolve with non-expert interactions with environments. |
| Approach: | They propose a novel learning paradigm that generates robots’ executable actions in the form of text, derived solely from visual observations. |
| Outcome: | The proposed learning paradigm surpasses baselines and can adapt to the target tasks effectively. |
SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM (2024.findings-emnlp)
Copied to clipboard
Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul Crook, Yifan Xu, Babak Damavandi, Xin Dong, Christos Faloutsos, Lei Li, Seungwhan Moon
| Challenge: | Vision-extended LLMs have made significant strides in VQA, but they still encounter significant difficulties in handling queries involving long-tail entities. |
| Approach: | They propose a benchmark to test models' ability to identify entities and provide detailed, entity-specific knowledge by combining 10 images and 10 knowledge-intensive QA pairs. |
| Outcome: | The proposed model outperforms existing methods on the SnapNTell dataset, achieving a 66.5% improvement in the BELURT score. |
SCCS: Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment (2023.findings-acl)
Copied to clipboard
Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin
| Challenge: | Existing methods for multimodal summarization ignore the structure and semantics of the whole video and article. |
| Approach: | They propose a semantic-consistent cross-domain summarization model that extracts features from video and article and uses fusion methods to select representative one. |
| Outcome: | The proposed model produces high-quality multimodal summaries on three MSMO datasets. |
MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models (2025.emnlp-demos)
Copied to clipboard
Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, Caiming Xiong
| Challenge: | Existing evaluation frameworks suffer from limitations such as static task benchmarks, limited scope, and inadequate integration with practical applications. |
| Approach: | They propose an open-source, Model Context Protocol-based evaluation framework specifically tailored for comprehensive and systematic assessment of LLM-powered agents. |
| Outcome: | The proposed framework uncovers nuanced performance patterns and identify domain-specific strengths and weaknesses, providing valuable insights beyond traditional binary success metrics. |
Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models? (2023.findings-eacl)
Copied to clipboard
Jielin Qiu, William Han, Jiacheng Zhu, Mengdi Xu, Michael Rosenberg, Emerson Liu, Douglas Weber, Ding Zhao
| Challenge: | Recent advances in Large Language Models (LLMs) have shown powerful ability in various downstream applications. |
| Approach: | They propose an approach for cardiovascular disease diagnosis and automatic ECG diagnosis report generation. |
| Outcome: | The proposed approach generates high-quality cardiac diagnosis reports and achieves competitive zero-shot classification performance even compared with supervised baselines. |
Can Brain Signals Reveal Inner Alignment with Human Languages? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Brain Signals, such as Electroencephalography, and human languages have been explored independently for many downstream tasks, however, the connection between them has not been well explored. |
| Approach: | They introduce a multimodal transformer alignment model to observe coordinated representations between EEG and language. |
| Outcome: | The proposed method achieved an F1-score improvement of 1.7% on ZuCo and 9.3% on Zuco datasets for sentiment analysis, and 7.4% on ZuCO for relation detection. |