Papers by Siyu Zhu
Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained masked language models inherit a considerable amount of relational knowledge from the source corpora. |
| Approach: | They propose to specialize pretrained masked language models into relational models from the perspective of network pruning. |
| Outcome: | The proposed model can represent grounded commonsense relations at non-trivial sparsity while being generalizable . the proposed model improves on a wealth of NLP tasks, but we know little about how much knowledge it imparts . |
Multi-turn Response Selection using Dialogue Dependency Relations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-turn response selection ignore the dependencies between the turns. |
| Approach: | They propose a dialogue extraction algorithm to transform a dialog history into threads based on their dependency relations. |
| Outcome: | The proposed model outperforms the state-of-the-art models on DSTC7 and DSTF8* with competitive results on UbuntuV2 . |
Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing text-image approaches use pre-trained vision-language representations for text retrieval . however, these models pose non-trivial memory requirements and substantial indexing time . |
| Approach: | They propose a framework to compress large pre-trained dual-encoders for lightweight text-image retrieval. |
| Outcome: | The proposed model performs better on Flickr30K and MSCOCO benchmarks than the original full model on mobile devices. |
Modeling Multi-Dimensional Cognitive States in Large Language Models under Cognitive Crowding (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) mainly address isolated tasks such as emotion analysis or stance detection. |
| Approach: | They propose a large-scale model that combines large-level annotations with hyperbolic space to model human cognitive states. |
| Outcome: | The proposed model outperforms baseline models on cognitive dimensions on single dimension tasks while retaining strong hierarchical structure. |
LLM×MapReduce-V3: Enabling Interactive In-Depth Survey Generation through a MCP-Driven Hierarchically Modular Agent System (2025.emnlp-demos)
Copied to clipboard
Yu Chao, Siyu Lin, Xiaorong Wang, Zhu Zhang, Zihan Zhou, Haoyu Wang, Shuo Wang, Jie Zhou, Zhiyuan Liu, Maosong Sun
| Challenge: | Generating high-quality long-form survey articles poses significant challenges to AI Agent systems. |
| Approach: | They propose a hierarchically modular agent system for long-form survey generation . they use atomic models to implement skeleton initialization, digest construction, and skelet refinement . human evaluations demonstrate system surpasses representative baselines . |
| Outcome: | The proposed system surpasses representative baselines in both content depth and length, highlighting the strength of MCP-based modular planning. |
Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems (2025.emnlp-industry)
Copied to clipboard
Kayhan Behdin, Ata Fatahibaarzi, Qingquan Song, Yun Dai, Aman Gupta, Zhipeng Wang, Hejian Sang, Shao Tang, Gregory Dexter, Sirou Zhu, Siyu Zhu, Tejas Dharamsi, Vignesh Kothapalli, Zhoutong Fu, Yihan Cao, Pin-Lun Hsu, Fedor Borisyuk, Natesh S. Pillai, Luke Simon, Rahul Mazumder
| Challenge: | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications. |
| Approach: | They propose two techniques for training and deploying small language models that deliver high performance for a variety of industry use cases. |
| Outcome: | The proposed techniques retain much of the quality of larger models while reducing training/serving costs and latency. |
DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing datasets focus on sentence-level event extraction, but document-level EE is limited due to the lack of large-scale and practical training and evaluation datasets. |
| Approach: | They propose a document-level event extraction dataset with 27,000+ events and 180,000+ arguments. |
| Outcome: | The proposed dataset includes 27,000+ events, 180,000+ arguments and large-scale manual annotations, fine-grained argument types and application-oriented settings. |
MIO: A Foundation Model on Multimodal Tokens (2025.emnlp-main)
Copied to clipboard
Zekun Moore Wang, King Zhu, Chunpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jessie Wang, Ning Shi, Siyu Li, Yizhi Li, Haoran Que, Zhaoxiang Zhang, Yuanxing Zhang, Ge Zhang, Ke Xu, Jie Fu, Wenhao Huang
| Challenge: | Existing models lack multimodal understanding capabilities, resulting in closed-source model that does not support multimodal interleaved sequences. |
| Approach: | They propose a foundation model built on multimodal tokens capable of understanding and generating speech, text, images, and videos in an end-to-end, autoregressive manner. |
| Outcome: | The proposed model is able to understand speech, text, images, and videos in an end-to-end, autoregressive manner. |
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work evaluates faithfulness using models trained on related tasks or in-domain synthetic data. |
| Approach: | They propose to do zero-shot faithfulness evaluation with a foundation language model. |
| Outcome: | The proposed model outperforms ChatGPT on faithfulness and inconsistency detection with 24x fewer parameters and is competitive with existing models. |
Pruning Pre-trained Language Models with Principled Importance and Self-regularization (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models often contain a vast amount of parameters, posing nontrivial requirements for storage and computation. |
| Approach: | They propose a pruning method where model prediction is regularized by the latest checkpoint with increasing sparsity throughout pruning. |
| Outcome: | The proposed approach is effective at sparsity levels, and can be applied to natural language understanding, question answering, and data-to-text generation tasks. |
T⋆: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning (2026.acl-short)
Copied to clipboard
| Challenge: | Autoregressive (AR) modeling via next-token prediction dominates scaling practice and deployed systems. |
| Approach: | They propose a TraceRL-based curriculum for progressive block-size scaling in masked diffusion language models. |
| Outcome: | The proposed curriculum outperforms direct large-block TraceRL on two SDAR scales and three benchmarks and retains block-size-specific non-monotone updates while improving accuracy. |
Low-Rank Prune-And-Factorize for Language Model Compression (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to reduce parameter redundancy in pre-processed language models fail to retain satisfactory performance under moderate to high compression rates. |
| Approach: | They propose to use network pruning to extract low-rank sparsity pattern desirable to matrix factorization. |
| Outcome: | The proposed method has a superior compression-performance trade-off compared to existing methods. |
ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use (2025.acl-long)
Copied to clipboard
Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiecao Chen
| Challenge: | Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models. |
| Approach: | They propose a dataset that provides rigorous evaluation of multi-hop tool use. |
| Outcome: | The proposed model achieves 49.04% accuracy across five model families. |
Context Compression for Auto-regressive Transformers with Sentinel Tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformer-based LLMs have limited performance due to complexity of attention module . key-value cache is the major memory footprint and inference latency problem . |
| Approach: | They propose a plug-and-play approach that incrementally compresses token activation into compact ones . they also profile the benefit of context compression on improving the system throughout . |
| Outcome: | The proposed approach reduces memory footprint and inference latency by compressing tokens into compact ones. |