Papers by Siyu Ren
Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained masked language models inherit a considerable amount of relational knowledge from the source corpora. |
| Approach: | They propose to specialize pretrained masked language models into relational models from the perspective of network pruning. |
| Outcome: | The proposed model can represent grounded commonsense relations at non-trivial sparsity while being generalizable . the proposed model improves on a wealth of NLP tasks, but we know little about how much knowledge it imparts . |
Multi-turn Response Selection using Dialogue Dependency Relations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing models for multi-turn response selection ignore the dependencies between the turns. |
| Approach: | They propose a dialogue extraction algorithm to transform a dialog history into threads based on their dependency relations. |
| Outcome: | The proposed model outperforms the state-of-the-art models on DSTC7 and DSTF8* with competitive results on UbuntuV2 . |
Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing text-image approaches use pre-trained vision-language representations for text retrieval . however, these models pose non-trivial memory requirements and substantial indexing time . |
| Approach: | They propose a framework to compress large pre-trained dual-encoders for lightweight text-image retrieval. |
| Outcome: | The proposed model performs better on Flickr30K and MSCOCO benchmarks than the original full model on mobile devices. |
Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language. |
| Approach: | They propose a model that integrates symbolic data into LLM training without loss of generality ability. |
| Outcome: | The proposed model performs better on symbol- and NL-centric tasks. |
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work evaluates faithfulness using models trained on related tasks or in-domain synthetic data. |
| Approach: | They propose to do zero-shot faithfulness evaluation with a foundation language model. |
| Outcome: | The proposed model outperforms ChatGPT on faithfulness and inconsistency detection with 24x fewer parameters and is competitive with existing models. |
Pruning Pre-trained Language Models with Principled Importance and Self-regularization (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models often contain a vast amount of parameters, posing nontrivial requirements for storage and computation. |
| Approach: | They propose a pruning method where model prediction is regularized by the latest checkpoint with increasing sparsity throughout pruning. |
| Outcome: | The proposed approach is effective at sparsity levels, and can be applied to natural language understanding, question answering, and data-to-text generation tasks. |
EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction (2025.naacl-long)
Copied to clipboard
| Challenge: | EASYTOOL combines tools from diverse tool documentation into a single tool instruction. |
| Approach: | They propose a framework that transforms tool documentation into a unified tool instruction. |
| Outcome: | EASYTOOL combines extensive tool documentation into a concise tool instruction . it reduces token consumption and improves performance of LLM-based agents . |
Low-Rank Prune-And-Factorize for Language Model Compression (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to reduce parameter redundancy in pre-processed language models fail to retain satisfactory performance under moderate to high compression rates. |
| Approach: | They propose to use network pruning to extract low-rank sparsity pattern desirable to matrix factorization. |
| Outcome: | The proposed method has a superior compression-performance trade-off compared to existing methods. |
Context Compression for Auto-regressive Transformers with Sentinel Tokens (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing Transformer-based LLMs have limited performance due to complexity of attention module . key-value cache is the major memory footprint and inference latency problem . |
| Approach: | They propose a plug-and-play approach that incrementally compresses token activation into compact ones . they also profile the benefit of context compression on improving the system throughout . |
| Outcome: | The proposed approach reduces memory footprint and inference latency by compressing tokens into compact ones. |