Papers by Shuyang Cao
AWESOME: GPU Memory-constrained Long Document Summarization using Memory Mechanism and Global Salient Content (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing solutions focus on efficient attentions or divide-and-conquer strategies, but these methods sacrifice global context, leading to incoherent and uninformative summaries. |
| Approach: | They propose to leverage the memory-efficient nature of divide-and-conquer methods while preserving global context. |
| Outcome: | The proposed framework improves informativeness, faithfulness, and coherence over baselines on government reports, meeting transcripts, screenplays, scientific papers, and novels. |
HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Document structure is critical for efficient information consumption, but it is difficult to encode it efficiently into the modern Transformer architecture. |
| Approach: | They propose a task which injects Hierarchical Biases foR Incorporating Document Structure into attention score calculation. |
| Outcome: | The proposed model produces better question-summary hierarchies than comparisons on hierarchy quality and content coverage, the authors show . |
Zero-shot Generalization in Dialog State Tracking through Generative Question Answering (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for Dialog State Tracking do not generalize well to new domains and unseen slots. |
| Approach: | They propose an ontology-free framework that queries for unseen constraints and slots in multi-domain task-oriented dialogs using a conditional language model pre-trained on substantive English sentences. |
| Outcome: | The proposed framework improves goal accuracy in zero-shot domain adaptation settings by up to 9% over the previous state-of-the-art on the MultiWOZ 2.1 dataset. |
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for generating abstractive summarization are inconsistent and rely on heuristically created data for error handling. |
| Approach: | They propose a contrastive learning formulation that leverages both positive and negative summaries to train summarization systems that are better at distinguishing between them. |
| Outcome: | The proposed learning framework produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training. |
Efficient Attentions for Long Document Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models that use full attentions have quadratic computational and memory complexities, and are too costly for long documents. |
| Approach: | They propose an efficient encoder-decoder attention with head-wise positional strides to effectively pinpoint salient information from the source. |
| Outcome: | The proposed model can process ten times more tokens than current models that use full attentions. |
Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior work has shown that in-context learning (ICL) with retriever augmentation can help LLMs better capture long-tail knowledge, reducing their reliance on pre-trained data. |
| Approach: | They propose a reinforcement learning-based dynamic uncertainty ranking method that accounts for the varying impact of each retrieved sample on LLM predictions. |
| Outcome: | The proposed method outperforms baseline models on question-answering datasets by 2.76% and 5.96% on long-tail questions that elude zero-shot inference. |
BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics (2023.acl-long)
Copied to clipboard
Liang Ma, Shuyang Cao, Robert L Logan IV, Di Lu, Shihao Ran, Ke Zhang, Joel Tetreault, Alejandro Jaimes
| Challenge: | Existing benchmarks measure the correlation with human judgements of faithfulness on model-generated summaries, but they are insufficient for diagnosing whether metrics are consistent, effective on human-written texts, and sensitive to different error types. |
| Approach: | They propose to use unfaithful minimal pairs to measure the consistency of automatic faithfulness metrics by comparing human-written summary pairs with a dataset of 889 human-writing, minimally different summary pairs. |
| Outcome: | The proposed benchmarks show that the most discriminative metrics tend not to be the most consistent, and that the best performing metrics are sensitive to errors. |
EpiGEN: An Efficient Multi-Api Code GENeration Framework under Enterprise Scenario (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to large language models fail to meet expectations for code generation tasks . existing approaches are faced with drawbacks of high resource consumption and inadequate handling of multi-API tasks. |
| Approach: | They propose an Efficient multi-Api code GENeration framework that uses private APIs to pre-train LLMs. |
| Outcome: | The proposed framework shows good acceptability and readability on single-GPU tasks compared to fully fine-tuned LLMs with a larger number of parameters. |
Controllable Open-ended Question Generation with A New Question Type Ontology (2021.acl-long)
Copied to clipboard
| Challenge: | Existing question types are limited to generating multiple-sense questions . we present a question type-aware question generation framework to generate open-ended questions based on multiple-phrase questions - a task that is less explored . |
| Approach: | They propose a question type-aware question generation framework which predicts question focuses and produces the question. |
| Outcome: | The proposed model improves question quality over competitive comparisons on large-scale datasets. |
Attention Head Masking for Inference Time Content Selection in Abstractive Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing studies show that multi-heads attentions at the same layer collectively guide the summarization. |
| Approach: | They propose an inference-time attention head masking mechanism that works on encoder-decoder attentions to pinpoint salient content at inference time. |
| Outcome: | The proposed technique outperforms state-of-the-art models on CNN/DailyMail and New York Times datasets and is data-efficient. |
Verifiable Generation with Subsentence-Level Fine-Grained Citations (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing work mainly targets the generation of sentence-level citations, lacking specificity about which parts of a sentence are backed by the cited sources. |
| Approach: | They propose to use subsentence-level fine-grained citations to generate more precise location of generated content supported by the cited sources. |
| Outcome: | The proposed model improves the accuracy and trustworthiness of large language models by allowing users to trace the information back to its source and verify its correctness. |
Inference Time Style Control for Summarization (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to generate summaries of different styles without training separate models are lacking parallel data and expensive (re)training. |
| Approach: | They propose two methods that can be deployed during summary decoding on any pre-trained Transformer-based summarization model. |
| Outcome: | The proposed methods generate news headlines with various ideological leanings while still informative. |
To See a World in a Spark of Neuron: Disentangling Multi-Task Interference for Training-Free Model Merging (2025.emnlp-main)
Copied to clipboard
Zitao Fang, Guodong Du, Shuyang Yu, Yifei Guo, Yiwei Zhang, Yiyao Cao, Jing Li, Ho-Kin Tang, Sim Kuan Goh
| Challenge: | Existing approaches to model merging ignore the fundamental roles of neurons, connectivity and activation. |
| Approach: | They propose a framework that relies on neuronal mechanisms to mitigate task interference . they decomposed task-specific representations into two complementary subspaces . their results offer new insights into mitigating task interference and improving knowledge fusion . |
| Outcome: | The proposed framework reduces task interference within neurons and improves knowledge fusion. |
SYNC: A Synthetic Long-Context Understanding Benchmark for Controlled Comparisons of Model Capabilities (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing synthetic tasks target narrow skill sets, limiting their ability to comprehensively assess model capabilities. |
| Approach: | They propose a new evaluation suite of synthetic tasks spanning domains including graph understanding and translation that test a wide range of capabilities. |
| Outcome: | The evaluation suite of synthetic tasks spanning domains including graph understanding and translation shows that the tasks perform significantly better on more challenging tasks. |
Instilling Type Knowledge in Language Models via Multi-Task QA (2022.findings-naacl)
Copied to clipboard
| Challenge: | Current methods to learn entity types rely on coarse, noisy labels . current methods rely only on text-to-text pre-training on type-centric questions . |
| Approach: | They propose to instill fine-grained type knowledge in language models by pre-training on type-centric questions. |
| Outcome: | The proposed model achieves state-of-the-art in zero-shot dialog state tracking benchmarks and can accurately infer entity types in Wikipedia articles. |
Multi-View Source Ablation for Faithful Summarization (2023.findings-eacl)
Copied to clipboard
| Challenge: | MuFaSSa is a metric for evaluating faithfulness of abstractive summaries . it uses different strategies to remove information from source document to form multiple ablated views . |
| Approach: | They propose a metric for evaluating faithfulness of abstractive summaries using multiple ablated views. |
| Outcome: | The proposed metric outperforms existing models on summarization tasks and human-annotated faithfulness labels. |
Time-aware Prompting for Text Generation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | a new study investigates the effects of incorporating timestamps into generation systems . textual prompts focus more on non-temporal information and are less sensitive to given timestams . |
| Approach: | They propose a data-to-text generation dataset that includes chronologically ordered revisions of biographical articles from English Wikipedia. |
| Outcome: | The proposed models improve the quality of the data-to-text generation dataset TempWikiBio . the proposed models are more sensitive to time-aware prompts than textual prompts . |