Papers by Xingwu Sun
TITA: A Two-stage Interaction and Topic-Aware Text Matching Model (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing models for keyword-document matching do not define topic-aware relevance clearly. |
| Approach: | They propose a two-stage interaction and topic-aware text matching model to solve this problem . they propose to combine latent topic of document with deep neural representation to model complex interactions between keyword and document. |
| Outcome: | The proposed model outperforms other well-designed baselines and shows excellent performance in the recommendation system. |
Continuous Speech Tokenizer in Text To Speech (2025.findings-naacl)
Copied to clipboard
| Challenge: | Autoregressive modeling is a common method for processing language sequences and is effective in token prediction. |
| Approach: | They propose a text-to-speech model based on continuous speech tokens and a continuous tokenizer for speech compression. |
| Outcome: | The proposed model has better continuity and higher estimated Mean Opinion Scores (MoS) this is attributed to better information preservation rate across low and high frequencies in the frequency domain. |
Exploring Forgetting in Large Language Model Pre-Training (2025.acl-long)
Copied to clipboard
| Challenge: | Existing research on task-level forgetting in LLMs has focused on pretraining . but, there is limited attention to finer-grained forgetting during training . |
| Approach: | They investigated the existence and measurement of forgetting in pre-training . they examined low-cost, straightforward methods to mitigate forgetting during the pre- training phase . |
| Outcome: | The proposed methods could be used to mitigate forgetting during the pre-training phase and offer insights into the dynamics of forgetting. |
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities (2023.acl-demo)
Copied to clipboard
Zhe Zhao, Yudong Li, Cheng Hou, Jing Zhao, Rong Tian, Weijie Liu, Yiren Chen, Ningyuan Sun, Haoyan Liu, Weiquan Mao, Han Guo, Weigang Gou, Taiqiang Wu, Tao Zhu, Wenhang Shi, Chen Chen, Shan Huang, Sihong Chen, Liqun Liu, Feifei Li, Xiaoshuai Chen, Xingwu Sun, Zhanhui Kang, Xiaoyong Du, Linlin Shen, Kimmo Yan
| Challenge: | Several pre-training models of different modalities are showing a rising trend of homogeneity in their model structures. |
| Approach: | They propose a toolkit that supports pre-training models of different modalities. |
| Outcome: | The proposed toolkit can match the performance of the original implementations on text, vision, and audio benchmarks. |
Language Models “Grok” to Copy (2025.naacl-short)
Copied to clipboard
| Challenge: | We examine the pre-training dynamics of language models, focusing on their ability to copy text from preceding context. |
| Approach: | They propose that Transformer-based language models develop copying abilities similarly to grokking . they argue that the connection between groking and context copying can improve in-context performance. |
| Outcome: | The proposed model development is similar to grokking, but the speed is independent of tokens trained. |
LightVLP: A Lightweight Vision-Language Pre-training via Gated Interactive Masked AutoEncoders (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing vision-language pre-training models use multi-modal encoders to encode image and text, causing noisy training corpora. |
| Approach: | They propose a vision-language pre-training framework with two autoencoders for efficient training . they propose masked tokens and a gated interaction mechanism to cope with noise . |
| Outcome: | The proposed model achieves 2.2% R@1 gains on COCO Text Retrieval and 1.1% on refCOCO+ on six datasets. |
Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval (2021.acl-long)
Copied to clipboard
| Challenge: | Existing retrieval models based on dense representations show better performance than sparse representations. |
| Approach: | They propose a method to mimic the queries to each of the documents by an iterative clustering process and represent the documents using multiple pseudo queries. |
| Outcome: | The proposed model achieves state-of-the-art results on a large dataset while remaining high efficiency. |
Enhanced Language Representation with Label Knowledge for Span Extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extract text spans from plain text do not fully exploit label knowledge. |
| Approach: | They propose a model to integrate label knowledge into text representations by encoding texts and annotations independently and then integrating label knowledge with an elaborate-designed semantics fusion module. |
| Outcome: | The proposed model achieves state-of-the-art performance on four benchmarks and reduces training time and inference time by 76% and 77% on average compared with the existing paradigm. |
Answer-focused and Position-aware Neural Question Generation (D18-1)
Copied to clipboard
| Challenge: | Recent neural network-based approaches generate interrogative words that do not match the answer type. |
| Approach: | They propose an answer-focused and position-aware neural question generation model to address these issues. |
| Outcome: | The proposed model outperforms the baseline and outperformed the state-of-the-art system. |
HMoE: Heterogeneous Mixture of Experts for Language Modeling (2025.emnlp-main)
Copied to clipboard
An Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, Weidong Han, Zhanhui Kang, Di Wang, Naoaki Okazaki, Cheng-zhong Xu
| Challenge: | Mixture of Experts (MoE) models use homogeneous experts with diverse capacities, resulting in a lack of expert specialization and parameter utilization. |
| Approach: | They propose a framework where experts differ in size and possess diverse capacities . they propose HMoE to encourage frequent activation of smaller experts . |
| Outcome: | The proposed framework outperforms homogeneous homogenous MoE models on evaluation benchmarks and achieves lower loss rate with fewer activated parameters. |
Sparsifying Mamba (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing attempts to integrate sparsification with Mamba fail to leverage Mamba's internal structure for fine-grained sparsifying. |
| Approach: | They propose to use Mamba to integrate sparsification into Mamba and propose a flexible and effective mechanism for parameter scalability. |
| Outcome: | The proposed framework can independently achieve parameter scalability and has stronger performance. |
Enhancing Document Ranking with Task-adaptive Training and Segmented Token Recovery Mechanism (2021.emnlp-main)
Copied to clipboard
| Challenge: | Document Retrieval (DR) requires the machine to retrieve and rank documents according to their relevance with the query. |
| Approach: | They propose a ranking model DR-BERT which improves the Document Retrieval task by a task-adaptive training process and a Segmented Token Recovery Mechanism. |
| Outcome: | The proposed ranking model keeps in the top three on the MS MARCO leaderboard since 2020. |
The Security Threat of Compressed Projectors in Large Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Mainstream VLPs have significant security implications, but their security implications have not been thoroughly examined. |
| Approach: | a study evaluates the security of visual language projectors by comparing them to uncompressed projector. |
| Outcome: | The evaluation reveals significant differences in security profiles between compressed and uncompressed projectors. |
An Anchor-based Relative Position Embedding Method for Cross-Modal Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Position Embedding (PE) is essential for transformer to capture the sequence ordering of input tokens. |
| Approach: | They propose a unified position embedding method that bridges the semantic gap between modalities and embeds the anchor-based distance to guide computation of cross-attention. |
| Outcome: | The proposed method obtains new SOTA results on a wide range of benchmarks. |
QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Currently, large vision-language models are limited in their ability to provide correct answers for multimodal tasks . however, they can still provide correct responses for multiple images associated with a single image . a query-agnostic visual attack (QAVA) provides robust adversarial examples that generate incorrect responses to unspecified and unknown questions. |
| Approach: | They propose a query-agnostic visual attack to create adversarial examples that generate incorrect answers to unspecified and unknown questions. |
| Outcome: | The proposed model improves performance on images when the question is unknown compared to known target questions . |
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) are known to hallucinate, which limits their practical applications. |
| Approach: | They propose a method that uses three types of preference pairs to target hallucinations from their diverse forms and causes. |
| Outcome: | The proposed method surpasses most state-of-the-art methods and shows potential for further improvements. |