Papers by Mingxuan Ju
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | a strict sum-to-one constraint forces attention sinks on irrelevant tokens, while probability mass disperses as sequence lengths increase. |
| Approach: | They propose a sink-free attention mechanism that achieves ultra-sparsity and improved robustness at longer sequence lengths without the computational overhead of projection methods. |
| Outcome: | The proposed mechanism produces >99 % exact zeros and eliminates attention sinks while maintaining competitive performance on standard and long-context benchmarks. |
Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings (2026.acl-long)
Copied to clipboard
Xueying Ding, Xingyue Huang, Mingxuan Ju, Liam Collins, Yozen Liu, Leman Akoglu, Neil Shah, Tong Zhao
| Challenge: | Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, harming performance. |
| Approach: | They propose a method that prepending a single summary token to reduce attention-level compression by partitioning the input into blocks and prepending blocks to subsequent blocks. |
| Outcome: | The proposed method achieves consistent performance gains across 11 retrieval datasets and 30 general embedding benchmarks. |
Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Open-domain question answering (QA) models employ a retriever-reader pipeline . however, state-of-the-art readers fail to capture complex relationships between entities . |
| Approach: | They propose a knowledge graph enhanced passage reader that captures entities in questions and retrieved passages. |
| Outcome: | The proposed knowledge graph enhanced passage reader improves on open-domain QA benchmarks by up to 2.2 exact match scores. |
Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing studies on the ability of a model to make consistently correct predictions in the presence of perturbations have not been conducted in open-domain question answering (OpenQA). |
| Approach: | They propose a query-side contrastive loss to improve the dense passage retriever (DPR) to improve DPR training. |
| Outcome: | The proposed approach improves the density of the dense passage retriever (DPR) training set without sacrificing accuracy on standard test sets. |
MemRec: Collaborative Memory-Augmented Agentic Recommender System (2026.acl-long)
Copied to clipboard
Weixin Chen, Yuhan Zhao, Jingyuan Huang, Zihe Ye, Mingxuan Ju, Tong Zhao, Neil Shah, Li Chen, Yongfeng Zhang
| Challenge: | Existing recommender systems rely on semantic user and item memories to make predictions, but these memories are kept in isolation. |
| Approach: | They propose a framework that architecturally decouples memory management from reasoning to decouple memory management and reasoning from the user and item memories. |
| Outcome: | The proposed framework decouples memory management from reasoning and achieves state-of-the-art performance on four benchmarks. |