Papers by Neil Shah
Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modeling (2026.acl-long)
Copied to clipboard
| Challenge: | a strict sum-to-one constraint forces attention sinks on irrelevant tokens, while probability mass disperses as sequence lengths increase. |
| Approach: | They propose a sink-free attention mechanism that achieves ultra-sparsity and improved robustness at longer sequence lengths without the computational overhead of projection methods. |
| Outcome: | The proposed mechanism produces >99 % exact zeros and eliminates attention sinks while maintaining competitive performance on standard and long-context benchmarks. |
Are Message Passing Neural Networks Really Helpful for Knowledge Graph Completion? (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge graphs are far from complete with large portions of triplets missing. |
| Approach: | They propose to use Graph Neural Networks to learn powerful embeddings to improve model performance. |
| Outcome: | The proposed models achieve comparable performance to MLP models, suggesting that MP may not be as crucial as previously thought. |
Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings (2026.acl-long)
Copied to clipboard
Xueying Ding, Xingyue Huang, Mingxuan Ju, Liam Collins, Yozen Liu, Leman Akoglu, Neil Shah, Tong Zhao
| Challenge: | Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, harming performance. |
| Approach: | They propose a method that prepending a single summary token to reduce attention-level compression by partitioning the input into blocks and prepending blocks to subsequent blocks. |
| Outcome: | The proposed method achieves consistent performance gains across 11 retrieval datasets and 30 general embedding benchmarks. |
ParrotTTS: Text-to-speech synthesis exploiting disentangled self-supervised representations (2024.findings-eacl)
Copied to clipboard
| Challenge: | ParrotTTS can train a multi-speaker variant using transcripts from a single speaker in low resource setup and generalizes to languages not seen while training the self-supervised backbone. |
| Approach: | They propose a modular text-to-speech synthesis model that can train a multi-speaker variant using transcripts from a single speaker. |
| Outcome: | The proposed model outperforms state-of-the-art multi-lingual text-to-speech models using only a fraction of paired data as latter. |
MemRec: Collaborative Memory-Augmented Agentic Recommender System (2026.acl-long)
Copied to clipboard
Weixin Chen, Yuhan Zhao, Jingyuan Huang, Zihe Ye, Mingxuan Ju, Tong Zhao, Neil Shah, Li Chen, Yongfeng Zhang
| Challenge: | Existing recommender systems rely on semantic user and item memories to make predictions, but these memories are kept in isolation. |
| Approach: | They propose a framework that architecturally decouples memory management from reasoning to decouple memory management and reasoning from the user and item memories. |
| Outcome: | The proposed framework decouples memory management from reasoning and achieves state-of-the-art performance on four benchmarks. |
Explainability and Hate Speech: Structured Explanations Make Social Media Moderators Faster (2024.acl-short)
Copied to clipboard
Agostina Calabrese, Leonardo Neves, Neil Shah, Maarten Bos, Björn Ross, Mirella Lapata, Francesco Barbieri
| Challenge: | Existing studies have shown that explanations can support content moderators to make faster decisions, but the benefits of such models have not been studied. |
| Approach: | They propose to use structured explanations to support content moderators to make faster decisions by 7.4%. |
| Outcome: | The proposed models lower the speed of real-world moderators by 7.4% compared to generic explanations and are often ignored . previous studies have shown that explanations can support moderator's decision making by detecting violations of policies but the benefits have not been studied . |
The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks (2020.coling-main)
Copied to clipboard
| Challenge: | Contextual embeddings have shown state-of-the-art performance for various tasks such as question answering, sentiment analysis, and textual similarity. |
| Approach: | They propose to integrate transformer-based neural language models into their models to achieve relative improvements of up to 36% on granular tasks. |
| Outcome: | The proposed model outperforms baselines for more granular tasks while outperforming TF-IDF for more complex tasks. |