Papers by Meng Tong
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments (2025.acl-long)
Copied to clipboard
| Challenge: | Quantization has shown promise for Large Language Models, but current methods require lengthy training to alleviate quantization loss. |
| Approach: | They propose to decouple weights and incorporate Low-Rank adapters to reduce weight sharing . they validate the approach on LLaMA2 families and Mistral on downstream evaluation . |
| Outcome: | The proposed approach shows high performance while reducing deployment time faced with multiple scenarios. |
Cross-layer Attention Sharing for Pre-trained Large Language Models (2026.tacl-1)
Copied to clipboard
Yongyu Mu, Yuzhang Wu, Yuchun Fan, Chenglong Wang, Hengyu Li, Jiali Zeng, Qiaozhi He, Murun Yang, Fandong Meng, Jie Zhou, Tong Xiao, Jingbo Zhu
| Challenge: | Existing studies focus on compressing the Key-Value cache or grouping attention heads, while overlooking redundancy between layers. |
| Approach: | They propose a lightweight substitute for self-attention in well-trained LLMs that uses feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. |
| Outcome: | The proposed model reduces redundancy by sharing weights across layers while maintaining high response quality while reducing redundant calculations within 53% 84% of the total layers. |
Modeling Localness for Self-Attention Networks (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to model locality for self-attention networks have shown great value for capturing global dependencies. |
| Approach: | They propose to model localness for self-attention networks to capture local context . they cast localness modeling as a learnable Gaussian bias, which indicates the central and scope of the local region to be paid more attention. |
| Outcome: | The proposed model improves the ability to capture local context and improves accuracy. |
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent years have seen a surge of interest in improving the generation quality of commonsense reasoning tasks. |
| Approach: | They propose a method that diversifies the generative reasoning by a mixture of expert strategy on commonsense knowledge graphs to encourage various generation outputs. |
| Outcome: | The proposed method improves diversity while achieving on par performance on two GCR benchmarks, based on both automatic and human evaluations. |
Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders (2025.emnlp-main)
Copied to clipboard
Weiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo, Chen Xu, Xiaofeng Zhao, Long Meng, Yunfei Lu, Min Zhang, Hao Yang, Tong Xiao, JingBo Zhu
| Challenge: | Existing work on integrating audio encoders with large language models (LLMs) has focused on semantic understanding tasks, but different tasks may require distinct features that emphasize either semantic or acoustic aspects. |
| Approach: | They propose to use a prompt-aware mixture to enhance the Speech LLM that uses multiple audio encoders to extract different features based on the prompt. |
| Outcome: | The proposed approach outperforms all single-encoder Speech LLMs on ASR, speaker number verification, and AC tasks. |
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained masked language models can be factual knowledge bases. |
| Approach: | They conduct a rigorous study to explore the underlying predicting mechanisms of MLMs . they find that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts a . |
| Outcome: | The proposed model improves on illustrative cases and external contexts . the results question the previous findings that MLMs can be reliable factual knowledge bases . |
GUIDE: Towards Scalable Advising for Research Ideas (2026.acl-long)
Copied to clipboard
| Challenge: | Existing systems that provide detailed, constructive feedback on academic papers struggle with review fidelity. |
| Approach: | They explore factors that underlie the development of robust advising systems . large language models have shown remarkable progress in tasks from text generation to code synthesis . |
| Outcome: | The proposed model outperforms general-purpose language models in acceptance rates for self-ranked top-30% submissions to ICLR 2025. |
An Empirical Study on Neural Keyphrase Generation (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent years have seen a flourishing of neural keyphrase generation (KPG) works, including the release of several large-scale datasets and a host of new models to tackle them. |
| Approach: | They propose to compare the generalizability of KPG models with other models by analyzing the most crucial factors that may affect their generalizarability. |
| Outcome: | The proposed model can be used to predict keyphrases from a set of input sequences, and it can be compared with existing models. |
Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation (2024.emnlp-main)
Copied to clipboard
Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Mahong Xia, Zhang Li, Boxing Chen, Hao Yang, Bei Li, Tong Xiao, JingBo Zhu
| Challenge: | Existing methods for instruction data selection have limitations such as relying on fragile external APIs, being affected by biases in GPT models, or reducing the diversity of the selected instruction dataset. |
| Approach: | They propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR). |
| Outcome: | The proposed method outperforms Alpaca's existing methods by 32.1% in GPT-4 evaluations. |
RankNAS: Efficient Neural Architecture Search by Pairwise Ranking (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods require training millions of architectures to estimate the accuracy of the search results. |
| Approach: | They propose a performance ranking method (RankNAS) that uses pairwise ranking and search space pruning to enlarge the search space. |
| Outcome: | The proposed method significantly accelerates NAS through pairwise ranking and search space pruning. |
Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues (2022.acl-long)
Copied to clipboard
Qingxiu Dong, Ziwei Qin, Heming Xia, Tian Feng, Shoujie Tong, Haoran Meng, Lin Xu, Zhongyu Wei, Weidong Zhan, Baobao Chang, Sujian Li, Tianyu Liu, Zhifang Sui
| Challenge: | Existing work in vision language cross-modal reasoning uses binary or multi-choice classification based on source image and textual query. |
| Approach: | They propose a task where a textual premise is the background presumption on each source image. |
| Outcome: | The proposed task is based on a dataset of 15,360 movie screenshots and human-curated premise templates from 6 pre-defined categories. |
Collaborative Policy Learning for Open Knowledge Graph Reasoning (D19-1)
Copied to clipboard
| Challenge: | Existing models of knowledge graph reasoning suffer from limited performance when working on sparse and incomplete graphs due to the lack of evidential paths that can reach target entities. |
| Approach: | They propose a framework to train two collaborative agents to reason for missing facts over a graph augmented by a text corpus. |
| Outcome: | Experiments on two public datasets show the proposed approach is effective on a knowledge graph reasoning task. |
Sentence-Permuted Paragraph Generation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing models produce similar contents from homogenized contexts due to the fixed left-to-right sentence order. |
| Approach: | They propose a framework permuting sentence orders to improve content diversity of multi-sentence paragraphs by permutating the sentence orders. |
| Outcome: | The proposed framework produces more diverse outputs with higher quality than existing models. |
On the Vulnerability of Text Sanitization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing reconstruction attacks on text sanitization are not able to accurately assess their effectiveness. |
| Approach: | They propose to use ASR to measure the effectiveness of reconstruction attacks to evaluate sanitization performance. |
| Outcome: | The proposed reconstruction attacks achieve a 46.4% improvement in ASR over the state-of-the-art baseline with a privacy budget of =4.0 on the SST-2 dataset. |
Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documents (2021.acl-short)
Copied to clipboard
| Challenge: | Faceted summarization provides briefings of a document from different perspectives. |
| Approach: | They propose a faceted summarization benchmark built on Emerald journal articles . they propose faceted models that bring structure into faceted documents . |
| Outcome: | The proposed benchmark is based on Emerald journal articles and covers a diverse range of domains. |
Multi-Input Multi-Output Sequence Labeling for Joint Extraction of Fact and Condition Tuples from Scientific Text (D19-1)
Copied to clipboard
| Challenge: | Existing methods to extract factual tuples from scientific text do not consider conditions. |
| Approach: | They propose a new sequence labeling framework to jointly extract fact and condition tuples from scientific sentences. |
| Outcome: | The proposed framework improves F1 score relative to existing methods by 4.2% and 6.2% on bioNLP2013. |
Unlocking the Effectiveness of LoRA-FP for Seamless Transfer Implantation of Fingerprints in Downstream Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | lightweight plug-and-play framework that encodes backdoor fingerprints into LoRA adapters . |
| Approach: | proposed framework encodes backdoor fingerprints into LoRA adapters via constrained fine-tuning . enables seamless fingerprint transplantation through parameter fusion, eliminating full-parameter updates while maintaining integrity. |
| Outcome: | The proposed framework achieves superior robustness against various scenarios while reducing computational overhead compared to traditional approaches. |
One Size Does Not Fit All: Generating and Evaluating Variable Number of Keyphrases (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for keyphrase generation do not provide a desideratum for the number of keyphrases in texts. |
| Approach: | They propose a recurrent generative model that generates multiple keyphrases as delimiter-separated sequences. |
| Outcome: | The proposed model outperforms baseline models on all datasets. |
Competence-based Curriculum Learning for Multilingual Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual machine translation models face an imbalance problem due to the different learning competencies of different languages. |
| Approach: | They propose Competence-based Curriculum Learning for Multilingual Machine Translation, named CCL-M, to help schedule the high resource languages and low resource languages. |
| Outcome: | The proposed approach achieves a steady and significant performance gain compared to the previous state-of-the-art approach on the TED talks dataset. |
Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on keyphrase generation on non-English languages haven’t been vastly investigated. |
| Approach: | They propose a retrieval-augmented method for multilingual keyphrase generation that leverages keyphrase annotations in English datasets to facilitate generating keyphrases in low-resource languages. |
| Outcome: | The proposed model outperforms baselines on non-English keyphrase generation datasets and the proposed model is scalable. |
General-to-Specific Transfer Labeling for Domain Adaptable Keyphrase Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Large distribution shifts among different domains hinder transferability of keyphrase generation models. |
| Approach: | They propose a pipeline which guides KPG models’ learning focus from general syntactical features to domain-related semantics in a data-efficient manner. |
| Outcome: | The proposed pipeline can produce good quality keyphrases in new domains and achieve consistent improvements after adaptation with limited in-domain annotated data. |