Papers by Hongyuan Mei
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)
Copied to clipboard
| Challenge: | Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks . |
| Approach: | They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost. |
| Outcome: | The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states . |
Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to transfer a pretrained language model include fine-tuning all the parameters in the language model and adapting all its subsets. |
| Approach: | They propose to select layers based on the variability of their hidden states given a task-specific corpus. |
| Outcome: | The proposed model reduces the computational cost of transfer learning methods without sacrificing performance. |
Robustness of Learning from Task Instructions (2023.findings-acl)
Copied to clipboard
| Challenge: | traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples. |
| Approach: | a new study investigates the system robustness when instructions are manipulated and paraphrased . task instructions give the model the definition of the task and allow it to output the appropriate answer . |
| Outcome: | a new study shows that supervised learning is robust when instructions are manipulated, paraphrased or iii from different levels of conciseness. |
On the Idiosyncrasies of the Mandarin Chinese Classifier System (N19-1)
Copied to clipboard
| Challenge: | idiosyncrasies of the Chinese classifier system have been studied, but little work has been done to quantify them with statistical methods. |
| Approach: | They propose an information-theoretic approach to measuring idiosyncrasies in Mandarin Chinese by calculating the mutual information between the distribution over classifiers and distributions over other linguistic quantities. |
| Outcome: | The proposed method reduces uncertainty in Mandarin Chinese classifiers by knowing semantic information about nouns that they modify. |
Explicit Planning Helps Language Models in Logical Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems that use pre-trained large language models to perform multi-step logical reasoning have been unable to perform this task. |
| Approach: | They propose a system that uses language models to perform multi-step logical reasoning and incorporates explicit planning into the inference procedure. |
| Outcome: | The proposed system outperforms other competing methods on multiple datasets and significantly outperformed chain-of-thought prompting on the PrOntoQA dataset. |
CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions (2025.findings-naacl)
Copied to clipboard
| Challenge: | CaseSumm is a dataset for long-context summarization in the legal domain . human groundtruth summaries are often not available for legal summarizing . |
| Approach: | They propose a dataset for long-context summarization that includes SCOTUS opinions and their official summaries. |
| Outcome: | The proposed dataset is the largest open legal case summarization dataset . it outperforms larger models on automatic metrics and human evaluation . |
Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) outperform existing benchmarks in both natural language and coding domains. |
| Approach: | They propose a scalable benchmark that integrates vision and language modalities to address this gap by eliminating textual shortcuts. |
| Outcome: | The new benchmark outperforms existing benchmarks in both natural language and coding domains. |