Papers by Yongwei Zhou
UniRPG: Unified Discrete Reasoning over Table and Text as Program Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for question answering using knowledge resources are mixed-of-experts and semantic parsing-based. |
| Approach: | They propose a semantic-parsing-based approach to perform Unified discrete Reasoning over heterogeneous knowledge resources as Program Generation. |
| Outcome: | The proposed approach improves interpretability and scalability over table and text . it achieves promising performance on the TAT-QA dataset without annotation . |
FRAME: Boosting LLMs with A Four-Quadrant Multi-Stage Pretraining Strategy (2025.findings-acl)
Copied to clipboard
Xuemiao Zhang, Feiyu Duan, Xu Liangyu, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, Xunliang Cai
| Challenge: | Multi-stage pretraining methods lack quantitative criteria for data partitioning and instead rely on intuitive heuristics. |
| Approach: | They propose a Four-quadRAnt Multi-stage prEtraining strategy that partitions data into four quadrants to achieve significant loss reductions four times. |
| Outcome: | The proposed strategy achieves 16.8% improvement over random across MMLU and CMMLU for the 3B model. |
QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm (2025.findings-acl)
Copied to clipboard
Qirui Zhou, Shaohui Peng, Weiqiang Xiong, Haixin Chen, Yuanbo Wen, Haochen Li, Ling Li, Qi Guo, Yongwei Zhao, Ke Gao, Ruizhi Chen, Yanjun Wu, Zhao Chen, Yunji Chen
| Challenge: | Existing LLMs cannot comprehend the complex data flow and computation process of the attention operator and utilize low-level primitive to exploit GPU performance. |
| Approach: | They propose an LLM-friendly Thinking Language (LLM-TL) that can decouple the generation of high-level optimization logic and low-level implementation on GPU and enhance LLMs’ understanding of attention operator. |
| Outcome: | The proposed method outshines existing LLMs on A100, RTX8000, and T4 GPUs, achieving a speed-up of up to 35.16. |
RoR: Read-over-Read for Long Document Machine Reading Comprehension (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension are limited to individual chunks due to encoding length constraint. |
| Approach: | They propose a read-over-read method that expands the reading field from chunk to document by predicting regional answers for each chunk. |
| Outcome: | Extensive experiments on QuAC and TriviaQA show that the proposed model performs well for long document reading. |
OPERA: Operation-Pivoted Discrete Reasoning over Text (2022.naacl-main)
Copied to clipboard
Yongwei Zhou, Junwei Bao, Chaoqun Duan, Haipeng Sun, Jiahui Liang, Yifan Wang, Jing Zhao, Youzheng Wu, Xiaodong He, Tiejun Zhao
| Challenge: | Existing methods to predict logical forms ignore the utilization of symbolic operations and lack reasoning ability and interpretability. |
| Approach: | They propose an operation-pivoted discrete reasoning framework that uses symbolic operations as neural modules to facilitate reasoning ability and interpretability. |
| Outcome: | Extensive experiments on DROP and RACENum datasets show the reasoning ability of OPERA. |
Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data (2025.findings-acl)
Copied to clipboard
Xuemiao Zhang, Xu Liangyu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, Xunliang Cai
| Challenge: | Existing methods of uniformly sampling data throughout the pretraining process are suboptimal because they overlook the model's evolving data preferences. |
| Approach: | They propose a Perplexity Difference (PD) based Preference Curriculum learning framework which perceives and uses the data preferred by LLMs as their capabilities improve . they propose PDPC to complete the arrangement of the dataset offline and ensure continuous training without interruption. |
| Outcome: | The proposed framework surpasses baselines on 1.3B and 3B models and achieves an increased average accuracy of over 8.1% across MMLU and CMMLU. |