Papers by Fangyu Lei
Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for numerical reasoning are not flexible enough to handle diverse expressions. |
| Approach: | They propose a Relational Graph enhanced Hybrid table-text Numerical reasoning model with Tree decoder which captures relationship between numerical value, table schema, and text information on the encoder side. |
| Outcome: | The proposed model outperforms the baseline model and achieves state-of-the-art results on the publicly available tabletext hybrid QA benchmark. |
MenatQA: A New Dataset for Testing the Temporal Comprehension and Reasoning Abilities of Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown nearly saturated performance on many NLP tasks. |
| Approach: | They construct multiple sensitive factors time QA which encompasses three temporal factors . they test current mainstream LLMs with different parameter sizes . |
| Outcome: | The proposed model incorporates three temporal factors with 2,853 samples . the results show that LLMs fall behind smaller models on these factors . |
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to evaluate extremely long-context LLMs or analyze their limitations. |
| Approach: | They propose a Synthetic, Scalable, Systematic evaluation suite for LLMs using SQL execution. |
| Outcome: | The proposed evaluation suite is able to scale text length and difficulty across scenarios and provides strong correlations with real-world benchmarks. |
Teaching Small Language Models to Reason for Knowledge-Intensive Multi-Hop Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can teach small language models to solve complex reasoning tasks by Chain-of-thought Distillation (CoTD) e.g., mathematical question answering. |
| Approach: | They propose a method that distills two student models to solve a multi-hop question . they use chain-of-thought distillation to generate step-by-step reasoning paths . |
| Outcome: | The proposed method surpasses existing methods on knowledge-intensive multi-hop questions. |
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained MLMs produce an anisotropic distribution of token representations . this is not ideal for tasks that require discriminative semantic meanings of distinct tokens - a problem that exists in pre-training models . |
| Approach: | They propose a continual pre-training approach that encourages BERT to learn an isotropic distribution of token representations. |
| Outcome: | The proposed approach improves on a wide range of English and Chinese benchmarks. |
GATE: Graph-based Adaptive Tool Evolution Across Diverse Tasks (2026.acl-long)
Copied to clipboard
Jianwen Luo, Yiming Huang, Jinxiang Meng, Fangyu Lei, Shizhu He, Xiao Liu, Shanshan Jiang, Bin Dong, Jun Zhao, Kang Liu
| Challenge: | Existing toolsets that use large language models are limited to single-task settings. |
| Approach: | They propose a framework that dynamically constructs and evolves a hierarchical graph of reusable tools across multiple scenarios. |
| Outcome: | The proposed framework achieves up to 4.3 faster milestone completion in Minecraft compared to the previous state-of-the-art method and provides an average improvement of 9.23% over existing tool-making methods in code generation tasks and 10.03% in agent tasks. |
S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering (2023.acl-short)
Copied to clipboard
| Challenge: | Existing question answering systems use a retriever-reader framework to answer multi-hop questions . existing models lack retrieval, selector, and reasoner capabilities . |
| Approach: | They propose a three-stage text tableQA framework which comprises of retriever, selector, and reasoner. |
| Outcome: | The proposed framework outperforms baseline methods in the few-shot setting and ranks first on the HybridQA leaderboard. |
DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models (2024.emnlp-main)
Copied to clipboard
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, Kang Liu
| Challenge: | DA-Code is a code generation benchmark designed to assess LLMs on agent-based data science tasks. |
| Approach: | They propose a code generation benchmark specifically designed for LLMs on agent-based data science tasks. |
| Outcome: | The benchmark performs better than existing frameworks, but lacks accuracy . it is based on real-world data, and includes examples that cover a wide range of tasks . |
Neeko: Leveraging Dynamic LoRA for Efficient Multi-Character Role-Playing Agent (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have revolutionized open-domain dialogue agents but face challenges in multi-character role-playing (MCRP) scenarios. |
| Approach: | They propose a framework for efficient multi-character role-playing that employs a dynamic low-rank adapter strategy and distinct LoRA blocks for each character. |
| Outcome: | Neeko employs a dynamic low-rank adapter (LoRA) strategy, enabling it to adapt seamlessly to diverse characters. |
Competition-Level Problems are Effective LLM Evaluators (2024.findings-acl)
Copied to clipboard
Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen
| Challenge: | Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about their capabilities and the potential data contamination problem. |
| Approach: | They propose to evaluate the reasoning capabilities of large language models in solving recent competition-level programming problems in Codeforces. |
| Outcome: | The proposed model has experienced a cliff-like decline in problems after September 2021, which shows the potential data contamination and the challenges for any existing LLM to solve unseen complex reasoning problems. |