Papers by Weizhu Chen

HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability (2021.acl-long)

Challenge:	Large language models have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning, but their strictly sequential nature constrains test-time scalability.
Approach:	They propose an end-to-end reinforcement learning framework to enhance LLMs' DAC-style reasoning capacity by decomposing a problem into subproblems and solving them sequentially.
Outcome:	The proposed model surpasses CoT by 8.6% and 6.3% on competition-level benchmarks and is available at the [github.com/MasterVito/DAC-RL].

Copied to clipboard

Jiaao Chen, Dinghan Shen, Weizhu Chen, Diyi Yang

Responsible & Trustworthy NLP

Challenge:	Using data augmentation to fine-tune pre-trained models with task-specific data has been shown to be ineffective and redundant during fine-timing.
Approach:	They propose a data augmentation technique to regularize pre-trained models and encourage them to learn more generalizable features by dropping contiguous spans during training.
Outcome:	The proposed method outperforms state-of-the-art methods on the GLUE benchmark and consistently exhibits superior generalization performances on out-of distribution and challenging counterexamples.

Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach (2021.emnlp-main)

Copied to clipboard

Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, Tuo Zhao

Robustness in NLP

Challenge:	Existing approaches to adversarial regularization treat adversarials and defending players equally, which is undesirable because only the defending player contributes to the generalization performance.
Approach:	They propose a method which formulates adversarial regularization as a Stackelberg game and induces a competition between a leader and a follower.
Outcome:	The proposed method outperforms existing adversarial regularization baselines on a set of machine translation and natural language understanding tasks.

The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020.acl-demos)

Copied to clipboard

Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao

Language Models can be Deductive Solvers (2024.findings-naacl)

Challenge:	MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models .
Approach:	They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop.
Outcome:	The proposed model can significantly compress a large model without significant performance drop.

Copied to clipboard

Jiazhan Feng, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, Weizhu Chen

Making Language Models Better Reasoners with Step-Aware Verifier (2023.acl-long)

Challenge:	Recent advances have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge.
Approach:	They propose a novel language model that internalizes and emulates the reasoning processes of logical solvers and avoids parsing errors by learning strict adherence to solver syntax and grammar.
Outcome:	The proposed model outperforms state-of-the-art solver-augmented language models and few-shot prompting methods on public deductive reasoning benchmarks.

Copied to clipboard

Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen

Few-Shot Named Entity Recognition: An Empirical Baseline Study (2021.emnlp-main)

Challenge:	Large language models have made impressive progress in few-shot learning but still face difficulties in reasoning tasks such as GSM8K.
Approach:	They propose a new approach that uses a verifier to filter out incorrect answers based on a weighted voting scheme to improve reasoning ability of language models.
Outcome:	The proposed approach improves GSM8K reasoning rate by 17.9% to 58.1%.

Copied to clipboard

Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, Jiawei Han

Reasoning Like Program Executors (2022.emnlp-main)

Challenge:	Existing methods to build named entity recognition systems with limited labeled data are lacking.
Approach:	They propose three orthogonal schemes to build named entity recognition systems when labeled data is limited.
Outcome:	The proposed NER systems outperform existing methods on few-shot and training-free settings.

Copied to clipboard

Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, Weizhu Chen

Can LLMs Learn From Mistakes? An Empirical Study on Reasoning Tasks (2024.findings-emnlp)

Challenge:	Existing language models are inadequate in reasoning, according to studies . a new reasoning pre-training paradigm is based on pretraining language models with programs .
Approach:	They propose a reasoning pre-training paradigm that empowers language models to harvest reasoning knowledge possessed by program executors.
Outcome:	The proposed reasoning pre-training paradigm can boost models' reasoning skills . it can be instantiated by different kinds of program executors and run on a single database .

Copied to clipboard

Shengnan An, Zexiong Ma, Siqi Cai, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, Weizhu Chen

Token-wise Curriculum Learning for Neural Machine Translation (2021.findings-emnlp)

Challenge:	Existing work has shown that simple learning can enhance the chain-of-thought (CoT) reasoning of large language models.
Approach:	They construct mistake-correction datasets to identify and correct mistakes in CoTs . they conclude that LLMs can learn from mistakes to enhance their CoT reasoning .
Outcome:	The proposed datasets show that LLMs can learn from mistakes to enhance their CoT reasoning performance.

Copied to clipboard

Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Tuo Zhao

Machine Translation

Challenge:	Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of “easy” samples from training data at the early stage of training.
Approach:	They propose a token-wise curriculum learning approach that creates sufficient amounts of easy samples from training data.
Outcome:	The proposed approach outperforms baselines on five language pairs on low-resource languages.

Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization (2021.acl-long)

Copied to clipboard

Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, Weizhu Chen

Parameter-free Sentence Embedding via Orthogonal Basis (D19-1)

Challenge:	'lottery tickets' can be trained to match the performance of a full model . subnetwork training can also outperform random sampled subnetworks of the same size .
Approach:	They propose to train a subnetwork of 'lottery tickets' to match the full model's performance.
Outcome:	The proposed model outperforms subnetworks of the same size in a phase transition phenomenon . the proposed model improves single task fine-tuning by 0.9 points on BERT-base and 1.0 points on GLUE large .

Copied to clipboard

Ziyi Yang, Chenguang Zhu, Weizhu Chen

Representation Learning

Challenge:	Existing methods to build sentence embeddings are parameterized and require training to optimize their parameters.
Approach:	They propose a non-parameterized method to combine pre-trained word embeddings into sentence representations using an orthogonal basis of the word vector subspace and its surrounding context.
Outcome:	The proposed method shows superior performance on 11 downstream NLP tasks and is competitive to other methods relying on large amounts of labelled data or prolonged training time.

Code Execution with Pre-trained Language Models (2023.findings-acl)

Copied to clipboard

Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, Nan Duan

Joint Generator-Ranker Learning for Natural Language Generation (2023.findings-acl)

Challenge:	Pre-trained code intelligence models ignore the execution trace and only rely on source code and syntactic structures to understand code execution.
Approach:	They develop a mutation-based data augmentation technique to create a Python dataset and task for code execution that challenges existing models.
Outcome:	The proposed model outperforms existing models on code execution and shows its potential for zero-shot code-to-code search and text-to code generation.

Copied to clipboard

Weizhou Shen, Yeyun Gong, Yelong Shen, Song Wang, Xiaojun Quan, Nan Duan, Weizhu Chen

Text Generation

Challenge:	Existing methods for text generation train the generator and ranker individually . existing methods neglect the mutual feedback that could enhance the quality of outputs .
Approach:	They propose a joint training algorithm that integrates the generator and ranker in a single framework.
Outcome:	The proposed algorithm surpasses existing methods on four public datasets across three common generation scenarios.

ARCH: Efficient Adversarial Regularized Training with Caching (2021.findings-emnlp)

Copied to clipboard

Simiao Zuo, Chen Liang, Haoming Jiang, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao

Responsible & Trustworthy NLP

Challenge:	Existing approaches to regularize models require generating a perturbation for each sample in each epoch.
Approach:	They propose an adversarial regularization method where perturbations are generated and cached once every several epochs.
Outcome:	The proposed method significantly eases the computational burden (saves up to 70% of computational time) it produces a notably better (in most of the tasks) or comparable model generalization.

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022.acl-long)

Copied to clipboard

Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren

Finding the Dominant Winning Ticket in Pre-Trained Language Models (2022.findings-acl)

Challenge:	Recent few-shot learning models such as GPT3 are expensive and slow to deploy for real-world applications.
Approach:	They propose a prompt-based low-resource learning method for VL tasks with a few examples . they pre-train a sequence-to-sequence transformer model with prefix and masked language modeling .
Outcome:	The proposed method outperforms Frozen on vision-language tasks with prompt-based learning by 18.2% point.

Copied to clipboard

Zhuocheng Gong, Di He, Yelong Shen, Tie-Yan Liu, Weizhu Chen, Dongyan Zhao, Ji-Rong Wen, Rui Yan

Memory-Efficient Differentiable Transformer Architecture Search (2021.findings-acl)

Challenge:	Existing studies on pre-trained language models show that they can fine-tune parameters but achieve good downstream performance.
Approach:	They find that a dominant winning ticket takes up 0.05% of the parameters and is transferable across different tasks.
Outcome:	The proposed model can achieve comparable performance with the full-parameter model, the authors show . the dominant winning ticket takes up 0.05% of the parameters, and the model is transferable across tasks, they show - the authors conclude .

Copied to clipboard

Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Weizhu Chen

Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020.emnlp-main)

Challenge:	Current neural architecture search methods suffer from huge computational cost.
Approach:	They propose a reversible recursive backpropagation algorithm that uses the last layer to store the outputs of the network.
Outcome:	The proposed algorithm outperforms standard Transformers on three sequence-to-sequence datasets.

Copied to clipboard

Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, Weizhu Chen

Controllable Natural Language Generation with Contrastive Prefixes (2022.findings-acl)

Challenge:	Existing methods for integrating knowledge graphs into pre-trained language models have been poorly implemented.
Approach:	They propose a self-supervised entity masking scheme that exploits relational knowledge underlying the text.
Outcome:	The proposed model achieves improved performance on five benchmarks, including question answering and knowledge base completion.

Copied to clipboard

Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen

ALLSH: Active Learning Guided by Local Sensitivity and Hardness (2022.findings-naacl)

Challenge:	Existing work on controllable natural language generation has focused on fine-tuning existing models or using attribute discriminators.
Approach:	They propose a lightweight framework for controllable GPT2 generation that utilizes attribute-specific vectors to steer natural language generation.
Outcome:	The proposed framework can guide generation towards desired attributes while keeping high linguistic quality.

Copied to clipboard

Shujian Zhang, Chengyue Gong, Xingchao Liu, Pengcheng He, Weizhu Chen, Mingyuan Zhou

Low-Resource NLP

Challenge:	Existing studies show that labeling in crowdsourcing annotations is not an annotation artifact but rather a core linguistic phenomenon.
Approach:	They propose to retrieve unlabeled data with a local sensitivity and hardness-aware acquisition function.
Outcome:	The proposed method achieves consistent gains over the commonly used active learning strategies in various classification tasks.

GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)

Copied to clipboard

Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan

UnitedQA: A Hybrid Approach for Open Domain Question Answering (2021.acl-long)

Challenge:	Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models.
Approach:	They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks.
Outcome:	The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages.

Copied to clipboard

Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

Natural Language Interfaces

Challenge:	Recent work on open-domain question answering focuses on either extractive or generative readers exclusively.
Approach:	They propose a hybrid approach to extractive and generative readers that leverages both models.
Outcome:	The proposed approach outperforms state-of-the-art models on NaturalQuestions and TriviaQA respectively.

Skill-Based Few-Shot Selection for In-Context Learning (2023.emnlp-main)

Copied to clipboard

Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, Jian-Guang Lou

Finetuning Pretrained Transformers into RNNs (2021.emnlp-main)

Challenge:	Existing methods based on pre-trained embeddings can be easily biased by surface features that are not important for the target task.
Approach:	They propose a skill-based few-shot selection method for in-context learning . it generates skill-specific descriptions for each test case and candidate example .
Outcome:	The proposed method significantly outperforms existing methods in five cross-domain semantic parsing datasets and six backbone models.

Copied to clipboard

Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen, Noah A. Smith

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy (2023.findings-emnlp)

Challenge:	Efficient transformers outperform recurrent neural networks in natural language generation, but this comes with significant computational cost and memory footprint during generation.
Approach:	They propose to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy.
Outcome:	The proposed transformers outperform recurrent neural networks in natural language generation but come with significant computational and memory footprint during generation.

Copied to clipboard

Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, Weizhu Chen

A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2022.acl-long)

Challenge:	Recent work has proposed to improve relevance modeling by having large language models actively involved in retrieval, i.e., to guide retrieval with generation.
Approach:	They propose to have large language models actively involved in retrieval to guide retrieval with generation.
Outcome:	The proposed method synergizes retrieval and generation in an iterative manner, and can generate better results in subsequent iterations.

Copied to clipboard

Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, Bill Dolan

DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models (2023.acl-long)

Challenge:	Existing work on pre-trained generative models often fails to detect non-existent or incorrect content . Existing studies have attempted to detect hallucinations based on oracle references .
Approach:	They propose a token-level, reference-free hallucination detection task based on Wikipedia annotations to detect non-existent or incorrect content.
Outcome:	The proposed task is token-level, reference-free hallucination detection task and dataset . authors argue that the proposed task can be used in real-time to detect hallucines .

Copied to clipboard

Xuxi Chen, Tianlong Chen, Weizhu Chen, Ahmed Hassan Awadallah, Zhangyang Wang, Yu Cheng

Generation-Augmented Retrieval for Open-Domain Question Answering (2021.acl-long)

Challenge:	Conventional fine-tuning works through updating all of the parameters in the pre-trained model, but as the size of pre-train models grows, it can be time-consuming and computationally expensive.
Approach:	They propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights.
Outcome:	The proposed framework saves 25% inference FLOPs while maintaining competitive downstream performance.

Copied to clipboard

Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, Weizhu Chen

Natural Language Interfaces

Challenge:	Existing approaches to answer open-domain questions use sparse representations and sparsity.
Approach:	They propose a method which augments a query by generating relevant contexts from heuristically discovered contexts without external supervision.
Outcome:	The proposed approach outperforms state-of-the-art dense retrieval methods on natural questions and triviaQA datasets.

A Hybrid Neural Network Model for Commonsense Reasoning (D19-60)

Copied to clipboard

Pengcheng He, Xiaodong Liu, Weizhu Chen, Jianfeng Gao

MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation (2022.naacl-main)

Challenge:	a hybrid neural network (HNN) model for commonsense reasoning is proposed . it combines language models and semantic similarity models to achieve new state-of-the-art results .
Approach:	They propose a hybrid neural network model for commonsense reasoning . it combines a masked language model and a semantic similarity model .
Outcome:	The proposed model outperforms the WNLI, WSC and PDP60 benchmarks on three commonsense reasoning tasks.

Copied to clipboard

Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, Weizhu Chen

OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering (2022.naacl-main)

Challenge:	Existing methods for training pre-trained language models have limited practicality due to latency requirements.
Approach:	They propose a method that uses a Mixture-of-Experts structure to increase model capacity and inference speed.
Outcome:	The proposed method outperforms existing distillation methods on natural language understanding and question answering tasks.

Copied to clipboard

Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, Weizhu Chen

Natural Language Interfaces

Challenge:	a table-based question answering system requires complex reasoning and alignment between questions and tables.
Approach:	They propose a table-based QA model that consumes both natural and synthetic data . they combine retrieval with masking to pair natural sentences with QA .
Outcome:	The proposed model outperforms existing models in few-shot and full settings and on WikiTableQuestions.

Gold-Medal-Level Olympiad Geometry Solving with Efficient Heuristic Auxiliary Constructions (2026.acl-long)

Copied to clipboard

Boyan Duan, Xiao Liang, Shuai Lu, Yaoxiang Wang, Yelong Shen, Kai-Wei Chang, Ying Nian Wu, Mao Yang, Weizhu Chen, Yeyun Gong

Green & Sustainable NLP

Challenge:	Existing methods for geometry theorem proving in Euclidean geometry are challenging and require a neural network to perform.
Approach:	They propose a method for adding auxiliary points in geometry that runs on CPUs without relying on neural network-based inference.
Outcome:	The proposed method achieves silver-medal-level human performance on IMO-30 benchmark.

Learning to Attend On Essential Terms: An Enhanced Retriever-Reader Model for Open-domain Question Answering (N19-1)

Copied to clipboard

Jianmo Ni, Chenguang Zhu, Weizhu Chen, Julian McAuley

Question Answering

Challenge:	Existing approaches to open-domain question answering struggle to retrieve indirectly related evidence when no direct evidence is provided.
Approach:	They propose a retriever-reader model that learns to attend on essential terms during the question answering process.
Outcome:	The proposed model achieves the state-of-the-art on multiple open-domain QA datasets and achieves a 'reader-reader' level.

Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)

Copied to clipboard

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)

Challenge:	Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training.
Approach:	They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model.
Outcome:	The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement)

Copied to clipboard

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan

CodeRetriever: A Large Scale Contrastive Pre-Training Method for Code Search (2022.emnlp-main)

Challenge:	Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks.
Approach:	They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions.
Outcome:	The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam.

Copied to clipboard

Xiaonan Li, Yeyun Gong, Yelong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, Nan Duan

Soft-Labeled Contrastive Pre-Training for Function-Level Code Representation (2022.findings-emnlp)

Challenge:	Existing code pre-training approaches often adopt (masked) language modeling as the training objective which targets on learning to predict (macked) tokens in a given code context.
Approach:	They propose a code-text contrastive learning model which learns function-level code semantic representations through large-scale code corpus.
Outcome:	The proposed model achieves new state-of-the-art with significant improvement over existing pre-trained models on eleven domain/language-specific code search tasks with six programming languages in different code granularity.

Copied to clipboard

Xiaonan Li, Daya Guo, Yeyun Gong, Yun Lin, Yelong Shen, Xipeng Qiu, Daxin Jiang, Weizhu Chen, Nan Duan

Automatic Instruction Evolving for Large Language Models (2024.emnlp-main)

Challenge:	Existing methods for contrastive pre-training ignore the relevance between codes in large code corpus.
Approach:	They propose a Soft-labeled contrastive pre-training framework with positive sample construction methods to learn functional-level code representation.
Outcome:	The proposed framework can obtain fine-grained soft-labels through an iterative adversarial manner and use them to learn better code representation.

Copied to clipboard

Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, Weizhu Chen

CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing (2022.acl-long)

Challenge:	Evol-Instruct is an end-to-end framework that evolves instruction datasets without human effort.
Approach:	They propose an end-to-end framework that evolves instruction datasets without human effort by analyzing and analyzing evolutionary strategies for the given instruction data.
Outcome:	The proposed method outperforms human-designed methods on various benchmarks including MT-Bench, AlpacaEval, GSM8K, and HumanEval.

Copied to clipboard

Chen Liang, Pengcheng He, Yelong Shen, Weizhu Chen, Tuo Zhao

RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023.emnlp-main)

Challenge:	Existing work has resorted to sharing weights among models, but results are not affordable for real-world deployment.
Approach:	They propose a consistency-regularized ensemble learning approach based on perturbed models to retain ensemble benefits while maintaining a low memory cost.
Outcome:	The proposed approach outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0.7 with a significantly smaller model size.

Copied to clipboard

Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, Weizhu Chen

Text Generation

Challenge:	RepoCoder is a repository-level code completion framework that utilizes the useful information scattered in files.
Approach:	They propose a repository-level code completion framework called RepoCoder . it integrates a similarity-based retriever and a pre-trained code language model . they propose 'repoBench' benchmark to validate the framework's effectiveness .
Outcome:	The proposed framework outperforms the vanilla retrieval-augmented code completion approach in the real-world.

Competition-Level Problems are Effective LLM Evaluators (2024.findings-acl)

Copied to clipboard

Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, Weizhu Chen