Papers by Jie Zeng

Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration (2024.findings-acl)

Challenge:	Existing studies focus on compressing the Key-Value cache or grouping attention heads, while overlooking redundancy between layers.
Approach:	They propose a lightweight substitute for self-attention in well-trained LLMs that uses feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights.
Outcome:	The proposed model reduces redundancy by sharing weights across layers while maintaining high response quality while reducing redundant calculations within 53% 84% of the total layers.

Copied to clipboard

Kejuan Yang, Xiao Liu, Kaiwen Men, Aohan Zeng, Yuxiao Dong, Jie Tang

ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoning (2025.emnlp-main)

Challenge:	Existing methods for extending the maximum context lengths of language models are lacking a strong baseline for in-context few-shot classification and on more challenging Chain-of-Thought reasoning, such as HotpotQA, deteriorate question miscomprehension and false inference.
Approach:	They propose to harness window-wise attention and positional embedding techniques to extend the maximum context lengths of language models.
Outcome:	The proposed method is able to extend the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques.

Copied to clipboard

Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Guanbo Wang, Fandong Meng, Jie Zhou, Ju Ren, Yaoxue Zhang

Data Efficient RLVR via Off-Policy Influence Guidance (2026.acl-long)

Challenge:	Existing methods for fine-tuning-based compression suffer from verbose outputs, increasing computational overhead.
Approach:	They propose a framework to generate concise reasoning chains using Confidence Injection and Early Stopping.
Outcome:	The proposed framework reduces the length of the model by up to 50% while maintaining high task accuracy.

Copied to clipboard

Erle Zhu, Dazhi Jiang, Yuan Wang, Xujun Li, Jiale Cheng, Yuxian Gu, Yilin Niu, Aohan Zeng, Jie Tang, Minlie Huang, Hongning Wang

Green & Sustainable NLP

Challenge:	Existing data selection methods for RLVR are heuristic-based, lacking theoretical guarantees and generalizability.
Approach:	They propose an off-policy influence estimation method that approximates data influence using offline trajectories.
Outcome:	The proposed method reduces the computational cost of policy rollouts and improves storage and computation efficiency.

Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction (2024.acl-long)

Copied to clipboard

Yice Zhang, Jie Zeng, Weiming Hu, Ziyi Wang, Shiwei Chen, Ruifeng Xu

Sentiment Analysis

Challenge:	Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review.
Approach:	They propose a self-training framework with a pseudo-label scorer to assess the match between reviews and their pseudo-labels and train a generative model on it.
Outcome:	The proposed framework can predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review, and it can significantly improve self-training.

UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models (2026.acl-demo)

Copied to clipboard

Qundong Shi, Jie Zhou, Biyuan Lin, Junbo Cui, Guoyang Zeng, Yixuan Zhou, Ziyang Wang, Xin Liu, Zhen Luo, Yudong Wang, Zhiyuan Liu

Multimodality

Challenge:	Existing evaluation frameworks for audio foundation models are heavily reliant on English, making it difficult to objectively assess models’ performance on Chinese.
Approach:	They propose a unified framework that supports 10 languages, 14 task categories, 24 models, and 36 benchmarks with one-command evaluation and real-time leaderboards.
Outcome:	The proposed framework supports 10 languages, 14 task categories, 24 models, and 36 benchmarks with one-command evaluation and real-time leaderboards.

RTADev: Intention Aligned Multi-Agent Framework for Software Development (2025.findings-acl)

Copied to clipboard

Jie Liu, Guohua Wang, Ronghui Yang, Jiajie Zeng, Mengchen Zhao, Yi Cai

MTP-RL: Acceleration of Reinforcement Learning Rollouts with Policy-Aligned Multi-Token Prediction (2026.findings-acl)

Challenge:	Large Language Models (LLMs) are efficient assistants to humans in software development tasks, but they can cause errors during the development process.
Approach:	They propose an intention aligned multi-agent framework that ensures that all agents work based on a consensus.
Outcome:	The proposed framework reduces errors and improves the quality of generated software code.

Copied to clipboard

Ke Wang, Aohan Zeng, Zhengxiao Du, Yuxuan Hu, Bohan Zhang, Xinyi Wang, Jie Tang, Jing Zhang

LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information (2025.findings-acl)

Challenge:	Reinforcement learning (RL) is widely applied to boost the performance of pretrained models, yet its training efficiency is severely constrained by rollout generation.
Approach:	They propose a framework that accelerates the rollout phase for diverse models by equipping a pipeline to equip the multi-layer parameter-sharing MTP for all models and an advantage-aware MTP optimization strategy.
Outcome:	The proposed framework achieves stable growth of acceptance length during RL training, and also accelerates RL rollouts, achieving an average 23.1%–55.3% reduction in rollout time compared to baselines.

Copied to clipboard

Bowen Ping, Jiali Zeng, Fandong Meng, Shuo Wang, Jie Zhou, Shanghang Zhang

Challenge:	Recent advances in large language models have improved their capacity to handle long text inputs, but current models still exhibit unsatisfactory performance in long-form generation.
Approach:	They propose a method to enhance long-form text generation through step-level supervision by leveraging Monte Carlo Tree Search to collect stepwise preference pairs and employ a global memory pool to maintain factual accuracy.
Outcome:	The proposed method improves performance on long-form generation benchmarks while maintaining lossless performance on several general benchmarks.

AgentTuning: Enabling Generalized Agent Abilities for LLMs (2024.findings-acl)

Copied to clipboard

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, Jie Tang

MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization (2021.naacl-main)

Challenge:	Open large language models (LLMs) with great performance in various tasks are far inferior to commercial models such as ChatGPT and GPT-4 when acting as agents to tackle complex tasks in the real world.
Approach:	They propose a method to enhance the agent capabilities of LLMs while maintaining their general abilities.
Outcome:	The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent tasks, demonstrating generalized agent capabilities.

Copied to clipboard

Chenguang Zhu, Yang Liu, Jie Mei, Michael Zeng

Dialogue Systems & Conversational Agents

Challenge:	Existing datasets for dialogue summarization are limited to their small sizes and are built from a narrow domain.
Approach:	They propose a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries.
Outcome:	The proposed dataset is larger and contains multi-party conversations from multiple domains.

Trust in Internal or External Knowledge? Generative Multi-Modal Entity Linking with Knowledge Retriever (2024.findings-acl)

Copied to clipboard

Xinwei Long, Jiali Zeng, Fandong Meng, Jie Zhou, Bowen Zhou

Multimodality

Challenge:	Existing generative approaches struggle with the knowledge gap between visual entity information and the intrinsic parametric knowledge of LLMs.
Approach:	They propose a knowledge retrieval method that leverages external sources to enhance visual entity information and a prioritization scheme that handles noisy retrieval results.
Outcome:	The proposed method shows improvements of 3.0% to 6.5% across all evaluation metrics compared to baselines.

Emergent Modularity in Pre-trained Transformers (2023.findings-acl)

Copied to clipboard

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou

Improving Machine Translation with Large Language Models: A Preliminary Study with Cooperative Decoding (2024.findings-acl)

Challenge:	Existing studies on pre-trained Transformers show that they learn fine-grained neuron functions.
Approach:	They examine the presence of modularity in pre-trained Transformers . they focus on Mixture-of-Experts, a promising candidate for modularity .
Outcome:	The proposed structure stabilizes at the early stage, which is faster than neuron stabilization.

Copied to clipboard

Jiali Zeng, Fandong Meng, Yongjing Yin, Jie Zhou

RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios (2022.naacl-demo)

Challenge:	Commercial machine translation engines are proficient in addressing the majority of translation requirements.
Approach:	They propose to combine NMT and MT-oriented LLMs to achieve superior translation quality by combining their strengths.
Outcome:	The proposed model can handle complex scenarios beyond the capability of NMT alone.

Copied to clipboard

Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Hannan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghazaleh Kazeminejad, Zhenhailong Wang, Chris Callison-Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, Heng Ji

Information Extraction & Text Mining

Challenge:	Existing methods for event prediction are incomplete and noisy.
Approach:	They propose to use news-related event schemas to extract newsworthy events . they build a demo website and include a video demonstrating the framework .
Outcome:	The proposed framework can be applied to a wide variety of newsworthy scenarios.

Exploring Memorization in Fine-tuned Language Models (2024.acl-long)

Copied to clipboard

Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, Dawei Yin

Semantic Text Processing

Challenge:	Existing studies have shown that pre-trained langauge models tend to memorize and regenerate segments of their pre-training corpus when prompted appropriately.
Approach:	They conduct the first comprehensive analysis to explore language models’ memorization during fine-tuning across tasks.
Outcome:	The proposed analysis shows that memorization presents a strong disparity among different fine-tuning tasks.

The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG) (2024.findings-acl)

Copied to clipboard

Shenglai Zeng, Jiankun Zhang, Pengfei He, Yiding Liu, Yue Xing, Han Xu, Jie Ren, Yi Chang, Shuaiqiang Wang, Dawei Yin, Jiliang Tang

ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline (2024.findings-emnlp)

Challenge:	Retrieval-augmented generation (RAG) is a powerful technique to facilitate language model generation with proprietary and private data, where data privacy is . a privacy issue that is currently under-explored, is posed by RAG.
Approach:	They propose to use retrieval-augmented generation (RAG) to facilitate language model generation with proprietary and private data where data privacy is a pivotal concern.
Outcome:	The proposed attack methods demonstrate that RAG can mitigate the old risks, i.e., leakage of the LLMs’ training data.

Copied to clipboard

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Zhao Wenyi, Jie Tang, Yuxiao Dong

Numerical Reasoning

Challenge:	Large language models (LLMs) have shown excellent mastering of human language but struggle in real-world applications that require mathematical problem-solving.
Approach:	They propose a pipeline to train a general Math-Critique model from the LLM itself to provide feedback signals and employ rejective fine-tuning and direct preference optimization over the Llm's own generations for data collection.
Outcome:	The proposed pipeline outperforms existing LLMs that could be two times larger.

Towards Multiple References Era – Addressing Data Leakage and Limited Reference Diversity in Machine Translation Evaluation (2024.findings-acl)

Copied to clipboard

Xianfeng Zeng, Yijin Liu, Fandong Meng, Jie Zhou

BMInf: An Efficient Toolkit for Big Model Inference and Tuning (2022.acl-demo)

Challenge:	Recent research shows a weak correlation between n-gram-based metrics and human evaluations in machine translation tasks.
Approach:	They propose to use multiple references generated by LLMs to improve alignment between automatic metrics and human evaluations.
Outcome:	The proposed approach improves the alignment between automatic metrics and human evaluations on the WMT22 benchmark with 4 languages and achieves a maximum accuracy gain of 9.5%.

Copied to clipboard

Xu Han, Guoyang Zeng, Weilin Zhao, Zhiyuan Liu, Zhengyan Zhang, Jie Zhou, Jun Zhang, Jia Chao, Maosong Sun

Green & Sustainable NLP

Challenge:	Recent years, pre-trained language models (PLMs) have achieved promising results on various NLP tasks.
Approach:	They propose an open-source toolkit for big model inference and tuning which can support big model tuning at extremely low computation cost.
Outcome:	The proposed toolkit can support big model inference and tuning at extremely low computation cost.

A General Framework to Enhance Fine-tuning-based LLM Unlearning (2025.findings-acl)

Copied to clipboard

Jie Ren, Zhenwei Dai, Xianfeng Tang, Hui Liu, Jingying Zeng, Zhen Li, Rahul Goutam, Suhang Wang, Yue Xing, Qi He, Hui Liu

RU22Fact: Optimizing Evidence for Multilingual Explainable Fact-Checking on Russia-Ukraine Conflict (2024.lrec-main)

Challenge:	Existing approaches to remove copyrighted and privacy-sensitive data from Large Language Models (LLMs) have been proposed to remove specific data from LLMs without requiring full retraining.
Approach:	They propose a general framework that enhances the utility of fine-tuning-based methods by distinguishing target data and suppressing related generations.
Outcome:	The proposed framework improves the unlearning and utility of fine-tuning-based methods by distinguishing the target data and suppressing related generations.

Copied to clipboard

Yirong Zeng, Xiao Ding, Yi Zhao, Xiangyu Li, Jie Zhang, Chao Yao, Ting Liu, Bing Qin

From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models (2024.findings-emnlp)

Challenge:	Existing methods to verify factuality of claims do not provide sufficient evidence for explainable fact-checking systems.
Approach:	They propose a method to automatically retrieve and summarize evidence from the Web and a novel multilingual explainable fact-checking dataset on the Russia-Ukraine conflict in 2022.
Outcome:	The proposed method can retrieve and summarize evidence from the Web and generate explanations in 16 languages.

Copied to clipboard

Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, Yanghua Xiao

Plug-and-Play Knowledge Injection for Pre-trained Language Models (2023.acl-long)

Challenge:	Large language models (LLMs) follow instructions with elaborate requirements, yet it remains under-explored how to enhance their ability to follow complex instructions with multiple constraints.
Approach:	They propose a method to obtain and utilize effective training data to enhance LLMs' ability to follow complex instructions with multiple constraints.
Outcome:	The proposed framework improves models' ability to follow instructions generally and generalize effectively across out-of-domain, in domain, and adversarial settings while maintaining general capabilities.

Copied to clipboard

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

Semantic Text Processing

Challenge:	Existing knowledge injection methods are not suitable for enhancing pre-trained language models with external knowledge bases.
Approach:	They propose a plug-and-play knowledge injection method where knowledge bases are injected into frozen existing downstream models by a knowledge plugin.
Outcome:	The proposed method improves the performance of knowledge injection on knowledge-driven tasks while keeping model parameters frozen.

Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation (2025.emnlp-main)

Copied to clipboard

Jiankun Zhang, Shenglai Zeng, Jie Ren, Tianqi Zheng, Hui Liu, Xianfeng Tang, Hui Liu, Yi Chang

Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models (2025.findings-acl)

Challenge:	Large Multimodal Models (LMMs) can process text, images, and audio, but they introduce privacy vulnerabilities.
Approach:	They propose a compositional structured prompt attack to exploit MRAG privacy vulnerabilities . they show that LMMs can generate outputs resembling retrieved content .
Outcome:	The proposed approach generates outputs resembling retrieved content and exposes sensitive information.

Copied to clipboard

Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, Fei Yu

Enhancing Large Vision-Language Models with Ultra-Detailed Image Caption Generation (2025.emnlp-main)

Challenge:	In real-world scenarios, user instructions often contain soft constraints, which are semantically related and cannot be rule-based verified, posing challenges for large language models.
Approach:	They propose a pipeline to construct datasets with high-quality outputs for instructions containing soft constraints automatically and use Direct Preference Optimization (DPO) as the training method.
Outcome:	The proposed model improves the LLMs' soft constraint following ability by using direct preference optimization (DPO) and constraint quantity.

Copied to clipboard

Yu Zeng, Yukun Qi, Yiming Zhao, Xikun Bao, Lin Chen, Zehui Chen, Shiting Huang, Jie Zhao, Feng Zhao

Multimodality

Challenge:	Existing pipelines for generating high-quality, ultra-detailed image captions are limited by the scarcity of image caption data.
Approach:	They propose a pipeline for generating high-quality, ultra-detailed image captions that integrates both pre-processing and post-processor stages.
Outcome:	The proposed pipeline improves LVLMs' perception and cognitive abilities across multiple vision-language benchmarks.

On the Generalization of Training-based ChatGPT Detection Methods (2024.findings-emnlp)

Copied to clipboard

Han Xu, Jie Ren, Pengfei He, Shenglai Zeng, Yingqian Cui, Amy Liu, Hui Liu, Jiliang Tang

Understanding and Addressing the Under-Translation Problem from the Perspective of Decoding Objective (2024.acl-long)

Challenge:	Existing studies show that training-based methods are ineffective to detect LLM generated texts from unseen tasks or topics which are not collected during training.
Approach:	They propose to train classification models to distinguish LLMs from human texts by a distribution shift caused by prompts, text lengths, topics, and language tasks.
Outcome:	The proposed methods can detect LLMs from black-box models, but they suffer from distribution shifts due to a wide range of factors, including prompts, text lengths, topics, and language tasks.

Copied to clipboard

Chenze Shao, Fandong Meng, Jiali Zeng, Jie Zhou

Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following (2026.acl-long)

Challenge:	Neural Machine Translation (NMT) has made remarkable progress over the past years, but under-translation and over-translatation remain challenging obstacles faced by NMT systems.
Approach:	They propose to employ the confidence of predicting the end of sentence (EOS) as a detector for under-translation and strengthen the confidence-based penalty to penalize candidates with a high risk of under-translated.
Outcome:	The proposed method can detect and rectify under-translated outputs, with minor impact on other correct translations.

Copied to clipboard

Qingyu Ren, Qianyu He, Powei Chang, Jie Zeng, Zeye Sun, Fei Yu, Jiaqing Liang, Yanghua Xiao

SeDev: Structured Semantic Exploration for LLM-Driven Code Generation (2026.acl-long)

Challenge:	Existing reinforcement learning approaches suffer from dependency on external supervision and sparse reward signals from multi-constraint tasks.
Approach:	They propose a self-supervised reinforcement learning framework that eliminates dependency on external supervision by deriving reward signals directly from instructions and generating pseudo-labels for reward model training.
Outcome:	The proposed framework achieves strong improvements across 3 in-domain and 5 out-of-domain datasets while maintaining computational efficiency.

Copied to clipboard

Ronghui Yang, Jie Liu, Jiajie Zeng, Jiexin Wang, Jiuchuan Jiang, Bo An, Yi Cai, Mengchen Zhao

Programming Languages in NLP

Challenge:	Large Language Models (LLMs) have shown remarkable capabilities in automating code generation, but they suffer from insufficient exploration of the vast solution space.
Approach:	They propose a large-scale LLM-driven code generation framework that efficiently finds high-quality solutions in only a few iterations.
Outcome:	The proposed framework outperforms baselines while maintaining reasonable time and computational costs.

BranchNorm: Robustly Scaling Extremely Deep Transformers (2024.findings-acl)

Copied to clipboard

Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie Zhou

Robustness in NLP

Challenge:	Recent work on DeepNorm scales Transformers into extremely deep (1000 layers) due to the training instability of Transformers, the depths of these SOTA models are still relatively shallow.
Approach:	They propose a branch-rescaled model which dynamically rescales the non-residual branch of Transformer in accordance with the training period.
Outcome:	The proposed approach significantly outperforms existing shallow models on multiple translation tasks and achieves better training stability and convergent performance.

TasTe: Teaching Large Language Models to Translate through Self-Reflection (2024.acl-long)

Copied to clipboard

Yutong Wang, Jiali Zeng, Xuebo Liu, Fandong Meng, Jie Zhou, Min Zhang

Joint Optimization of Training Data and Policy in RLHF (2026.findings-acl)

Challenge:	Existing approaches to enhance LLMs' performance in machine translation are unable to fully exploit their instruction-following capabilities.
Approach:	They propose a framework for translating through self-reflection that involves two stages of inference . they propose to use the framework to refine LLMs' preliminary translations .
Outcome:	The proposed framework can produce translation outputs that match the quality of NMT systems.

Copied to clipboard

Zhuohao Yu, Jiali Zeng, Weizheng Gu, Mengyuan Sun, Yidong Wang, Fandong Meng, Jie Zhou, Shikun Zhang, Wei Ye

Challenge:	JODP optimizes policies on fixed training inputs, limiting the diversity of learning signals.
Approach:	They propose a framework where policy generates improved variants of training problems to enhance its own learning.
Outcome:	The proposed framework improves on safety alignment tasks by allowing 4B models to reach 8B model performance with less than 1% additional computational overhead.

Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data (2025.emnlp-main)

Copied to clipboard

Shenglai Zeng, Jiankun Zhang, Pengfei He, Jie Ren, Tianqi Zheng, Hanqing Lu, Han Xu, Hui Liu, Yue Xing, Jiliang Tang

Instruction Position Matters in Sequence Generation with Large Language Models (2024.findings-acl)

Challenge:	Existing literature suggests that RAG systems may face privacy issues when the retrieval process involves private data.
Approach:	They propose a two-stage synthetic data generation paradigm that uses attributes to preserve contextual information from the original data.
Outcome:	The proposed approach preserves key contextual information from the original data while reducing privacy risks.

Copied to clipboard

Yijin Liu, Xianfeng Zeng, Chenze Shao, Fandong Meng, Jie Zhou

Text Generation

Challenge:	Large language models (LLMs) can perform conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning.
Approach:	They propose to shift the position of task instructions after the input sentences to enhance the model's instruction-following capability.
Outcome:	The proposed method outperforms traditional settings across various model scales (1B / 7B & 13B) and different sequence generation tasks (translation and summarization) without any additional data or annotation costs.

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (2024.acl-long)

Copied to clipboard

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

Consistency Regularization Training for Compositional Generalization (2023.acl-long)

Challenge:	Large language models (LLMs) can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases.
Approach:	They propose a bilingual, multi-task benchmark for long context understanding that extends context windows and more sophisticated memory mechanisms to improve models' long context capabilities.
Outcome:	The proposed model outperforms open-source models but struggles on longer contexts.

Copied to clipboard

Yongjing Yin, Jiali Zeng, Yafu Li, Fandong Meng, Jie Zhou, Yue Zhang

Semantic Text Processing

Challenge:	Existing neural models have difficulty generalizing to unseen combinations of seen components.
Approach:	They propose to improve the capability of Transformer on compositional generalization by consistency regularization training without modifying model architectures.
Outcome:	The proposed model performs well on semantic parsing and machine translation benchmarks.

CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024.acl-long)

Copied to clipboard

Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang