PRE-

Retrieval-augmented Generation across Heterogeneous Knowledge (2022.naacl-srw)

Challenge:	Pre-training on high-resource automatic speech recognition (ASR) tasks improves ST performance even when source language is low-resourced.
Approach:	They propose a method to improve direct speech-to-text translation when source language is low-resource . they pre-train model on high-res automatic speech recognition task and fine-tune parameters for ST .
Outcome:	The proposed approach improves Spanish English ST even when the source language is low-resource . the pre-trained encoder accounts for most of the improvement, the authors show .

Copied to clipboard

Wenhao Yu

Information Retrieval

Challenge:	Existing methods for retrieving knowledge from a single source homogeneous corpus have been gaining increasing attention in the field of natural language processing (NLP) however, they still suffer from the following drawbacks: (i) They are usually trained offline, making the model agnostic to the latest information, e.g., asking a chat-bot about COVID-19.
Approach:	They propose to use a single-source homogeneous corpus to generate retrieval-augmented generation models that can learn from the pre-training corpus.
Outcome:	The proposed methods have been applied to various knowledge-intensive NLP tasks, but most of the work has focused on retrieving unstructured text documents from Wikipedia.

A Practical Toolkit for Multilingual Question and Answer Generation (2023.acl-demo)

Copied to clipboard

Asahi Ushio, Fernando Alva-Manchego, Jose Camacho-Collados

Effects of Pre- and Post-Processing on type-based Embeddings in Lexical Semantic Change Detection (2021.eacl-main)

Challenge:	Generating questions and answers from text is a challenging task due to the expected structured output.
Approach:	They propose an online service for multilingual QAG along with a python package for model fine-tuning, generation, and evaluation.
Outcome:	The proposed model is available in eight languages and can be used online or locally via lmqg.

Copied to clipboard

Jens Kaiser, Sinan Kurtyigit, Serge Kotchourko, Dominik Schlechtweg

How does the pre-training objective affect what large language models learn about linguistic properties? (2022.acl-short)

Challenge:	Lexical semantic change detection is a new and innovative research field.
Approach:	They propose to pre-train on large corpora and refine on diachronic target corpors to improve performance.
Outcome:	The proposed models improve on large corpora and diachronic target corpors . the proposed models are compared with existing models in a variety of learning scenarios .

Copied to clipboard

Ahmed Alajrami, Nikolaos Aletras

FPI: Failure Point Isolation in Large-scale Conversational Assistants (2022.naacl-industry)

Challenge:	Several pre-training objectives have been proposed to pre-train language models . but, to our knowledge, no studies have investigated how different pre- training objectives affect what BERT learns about linguistic properties.
Approach:	They propose to use masked language modeling to pre-train language models . they propose to optimize a mangled language modeling objective to learn linguistic information .
Outcome:	The proposed objectives improve BERT's learning of linguistic properties compared to non-linguistically motivated objectives.

Copied to clipboard

Rinat Khaziev, Usman Shahid, Tobias Röding, Rakesh Chada, Emir Kapanci, Pradeep Natarajan

Naturalistic Causal Probing for Morpho-Syntax (2023.tacl-1)

Challenge:	Large-scale conversational assistants can cause errors in their modules . a machine learning system can analyze large volumes of data and isolate the source of error .
Approach:	They propose a machine learning system that embeds incoming request and context using pre-trained transformer models and encodes additional metadata features to output failure point predictions.
Outcome:	The proposed system obtains 92.2% of human performance while scaling to analyze the entire traffic in 8 different languages of a large-scale conversational assistant.

Copied to clipboard

Afra Amini, Tiago Pimentel, Clara Meister, Ryan Cotterell

ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation (2021.acl-demo)

Challenge:	Existing methods for probing are limited and lack understanding of their limitations and weaknesses.
Approach:	They propose a strategy for input-level intervention on naturalistic sentences . they use morpho-syntactic features of a sentence to intervene on the rest of the sentence .
Outcome:	The proposed approach allows for input-level intervention on naturalistic sentences while keeping the rest of the sentence unchanged.

Copied to clipboard

Weizhen Qi, Yeyun Gong, Yu Yan, Can Xu, Bolun Yao, Bartuer Zhou, Biao Cheng, Daxin Jiang, Jiusheng Chen, Ruofei Zhang, Houqiang Li, Nan Duan

Cross-Thought for Sentence Encoder Pre-training (2020.emnlp-main)

Challenge:	Existing models for pre-training are not convenient for users to find and set them up.
Approach:	They propose to extend ProphetNet into other domains and languages by pre-training models . they pre-train a cross-lingual generation model ProphetNet-Multi and a Chinese generation model .
Outcome:	The proposed models achieve new state-of-the-art on 10 benchmarks.

Copied to clipboard

Shuohang Wang, Yuwei Fang, Siqi Sun, Zhe Gan, Yu Cheng, Jingjing Liu, Jing Jiang

Cross-Domain Generalization of Neural Constituency Parsers (P19-1)

Challenge:	Existing models to pretrain sentence encoders with large unlabeled corpus are lacking in linguistic information retrieval.
Approach:	They propose a novel approach to pre-training sequence encoder using transformers . they propose to train a Transformer-based sequence encoded over a large set of short sequences based on a set of masked words .
Outcome:	The proposed approach outperforms state-of-the-art encoders on hotpotQA by improving intermediate information retrieval performance.

Copied to clipboard

Daniel Fried, Nikita Kitaev, Dan Klein

Syntactic Text Processing

Challenge:	Neural parsers perform well on in-domain benchmarks, but their performance degrades in well-understood ways.
Approach:	They analyze generalization on English and Chinese corpora to see if they can generalize to other domains.
Outcome:	The proposed neural parsers perform better on in-domain benchmarks than on out-of-domain corpora.

UER: An Open-Source Toolkit for Pre-training Models (D19-3)

Copied to clipboard

Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, Xiaoyong Du

Repetition Neurons: How Do Language Models Produce Repetitions? (2025.naacl-short)

Challenge:	Existing work on pre-training models have shown that it is important to use a framework to deploy various pre- training models efficiently.
Approach:	They propose an assemble-on-demand pre-training toolkit that assembles pre-trained models on demand and encapsulates them with rich modules.
Outcome:	The proposed framework can reproduce state-of-the-art models or develop models that remain unexplored.

Copied to clipboard

Tatsuya Hiraoka, Kentaro Inui

Re-appraising the Schema Linking for Text-to-SQL (2023.findings-acl)

Challenge:	Existing studies on text generation with LLMs focus on attention heads, framing repetition as a key mechanism for in-context learning.
Approach:	They introduce repetition neurons that are regarded as "skill neurons" responsible for the repetition problem in text generation tasks.
Outcome:	The authors identify repetition neurons as "skill neurons" that perceive repetition as a task to copy the previous context repeatedly, similar to in-context learning.

Copied to clipboard

Yujian Gan, Xinyun Chen, Matthew Purver

Programming Languages in NLP

Challenge:	Recent work has shown that schema linking reduces robustness of text-to-SQL models . EMSL is used to correlate natural language queries with the given database schema .
Approach:	They propose a grammar linking module to help model align grammar references with SQL keywords.
Outcome:	The proposed language model improves performance without using EMSL, the authors show . their language model is more robust, and the proposed grammar linking improves interoperability .

LongT5: Efficient Text-To-Text Transformer for Long Sequences (2022.findings-naacl)

Copied to clipboard

Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang

Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training (2022.emnlp-industry)

Challenge:	Recent work has shown that increasing the input length or increasing model size can improve the performance of Transformer-based neural models.
Approach:	They propose a model that integrates attention ideas from long-input transformers and adopts pre-training strategies from summarization pre-train into the scalable T5 architecture.
Outcome:	The proposed model outperforms the original T5 models on several summarization and question answering tasks and achieves state-of-the-art results.

Copied to clipboard

Taolin Zhang, Junwei Dong, Jianing Wang, Chengyu Wang, Ang Wang, Yinghui Liu, Jun Huang, Yong Li, Xiaofeng He

Top-down Discourse Parsing via Sequence Labelling (2021.eacl-main)

Challenge:	Existing knowledge-enhanced pre-trained language models (KEPLMs) can capture internal knowledge, but can't understand external background knowledge.
Approach:	They propose to use Chinese knowledge-enhanced pre-trained language models to improve context-aware representations via learning from structured relations in knowledge bases.
Outcome:	Experiments show that Chinese knowledge-enhanced pre-trained language models outperform strong baselines over various benchmark NLP tasks and in different model sizes.

Copied to clipboard

Fajri Koto, Jey Han Lau, Timothy Baldwin

Learning Rich Representation of Keyphrases from Text (2022.findings-naacl)

Challenge:	Discourse analysis is a systematic way to understand how texts are segmented hierarchically into discourse units.
Approach:	They propose a top-down approach to discourse parsing that is conceptually simpler than its predecessors.
Outcome:	The proposed model eliminates the decoder and reduces the search space for splitting points.

Copied to clipboard

Mayank Kulkarni, Debanjan Mahata, Ravneet Arora, Rajarshi Bhowmik

Going beyond research datasets: Novel intent discovery in the industry setting (2023.findings-eacl)

Challenge:	Prior work has referred to extractive (part of document) or abstractive (not part of document).
Approach:	They propose to use a new pre-training objective to introduce keyphrases into transformer language models in discriminative and generative settings.
Outcome:	The proposed model improves performance in discriminative and generative settings and also improves on named entity recognition, question answering, relation extraction and abstractive summarization tasks.

Copied to clipboard

Aleksandra Chrabrowa, Tsimur Hadeliya, Dariusz Kajtoch, Robert Mroczkowski, Piotr Rybak

What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognition (2021.tacl-1)

Challenge:	Novel intent discovery automates grouping of similar messages to identify previously unknown intents.
Approach:	They propose to use question-only data to improve the intent discovery pipeline . they propose to utilize conversational structure of real-life datasets for clustering .
Outcome:	The proposed method gives 33pp performance boost over state-of-the-art model for question only . it also gives 13pp performance increase over the naive baseline model .

Copied to clipboard

Piotr Żelasko, Raghavendra Pappagari, Najim Dehak

Benchmarking Long-tail Generalization with Likelihood Splits (2023.findings-eacl)

Challenge:	Existing punctuation in the transcripts has a massive effect on the models’ performance, and specific label set specificity does not affect dialog act segmentation performance.
Approach:	They apply two pre-trained transformer models to a conversation transcript as a sequence of dialog acts and achieve strong results on Switchboard Dialog Act and Meeting Recorder Dialog Act corpora.
Outcome:	The proposed models achieve 8.4% and 14.2% error rates on the Switchboard Dialog Act and Meeting Recorder Dialog Act corpora.

Copied to clipboard

Ameya Godbole, Robin Jia

Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs (2025.findings-naacl)

Challenge:	Existing methods to test out of distribution generalization have their own strengths and weaknesses.
Approach:	They propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets.
Outcome:	The proposed approach can be customized to construct meaningful splits for a wide range of tasks.

Copied to clipboard

Itai Mondshine, Tzuf Paz-Argaman, Reut Tsarfaty

Multilinguality

Challenge:	Current LLMs are primarily trained on English data but also include data from other languages.
Approach:	They propose to use a pre-translation strategy to translate a task prompt into English before inference . they use 'a modular entity' that could be translated into four different languages .
Outcome:	The proposed strategies are based on a set of pre-trained data across 35 languages covering both low and high-resource languages.

MCP: Self-supervised Pre-training for Personalized Chatbots with Multi-level Contrastive Sampling (2022.findings-emnlp)

Copied to clipboard

Zhaoheng Huang, Zhicheng Dou, Yutao Zhu, Zhengyi Ma

MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solving (2022.findings-naacl)

Challenge:	Existing studies focus on generating implicit user profiles from the user’s dialogue history, thus it suffers from data sparsity and performance degradation.
Approach:	They propose a self-supervised learning framework MCP for capturing better representations from users’ dialogue history for personalized chatbots.
Outcome:	The proposed model improves on two real-world datasets.

Copied to clipboard

Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, Xiangliang Zhang

Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications? (2024.naacl-short)

Challenge:	Existing work on math word problem solvers replace real numbers with symbolic placeholders to focus on logic reasoning.
Approach:	They propose to inject numerical properties into symbolic placeholders with contextualized representation learning schema to solve number representation dilemma.
Outcome:	The proposed model can solve MWP problems on English and Chinese benchmarks.

Copied to clipboard

Yotam Intrator, Matan Halfon, Roman Goldenberg, Reut Tsarfaty, Matan Eyal, Ehud Rivlin, Yossi Matias, Natalia Aizenberg

Multilinguality

Challenge:	Existing studies have focused on pre-translation, but there is still need for it . authors say that it is not universally necessary to translate large language models .
Approach:	They re-evaluate the need for pre-translation in the context of PaLM2 models . authors found that PaLM2-L consistently outperforms pre-translated in 94 out of 108 languages .
Outcome:	The proposed model outperforms pre-translation in 94 out of 108 languages and 6 benchmarks . authors argue that pre-translated inputs can be used to improve performance .

AMUSED: A Multi-Stream Vector Representation Method for Use in Natural Dialogue (2020.lrec-1)

Copied to clipboard

Gaurav Kumar, Rishabh Joshi, Jaspreet Singh, Promod Yenigalla

Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts (2022.findings-emnlp)

Challenge:	Current architectures only take care of semantic and contextual information for a given query and fail to fully account for syntactic and external knowledge which are crucial for generating responses in a chit-chat system.
Approach:	They propose a multi-stream deep learning architecture that learns unified embeddings for query-response pairs by incorporating Graph Convolution Networks over their dependency parse.
Outcome:	The proposed architecture improves on the next sentence prediction task and significantly improves existing techniques.

Copied to clipboard

Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, Xipeng Qiu

Language Model Pre-Training with Sparse Latent Typing (2022.emnlp-main)

Challenge:	Prompt tuning is parameter-efficient but lags behind other state-of-the-art methods.
Approach:	They propose a parameter-efficient tuning method that only optimizes a soft prompt to adapt PTMs to downstream tasks.
Outcome:	The proposed method is parameter-efficient but lags behind other state-of-the-art methods.

Copied to clipboard

Liliang Ren, Zixuan Zhang, Han Wang, Clare Voss, ChengXiang Zhai, Heng Ji

Capturing Row and Column Semantics in Transformer Based Question Answering over Tables (2021.naacl-main)

Challenge:	Modern large-scale Pre-trained Language Models focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences.
Approach:	They propose a new pre-training objective that enables the model to learn latent types . the objective allows the model a self-supervised way to extract sentence-level keywords .
Outcome:	The proposed model learns interpretable latent type categories without external knowledge and improves downstream tasks.

Copied to clipboard

Michael Glass, Mustafa Canim, Alfio Gliozzo, Saneem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avi Sil, Feifei Pan, Samarth Bharadwaj, Nicolas Rodolfo Fauceglia

Complicate Then Simplify: A Novel Way to Explore Pre-trained Models for Text Classification (2022.coling-1)

Challenge:	Existing transformer based approaches have been used to answer questions over tables.
Approach:	They propose a transformer based architecture that independently classifies rows and columns to identify relevant cells and a model that incorporates existing tables to improve efficiency.
Outcome:	The proposed model outperforms the state-of-the-art transformer based approaches on WikiSQL lookup questions and achieves 3.4% and 18.86% additional precision improvement on the standard WikisQL benchmark.

Copied to clipboard

Xu Zhang, Zejie Liu, Yanzheng Xiang, Deyu Zhou

Information Retrieval

Challenge:	Existing frameworks for text classification employing pre-trained models are constrained by the difficulty of the task.
Approach:	They propose a framework which implements a two-stage training strategy to fully exploit the knowledge in pre-trained models.
Outcome:	The proposed framework outperforms state-of-the-art classification models on six text classification corpora.

S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers (2022.findings-acl)

Copied to clipboard

Binyuan Hui, Ruiying Geng, Lihan Wang, Bowen Qin, Yanyang Li, Bowen Li, Jian Sun, Yongbin Li

Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction (2021.acl-short)

Challenge:	Existing graph-based encoders for text-to-SQL do not model the syntax of natural language questions.
Approach:	They propose to inject Syntax to question-Schema graph encoder for text-to-SQL parsers and employ the decoupling constraint to induce diverse relational edge embedding.
Outcome:	The proposed approach outperforms all existing methods when pre-training models are used, resulting in a performance ranks first on the Spider leaderboard.

Copied to clipboard

Ming Shen, Pratyay Banerjee, Chitta Baral

Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and NER (2025.emnlp-industry)

Challenge:	Masked Noun-Phrase Prediction (MNPP) is a pre-training strategy to tackle pronoun resolution in a fully unsupervised setting.
Approach:	They propose a pre-training strategy to tackle pronoun resolution in an unsupervised setting by fine-tuning a large pre-trained model on a human-labeled dataset and then transferring to a smaller dataset such as Winograd Schema Challenge (WSC).
Outcome:	The proposed method outperforms all previous unsupervised methods on all datasets by large margins.

Copied to clipboard

Junyi Zhu, Savas Ozkan, Andrea Maracani, Sinan Mutlu, Cho Jung Min, Mete Ozay

Word Order Matters When You Increase Masking (2022.emnlp-main)

Challenge:	nave multitask pre-finetuning introduces conflicting optimization signals that degrade overall performance.
Approach:	They propose a framework that enables a single shared encoder backbone with modular adapters.
Outcome:	The proposed framework achieves comparable performance to individual pre-finetuning while meeting practical deployment constraint.

Copied to clipboard

Karim Lasri, Alessandro Lenci, Thierry Poibeau

SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)

Challenge:	Recent work shows that word order information is not always useful for pre-trained models to solve downstream tasks.
Approach:	They examine the effect of removing position encodings on the pre-training objective itself to see whether models can reconstruct position information from co-occurrences.
Outcome:	The proposed model can reconstruct position information from co-occurrences without explicit encoding, but not on some tasks.

Copied to clipboard

Haejun Lee, Drew A. Hudson, Kangwook Lee, Christopher D. Manning

DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations (2025.emnlp-industry)

Challenge:	Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them.
Approach:	They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering.
Outcome:	The proposed model improves the original BERT model on downstream tasks by large margins.

Copied to clipboard

Elena Khasanova, Harsh Saini, Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, Shashi Bhushan Tn

Entity-Based Semantic Adequacy for Data-to-Text Generation (2021.findings-emnlp)

Challenge:	Large Language Models (LLMs) have been used in real-world industrial scenarios for various natural language processing tasks, but their high inference cost makes their deployment impractical, necessitating the use of smaller models.
Approach:	They propose a continual pre-training technique that generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization.
Outcome:	The proposed technique improves small LLMs’ domain adaptability for business conversational tasks, compared with traditional methods that rely on next-token prediction.

Copied to clipboard

Juliette Faille, Albert Gatt, Claire Gardent

MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories (2021.naacl-main)

Challenge:	Existing pre-trained language models have improved the fluency of text generation systems, but semantic adequacy remains an unsolved issue.
Approach:	They propose an automatic evaluation metric to assess to what extent models that verbalise RDF graphs produce text that contains mentions of entities occurring in the input.
Outcome:	The proposed metric can be used to assess to what extent generation models verbalise RDF graphs produce text that contains mentions of the entities occurring in the input.

Copied to clipboard

Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, Jongwuk Lee

Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa (2021.naacl-main)

Challenge:	Existing studies have developed computational models to recognize metaphorical words in sentences.
Approach:	They propose a model that leverages contextualized word representation and linguistic metaphor identification theories to detect whether the target word is metaphorical.
Outcome:	The proposed model outperforms baseline models on four benchmark datasets . it leverages contextualized word representation and linguistic metaphor identification theories to detect whether the target word is metaphorical.

Copied to clipboard

Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, Xipeng Qiu

APGN: Adversarial and Parameter Generation Networks for Multi-Source Cross-Domain Dependency Parsing (2021.findings-emnlp)

Challenge:	Aspect-based sentiment analysis (ABSA) is a fine-grained task in sentiment analysis.
Approach:	They compare a model with a dependency parser and a tree from a fine-tuned RoBERTa model to find the polarities for aspects in a sentence.
Outcome:	The proposed model outperforms the parser-provided tree on six datasets across four languages.

Copied to clipboard

Ying Li, Meishan Zhang, Zhenghua Li, Min Zhang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan

Robustness in NLP

Challenge:	Existing models for dependency parsing use labeled training data for several fixed domains, but performance drops when labeles only exist for several out-domains.
Approach:	They propose a model for multi-source cross-domain dependency parsing that uses a parameter generation network and adversarial network for learning domain-invariant representations.
Outcome:	The proposed model improves cross-domain parsing performance by about 2 points over strong BERT-enhanced baselines over a recently released dataset for multi-domain dependency parse.

Are Prompt-based Models Clueless? (2022.acl-long)

Copied to clipboard

Pride Kavumba, Ryo Takahashi, Yusuke Oda

RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER (2020.coling-main)

Challenge:	Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective.
Approach:	They propose to examine whether few-shot prompt-based models exploit superficial cues by reusing the model head and formatting the input to match the pre-training objective.
Outcome:	The proposed models perform well on instances with superficial cues, but often outperform random accuracy on instances without superficial cuing.

Copied to clipboard

Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng, Yuxuan Sun, Zengwei Zheng, Yuanyi Chen

Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies (2021.eacl-main)

Challenge:	Named entity recognition (MNER) for tweets is a key task of many applications.
Approach:	They propose a pre-trained multimodal named entity recognition model based on Relationship Inference and Visual Attention (RIVA) for tweets.
Outcome:	The proposed model improves on the multimodal named entity recognition (MNER) task on tweets with the aid of visual clues.

Copied to clipboard

Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, Yulan He

Different Strokes for Different Folks: Investigating Appropriate Further Pre-training Approaches for Diverse Dialogue Tasks (2021.emnlp-main)

Challenge:	Biomedical question-answering (QA) provides users with high-quality information from a vast scientific literature.
Approach:	They propose to use a biomedical entity-aware masking strategy to fine-tune masked language models to their domains.
Outcome:	The proposed approach is an adaptation process for masked LMs, not memory or components.

Copied to clipboard

Yao Qiu, Jinchao Zhang, Jie Zhou

Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)

Challenge:	Pre-trained models can be fine-tuned on domain-specific unlabeled data . however, most further pre-training works just keep running the conventional pre- training task .
Approach:	They propose to add a further pre-training phase to the model to improve downstream tasks . they propose to use a domain-adaptive pre-tuning phase to fine-tune the models on unlabeled data .
Outcome:	The proposed method improves multiple task-oriented dialogue downstream tasks.

Copied to clipboard

Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, Juanzi Li

CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification (2024.eacl-long)

Challenge:	Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression.
Approach:	They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM .
Outcome:	The proposed model can be used to distill a 1.9B student model using a teacher LLM.

Copied to clipboard

Yang Li, Canran Xu, Guodong Long, Tao Shen, Chongyang Tao, Jing Jiang

Paragraph-based Transformer Pre-training for Multi-Sentence Inference (2022.naacl-main)

Challenge:	Existing methods to improve pre-trained language models for many-class classification suffer from verbalizer ambiguity . a significant disparity exists between the pre-training and fine-tuning stages of the model .
Approach:	They propose a method to tune pre-trained language models to a broad spectrum of tasks . they use an instance-dependent soft prefix to complement language verbalizers in many-class classification .
Outcome:	The proposed method outperforms baselines on many-class datasets.

Copied to clipboard

Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti

PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning (2026.findings-acl)

Challenge:	Recent studies show that pre-trained transformers perform poorly for multi-candidate inference tasks.
Approach:	They propose a pre-training objective that models paragraph-level semantics across multiple input sentences.
Outcome:	The proposed model outperforms existing models on three AS2 and one fact verification datasets.

Copied to clipboard

Wenfeng Feng, Penghong Zhao, Guochao Jiang, Chuzhan Hao, Guohua Liu, Yuewei Zhang

Transition-based Opinion Generation for Aspect-based Sentiment Analysis (2024.findings-acl)

Challenge:	extending grouping-based methods to agentic reasoning presents unique challenges . frequent environment interactions and tool invocations render intra-group advantage estimation unstable .
Approach:	They propose a grouping-based method that uses a single round of rollouts to stabilize advantage estimation.
Outcome:	a new RL framework outperforms grouping-based methods in retrieval tasks and advanced mathematical reasoning benchmarks.

Copied to clipboard

Tianlai Ma, Zhongqing Wang, Guodong Zhou

DocQueryNet: Value Retrieval with Arbitrary Queries for Form-like Documents (2022.coling-1)

Challenge:	Existing approaches to aspect-based sentiment analysis often overlook the importance of explicitly modeling structure among sentiment elements.
Approach:	They propose to integrate general pre-trained sequence-to-sequence language models with a structure-aware transition-based approach to model sentiment structure.
Outcome:	The proposed model improves the state-of-the-art performance on several benchmark datasets.

Copied to clipboard

Mingfei Gao, Le Xue, Chetan Ramaiah, Chen Xing, Ran Xu, Caiming Xiong

A Language Model-based Generative Classifier for Sentence-level Discourse Parsing (2021.emnlp-main)

Challenge:	Existing methods that only address a fixed set of fields are difficult to use for different form types.
Approach:	They propose a value retrieval method with arbitrary queries for form-like documents . they propose 'docQueryNet' to predict target value based on understanding of layout and semantics of a form .
Outcome:	The proposed method outperforms existing methods on value retrieval . it improves document understanding on large-scale model pre-training by 17% .

Copied to clipboard

Ying Zhang, Hidetaka Kamigaito, Manabu Okumura

Improving AMR Parsing with Sequence-to-Sequence Pre-training (2020.emnlp-main)

Challenge:	Existing methods to consider textual coherence are limited in labeled data.
Approach:	They propose a language model-based generative classifier that uses labels as input and embeds labels into their representations.
Outcome:	The proposed classifier achieves state-of-the-art in discourse segmentation and relation F1 scores with gold boundaries and automatically segmented boundaries.

Copied to clipboard

Dongqin Xu, Junhui Li, Muhua Zhu, Min Zhang, Guodong Zhou

Exploiting Definitions for Frame Identification (2021.eacl-main)

Challenge:	Abstract meaning representation (AMR) parsing is limited by the size of curated datasets.
Approach:	They propose a seq2seq pre-training approach to build pre-trained models on three relevant tasks.
Outcome:	The proposed model improves performance on three relevant tasks while maintaining the response of pre-trained models.

Copied to clipboard

Tianyu Jiang, Ellen Riloff

MANTa: Efficient Gradient-Based Tokenization for End-to-End Robust Language Modeling (2022.findings-emnlp)

Challenge:	a frame-semantic parsing task is to determine which frame best captures the meaning of a word or phrase in a sentence.
Approach:	They propose a frame identification model that generates representations for frames and lexical units (senses) they evaluate the model on three data sets and show it consistently achieves better performance than previous systems.
Outcome:	The proposed model consistently outperforms previous systems on three data sets.

Copied to clipboard

Nathan Godey, Roman Castagné, Éric de la Clergerie, Benoît Sagot

Unsupervised Extractive Summarization using Pointwise Mutual Information (2021.eacl-main)

Challenge:	Subword tokenization algorithms have been an essential component of language modeling but their static nature results in important flaws that degrade the models’ downstream performance and robustness.
Approach:	They propose a module for Adaptive Neural TokenizAtion that is differentiable and trained end-to-end with the language model.
Outcome:	The proposed tokenizer improves robustness to character perturbations and out-of-domain data.

Copied to clipboard

Vishakh Padmakumar, He He

Normalizing without Modernizing: Keeping Historical Wordforms of Middle French while Reducing Spelling Variants (2024.findings-naacl)

Challenge:	Unsupervised approaches to extractive summarization rely on notion of sentence importance defined by semantic similarity between a sentence and the document.
Approach:	They propose a method to measure relevance and redundancy using PMI between sentences.
Outcome:	The proposed method outperforms similarity-based methods on news, medical journal articles, and personal anecdotes.

Copied to clipboard

Raphael Rubino, Johanna Gerlach, Jonathan Mutal, Pierrette Bouillon

Green & Sustainable NLP

Challenge:	a new method to normalize orthographic variations of historical documents is needed for digital humanities and diachronic studies.
Approach:	They propose to normalize orthographic wordforms found in Middle French archives . authors say it improves accuracy and accuracy over a strong baseline .
Outcome:	The proposed methods normalize orthographic variations of historical documents without modernizing them.

Breaking the Corpus Bottleneck for Context-Aware Neural Machine Translation with Cross-Task Pre-training (2021.acl-long)

Copied to clipboard

Linqing Chen, Junhui Li, Zhengxian Gong, Boxing Chen, Weihua Luo, Min Zhang, Guodong Zhou

Machine Translation

Challenge:	Context-aware neural machine translation (NMT) remains challenging due to the lack of large-scale document-level parallel corpora.
Approach:	They propose to use large-scale parallel datasets and source-side monolingual documents to improve context-aware neural machine translation.
Outcome:	The proposed model can be used to translate both sentences and documents on four translation tasks.

Improving Commonsense Question Answering by Graph-based Iterative Retrieval over Multiple Knowledge Sources (2020.coling-main)

Copied to clipboard

Qianglong Chen, Feng Ji, Haiqing Chen, Yin Zhang

Formulating Few-shot Fine-tuning Towards Language Model Pre-training: A Pilot Study on Named Entity Recognition (2022.findings-emnlp)

Challenge:	Existing methods to facilitate natural language understanding rarely involve commonsense or background knowledge.
Approach:	They propose a question-answering method that integrates multiple knowledge sources to boost performance.
Outcome:	The proposed method outperforms other competing methods on the CommonsenseQA dataset and achieves the new state-of-the-art.

Copied to clipboard

Zihan Wang, Kewen Zhao, Zilong Wang, Jingbo Shang

Rethinking Denoised Auto-Encoding in Language Pre-Training (2021.emnlp-main)

Challenge:	Existing methods for fine-tuning pre-trained language models are limited . we propose a few-shot fine-uning framework for NER .
Approach:	They propose a few-shot fine-tuning framework for named entity recognition (NER) they propose three new types of tokens, "is-entity", "which-type" and "bracket"
Outcome:	The proposed framework improves on pre-trained language models on several benchmark datasets.

Copied to clipboard

Fuli Luo, Pengcheng Yang, Shicheng Li, Xuancheng Ren, Xu Sun, Songfang Huang, Fei Huang

Improving Named Entity Recognition via Bridge-based Domain Adaptation (2023.findings-acl)

Challenge:	Pre-trained models such as BERT have achieved success in learning sequence representations, but they tend to learn representations that are covariant with the noise of pre-training.
Approach:	They propose to train self-trained models to learn noise invariant sequence representations . they encourage consistency between original sequence and corrupted version via unsupervised instance-wise training signals.
Outcome:	The proposed model improves on 11 natural language understanding and cross-modal tasks and achieves 0.6% gain on GLUE benchmarks and 0.8% increment on NLVR2 .

Copied to clipboard

Jingyun Xu, Changmeng Zheng, Yi Cai, Tat-Seng Chua

Upstream Mitigation Is Not All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models (2022.acl-long)

Challenge:	Existing methods for named entity recognition use pre-training language models to represent words, leading to entity type misclassification.
Approach:	They propose a model-agnostic framework called MoCL for cross-domain named entity recognition to refine the original representations and combine it with two distinct cross- domain NER methods and two pre-training language models to explore its generalization ability.
Outcome:	The proposed framework is model-agnostic and can be used to generalize and refine existing models.

Copied to clipboard

Ryan Steed, Swetasudha Panda, Ari Kobren, Michael Wick

On the Use of Bert for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation (2022.naacl-main)

Challenge:	Large language models and other massively pre-trained "foundation" models can easily adapt to a wide variety of downstream tasks in a process called finetuning.
Approach:	They propose to use the bias transfer hypothesis to reduce social biases internalized by large language models during pre-training into harmful task-specific behavior after fine-tuning.
Outcome:	The bias transfer hypothesis is the theory that social biases internalized by large language models during pre-training transfer into harmful task-specific behavior after fine-tuning.

Copied to clipboard

Yongjie Wang, Chuang Wang, Ruobing Li, Hui Lin

Pre- and In-Parsing Models for Neural Empty Category Detection (P18-1)

Challenge:	Pre-trained models have not been used to outperform other deep learning models such as CNN in Automated Essay Scoring (AES).
Approach:	They propose a novel multi-scale essay representation for BERT that can be jointly learned . they employ multiple losses and transfer learning from out-of-domain essays to further improve performance .
Outcome:	The proposed model outperforms existing models in the area of automated essay scoring . the proposed model generalizes well to the CommonLit Readability Prize data set .

Copied to clipboard

Yufei Chen, Yuanyuan Zhao, Weiwei Sun, Xiaojun Wan

Syntactic Text Processing

Challenge:	Existing studies on empty category detection have shown positive effects on syntactic parsing . empty categories are used to indicate long-distance dependencies, discontinuous constituents, and certain dropped elements.
Approach:	They propose to use ECD to detect empty categories without syntactic analysis.
Outcome:	The proposed models outperform the prior state-of-the-art by significant margins.

Semantic Specialization for Knowledge-based Word Sense Disambiguation (2023.eacl-main)

Copied to clipboard

Sakae Mizuki, Naoaki Okazaki

DP-Rewrite: Towards Reproducibility and Transparency in Differentially Private Text Rewriting (2022.coling-1)

Challenge:	Existing methods for knowledge-based Word Sense Disambiguation (WSD) use only lexical knowledge to adapt contextualized embeddings.
Approach:	They propose a semantic specialization where contextualized embeddings are adapted to the WSD task using only lexical knowledge.
Outcome:	The proposed method outperforms previous studies that adapt contextualized embeddings while controlling deviations from the original embeddables.

Copied to clipboard

Timour Igamberdiev, Thomas Arnold, Ivan Habernal

Visually Guided Generative Text-Layout Pre-training for Document Intelligence (2024.naacl-long)

Challenge:	Existing systems for differentially private text rewriting lack the means to validate privacy-preserving claims.
Approach:	They propose an open-source framework for differentially private text rewriting which is modular, extensible and highly customizable.
Outcome:	The proposed framework provides a way to lead and validate private text rewriting research.

Copied to clipboard

Zhiming Mao, Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, Kam-Fai Wong

Populating Legal Ontologies using Semantic Role Labeling (2020.lrec-1)

Challenge:	Prior work shows that pre-training techniques can boost the performance of visual document understanding (VDU) . Xu et al., 2020;; Gu e t al, 2021;; Appalaraju e al. 2022)
Approach:	They propose a visually guided generative text-layout pre-training method that optimizes hierarchical language and layout modeling objectives to generate interleaved text and layout sequences.
Outcome:	The proposed model can process word-intensive documents of any length and achieves competitive performance over baselines on VDU tasks.

Copied to clipboard

Llio Humphreys, Guido Boella, Luigi Di Caro, Livio Robaldo, Leon van der Torre, Sepideh Ghanavati, Robert Muthuri

STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering (2020.emnlp-main)

Challenge:	This paper is concerned with the ‘resource consumption bottleneck’ of creating semantic technologies manually.
Approach:	They propose to combine general-purpose NLP modules with pre- and post-processing using rules based on domain knowledge to solve the acquisition paradox.
Outcome:	The proposed system extracts norms from legislation and represents them as structured norms in legal ontologies.

Copied to clipboard

Hrituraj Singh, Sumit Shekhar

Knowledge-Grounded Dialogue Generation with Pre-trained Language Models (2020.emnlp-main)

Challenge:	Chart Question Answering (CQA) is a task of answering natural language questions about visualisations in the chart image.
Approach:	They propose a method for Chart Question Answering which improves the question/answering through sequential elements localization, question encoding and then, a structural transformer-based learning approach.
Outcome:	The proposed method outperforms state-of-the-art methods on various chart Q/A datasets while outperforming even human baseline.

Copied to clipboard

Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, Rui Yan

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption (2022.findings-acl)

Challenge:	Empirical results indicate that pre-trained language models can significantly outperform state-of-the-art methods in both automatic evaluation and human judgment.
Approach:	They propose to equip a pre-trained language model with a knowledge selection module to generate knowledge-grounded dialogues.
Outcome:	The proposed model outperforms state-of-the-art methods in evaluation and human judgment.

Copied to clipboard

Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei

Leaner and Faster: Two-Stage Model Compression for Lightweight Text-Image Retrieval (2022.naacl-main)

Challenge:	enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks.
Approach:	They propose an approximation approach for transformers which enables inference on ciphertext data.
Outcome:	The proposed approach can infer pre-trained models on encrypted data with negligible performance drop but enjoy theory-guaranteed privacy-preserving advantage.

Copied to clipboard

Siyu Ren, Kenny Zhu

BERT Busters: Outlier Dimensions that Disrupt Transformers (2021.findings-acl)

Challenge:	Existing text-image approaches use pre-trained vision-language representations for text retrieval . however, these models pose non-trivial memory requirements and substantial indexing time .
Approach:	They propose a framework to compress large pre-trained dual-encoders for lightweight text-image retrieval.
Outcome:	The proposed model performs better on Flickr30K and MSCOCO benchmarks than the original full model on mobile devices.

Copied to clipboard

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, Anna Rumshisky

FlauBERT: Unsupervised Language Model Pre-training for French (2020.lrec-1)

Challenge:	Existing studies show that pre-trained Transformers are remarkably robust to pruning.
Approach:	They show that pre-trained Transformer encoders are surprisingly fragile to pruning . they show that disabling them significantly degrades both the MLM loss and the downstream task performance.
Outcome:	The results show that the removal of features in pre-trained transformers significantly degrades both the MLM loss and the downstream task performance.

Copied to clipboard

Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, Didier Schwab

Fusion or Defusion? Flexible Vision-and-Language Pre-Training (2023.findings-acl)

Challenge:	Language models are a key step to achieve state-of-the-art results in many different Natural Language Processing (NLP) tasks.
Approach:	They propose to use a language model that is pre-trained on a large and heterogeneous French corpus to train continuous word representations.
Outcome:	The proposed model outperforms existing models on a large and heterogeneous French corpus.

Copied to clipboard

Rongyi Sun, Ziran Li, Yifeng Ding, Qifan Wang, Jingang Wang, Haitao Zheng, Wei Wu, Yunsen Xian

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations (2021.emnlp-main)

Challenge:	Existing approaches to vision-and-language pretraining (VLP) lack effectiveness and efficiency in downstream multimodal tasks.
Approach:	They propose a flexible vision-and-language pre-training model by incorporating cross-modal fusions into a dual-encoder architecture and a cross-module knowledge transfer strategy to guide the training process.
Outcome:	The proposed model is well-equipped with effectiveness and efficiency compared with other strong VLP models.

Copied to clipboard

Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, Zitao Liu

MIND: A Large-scale Dataset for News Recommendation (2020.acl-main)

Challenge:	Existing audio-language task-specific predictive approaches focus on building complicated late-fusion mechanisms.
Approach:	They propose a cross-modal transformer for audio-and-language that learns inter-modal connections between audio and language through two proxy tasks on a large amount of audio- and-language pairs.
Outcome:	The proposed model improves on multiple audio-and-language tasks and can be used in fine-tuning phase.

Copied to clipboard

Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, Ming Zhou

Are Pretrained Convolutions Better than Pretrained Transformers? (2021.acl-long)

Challenge:	Personalized news recommendation is an important technique for personalized news service.
Approach:	They propose to build a large-scale news recommendation dataset from Microsoft News . they demonstrate that news recommendation relies on the quality of news content understanding .
Outcome:	The proposed dataset contains 1 million users and more than 160k English news articles, each of which has rich textual content such as title, abstract and body.

Copied to clipboard

Yi Tay, Mostafa Dehghani, Jai Prakash Gupta, Vamsi Aribandi, Dara Bahri, Zhen Qin, Donald Metzler

BERT-like Models for Slavic Morpheme Segmentation (2025.acl-long)

Challenge:	Recent research has shown promise in entirely convolutional, or CNN, architectures, but they have not been explored using the pre-train-fine-tune paradigm.
Approach:	They propose to use the pre-train-fine-tune paradigm to study convolutional models.
Outcome:	The proposed architectures outperform Transformers in certain scenarios, but with caveats.

Copied to clipboard

Dmitry Morozov, Lizaveta Astapenka, Anna Glazkova, Timur Garipov, Olga Lyashevskaya

Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models (2022.acl-long)

Challenge:	Existing morpheme segmentation algorithms for Slavic languages have been improved but performance is still low for words with roots not present in training data.
Approach:	They propose to fine-tune BERT-like models for morpheme segmentation using data from Belarusian, Czech, and Russian to account for word semantics.
Outcome:	The proposed models outperform all previous approaches in Czech and Russian, with word-level accuracy of 92.5-95.1%.

Copied to clipboard

Biru Zhu, Yujia Qin, Fanchao Qi, Yangdong Deng, Zhiyuan Liu, Maosong Sun, Ming Gu

Robustness in NLP

Challenge:	Existing feature-based model selection methods are vulnerable to fine-tuning and are not reliable indicators for the PTM’s transferability.
Approach:	They propose feature-based model selection methods which assess PTMs’ transferability to a specific task in a fast way without fine-tuning.
Outcome:	The proposed methods can make FMS mistakenly judge transferability of models and can be combined with the backdoor attack to misguide the FMS to select poisoned models.

PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2022.acl-long)

Copied to clipboard

Wen Xiao, Iz Beltagy, Giuseppe Carenini, Arman Cohan

Generating Textual Explanations for Machine Learning Models Performance: A Table-to-Text Task (2022.lrec-1)

Challenge:	Existing pretrained models require domain-specific additional information to be effective.
Approach:	They propose a pre-trained model for multi-document representation with a focus on summarization that uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents.
Outcome:	PRIMERA outperforms current state-of-the-art models on most datasets with large margins . PRImerA uses efficient encoder-decoder transformers to simplify processing of concatenated input documents.

Copied to clipboard

Isaac Ampomah, James Burton, Amir Enshaei, Noura Al Moubayed

An Empirical Study of Pre-trained Transformers for Arabic Information Extraction (2020.emnlp-main)

Challenge:	Numerical tables are widely used to communicate or report the classification performance of machine learning models with respect to a set of evaluation metrics.
Approach:	They propose a task where neural models are trained to generate textual explanations based on the metrics’ scores reported in numerical tables.
Outcome:	The proposed model outperforms existing methods and can be used to explain the performance of ML models.

Copied to clipboard

Wuwei Lan, Yang Chen, Wei Xu, Alan Ritter

ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences (2024.acl-long)

Challenge:	Multilingual pre-trained Transformers have been shown to enable effective cross-lingual zero-shot transfer, but their performance on Arabic information extraction tasks is not well studied.
Approach:	They pre-train a bilingual BERT that is designed specifically for Arabic NLP and English-to-Arabic zero-shot transfer learning.
Outcome:	The pre-trained model significantly outperforms mBERT, XLM-RoBERTa, and AraBERT in both the supervised and zero-shot transfer settings.

Copied to clipboard

Yuanhe Tian, Ruyi Gan, Yan Song, Jiaxing Zhang, Yongdong Zhang

Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics (2022.findings-emnlp)

Challenge:	Current large language models (LLMs) are ineffective in learning domain knowledge and aligning with human preference.
Approach:	They propose a benchmark LLM for Chinese medical domain that uses pre-training, supervised fine-tuning and RLHF to train LLMs.
Outcome:	The proposed LLM performs better than existing LLMs in the Chinese medical domain.

Copied to clipboard

Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, Ahmad Beirami

Robustness in NLP

Challenge:	Recent studies have revealed the vulnerability of dialogue state tracking models to distributional shifts, resulting in poor performance.
Approach:	They present a toolkit for standardized and comprehensive dialogue state tracking diagnoses that provides a richer summary of strengths and weaknesses.
Outcome:	The proposed toolkit shows that different classes of DST models have clear strengths and weaknesses, while generation models are more promising for handling language variety and span-based classification models are robust to unseen entities.

PEER: Pre-training ELECTRA Extended by Ranking (2023.findings-acl)

Copied to clipboard

Ru He, Wei Wang, Songfang Huang, Fei Huang

Through the Lens of History: Methods for Analyzing Temporal Variation in Content and Framing of State-run Chinese Newspapers (2025.naacl-long)

Challenge:	Existing models for pre-training require expensive pre-trainer computation cost . ELECTRA model can perform replaced token detection (RTD) task with reduced pre- training cost compared to current models .
Approach:	They propose to extend a discriminator-based replaced token detection task into a ranker-based task . they propose to use a binary classifier to perform a more precise task with negligible additional computation cost.
Outcome:	The proposed model outperforms state-of-the-art models with ELECTRA in GLUE tasks given the same cost.

Copied to clipboard

Shijia Liu, David A. Smith

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning (2022.naacl-main)

Challenge:	State-run newspapers are believed to strategically select and frame news articles to align with the shifting political tides of the country.
Approach:	They analyze more than 50 years of articles from the People's Daily and Reference News to quantify differences in content and framing over time.
Outcome:	The proposed methods show that the changes in name mentions and sentiment in news articles are more significant in People’s Daily than in Reference News .

Copied to clipboard

Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir Radev

Summarization

Challenge:	Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization.
Approach:	They propose a typology of factual errors to better understand hallucinations generated by current models and a contrastive fine-tuning strategy to improve the factual consistency and overall quality of summaries.
Outcome:	The proposed model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarizing datasets.

Graph Pre-training for AMR Parsing and Generation (2022.acl-long)

Copied to clipboard

Xuefeng Bai, Yulong Chen, Yue Zhang

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2022.acl-long)

Challenge:	Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure.
Approach:	They propose two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-tuning to improve structure awareness.
Outcome:	The proposed model is superior to pre-trained language models on AMR parsing and AMR-to-text generation tasks.

Copied to clipboard

Ori Yoran, Alon Talmor, Jonathan Berant

Template-free Prompt Tuning for Few-shot NER (2022.naacl-main)

Challenge:	Large pre-trained language models struggle in tasks that require reasoning . recent work shows that they struggle in performing symbolic reasoning operations without substantial amounts of additional data.
Approach:	They propose to leverage semi-structured tables and generate at scale question-paragraph pairs where answering the question requires reasoning over multiple facts in the paragraph.
Outcome:	The proposed model outperforms T5, a popular pre-trained encoder-decoder model, on reasoning-focused reading comprehension datasets.

Copied to clipboard

Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, Xuanjing Huang

Operation-guided Neural Networks for High Fidelity Data-To-Text Generation (D18-1)

Challenge:	Prompt-based methods have been successfully applied in few-shot learning tasks . however, when applied to token-level labeling tasks, it would be time-consuming to enumerate the template queries over all potential entity spans.
Approach:	They propose a method to reformulate NER tasks as LM problems without templates.
Outcome:	The proposed method is 30.12 times faster than the template-based method under few-shot settings.

Copied to clipboard

Feng Nie, Jinpeng Wang, Jin-Ge Yao, Rong Pan, Chin-Yew Lin

PELMS: Pre-training for Effective Low-Shot Multi-Document Summarization (2024.naacl-long)

Challenge:	Recent neural models for data-to-text generation generate descriptions that are not consistent with structured data.
Approach:	They propose a framework for data-to-text generation that uses symbolic operations to generate texts from structured data.
Outcome:	The proposed framework improves the fidelity of the generated texts to the input structured data.

Copied to clipboard

Joseph Peper, Wenzhao Qiu, Lu Wang

MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2022.acl-long)

Challenge:	Existing methods for abstractive multi-document summarization fail to generate concise, reflective summaries.
Approach:	They propose a pre-trained abstractive multi-document summarization model that uses unlabeled multi-doctoral inputs to generate concise, reflective summaries.
Outcome:	The proposed model outperforms competing models on a wide range of MDS datasets.

Copied to clipboard

Zhixing Tan, Xiangwen Zhang, Shuo Wang, Yang Liu

Towards the Detection of a Semantic Gap in the Chain of Commonsense Knowledge Triples (2022.lrec-1)

Challenge:	Prompting has been shown to be a promising approach for applying pre-trained language models to perform downstream tasks.
Approach:	They propose a method that divides the translation process into three stages using pre-trained language models.
Outcome:	The proposed method significantly improves translation performance of pre-trained language models on three translation tasks.

Copied to clipboard

Yoshihiko Hayashi

Uncovering Hidden Consequences of Pre-training Objectives in Sequence-to-Sequence Models (2023.findings-acl)

Challenge:	a commonsense knowledge resource organizes common sense that is not necessarily correct all the time, but most people are expected to know or believe.
Approach:	They propose a machine learning-based approach to detect semantic gaps in a commonsense knowledge graph . they use a conceptNet dataset to test the validity of two adjacent triples .
Outcome:	The proposed approach detects a semantic gap in a commonsense knowledge graph . the proposed approach also provides insights into the effectiveness of sense embeddings .

Copied to clipboard

Tannon Kew, Rico Sennrich

Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization (2023.findings-emnlp)

Challenge:	Existing studies have found that some pre-training objectives have negligible impact on downstream performance, but they are not effective on all models.
Approach:	They propose a zero-shot control method that is only successful on a subset of models and experiment with different pre-training objectives.
Outcome:	The proposed method is only successful on a subset of models and unexpected interactions between the pre-training method and downstream controllability are found.

Copied to clipboard

Weixiao Zhou, Gengyao Li, Xianfu Cheng, Xinnian Liang, Junnan Zhu, Feifei Zhai, Zhoujun Li

Tell Me How to Ask Again: Question Data Augmentation with Controllable Rewriting in Continuous Space (2020.emnlp-main)

Challenge:	Existing methods for dialogue summarization only apply to specific scenarios and domains.
Approach:	They propose a pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization.
Outcome:	The proposed model significantly outperforms state-of-the-art models on three dialogue summarization datasets from different scenarios and domains.

Copied to clipboard

Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Jiancheng Lv, Nan Duan, Ming Zhou

On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)

Challenge:	Existing data augmentation techniques for natural language processing tasks are difficult to design.
Approach:	They propose a controllable rewriting based question data augmentation method for machine reading comprehension, question generation and question-answering natural language inference tasks.
Outcome:	The proposed method generates high-quality, high-quality question data samples on machine reading comprehension, question generation, and question-answering natural language inference tasks.

Copied to clipboard

Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro

Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking (2022.emnlp-main)

Challenge:	Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work .
Approach:	They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data .
Outcome:	The proposed models can achieve competitive or better performance than BERT under comparable conditions.

Copied to clipboard

Mingyu Lee, Jun-Hyung Park, Junho Kim, Kang-Min Kim, SangKeun Lee

LVLM-Aware Multimodal Retrieval for RAG-Based Medical Diagnosis with General-Purpose Models (2026.findings-acl)

Challenge:	Masked language modeling (MLM) has been widely used for pre-training effective bidirectional representations but comes at a substantial training cost.
Approach:	They propose a concept-based curriculum masking method that evaluates the MLM difficulty of each token based on a carefully-designed linguistic difficulty criterion.
Outcome:	The proposed method significantly improves pre-training efficiency with the original BERT model at half the training cost.

Copied to clipboard

Nir Mazor, Tom Hope

IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning (2023.findings-acl)

Challenge:	Using retrieval augmentation, large vision language models can be used for diagnostic accuracy, but multimodal retrieval-augmented diagnosis is challenging.
Approach:	They propose a lightweight mechanism for enhancing diagnostic performance of retrieval-augmented LVLMs by fine-tuning a multimodal retriever and general-purpose backbone models.
Outcome:	The proposed mechanism achieves competitive results without medical training compared to pre-trained models with extensive training.

Copied to clipboard

Zihang Xu, Ziqing Yang, Yiming Cui, Shijin Wang

MCLF: A Multi-grained Contrastive Learning Framework for ASR-robust Spoken Language Understanding (2023.findings-emnlp)

Challenge:	Existing systems for logical reasoning have surpassed the average performance of humans in many tasks like SQuAD but there is still a long way to go when it comes to logical reasoning.
Approach:	They propose an InDicator-Oriented Logic Pre-training task which logically strengthens pre-trained models with the help of 6 types of logical indicators and a logicalally rich dataset.
Outcome:	The proposed task achieves state-of-the-art on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC.

Copied to clipboard

Zhiqi Huang, Dongsheng Chen, Zhihong Zhu, Xuxin Cheng

Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (2023.emnlp-main)

Challenge:	Trending ASR-robust SLU systems have seen impressive improvements through global contrastive learning, but they can easily lead to severe semantic changes.
Approach:	They propose a two-stage multi-grained contrastive learning framework to improve ASR robustness . they first adapt pre-trained language models to downstream SLU datasets and then fine-tune it on the corresponding dataset.
Outcome:	The proposed framework improves on four datasets and four BERT-like backbone models.

Copied to clipboard

Ahmed Alajrami, Katerina Margatina, Nikolaos Aletras

Effective Sequence-to-Sequence Dialogue State Tracking (2021.emnlp-main)

Challenge:	Pre-trained language models capture semantic and syntactic information, but no study has examined how information loss in input token characters affects their performance.
Approach:	They address this gap by pre-training language models using small subsets of token characters.
Outcome:	The proposed model retains 90% and 77% of the full-token model in standard NLU benchmarks and probing tasks even under extreme settings.

Copied to clipboard

Jeffrey Zhao, Mahdis Mahdieh, Ye Zhang, Yuan Cao, Yonghui Wu

Autoregressive Reasoning over Chains of Facts with Transformers (2020.coling-main)

Challenge:	Using Sequence-to-Sequence models for dialogue state tracking remains an understudied topic.
Approach:	They propose to use a pre-training objective and a dialogue context representation to investigate this problem.
Outcome:	The proposed model is more effective than auto-regressive language modeling, the authors show . the proposed model may have a hard time recovering from earlier mistakes, they say .

Copied to clipboard

Ruben Cartuyvels, Graham Spinks, Marie-Francine Moens

Learning to Rank Context for Named Entity Recognition Using a Synthetic Dataset (2023.emnlp-main)

Challenge:	Existing work on this task either evaluates facts in isolation or artificially limits the possible chains of facts, thus limiting multi-hop inference.
Approach:	They propose an iterative inference algorithm that decomposes the selection of facts from a corpus autoregressively and conditioning the next iteration on previously selected facts.
Outcome:	The proposed method outperforms the previous state-of-the-art in terms of precision, training time and inference efficiency on the WorldTree dataset.

Copied to clipboard

Arthur Amalvy, Vincent Labatut, Richard Dufour

Fine-Grained Legal Argument-Pair Extraction via Coarse-Grained Pre-training (2024.lrec-main)

Challenge:	Existing models for named entity recognition have limited range when applied to long documents . lack of supervision for such a task means one has to settle for unsupervised approaches.
Approach:	They propose to train a neural context retrieval model based on an instruction-tuned large language model.
Outcome:	The proposed method outperforms baselines on an English literary dataset . pre-trained transformer-based models can perform named entity recognition (NER) with great accuracy, but limited range when applied to long documents such as whole novels.

Copied to clipboard

Chaojun Xiao, Yutao Sun, Yuan Yao, Xu Han, Wenbin Zhang, Zhiyuan Liu, Maosong Sun

Improving Grammatical Error Correction via Contextual Data Augmentation (2024.findings-acl)

Challenge:	Current methods conceptualize LAE as a supervised sentence-pair classification problem and necessitate extensive manual annotations.
Approach:	They propose a model that focuses on fine-grained alignment of argument pairs building upon coarse-grain complaint-defense pairs.
Outcome:	The proposed model outperforms baseline models by 3.7 and 2.4 points on average.

Copied to clipboard

Yixuan Wang, Baoxin Wang, Yijun Liu, Qingfu Zhu, Dayong Wu, Wanxiang Che

Low-Resource NLP

Challenge:	Increasing use of synthetic data due to inconsistent error distribution and noisy labels is limiting the use of these data.
Approach:	They propose a method for augmentation of synthetic data with a more consistent error distribution.
Outcome:	The proposed method outperforms strong baselines and achieves state-of-the-art with only a few synthetic data.

DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning (2025.emnlp-main)

Copied to clipboard

Zhuoyuan Mao, Mengjie Zhao, Qiyu Wu, Hiromi Wakaki, Yuki Mitsufuji

Korean-Specific Dataset for Table Question Answering (2022.lrec-1)

Challenge:	Recent advances in music large language models have significantly improved music understanding tasks, but the potential of incorporating additional modalities such as images, videos and textual music features remains unexplored.
Approach:	They propose a multimodal music understanding LLM fine-tuned via multi-way instruction tuning with multi-ways aligned music, text, image, and video data.
Outcome:	The proposed model achieves state-of-the-art performance across six music understanding tasks and zero-shot scenarios.

Copied to clipboard

Changwook Jun, Jooyoung Choi, Myoseop Sim, Hyun Kim, Hansol Jang, Kyungkoo Min

Tree Transformer’s Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects (2024.acl-long)

Challenge:	Existing question answering systems mainly focus on text data, but few Korean datasets exist . a dataset for table question answering is written in English, but it lacks Korean-specific datasets .
Approach:	They construct Korean-specific datasets for table question answering using crowd-sourced workers . they then fine-tune the model with these datasets and report the evaluation results .
Outcome:	The proposed model is based on Korean datasets and is publicly available . the model is evaluated against other datasets from Korean question answering systems .

Copied to clipboard

Lingling Zhou, Suzan Verberne, Gijs Wijnholds

Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023.acl-long)

Challenge:	Prepositional phrase attachment ambiguity is structural in nature, while garden path constructions are incremental in nature.
Approach:	They pretrain and evaluate an unsupervised Transformer model that induces tree representations internally and compare it to a pretrained supervised BiLSTM model.
Outcome:	The Tree Transformer model induces tree representations internally, but its parsing ability is inferior to the supervised BiLSTM model, and it is not as sensitive to lexical cues as other large LSTM models.

Copied to clipboard

Artidoro Pagnoni, Alex Fabbri, Wojciech Kryscinski, Chien-Sheng Wu

BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification (2021.emnlp-main)

Challenge:	Existing methods to control document controllable summarization lack abundant labeled data.
Approach:	They propose a question-driven, unsupervised pretraining objective to improve controllability in document controllable summarization tasks.
Outcome:	The proposed method outperforms pre-finetuning approaches on QMSum and SQuALITY.

Copied to clipboard

Zeguan Xiao, Jiarun Wu, Qingliang Chen, Congjian Deng

Pre-training Language Models for Comparative Reasoning (2023.emnlp-main)

Challenge:	Existing approaches to Aspect-based sentiment classification ignore sequential features of context and lack syntactic knowledge of sentences.
Approach:	They propose a model which integrates sequential grammatical features from context and syntactic knowledge from dependency graphs to augment GCN to better encode dependency graph outputs.
Outcome:	The proposed model outperforms state-of-the-art models when equipped with contextual word embedding from pre-training language models.

Copied to clipboard

Mengxia Yu, Zhihan Zhang, Wenhao Yu, Meng Jiang

Pre-training Language Models with Deterministic Factual Knowledge (2022.emnlp-main)

Challenge:	Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions.
Approach:	They propose a framework to pre-train language models for enhancing comparative reasoning abilities . they collect scalable data for text-based entity comparison .
Outcome:	The proposed framework significantly improves comparative reasoning abilities under low-resource conditions on downstream tasks.

Copied to clipboard

Shaobo Li, Xiaoguang Li, Lifeng Shang, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, Qun Liu

Discourse Comprehension: A Question Answering Framework to Represent Sentence Connections (2022.emnlp-main)

Challenge:	Existing studies show that Pre-trained Language Models fail to capture factual knowledge robustly.
Approach:	They propose to let PLMs learn the deterministic relationship between context and masked content to improve their ability to capture factual knowledge.
Outcome:	The proposed methods improve accuracy and consistency of factual knowledge capturing and boost performance of other knowledge-intensive tasks.

Copied to clipboard

Wei-Jen Ko, Cutter Dalton, Mark Simmons, Eliza Fisher, Greg Durrett, Junyi Jessy Li

Question Answering

Challenge:	Existing systems for text comprehension are inadequate for more holistic comprehension of a discourse.
Approach:	They propose a new paradigm that captures both discourse and semantic links between sentences in the form of free-form, open-ended questions.
Outcome:	The proposed model captures discourse and semantic links between sentences in the form of free-form, open-ended questions.

IndoNLI: A Natural Language Inference Dataset for Indonesian (2021.emnlp-main)

Copied to clipboard

Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, Clara Vania

LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding (2023.acl-long)

Challenge:	XLM-R model outperforms other pre-trained models in annotated data.
Approach:	They adapt the data collection protocol for MNLI and collect 18K sentence pairs annotated by crowd workers and experts.
Outcome:	The proposed dataset outperforms other pre-trained models on the expert-annotated data.

Copied to clipboard

Yi Tu, Ya Guo, Huan Chen, Jinyang Tang

Nearest Neighbor Machine Translation is Meta-Optimizer on Output Projection Layer (2023.emnlp-main)

Challenge:	Pre-trained models on document images with transformer-based backbones have led to significant performance gains in this field.
Approach:	They propose a multi-modal pre-training model that combines text, layout and image . they propose to use local 1D position instead of global 1D positions as layout input .
Outcome:	The proposed model can achieve state-of-the-art results on a wide variety of VrDU problems.

Copied to clipboard

Ruize Gao, Zhirui Zhang, Yichao Du, Lemao Liu, Rui Wang

mForms : Multimodal Form Filling with Question Answering (2024.lrec-main)

Challenge:	Nearest Neighbor Machine Translation (kNN-MT) is a powerful domain adaptation tool . the reasons for its success have not been thoroughly investigated .
Approach:	They propose to integrate pre-trained Neural Machine Translation models with token-level retrieval . they propose to implicitly execute gradient descent on the output projection layer of NMT .
Outcome:	The proposed approach outperforms model fine-tuning on in-domain tests while achieving better performance on out-of-domain sets.

Copied to clipboard

Larry Heck, Simon Heck, Anirudh S. Sundar

Text-Guided Multi-Scale Frequency Representation Adaptation (2026.acl-long)

Challenge:	The paper presents a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA).
Approach:	They propose a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA) the paper introduces a multimodal form-filled dataset and an extension of the popular ATIS dataset to support future research and experimentation.
Outcome:	The proposed approach maintains robust accuracy for sparse training conditions and achieves state-of-the-art F1 of 0.97 on ATIS with approximately 1/10th the training data.

Copied to clipboard

Weicai Yan, Xinhua Ma, Wang Lin, Tao Jin

LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers (2025.findings-acl)

Challenge:	Existing methods for fine-tuning visual signals are limited by their size and complexity.
Approach:	They propose a multi-scale frequency-based fine-tuning method that integrates textual information and performs multi-level fine- tuning of visual signals in the frequency domain.
Outcome:	Extensive experiments on multimodal models, including CLIP and LLaVA, demonstrate that the proposed method significantly improves performance and efficiency with minimal cost and fast convergence within one epoch.

Copied to clipboard

Shantanu Ghosh, Rayan Syed, Chenyu Wang, Vaibhav Choudhary, Binxu Li, Clare B Poynton, Shyam Visweswaran, Kayhan Batmanghelich

Pre-Training Curriculum for Multi-Token Prediction in Language Models (2025.acl-long)

Challenge:	Current slice discovery methods in computer vision rely on converting input images into sets of attributes and testing hypotheses about configurations of pre-computed attributes associated with elevated error patterns.
Approach:	They propose a method to identify systematic biases in the mistakes of pre-trained vision models by converting input images into sets of attributes and testing hypotheses about configurations of these attributes.
Outcome:	The proposed method outperforms existing methods on 3 natural and 3 medical imaging datasets and generates pseudo-labels for each identified bias.

Copied to clipboard

Ansar Aynetdinov, Alan Akbik

SciMRC: Multi-perspective Scientific Machine Reading Comprehension (2024.lrec-main)

Challenge:	Multi-token prediction (MTP) is a pre-training objective for language models . prior work has shown that smaller language models struggle with the MTP objective .
Approach:	They propose a curriculum learning strategy that uses multiple prediction heads to predict the next tokens at each prediction step.
Outcome:	The proposed curriculum improves performance and output quality while retaining the benefits of self-speculative decoding.

Copied to clipboard

Xiao Zhang, Heqi Zheng, Yuxiang Nie, Heyan Huang, Xian-Ling Mao

Machine Reading Comprehension

Challenge:	Existing datasets focused on single-perspective question-answer pairs overlooking inherent variation in comprehension levels among different readers.
Approach:	They propose a multi-perspective scientific machine reading comprehension dataset . their dataset comprises 741 scientific papers and 6,057 question-answer pairs .
Outcome:	The proposed dataset includes questions from beginners, students, and experts.

Stance Reasoner: Zero-Shot Stance Detection on Social Media with Explicit Reasoning (2024.lrec-main)

Copied to clipboard

Maksym Taranukhin, Vered Shwartz, Evangelos Milios

Unveiling the Potential of BERT-family: A New Recipe for Building Scalable, General and Competitive Large Language Models (2025.acl-long)

Challenge:	Stance Reasoner is a model for zero-shot stance detection on social media platforms that can be used to extract opinions from opinionated content.
Approach:	They propose a method that leverages explicit reasoning over background knowledge to guide the model’s inference about the document’s stance on a target.
Outcome:	The proposed model outperforms the current state-of-the-art models on 3 Twitter datasets, including fully supervised models.

Copied to clipboard

Yisheng Xiao, Juntao Li, Wenpeng Hu, Zhunchen Luo, Min Zhang

Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations (2025.emnlp-main)

Challenge:	Generative large language models (LLMs) have significantly influenced various aspects of society, reshaping how we access and interact with information and knowledge.
Approach:	They propose a pre-training task that helps BERT-family excel in wider applications . they also explore the integration of cutting-edge technologies into their models to further enhance their capabilities.
Outcome:	The proposed model exhibits performance levels comparable to current SOTA LLMs across a spectrum of tasks.

Copied to clipboard

Linyang He, Qiaolin Wang, Xilin Jiang, Nima Mesgarani