Papers with NLU
CogKTR: A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding (2022.emnlp-demos)
Copied to clipboard
Zhuoran Jin, Tianyi Men, Hongbang Yuan, Yuyang Zhou, Pengfei Cao, Yubo Chen, Zhipeng Xue, Kang Liu, Jun Zhao
| Challenge: | Existing knowledge-enhanced methods are limited to knowledge-intensive tasks. |
| Approach: | They propose a knowledge-enhanced text representation toolkit for natural language understanding . it combines knowledge acquisition, knowledge representation, knowledge injection and knowledge application . |
| Outcome: | The proposed toolkit supports knowledge acquisition, knowledge representation, knowledge injection, and knowledge application. |
Evaluating Cross-Lingual Transfer Learning Approaches in Multilingual Conversational Agent Models (2020.coling-industry)
Copied to clipboard
| Challenge: | Existing voice assistant models are developed for each region or language, requiring linear effort to develop and maintain. |
| Approach: | They propose a general multilingual model framework for natural language understanding models . they show multilingual models can reach same or better performance compared to monolingual models a . |
| Outcome: | The proposed model framework can bootstrap new language models faster and reduce effort . it can reach same or better performance compared to monolingual models across language-specific test data . |
Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on improving cross-lingual transferability of NMT model is under-explored. |
| Approach: | They propose a model that leverages a multilingual pretrained encoder to improve cross-lingual transferability. |
| Outcome: | The proposed model outperforms mBART and m2m-100 on a zero-shot cross-lingual transfer task. |
Event-Centric Natural Language Processing (2021.acl-tutorials)
Copied to clipboard
| Challenge: | This tutorial will provide an introduction to various methods for automating the extraction, conceptualization and prediction of events and their relations. |
| Approach: | This tutorial will provide an introduction to various methods for automating events and their relations, and a wide range of NLU and commonsense understanding tasks. |
| Outcome: | This tutorial will provide an introduction to various methods for automating extraction, conceptualization and prediction of events and their relations, and a wide range of NLU and commonsense understanding tasks. |
Identifying and Resolving Annotation Changes for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Annotation conflict resolution is crucial for machine learning, says a new study . past work on annotation conflict resolution assumed data is collected at once . a a supervised neural model can resolve conflicts in data annotation but requires access to high-quality data . |
| Approach: | They propose an approach to resolve annotation conflicts in a real-world context using a German dialog system. |
| Outcome: | The proposed approach improves on a real-world dataset with 3.5M utterances in German. |
AutoNLU: An On-demand Cloud-based Natural Language Understanding System for Enterprises (2020.aacl-demo)
Copied to clipboard
| Challenge: | AutoNLU is an on-demand cloud-based system that enables users to create and edit datasets and train and test different state-of-the-art NLU models. |
| Approach: | They introduce an on-demand cloud-based system that provides an easy-to-use interface . they build powerful keyphrase extraction models that achieve state-of-the-art results . |
| Outcome: | The proposed model achieves state-of-the-art on two public benchmarks and is easy to use and use. |
Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing work proposes dialect adaptation for encoder models or encoder-decoder models. |
| Approach: | They propose to use MD-3 to combine task adapters and dialect adapters to decoder models using a masked word game-playing conversation. |
| Outcome: | The proposed architecture outperforms baselines on Indian English and Nigerian English on a masked conversation with two models. |
Optimizing NLU Reranking Using Entity Resolution Signals in Multi-domain Dialog Systems (2021.naacl-industry)
Copied to clipboard
Tong Wang, Jiangning Chen, Mohsen Malmir, Shuyan Dong, Xin He, Han Wang, Chengwei Su, Yue Liu, Yang Liu
| Challenge: | In dialog systems, the Natural Language Understanding component makes the interpretation decision before the mentioned entities are resolved. |
| Approach: | They propose to leverage Entity Resolution (ER) features in NLU reranking to learn model weights . they propose a score distribution matching method to ensure the models are calibrated . |
| Outcome: | The proposed approach outperforms the baseline model on multiple domain evaluations. |
Input-specific Attention Subnetworks for Adversarial Detection (2022.findings-acl)
Copied to clipboard
| Challenge: | a new method to prune attention heads is proposed for adversarial detection . attention heads in models such as BERT are over-provisioned and can be pruned . |
| Approach: | They propose a method to construct input-specific attention subnetworks from which three features are extracted to discriminate between authentic and adversarial inputs. |
| Outcome: | The proposed method significantly improves state-of-the-art adversarial detection accuracy on 10 NLU datasets with 11 different adversarials. |
Crowd-sourcing annotation of complex NLU tasks: A case study of argumentative content annotation (D19-59)
Copied to clipboard
| Challenge: | Recent advances in machine reading and listening comprehension involve the annotation of long texts. |
| Approach: | They propose a way to perform a sentence-by-sentence annotation task with crowd annotators. |
| Outcome: | The proposed approach can be used to identify claims in a debate speech. |
Flowstorm: Open-Source Platform with Hybrid Dialogue Architecture (2022.naacl-demo)
Copied to clipboard
| Challenge: | Flowstorm is an open-source conversational AI platform that can be used as a web app. |
| Approach: | They propose a conversational AI platform called Flowstorm that uses a combination of tree structures and generative models to handle specific dialogue scenarios. |
| Outcome: | The proposed platform can be used as a web app or run on their own . it uses tree structures and generative models to create and analyze conversational applications . |
Semantic Diversity for Natural Language Understanding Evaluation in Dialog Systems (2020.coling-industry)
Copied to clipboard
| Challenge: | a dialog system is used to evaluate NLU models using aggregated metrics on a large number of utterances. |
| Approach: | They propose a method to generate a test set with high semantic diversity for NLU evaluation in dialog systems. |
| Outcome: | The proposed test sets are based on high diversity of utterances from different regions of the utteration embedding space. |
Don’t Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive Bayes With Attention (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) are becoming increasingly popular in business scenarios, but maintaining topic continuity is a challenge. |
| Approach: | They propose a topic continuity model that assesses whether a response aligns with the initial conversation topic using a Naive Bayes approach. |
| Outcome: | The proposed model outperforms existing models in handling lengthy and complex conversations. |
Primacy Effect of ChatGPT (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing machine learning models may lead to poor performance in discriminative natural language understanding tasks. |
| Approach: | They propose to use ChatGPT to query large amounts of human-written text to find the answer to a question. |
| Outcome: | The proposed model has a high chance to select labels at earlier positions as the answer. |
Continuous Model Improvement for Language Understanding with Machine Translation (2021.naacl-industry)
Copied to clipboard
Abdalghani Abujabal, Claudio Delli Bovi, Sungho Ryu, Turan Gojayev, Fabian Triefenbach, Yannick Versley
| Challenge: | a simple translation-test approach would fail the latency requirements of a live environment. |
| Approach: | They show that annotating unlabeled utterances offline can improve performance . they demonstrate that an extrinsic evaluation can improve the performance if manual data is available . |
| Outcome: | The proposed method improves performance in an extrinsic evaluation setting with real-world commercial dialog system in german. |
Machine Reading, Fast and Slow: When Do Models “Understand” Language? (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models of reading comprehension score highly on NLU benchmarks, but they are often 'read fast', i.e. rely on shallow patterns. |
| Approach: | They propose a definition for the reasoning steps expected from a system that would be 'reading slowly' they compare that behavior with five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. |
| Outcome: | The proposed model is compared with five models of the BERT family of various sizes, and compared using saliency scores and counterfactual explanations. |
P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022.acl-short)
Copied to clipboard
| Challenge: | Existing methods of prompt tuning cannot handle hard sequence labeling tasks. |
| Approach: | They propose to optimize prompt tuning to tune continuous prompts with a frozen language model. |
| Outcome: | The proposed method matches finetuning with prompt tuning while having only 0.1%-3% tuned parameters. |
Learning from Contrastive Prompts: An Automated Prompt Optimization Framework (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt optimization methods often underperform due to learning exclusively from incorrect samples. |
| Approach: | They propose a framework that leverages contrastive prompts to distinguish between high- and low-performing cases. |
| Outcome: | The proposed framework can generalize across open and proprietary models and NLU benchmarks. |
Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking (2022.starsem-1)
Copied to clipboard
Ronen Tamari, Kyle Richardson, Noam Kahlon, Aviad Sar-shalom, Nelson F. Liu, Reut Tsarfaty, Dafna Shahaf
| Challenge: | Controlled synthetic tasks are an important resource for diagnosing model behavior. |
| Approach: | They propose a framework that provides fine-grained control over task generation in bAbI. |
| Outcome: | The proposed framework provides fine-grained control over task generation in the bAbI benchmark. |
Delexicalized Paraphrase Generation (2020.coling-industry)
Copied to clipboard
| Challenge: | Using convolutional neural networks, we generate delexicalized sentences . 1.29% accuracy is achieved with the generated paraphrases . |
| Approach: | They propose a neural paraphrasing model that generates delexicalized sentences . they use convolutional neural networks to pool on slot values and use pointers to locate them . |
| Outcome: | The proposed model generates delexicalized sentences with high quality . it can be used for intent classification and named entity recognition tasks . |
Entity resolution for noisy ASR transcripts (D19-3)
Copied to clipboard
| Challenge: | Domain-agnostic Automatic Speech Recognition systems often mistranscribe domain-specific words and phrases. |
| Approach: | They propose a method for handling ASR errors in named entities, specifically person names, for a voice-based collaboration assistant. |
| Outcome: | The proposed method improves accuracy by 40.8% on a voice-based collaboration assistant. |
Efficient Semi-supervised Consistency Training for Natural Language Understanding (2022.naacl-industry)
Copied to clipboard
| Challenge: | Manually labeled training data is expensive, noisy, and often scarce . semi-supervised learning methods can be used to improve model performance . |
| Approach: | They explore different methods for consistency training on unlabeled data . they use human paraphrasing, back-translation, and dropout to augment unlabed data. |
| Outcome: | The proposed methods outperform purely supervised learning on unlabeled data. |
Multi-task Learning of Spoken Language Understanding by Integrating N-Best Hypotheses with Hierarchical Attention (2020.coling-industry)
Copied to clipboard
| Challenge: | Existing methods to integrate hypotheses into speech recognition systems are noisy and can cause information loss. |
| Approach: | They propose to integrate hypotheses into multi-task learning and transfer learning to improve performance. |
| Outcome: | The proposed model improves domain and intent classification by 19% and 37% compared to current methods . the proposed model could recover transcription and rewrite the query for a better understanding . |
Active Learning for New Domains in Natural Language Understanding (N19-2)
Copied to clipboard
| Challenge: | Existing approaches to improve the accuracy of new domains are lacking annotated live utterances. |
| Approach: | They propose an algorithm called Majority-CRF that uses an ensemble of classification models to guide the selection of relevant utterances and a sequence labeling model to prioritize informative examples. |
| Outcome: | The proposed algorithm achieves 6.6%-9% error rate reduction and statistically significant improvements on six new domains. |
Exploiting In-Domain Bilingual Corpora for Zero-Shot Transfer Learning in NLU of Intra-Sentential Code-Switching Chatbot Interactions (2022.emnlp-industry)
Copied to clipboard
Maia Aguirre, Manex Serras, Laura García-sardiña, Jacobo López-fernández, Ariane Méndez, Arantza Del Pozo
| Challenge: | Multilingual speakers outnumber monolingual speakers in the world . CS is a frequent habit in both spoken and written informal communications . |
| Approach: | They evaluate the efficacy of cross-lingual transfer learning with mBERT for NLU on a Basque-Spanish CS chatbot corpus. |
| Outcome: | The proposed model outperforms models trained on Basque and Spanish without CS on a basque-Spanish chatbot corpus. |
Infusing Finetuning with Semantic Dependencies (2021.tacl-1)
Copied to clipboard
| Challenge: | Several diagnostics help to localize the benefits of our approach. |
| Approach: | They apply convolutional graph encoders to integrate semantic parses into task-specific finetuning. |
| Outcome: | The proposed approach yields benefits to natural language understanding (NLU) tasks in the GLUE benchmark. |
VLUE: A New Benchmark and Multi-task Knowledge Transfer Learning for Vietnamese Natural Language Understanding (2024.findings-naacl)
Copied to clipboard
| Challenge: | a lack of standard evaluation metrics and benchmarks makes it difficult to identify strengths of Vietnamese NLP models. |
| Approach: | They propose to establish a standardized set of benchmarks for Vietnamese NLU . they propose to evaluate Vietnamese language understanding models using a pre-trained model . |
| Outcome: | The proposed model combines proficiency of a multilingual pre-trained model with Vietnamese linguistic knowledge. |
Unsupervised training data re-weighting for natural language understanding with local distribution approximation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | a distribution mismatch between offline training and live data can cause biases . cyclic seasonality shifts, and changing pool of users can contribute to this problem . |
| Approach: | They propose an unsupervised approach to mitigate offline training data sampling bias . they propose a local distribution approximation in the pre-trained embedding space . |
| Outcome: | The proposed approach mitigates the offline training data sampling bias in multiple NLU tasks without additional annotation. |
Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing curriculum learning approaches rely on manually defined difficulty metrics which may not accurately reflect the model’s own perspective. |
| Approach: | They propose a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) they evaluate four datasets covering binary and multi-class classification tasks. |
| Outcome: | The proposed model leads to faster convergence and improved performance compared to standard random sampling. |
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020.acl-demos)
Copied to clipboard
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, Samuel R. Bowman
| Challenge: | jiant is an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Approach: | They introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. |
| Outcome: | The proposed toolkit reproduces published performance on GLUE and SuperGLUE tasks. |
Extreme Model Compression for On-device Natural Language Understanding (2020.coling-industry)
Copied to clipboard
| Challenge: | Xu and Sarikaya et al., 2014) perform word-embedding compression with NLU task learning . their approach achieves a compression rate of 97.4% with less than 3.7% degradation in predictive performance. |
| Approach: | They propose a task-aware, end-to-end compression approach that performs word-embedding compression with NLU task learning. |
| Outcome: | The proposed approach outperforms baselines and achieves 97.4% compression rate with less than 3.7% degradation in predictive performance. |
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020.acl-demos)
Copied to clipboard
Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao
| Challenge: | MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models . |
| Approach: | They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. |
| Outcome: | The proposed model can significantly compress a large model without significant performance drop. |
Selecting Machine-Translated Data for Quick Bootstrapping of a Natural Language Understanding System (N18-3)
Copied to clipboard
| Challenge: | In recent years, there has been growing interest in voice-controlled devices, such as Amazon Alexa or Google home. |
| Approach: | They investigate the use of Machine Translation to bootstrap a natural language understanding system for a new language for the use case of a large-scale voice-controlled device. |
| Outcome: | The proposed method reduces the time and cost of getting annotated corpus for a new language while still providing a large enough coverage of user requests. |
InterpreT: An Interactive Visualization Tool for Interpreting Transformers (2021.eacl-demos)
Copied to clipboard
Vasudev Lal, Arden Ma, Estelle Aflalo, Phillip Howard, Ana Simoes, Daniel Korat, Oren Pereg, Gadi Singer, Moshe Wasserblat
| Challenge: | Using Transformer-based models for NLU/NLP tasks is a growing interest . but there are many open questions regarding the behavior of these models . |
| Approach: | They present an interactive visualization tool for interpreting Transformer-based models. |
| Outcome: | The tool can track and visualize token embeddings through each layer of a Transformer, highlight distances between certain token embeds, and identify task-related functions of attention heads using new metrics. |
Graph-Based Semi-Supervised Learning for Natural Language Understanding (D19-53)
Copied to clipboard
| Challenge: | Semi-supervised learning is an efficient method to augment training data from unlabeled data. |
| Approach: | They propose semi-supervised learning models and their inductive variants for NLU and use them to find similar utterances and construct a graph. |
| Outcome: | The proposed model improves the error rate of the model by 5% using publicly available NLU data and models. |
CRUISE: Cold-Start New Skill Development via Iterative Utterance Generation (P18-4)
Copied to clipboard
| Challenge: | Existing systems require developers to manually generate and annotate a large number of utterances. |
| Approach: | They propose a system that guides ordinary software developers to build a high quality NLU engine from scratch. |
| Outcome: | The proposed system shows that iterative pruning of incorrect utterances reduces human workload and cognitive load. |
BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems (2022.emnlp-demos)
Copied to clipboard
| Challenge: | evaluating and troubleshooting production TOD systems is still a largely manual process requiring large amount of human conversations with the systems. |
| Approach: | They propose a data-efficient end-to-end Bot SIMulation framework for commercial task-oriented dialog systems that can generate user queries and generate semantic-level dialog acts and entities from bot definitions. |
| Outcome: | The proposed framework is able to infer semantic-level dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing. |
Adaptive Natural Language Generation for Task-oriented Dialogue via Reinforcement Learning (2022.coling-1)
Copied to clipboard
| Challenge: | In task-oriented dialogue systems, the role of the natural language generation component is to convert a system's intentions, called dialogue acts (DAs), into natural language utterances and to convey DAs accurately to users. |
| Approach: | They propose a method for Adaptive Natural language generation for Task-Oriented dialogue via Reinforcement learning that incorporates a natural language understanding module into the objective function of RL. |
| Outcome: | The proposed method generates adaptive utterances against speech recognition errors and the different vocabulary levels of users in a multi-world task-oriented dialogue system. |
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing studies on the performance of pre-trained language models on natural language understanding tasks have focused on the natural language inference and textual entailment tasks. |
| Approach: | They propose to use corrupted data to fine-tune pre-trained language models to assess their language understanding capabilities. |
| Outcome: | The proposed transformations can be applied to all but one NLU task and show that understanding the meaning of utterances is not required for high performance. |
Iterative Stratified Testing and Measurement for Automated Model Updates (2022.emnlp-industry)
Copied to clipboard
Elizabeth Dekeyser, Nicholas Comment, Shermin Pei, Rajat Kumar, Shruti Rai, Fengtao Wu, Lisa Haverty, Kanna Shimizu
| Challenge: | Automating updates to machine learning systems is an important but understudied challenge in AutoML. |
| Approach: | They propose a framework that relies on iterative model building coupled with data-shape stratified model testing and improvement to improve model accuracy. |
| Outcome: | The proposed framework shows a 26% improvement in accuracy for new model use cases on a large-scale NLU system compared to a naive baseline and current cutting-edge methods. |
Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pretrained language models such as masked language model (MLM) have brought significant improvements to many NLU and NLG tasks. |
| Approach: | They propose a probabilistic masking scheme for the masked language model and a model with a uniform prior distribution on the masking ratio. |
| Outcome: | The proposed model outperforms BERT on a bunch of downstream NLG tasks. |
Lightweight Transformers for Conversational AI (2022.naacl-industry)
Copied to clipboard
| Challenge: | Commercial dialogue systems typically require a small footprint and fast execution time, but recent trends are in the other direction, resulting in difficulties in model deployment. |
| Approach: | They build Transformer-based Language Models from scratch on large corpora of conversational data and compare their performance against BERT and other strong baselines on dialogue probing tasks. |
| Outcome: | The proposed model outperforms existing models on dialogue probing tasks and can be fine-tuned on a single consumer GPU card. |
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pretraining frameworks do not perform well for all tasks of three main categories, such as natural language understanding (NLU), unconditional generation, and conditional generation. |
| Approach: | They propose a general language model based on autoregressive blank infilling to address this challenge. |
| Outcome: | The proposed model outperforms BERT, T5, and GPT on a wide range of tasks across NLU, conditional and unconditional generation tasks. |
Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | a number of languages are processed incrementally, but the best ones do not . we test five models on various datasets and compare their performance using three incremental evaluation metrics. |
| Approach: | They investigate how bidirectional LSTMs and Transformers behave under incremental interfaces . they propose to use bidirectional encoders in incremental mode while retaining non-incremental quality . |
| Outcome: | The proposed models perform better under incremental interfaces than the "omni-directional" BERT model, which achieves better non-incremental performance, but is impacted more by the incremental access. |
Semi-supervised Adversarial Text Generation based on Seq2Seq models (2022.emnlp-industry)
Copied to clipboard
| Challenge: | In contrast, adversarial training has been used in computer vision to improve models’ robustness due to the discrete nature of text. |
| Approach: | They propose a way to generate adversarial samples by using pseudo-labeled in-domain text data to train a seq2seq model for adversarials and combine it with paraphrase detection. |
| Outcome: | The proposed model generates realistic and relevant adversarial samples compared to other state-of-the-art models and recovers up to 70% of errors. |
Developing Production-Level Conversational Interfaces with Shallow Semantic Parsing (D18-2)
Copied to clipboard
| Challenge: | In this demo, we demonstrate an end-to-end approach for building conversational interfaces from prototype to production. |
| Approach: | They propose an end-to-end approach for building conversational interfaces from prototype to production that leverages shallow semantic parsing. |
| Outcome: | The proposed approach has proven to work well for a number of applications across diverse verticals. |
Multilingual Argument Mining: Datasets and Analysis (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Argument mining tasks in non-English languages are dominated by English . we use a pre-trained language model that supports 104 languages to train models . |
| Approach: | They propose a multilingual BERT model to address argument mining tasks in non-English languages . they use English datasets and machine translation to facilitate transfer learning . |
| Outcome: | The proposed model is well suited for classifying the stance of arguments and detecting evidence, but less so for assessing the quality of arguments. |
PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English (2023.acl-short)
Copied to clipboard
| Challenge: | Existing efforts to understand privacy policies are limited by processing the language in a way exclusive to a single task focusing on certain privacy practices. |
| Approach: | They propose a privacy policy language understanding evaluation benchmark to evaluate the understanding of privacy policies across multiple tasks. |
| Outcome: | The proposed framework improves the understanding of privacy policies across multiple tasks. |
Robust Natural Language Understanding with Residual Attention Debiasing (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing ensemble-based debiasing methods do not address unintended dataset biases . attention plays a crucial role in providing robust prediction in NLU models . |
| Approach: | They propose an end-to-end debiasing method that mitigates unintended biases from attention. |
| Outcome: | The proposed method improves the OOD performance of BERT-based models on three benchmarks. |
NepBERTa: Nepali Language Model Trained in a Large Corpus (2022.aacl-short)
Copied to clipboard
| Challenge: | Nepali is a low-resource language with more than 40 million speakers worldwide. |
| Approach: | They present a BERT-based natural language understanding model trained on the most extensive monolingual Nepali corpus ever. |
| Outcome: | The proposed model performs well in Nepali-specific NLP tasks including Named-Entity Recognition, Content Classification, POS Tagging, and Sequence Pair Similarity. |
Local Structure Matters Most in Most Languages (2022.aacl-short)
Copied to clipboard
| Challenge: | Recent perturbation studies have found unintuitive results on what does and does not matter when performing Natural Language Understanding (NLU) tasks in English. |
| Approach: | They replicate a study on the importance of local structure and relative unimportance of global structure in a multilingual setting. |
| Outcome: | The proposed model replicates a study on the importance of local structure and relative unimportance of global structure in a multilingual setting. |
Combining Weakly Supervised ML Techniques for Low-Resource NLU (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have improved the performance of virtual assistants . however, meager training data is often a key bottleneck in creating voice-enabled applications . |
| Approach: | They propose to use unsupervised and semi-supervised techniques to improve NLU accuracy . they incorporate anonymized, unlabeled and automatically transcribed user utterances into training . |
| Outcome: | The proposed methods improve NLU accuracy in low-resource settings by integrating unsupervised and SSL techniques. |
GLGE: A New General Language Generation Evaluation Benchmark (2021.findings-acl)
Copied to clipboard
Dayiheng Liu, Yu Yan, Yeyun Gong, Weizhen Qi, Hang Zhang, Jian Jiao, Weizhu Chen, Jie Fu, Linjun Shou, Ming Gong, Pengcheng Wang, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, Ruofei Zhang, Winnie Wu, Ming Zhou, Nan Duan
| Challenge: | Multi-task benchmarks focus on a range of Natural Language Understanding (NLU) tasks without considering the Natural Language Generation (NLG) models. |
| Approach: | They propose a multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. |
| Outcome: | The proposed benchmarks are based on GLUE and Su-perGLUE for English and several other languages. |
Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values (2023.acl-srw)
Copied to clipboard
| Challenge: | Large language models (LMs) have been shown to be highly effective for identifying harmful training instances, but dataset size and model complexity constraints limit the ability to apply Shapley-based data valuation to fine-tuning large pre-trained language models. |
| Approach: | They propose an algorithm that aggregates Shapley values from subsets for valuation of entire training set and a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model. |
| Outcome: | The proposed method outperforms existing methods on benchmark datasets and can filter fine-tuning data to increase language model performance compared to training with the full fine-uning dataset. |
Automatic Derivation of Semantic Representations for Thai Serial Verb Constructions: A Grammar-Based Approach (2024.acl-srw)
Copied to clipboard
| Challenge: | Using rich semantic representations for Thai Serial Verb Constructions (SVCs) is time-consuming and manual annotation is preferred. |
| Approach: | They propose to implement an HPSG analysis for Thai Serial Verb Constructions (SVCs) they use a DELPH-IN computational grammar to generate appropriate representations from syntactic features. |
| Outcome: | The proposed grammar increases verified coverage of Thai SVCs by 73% and decreases ambiguity by 46% on held-out data. |
AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models such as BERT have shown great power in natural language understanding . fine-grained tokenizations have advantages and disadvantages for learning of pre-tried models . |
| Approach: | They propose a pretrained language model based on both fine-grained and coarse-grain tokenizations . they propose to use both tokenization techniques to learn pre-trained models . |
| Outcome: | The proposed model outperforms BERT on benchmark datasets for Chinese and English . it can perform better with the same computational cost as BERT, the authors show . |
FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding (2022.acl-long)
Copied to clipboard
Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Chonghua Liao, Li Jian, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, Zhilin Yang
| Challenge: | Existing evaluation protocols for few-shot natural language understanding (NLU) tasks are inconsistent and hinder fair comparison and measuring progress. |
| Approach: | They propose an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. |
| Outcome: | The proposed framework improves evaluation procedures in three key aspects, i.e., performance, dev-test correlation, and stability. |
KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmark datasets for natural language inference and semantic textual similarity (STS) are not available in the Korean language. |
| Approach: | They construct and release new datasets for Korean NLI and STS . they machine-translate existing English training sets and manually translate development and test sets into Korean to accelerate research on Korean NLU. |
| Outcome: | The proposed datasets are available at https://github.com/kakaobrain/KorNLUDatasets. |
Industry Scale Semi-Supervised Learning for Natural Language Understanding (2021.naacl-industry)
Copied to clipboard
| Challenge: | Obtaining human annotation is expensive and time-consuming process. |
| Approach: | They propose a semi-supervised learning pipeline which leverages millions of unlabeled examples to improve natural language understanding tasks. |
| Outcome: | The proposed pipeline can be used to improve natural language understanding tasks. |
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for incorporating knowledge from multiple tasks suffer from catastrophic forgetting and difficulties in dataset balancing. |
| Approach: | They propose an algorithm that extracts and combine adapters in a knowledge composition step. |
| Outcome: | The proposed class outperforms traditional methods such as full fine-tuning and multi-task learning on 16 diverse NLU tasks. |
To Chat or Task: a Multi-turn Dialogue Generation Framework for Task-Oriented Dialogue Systems (2025.acl-industry)
Copied to clipboard
| Challenge: | Large language models (LLMs) are designed to handle complex task requests, but lack of specific datasets for training and evaluation of such systems . |
| Approach: | They propose a framework to generate a dataset for in-vehicle speech recognition systems . they train an in-car context sensor that correctly identifies the functional intent of the driver . |
| Outcome: | The proposed framework outperforms baseline models across experimental settings. |
PromptDA: Label-guided Data Augmentation for Prompt-based Few Shot Learners (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on prompt-based few-shot tuning focus on deriving proper label words with a verbalizer or generating prompt templates to elicit semantics from PLMs. |
| Approach: | They propose a framework that leverages label semantics for prompt-based tuning. |
| Outcome: | The proposed framework improves on few-shot text classification tasks by leveraging label semantics and data augmentation. |
ABEX: Data Augmentation for Low-Resource NLU via Expanding Abstract Descriptions (2024.acl-long)
Copied to clipboard
Sreyan Ghosh, Utkarsh Tyagi, Sonal Kumar, Chandra Kiran Evuru, Ramaneswaran S, S Sakshi, Dinesh Manocha
| Challenge: | ABEX is a novel and effective generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks. |
| Approach: | They propose a novel generative data augmentation methodology for low-resource Natural Language Understanding (NLU) tasks based on a paradigm for generating diverse forms of an input document . |
| Outcome: | The proposed method outperforms all baselines qualitatively with improvements of 0.04% - 38.8%. |
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2024.acl-long)
Copied to clipboard
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, Madian Khabsa
| Challenge: | Existing benchmarks for text comprehension only cover 30 languages, but lack of labeled data is a major obstacle to building functional systems in most languages. |
| Approach: | They present a multiple-choice machine reading comprehension dataset spanning 122 languages . they use it to evaluate the capabilities of multilingual masked language models and large language models . |
| Outcome: | The proposed dataset enables the evaluation of text models in high-, medium- and low-resource languages. |
SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Slot filling is a key subtask in spoken language understanding (SLU) . recent advent of speech-based large language models has opened new avenues for speech understanding . |
| Approach: | They propose to improve slot-filling task by creating an empirical upper bound for the task . they propose to use a speech-based large language model to integrate speech and text modalities . |
| Outcome: | The proposed model improves slot filling performance while reducing generalization gaps. |
Linking artificial and human neural representations of language (D19-1)
Copied to clipboard
| Challenge: | a pre-trained BERT architecture is used to fine-tune sentence encoding models on a variety of natural language understanding (NLU) tasks. |
| Approach: | They compare sentence encoding models with fMRI-based fMR predictions of the sentence . they use a pre-trained BERT architecture as a baseline and fine-tune it on a variety of natural language understanding (NLU) tasks. |
| Outcome: | The proposed model does not yield significant improvements in brain decoding performance on the natural language understanding (NLU) tasks. |
Investigating the Role and Impact of Disfluency on Summarization (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing studies have focused on disfluency detection and removal, with limited studies into its impact on downstream tasks. |
| Approach: | They propose to incorporate disfluency in summarization models to reduce the impact of replacement disfluencies on natural language processing tasks. |
| Outcome: | The proposed model improves on both public and real-life datasets and shows that it can handle disfluent data with up to 6.99-point degradation in Rouge-L score and replacement disfluencies have the highest negative impact. |
XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding (2022.acl-short)
Copied to clipboard
| Challenge: | XDBERT (cross-modal distilled BERT) outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG) benchmarks, and readability benchmarks. |
| Approach: | They propose to distill visual information from pretrained multimodal transformers to pretrained language encoders to cater to the language-heavy characteristics of NLU. |
| Outcome: | The proposed framework outperforms pretrained-BERT in general language understanding evaluation (GLUE), situations with adversarial generations (SWAG), and readability benchmarks. |
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification (2022.emnlp-industry)
Copied to clipboard
Jai Gupta, Yi Tay, Chaitanya Kamath, Vinh Tran, Donald Metzler, Shailesh Bavadekar, Mimi Sun, Evgeniy Gabrilovich
| Challenge: | a new method for classification of COVID-19 vaccination related search queries is proposed . the proposed method uses pretrained Transformers and dense features to generate search insights . |
| Approach: | They propose a machine learning model that detects COVID-19 vaccination related search queries . they use pretrained Transformers to consider dense features as memory tokens that the model can attend to . |
| Outcome: | The proposed model improves the Vaccine Search Insights task by +15% . the proposed model uses pretrained Transformers and traditional dense features . |
Leveraging Moment Injection for Enhanced Semi-supervised Natural Language Inference with Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing studies have used class-specific fine-tuned large language models to generate hypotheses and assign pseudo-labels but discarded many LLM-constructed samples to ensure the quality. |
| Approach: | They propose to leverage LLM-constructed samples by injecting the moments of labeled samples during training to properly adjust the level of noise. |
| Outcome: | The proposed method outperforms strong baselines on multiple NLI datasets in low-resource settings. |
Benchmarking Robustness of Machine Reading Comprehension Models (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks only evaluate models' robustness under test-time perturbations or adversarial attacks. |
| Approach: | They propose a model-agnostic benchmark to evaluate models' robustness under adversarial attacks. |
| Outcome: | The proposed model-agnostic benchmark evaluates models under four different types of adversarial attacks. |
Plug and Play Knowledge Distillation for kNN-LM with External Logits (2022.aacl-short)
Copied to clipboard
| Challenge: | Despite the promising evaluation results by knowledge distillation (KD) in natural language understanding (NLU) and sequence-to-sequence (seq2sequ) tasks, KD for causal language modeling (LM) remains a challenge. |
| Approach: | They propose to use external logits to improve a student's kNN-LM by leveraging teacher's knowledge at test time. |
| Outcome: | The proposed method improves a student's kNN-LM in multiple language modeling datasets and improves perplexity. |
STIL - Simultaneous Slot Filling, Translation, Intent Classification, and Language Identification: Initial Results using mBART on MultiATIS++ (2020.aacl-main)
Copied to clipboard
| Challenge: | Slot-filling, Translation, Intent classification, and Language identification (STIL) are tasks for multilingual Natural Language Understanding (NLU) . |
| Approach: | They propose to perform simultaneous slot filling and translation into a single output language (English in this case). |
| Outcome: | The proposed task performs better than the current state-of-the-art system for the languages tested, but with lower intent classification accuracy and lower slot F1 . |
Unified Contextual Query Rewriting (2023.acl-industry)
Copied to clipboard
Yingxue Zhou, Jie Hao, Mukund Rungta, Yang Liu, Eunah Cho, Xing Fan, Yanbin Lu, Vishal Vasudevan, Kellen Gillespie, Zeynab Raeesy
| Challenge: | Large-scale conversational AI agents such as Alexa, Siri, and Google Assistant are becoming increasingly popular in real-world applications to assist users in daily life. |
| Approach: | They propose a unified contextual query rewriting model that unifies QR for friction reduction and contextual carryover . they leverage the text-to-text unified framework which uses independent tasks with weighted loss to account for task importance . |
| Outcome: | The proposed model reduces friction and contextual carryover by using multiple auxiliary tasks. |
Scalable Prompt Generation for Semi-supervised Learning with Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prompt-based learning methods in semi-supervised learning (SSL) settings have been shown to be effective on multiple natural language understanding datasets and tasks. |
| Approach: | They propose to use a set of prompt tokens to create diverse prompt models and a varying number of soft prompt token to encourage language models to learn different prompts. |
| Outcome: | The proposed method achieves the best average accuracy of 71.5% in different few-shot learning settings. |
Towards Enhancing Health Coaching Dialogue in Low-Resource Settings (2022.coling-1)
Copied to clipboard
Yue Zhou, Barbara Di Eugenio, Brian Ziebart, Lisa Sharp, Bing Liu, Ben Gerber, Nikolaos Agadakos, Shweta Yadav
| Challenge: | Health coaching is cost-prohibitive due to its highly personalized nature. |
| Approach: | They propose to build a health coaching dialogue system that converses with patients . they propose to use simplified NLU and NLG frameworks and mechanism-conditioned empathetic response generation. |
| Outcome: | The proposed system generates more empathetic, fluent, and coherent responses . it outperforms the state-of-the-art in NLU tasks while requiring less annotations. |
Zero-shot Slot Filling in the Age of LLMs for Dialogue Systems (2025.coling-industry)
Copied to clipboard
| Challenge: | Existing methods for zero-shot slot filling focus on text data, overlooking conversational data. |
| Approach: | They propose a method for automatic data annotation with slot induction and black-box knowledge distillation from a teacher LLM to a smaller model. |
| Outcome: | The proposed method outperforms existing models on internal datasets by 26% relative increase in F1 score. |
Canary Extraction in Natural Language Understanding Models (2022.acl-short)
Copied to clipboard
| Challenge: | Recent literature has focused on Model Inversion Attacks (ModIvA) that can extract training data from model parameters. |
| Approach: | They propose an attack that extracts canaries from NLU training data and reconstructs them using non-sensitive tokens. |
| Outcome: | The proposed attack can reconstruct a four digit code in the training dataset with a probability of 0.5 in its best configuration. |
Compositional Zero-Shot Domain Transfer with Text-to-Text Models (2023.tacl-1)
Copied to clipboard
Fangyu Liu, Qianchu Liu, Shruthi Bannur, Fernando Pérez-García, Naoto Usuyama, Sheng Zhang, Tristan Naumann, Aditya Nori, Hoifung Poon, Javier Alvarez-Valle, Ozan Oktay, Stephanie L. Hyland
| Challenge: | Existing approaches to zero-shot domain transfer are limited by domain gap and lack of in-domain labels. |
| Approach: | They propose a compositional transfer learning framework (DoT51) that learns domain knowledge and task knowledge in a multi-task manner without access to in-domain labels. |
| Outcome: | The proposed framework outperforms the current state-of-the-art in zero-shot domain transfer by over 7 absolute points in accuracy on RadNLI. |
Towards Unsupervised Language Understanding and Generation by Joint Dual Learning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work exploits dual property between understanding and generation to improve performance of modular dialogue systems. |
| Approach: | They propose a dual supervised learning framework that exploits the dual property between understanding and generation. |
| Outcome: | The proposed framework improves both NLU and NLG performance by incorporating supervised and unsupervised learning algorithms. |
WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on weak supervision for NLU focus on a specific task or simulate weak supervision signals from ground-truth labels. |
| Approach: | They propose a benchmark to advocate and facilitate research on weak supervision for NLU . they use document-level and token-level prediction tasks as examples . |
| Outcome: | The proposed benchmark advocates and facilitates research on weak supervision for NLU tasks. |
Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU (2022.findings-naacl)
Copied to clipboard
| Challenge: | a recent study has shown that fine-tuning pre-trained models is parameter-inefficient and expensive. |
| Approach: | They propose a task-attuned token module which integrates pre-trained network representations into a pre-trainer. |
| Outcome: | The proposed model trains only 0.0009% of the parameters and is efficient during computation and scalable during deployment. |
LLM ContextBridge: A Hybrid Approach for Intent and Dialogue Understanding in IVSR (2025.coling-industry)
Copied to clipboard
| Challenge: | In-vehicle speech recognition systems struggle with interpreting user intent accurately due to limitations in contextual understanding and ambiguity resolution. |
| Approach: | They propose a hybrid architecture that integrates Pretrained Language Model-based intent classification with Large Language Models to enhance both command recognition and dialogue management. |
| Outcome: | The proposed architecture improves recognition accuracy and user experience in multi-turn dialogues. |
PATE: A Corpus of Temporal Expressions for the In-car Voice Assistant Domain (2020.lrec-1)
Copied to clipboard
| Challenge: | In the voice assistant domain, temporal expression recognition is a key module for AI voice assistants, but research on temporal recognition has focused on data from the news, the clinical domain, and social media. |
| Approach: | They propose a crowdsourcing method for eliciting natural-language commands containing temporal expressions for an AI voice assistant by using pictures and scenario descriptions. |
| Outcome: | The proposed method elicits natural-language commands containing temporal expressions using pictures and scenario descriptions. |
ParsiNLU: A Suite of Language Understanding Challenges for Persian (2021.tacl-1)
Copied to clipboard
Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, Yadollah Yaghoobzadeh
| Challenge: | Despite progress in natural language understanding, most progress is concentrated on resource-rich languages like English . despite high-quality benchmarks, there are few available NLU datasets for Persian language . |
| Approach: | They propose a benchmark for Persian language that includes a range of language understanding tasks . they present their results on monolingual and multilingual pre-trained language models . |
| Outcome: | The proposed benchmarks compare human performance with monolingual and multilingual models on Persian language with high quality evaluation datasets. |
Jointly Improving Language Understanding and Generation with Quality-Weighted Weak Supervision of Automatic Labeling (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural natural language generation and understanding models are data-hungry and require massive amounts of annotated data to be competitive. |
| Approach: | They propose a framework that automatically synthesizes weak labels from large-scale weakly-labeled data with a fine-tuned GPT-2 and adapts parameter updates to the models according to the estimated label-quality. |
| Outcome: | The proposed framework outperforms benchmark systems on the E2E and Weather datasets when 100% of the training data is used. |
Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU models (2021.naacl-main)
Copied to clipboard
Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, Xia Hu
| Challenge: | Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. |
| Approach: | They propose a shortcut mitigation framework to suppress NLU models from making overconfident predictions for samples with large shortcut degree. |
| Outcome: | The proposed framework suppresses the model from making overconfident predictions for samples with large shortcut degree. |
Exploring Zero and Few-shot Techniques for Intent Classification (2023.acl-industry)
Copied to clipboard
| Challenge: | Intent classification is the primary natural language understanding task for a virtual agent or a chatbot. |
| Approach: | They propose four different approaches to zero-shot intent classification with low-resource constraints . they use domain adaptation, data augmentation, and parametric fine-tuning to achieve this . |
| Outcome: | The proposed approaches perform well in low-resource settings for zero/few-shot intent classification . the proposed methods remove or substantially reduce the work to provide intent-utterances . |
Batch Prompting: Efficient Inference with Large Language Model APIs (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Performing inference on large volumes of samples can be computationally and financially costly. |
| Approach: | They propose a prompting approach that enables large language models to run inference in batches instead of one sample at a time. |
| Outcome: | The proposed prompting reduces both token and time costs while retaining downstream performance. |
Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on large-scale labeled support sets are not feasible in practical scenarios. |
| Approach: | They introduce a language model-based determinant point process that considers uncertainty and diversity of unlabeled instances for optimal selection. |
| Outcome: | The proposed method can effectively select canonical examples on 9 NLU and 2 Generation datasets. |
Multi-task Learning for Multilingual Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing multilingual neural machine translation systems rely on bitext training data, which is limited and costly to collect. |
| Approach: | They propose a multi-task learning framework that trains the model with the translation task on bitext data and two denoising tasks on monolingual data. |
| Outcome: | The proposed framework outperforms pre-training models for both NMT and cross-lingual transfer learning NLU tasks. |
Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding (2025.acl-short)
Copied to clipboard
| Challenge: | Prior research has found that large language models overlook input-label mapping information in ICL, relying more on their pre-trained knowledge. |
| Approach: | They propose a novel method that contrasts input-label mappings between positive and negative in-context examples to improve model performance. |
| Outcome: | The proposed method improves performance on 7 natural language understanding tasks without additional training. |
Multi-Task Networks with Universe, Group, and Task Feature Learning (P19-1)
Copied to clipboard
| Challenge: | In multi-task learning, multiple related tasks are learned together. |
| Approach: | They propose methods that take advantage of natural groupings of related tasks . they propose parallel and serial architectures that can learn different feature spaces . |
| Outcome: | The proposed methods improve performance on natural language understanding (NLU) tasks. |
Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning (2022.findings-acl)
Copied to clipboard
| Challenge: | Using distributed NLI, we show that models can capture human judgement distribution more effectively than the softmax baseline. |
| Approach: | They propose a new NLU task to predict the distribution of human judgements . they propose Monte Carlo, Deep Ensemble, Re-Calibration and Distribution Distillation methods to capture human judgement distributions. |
| Outcome: | The proposed methods perform better than the softmax baseline, but the results are still far below the estimated human upper-bound. |
Dialogue-AMR: Abstract Meaning Representation for Dialogue (2020.lrec-1)
Copied to clipboard
Claire Bonial, Lucia Donatelli, Mitchell Abrams, Stephanie M. Lukin, Stephen Tratz, Matthew Marge, Ron Artstein, David Traum, Clare Voss
| Challenge: | Abstract Meaning Representation (AMR) does not capture the illocutionary force or speaker’s intended contribution in the broader dialogue context. |
| Approach: | They propose a schema that enriches Abstract Meaning Representation (AMR) it provides a semantic representation for facilitating Natural Language Understanding (NLU) in dialogue systems. |
| Outcome: | The proposed schema provides a semantic representation for facilitating Natural Language Understanding (NLU) in human-robot dialogue systems. |
Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Task-adaptive pre-training (TAPT) and Self-training can be complementary with simple TFS protocol. |
| Approach: | They propose to use task-adaptive pre-training and self-training to combine TAPT and ST with a simple TFS protocol to achieve strong combined gains across six datasets. |
| Outcome: | The proposed method can achieve strong combined gains across six datasets covering sentiment classification, paraphrase identification, natural language inference, named entity recognition and dialogue slot classification. |
Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work attempts to apply incremental processing to NLUs but this is computationally expensive and does not scale efficiently for long sequences. |
| Approach: | They propose to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. |
| Outcome: | The proposed model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental quality. |
When Do You Need Billions of Words of Pretraining Data? (2021.acl-long)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are dominated by models that can encode billions of words. |
| Approach: | They use classifier probing, information-theoretic probing and unsupervised relative acceptability judgments to evaluate model ability. |
| Outcome: | The proposed models require only about 10M to 100M words to learn to encode most syntactic and semantic features. |
CoDA21: Evaluating Language Understanding Capabilities of NLP Models With Context-Definition Alignment (2022.acl-short)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. |
| Approach: | They propose a benchmark that measures natural language understanding (NLU) abilities of pretrained language models. |
| Outcome: | The proposed benchmark measures the ability of pretrained language models to perform on many tasks. |
AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale conversational systems typically generate unnatural, robotic responses using template-based approaches. |
| Approach: | They propose a data augmentation approach that combines a self-trained neural retrieval model with a few-shot learned NLU model to automatically create MR-to-Text data from open-domain texts. |
| Outcome: | The proposed approach outperforms the state-of-the-art methods on the FewshotWOZ data in both BLEU and Slot Error Rate. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2022.findings-naacl)
Copied to clipboard
Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, Rifat Shahriyar
| Challenge: | Bangla is a widely spoken yet low-resource language in the NLP literature. |
| Approach: | They propose a BERT-based natural language understanding model pretrainable in Bangla, a widely spoken yet low-resource language in the NLP literature. |
| Outcome: | The proposed model outperforms multilingual and monolingual models on four NLU tasks covering text classification, sequence labeling, and span prediction. |
SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Korean uses a featural writing system in which each character is composed of subcharacter units known as Jamo. |
| Approach: | They propose a model-agnostic module that injects subcharacter compositional knowledge into Korean language models. |
| Outcome: | a new module improves embeddings of Korean subwords with structural granularity . the module improve grammatical regularities and semantic cohesive variations . |
Metacognitive Prompting Improves Understanding in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in prompting have enhanced reasoning in logic-intensive tasks for LLMs, yet the nuanced understanding abilities of these models remain underexplored. |
| Approach: | They propose a strategy inspired by human introspective reasoning processes to enhance LLMs' understanding abilities. |
| Outcome: | The proposed method outperforms chain-of-thought prompting and its advanced versions on ten natural language understanding (NLU) datasets. |
Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models (2023.findings-acl)
Copied to clipboard
Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron Halfaker, Liqun Shao, Daniel McDuff, Ahmed Hassan Awadallah, Dragomir Radev, Jianfeng Gao
| Challenge: | Existing pre-trained language models that ignore the logical structures underlying natural language text often lack the ability to capture and encode key logical information in the input sequences. |
| Approach: | They propose to construct logic-aware input embeddings for transformer language models through logic detection, logic mapping and hierarchical logical projections and then develop a new modeling paradigm that can upgrade existing transformer language model into logical transformers to boost their performance. |
| Outcome: | The proposed model can achieve superior performance on four important and challenging tasks. |
KLEJ: Comprehensive Benchmark for Polish Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | Recent introduction of robust, general-purpose models for fine-tuning has enabled improvements in general natural language understanding (NLU) but such benchmarks are only available for a handful of languages. |
| Approach: | They propose a multi-task benchmark for the Polish language understanding with an online leaderboard . they also propose GLUE, a task for named entity recognition and sentiment analysis . |
| Outcome: | The proposed model performs best on three out of nine tasks in the Polish language . the proposed model is also used in an e-commerce domain to analyze the sentiments of users . |
Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn general representations of text can achieve sub-optimal performance in low-resource scenarios. |
| Approach: | They propose to use language model pre-training and multi-task learning to learn robust representations but these methods can achieve sub-optimal performance in low-resource scenarios. |
| Outcome: | The proposed model outperforms strong baselines on the GLUE benchmark and can be adapted to new tasks efficiently and effectively. |
ELQA: A Corpus of Metalinguistic Questions and Answers about English (2023.acl-long)
Copied to clipboard
| Challenge: | ELQA corpus is metalinguistic—it consists of language about language. |
| Approach: | They present a corpus of questions and answers in and about the English language . they use a free-form question answering task and multiple LLMs to analyze their capacity . |
| Outcome: | The ELQA corpus covers grammar, meaning, fluency, and etymology . the results can be used to investigate metalinguistic capabilities of NLU models . |
Debiasing Methods in Natural Language Understanding Make Bias More Accessible (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent debiasing methods in natural language understanding improve performance on out-of-distribution datasets by pressuring models into making unbiased predictions. |
| Approach: | They propose a general probing-based framework that allows for post-hoc interpretation of biases in language models and use an information-theoretic approach to measure the extractability of certain biase . |
| Outcome: | The proposed framework allows for post-hoc interpretation of biases in language models and measures the extractability of certain biase . |
How much pretraining data do language models need to learn syntax? (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining methods are convenient, but expensive in terms of time and resources. |
| Approach: | They investigate the impact of pretraining data size on the syntactic capabilities of RoBERTa by using syntaktic structural probes to determine whether models pretrained on more data encode a higher amount of syntastic information. |
| Outcome: | The proposed models perform better on part-of-speech tagging, dependency parsing and paraphrase identification. |
LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models are trained based on single-grained tokenization, making it hard to learn the precise meaning of coarse-grain words and phrases. |
| Approach: | They propose a language model pretraining method that incorporates multi-grained information of input text into pre-trained language models. |
| Outcome: | The proposed method improves performance on CLUE and SuperGLUE in Chinese and English with little extra inference cost. |
Modular Monolingual Adaptation using Pretrained Language Models (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches to building monolingual models for low-resource languages require a full model tuning process. |
| Approach: | They propose a modular approach to build monolingual models for low-resource languages by finetuning the whole model on the target language. |
| Outcome: | The proposed model improves on natural language understanding tasks on Scottish Gaelic, Irish, and Quechua with Quechuan being a very low-resource language. |
CASA-NLU: Context-Aware Self-Attentive Natural Language Understanding for Task-Oriented Chatbots (D19-1)
Copied to clipboard
| Challenge: | Prior work on contextual NLU has been limited in terms of the types of contextual signals used and the understanding of their impact on the model. |
| Approach: | They propose a context-aware self-attentive NLU model that uses multiple signals over a variable context window, such as previous intents, slots, dialog acts and utterances, in addition to the current user uttered. |
| Outcome: | The proposed model outperforms a baseline model on two conversational datasets yielding a gain of up to 7% on the IC task. |
D.Va: Validate Your Demonstration First Before You Use It (2025.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) heavily relies on selecting effective demonstrations to achieve outputs that better align with the expected results. |
| Approach: | They propose a method which integrates a demonstration validation perspective into this field and integrates it into the learning paradigm. |
| Outcome: | The proposed method surpasses all retrieval-based in-context learning techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks. |
Don’t Blame the Annotator: Bias Already Starts in the Annotation Instructions (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that data collected through crowdsourcing often exhibit various biases that lead to overestimation of model performance. |
| Approach: | They propose to model instruction bias in 14 recent NLU benchmarks by analyzing crowdsourcing instructions and analyzing their results. |
| Outcome: | The proposed model can be over-represented in datasets with a large number of examples, and the results are consistent with previous studies. |
RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge (2021.findings-acl)
Copied to clipboard
| Challenge: | a riddle-style commonsense questions require complex commonsensense reasoning and figurative language skills . there is currently no dataset aimed at testing these abilities . authors propose a new multiple-choice question answering task . |
| Approach: | They propose a new multiple-choice question answering task that uses a large dataset for riddlestyle commonsense questions. |
| Outcome: | The proposed task comes with the first large dataset for answering riddlestyle commonsense questions. |
mPMR: A Multilingual Pre-trained Machine Reader at Scale (2023.acl-short)
Copied to clipboard
| Challenge: | Existing mPLMs only transfer NLU capability from source to target languages . mPMR allows direct inheritance of multilingual NLU capabilities to downstream tasks . |
| Approach: | They propose a method to guide multilingual pre-trained language models to perform natural language understanding in multiple languages. |
| Outcome: | mPMR enables multilingual pre-trained language models to perform natural language understanding (NLU) in multiple languages. |
Towards Building a Spoken Dialogue System for Argument Exploration (2022.lrec-1)
Copied to clipboard
| Challenge: | Argumentative dialogue systems lack a robust natural language understanding framework for complex tasks . drop-down menus hinder the application of natural language learning approaches . |
| Approach: | They propose to integrate a natural language understanding framework into an argumentative dialogue system. |
| Outcome: | The proposed system is compared to a baseline system using a drop-down menu . the drop- down menu convinces, but the willingness to use it is significantly higher . |
Learning Slice-Aware Representations with Mixture of Attentions (2021.findings-acl)
Copied to clipboard
| Challenge: | Real-world machine learning systems are achieving excellent performance in terms of coarse-grained metrics like overall accuracy and F-1 score. |
| Approach: | They extend slice-based learning (SBL) with a mixture of attentions to learn slice-aware dual attentive representations. |
| Outcome: | The proposed approach outperforms the baseline method and the original SBL approach on monitored slices with two natural language understanding tasks. |
A Sequence-to-Sequence Approach to Dialogue State Tracking (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods for dialogue state tracking are still challenging, but they are improving . a new approach to dialogue state monitoring is proposed, called Seq2Seq-DU . |
| Approach: | They propose a new dialogue state tracking module that formalizes DST as a sequence-to-sequence problem. |
| Outcome: | The proposed method outperforms existing methods on benchmark datasets in different settings. |
Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Training (2024.eacl-long)
Copied to clipboard
| Challenge: | End-to-end (E2E) spoken language understanding models are constrained by the cost of collecting speech-semantics pairs. |
| Approach: | They propose a model that learns E2E SLU without speech-semantics pairs . they propose cross-modal selective self-training (CMSST) to address imbalance and noise issues . |
| Outcome: | The proposed model learns E2E SLU without speech-semantics pairs . the proposed model requires the domains of speech-text and text-sensitization to match . |
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods. |
| Approach: | They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness. |
| Outcome: | The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models. |
Intent Identification and Entity Extraction for Healthcare Queries in Indic Languages (2023.findings-eacl)
Copied to clipboard
| Challenge: | Currently, there is a lack of data and technology for resource-poor languages in developing countries like India. |
| Approach: | They propose to use two different datasets to analyze query intents and entities in healthcare. |
| Outcome: | The proposed model is useful to identify query intents and entities in real-world scenarios. |
Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to end-to-end questionanswering assume that pre-trained language can decompose complex tasks into more straightforward sub-tasks. |
| Approach: | They propose to use distant supervision to train decomposition-based transformers for large-scale parallel news. |
| Outcome: | The proposed model improves on semantic parsing and on hotpotQA and strategyQA datasets by 20% to 30%. |
Less Is Better: Recovering Intended-Feature Subspace to Robustify NLU Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to debiase datasets rely on knowledge of bias attributes . current approaches focus on how to leverage kinds of supervision effectively . |
| Approach: | They propose to extend the supervision on bias by extending it into feature space. |
| Outcome: | Empirical results show that a low-dimensional subspace with intended features can represent biased datasets. |
A New Concept of Deep Reinforcement Learning based Augmented General Tagging System (C18-1)
Copied to clipboard
| Challenge: | Existing systems for general sequence tagging/labeling are based on neural network architectures. |
| Approach: | They propose a deep neural network based sequence labeling model and a augmented tagger to improve system performance by modeling the data with minority tags. |
| Outcome: | The proposed system outperforms the current state-of-the-art model on ATIS and CoNLL-2003 datasets by 1.9% and 1.4%. |
To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning. (2022.coling-1)
Copied to clipboard
| Challenge: | Reasoning and knowledge-related skills are considered as fundamental skills for natural language understanding (NLU) tasks. |
| Approach: | They propose a method to diagnose correlations between an NLU dataset and a specific skill. |
| Outcome: | The proposed method is able to diagnose correlations between dataset and logical reasoning skill on 8 MRC and 3 NLI datasets. |
Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing calibration methods rescale posterior distributions of classifiers after training. |
| Approach: | They propose to use a noise contrastive estimation technique to train an energy-based model during finetuning of pretrained text encoders. |
| Outcome: | The proposed model can reach a better calibration competitive to strong baselines with little or no loss in accuracy. |
NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue (2022.findings-naacl)
Copied to clipboard
| Challenge: | NLU++ provides a more challenging evaluation environment for dialogue NLU models . Typical ToD systems still rely on a modular design . |
| Approach: | They propose to use NLU++ to provide a more challenging evaluation environment for dialogue NLU models. |
| Outcome: | The proposed dataset improves existing datasets and provides a much more challenging evaluation environment for dialogue NLU models. |
Leveraging fine-tuned Large Language Models with LoRA for Effective Claim, Claimer, and Claim Object Detection (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing work on identifying claims has focused on sentence level, neglecting supplementary attributes such as the claimer and claim object of the claim. |
| Approach: | They propose a novel approach to detect claims using large language models in natural language understanding and text generation. |
| Outcome: | The proposed approach transforms claim, claimer and claim object detection task into QA setting. |
A Two-Stage Prediction-Aware Contrastive Learning Framework for Multi-Intent NLU (2024.lrec-main)
Copied to clipboard
| Challenge: | Multi-intent natural language understanding (NLU) models lack the rich information between the shared intents, especially in low-data scenarios. |
| Approach: | They propose a two-stage framework for multi-intent natural language understanding to harness shared intent information by word-level pre-training and prediction-aware contrastive fine-tuning. |
| Outcome: | The proposed framework surpasses baselines on low-data and full-data scenarios. |
A Neural-Symbolic Approach to Natural Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have enabled deep neural networks to perform natural language understanding tasks, but their performance can drastically deteriorate when logical reasoning is needed. |
| Approach: | They propose a framework for NLU based on analogical reasoning based upon neural processing and logical reasoning using both neural and symbolic processing. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on two NLU tasks, question answering (QA) and natural language inference (NLI). |
Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue (2022.findings-acl)
Copied to clipboard
| Challenge: | Multi-SentAugment and LayerAgg are self-training methods that augment available training data with similar (automatically labelled) in-domain sentences from large monolingual Web-scale corpora. |
| Approach: | They propose to use multi-sentaugment and layeragg to improve dialogue natural language understanding across multiple languages. |
| Outcome: | The proposed methods generalise well in zero- and few-shot scenarios and leverage external unannotated data sources. |
Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation (2024.naacl-long)
Copied to clipboard
| Challenge: | Large-scale multilingual pretrained language models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLm. |
| Approach: | They propose to leverage the learned knowledge from well-performing languages to guide under-performing ones within the same mPLM. |
| Outcome: | The proposed model shows that it can guide under-performing languages while minimizing language-level performance disparities across different mPLMs. |
Towards Stable Natural Language Understanding via Information Entropy Guided Debiasing (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to debiase Natural Language Understanding models use dataset biases instead of learning the intended task. |
| Approach: | They propose a debiasing framework that detects and purifies dataset biases using information entropy. |
| Outcome: | The proposed framework improves the stability of performance on out-of-distribution datasets for a set of widely adopted NLU models. |
A Generative Model for Joint Natural Language Understanding and Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language understanding (NLU) and natural language generation (NLG) have opposite goals. |
| Approach: | They propose a generative model which couples NLU and NLG through a shared latent variable. |
| Outcome: | The proposed model achieves state-of-the-art performance on two dialogue datasets with flat and tree-structured formal representations. |
Training Dynamics for Curriculum Learning: A Study on Monolingual and Cross-lingual NLU (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches for NLU use CL to improve in-distribution data performance via heuristic-oriented or task-agnostic difficulties. |
| Approach: | They propose to use CL to improve in-distribution data performance by taking advantage of training dynamics as difficulty metrics instead of heuristic-oriented or task-agnostic difficulties. |
| Outcome: | The proposed model schedulers improve on in-distribution, out-of-distortion and zero-shot cross-lingual transfer datasets while being 20% faster on average. |
Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for creating vision-and-language models involve structural modifications and V&L pre-training. |
| Approach: | They propose to extend a language model through structural modifications and V&L pre-training to make it inherit the capability of natural language understanding from the original language model. |
| Outcome: | The proposed method improves performance of vision-and-language models by extending pre-trained models with the same pre-training. |
End-to-End Self-Debiasing Framework for Robust NLU Training (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing models incorporate dataset biases leading to strong performance on in-distribution test sets but poor performance on out-of-distortion (OOD) tests. |
| Approach: | They propose a debiasing framework where the shallow representations of the main model are used to derive a bias model and both models are trained simultaneously. |
| Outcome: | The proposed framework outperforms existing approaches on three well-studied NLU tasks while still delivering high in-distribution performance. |
AutoCAD: Automatically Generate Counterfactuals for Mitigating Shortcut Learning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating counterfactuals rely on human efforts or task-specific designs. |
| Approach: | They propose to use a fully automatic and task-agnostic CAD generation framework to generate diverse counterfactuals. |
| Outcome: | The proposed framework outperforms human-in-the-loop and task-specific CAD methods on multiple out-of-domain and challenge benchmarks. |
A Dog Is Passing Over The Jet? A Text-Generation Dataset for Korean Commonsense Reasoning and Evaluation (2022.findings-naacl)
Copied to clipboard
Jaehyung Seo, Seounghoon Lee, Chanjun Park, Yoonna Jang, Hyeonseok Moon, Sugyeong Eo, Seonmin Koo, Heuiseok Lim
| Challenge: | Korean pretrained language models struggle to generate short sentences with a given condition based on compositionality and commonsense reasoning. |
| Approach: | They propose a Korean text-generation dataset for Korean generative commonsense reasoning and language model evaluation using a semi-automatic dataset construction approach. |
| Outcome: | The proposed dataset is available at http://aihub.or.kr/opendata/korea-university. |
BasqueGLUE: A Natural Language Understanding Benchmark for Basque (2022.lrec-1)
Copied to clipboard
| Challenge: | Natural Language Understanding (NLU) benchmarks are costly to develop and language-dependent . basqueGLUE is the first benchmark for Basque, a less-resourced language . |
| Approach: | They propose a benchmark for Basque, a less-resourced language, using existing datasets. |
| Outcome: | The proposed benchmarks take into account a wide and diverse set of NLU tasks that require some form of language understanding beyond the detection of superficial clues. |
How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Popular datasets suffer from systematic spurious correlations that artificially inflate actual model performance. |
| Approach: | They propose a method for the automated creation of a challenging test set without relying on manual construction of artificial and unrealistic examples. |
| Outcome: | The proposed method reduces spurious correlations and improves model performance . examples labeled as having the highest difficulty show markedly decreased performance compared to the full dataset . |
Measuring and Mitigating Local Instability in Deep Neural Networks (2023.findings-acl)
Copied to clipboard
| Challenge: | Uncertain details like random initialization can change the outputs of a trained system with potentially disastrous consequences. |
| Approach: | They propose a model stability problem by studying how the predictions of a deep neural network change as a consequence of stochasticity in the training process. |
| Outcome: | The proposed method outperforms data-agnostic methods and is 90% cheaper than the gold standard. |
Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on dialogue modeling use pre-trained language models to encode dialogue history as successive tokens, which is insufficient in capturing the temporal characteristics of dialogues. |
| Approach: | They propose a bidirectional information decoupling network as a universal dialogue encoder which explicitly incorporates both the past and future contexts. |
| Outcome: | The proposed model incorporates past and future contexts and can be generalized to a wide range of dialogue-related tasks. |
Exploring Cross-Lingual Transfer Learning with Unsupervised Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | a new CLTL model is proposed to facilitate cross-linguistic transfer learning between distant languages . a key to CLTL is to learn a shared representation space for the given source-target language pair. |
| Approach: | They propose a new CLTL model that integrates machine translation with MT . they use an unannotated data technique to make use of the model's pre-training and fine-tuning . |
| Outcome: | The proposed model achieves better CLTL performance than the baseline model without more annotated data. |
Probing Linguistic Systematicity (2020.acl-main)
Copied to clipboard
| Challenge: | Existing evidence that deep natural language understanding models do not learn systematically is lacking. |
| Approach: | They examine whether deep natural language understanding models exhibit systematicity . they find that network architectures can generalize non-systematically . |
| Outcome: | The proposed model generalizes non-systematically, but is unsatisfactory, the authors argue . they show that the current state-of-the-art models do not generalize systematically . |
TAPE: Assessing Few-shot Russian Language Understanding (2022.findings-emnlp)
Copied to clipboard
Ekaterina Taktasheva, Tatiana Shavrina, Alena Fenogenova, Denis Shevelev, Nadezhda Katricheva, Maria Tikhonova, Albina Akhmetgareeva, Oleg Zinkevich, Anastasiia Bashmakova, Svetlana Iordanskaia, Alena Spiridonova, Valentina Kurenshchikova, Ekaterina Artemova, Vladislav Mikhailov
| Challenge: | Recent advances in zero-shot and few-shot learning have shown promise for a scope of research and practical purposes, but lacks standardized evaluation suites for non-English languages. |
| Approach: | They propose a novel benchmark that includes six more complex NLU tasks for Russian, covering multi-hop reasoning, ethical concepts, logic and commonsense knowledge. |
| Outcome: | The proposed benchmark includes six more complex NLU tasks for Russian, covering multi-hop reasoning, ethical concepts, logic and commonsense knowledge. |
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling (2026.acl-long)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method for large language models. |
| Approach: | They propose a drop-in extension that reparameterizes a rank-rtot update as a sum of K *static* low-rank experts. |
| Outcome: | Experiments on reasoning and knowledge-intensive benchmarks show consistent gains over matched-budget LoRA. |
Recursive Template-based Frame Generation for Task Oriented Dialog (2020.acl-main)
Copied to clipboard
| Challenge: | The input of an NLU component is a semantic frame that captures the intent and slot-labels provided by the user. |
| Approach: | They propose a recursive, hierarchical representation that captures the intent and slot-labels provided by the user and extend local tree-based loss functions with terms that provide global supervision. |
| Outcome: | The proposed representation improves on the widely used ATIS dataset and significantly improves the performance of the proposed framework. |
Towards preserving word order importance through Forced Invalidation (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies show pre-trained language models are insensitive to word order . performance on NLU tasks remains unchanged even after permuting the word . |
| Approach: | They propose a simple approach called Forced Invalidation to force the model to identify permuted sequences as invalid samples. |
| Outcome: | The proposed approach significantly improves the sensitivity of the models to word order on English NLU and QA tasks over BERT-based and attention-based models over word embeddings. |
Decorrelate Irrelevant, Purify Relevant: Overcome Textual Spurious Correlations from a Feature Perspective (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to debiase samples with biased features obstructs the model in learning from non-biased parts of the samples. |
| Approach: | They propose to eliminate spurious correlations in a fine-grained manner from a feature space perspective by using Random Fourier Features and weighted re-sampling to decorrelate dependencies between features. |
| Outcome: | The proposed method eliminates spurious correlations in a fine-grained manner from a feature space perspective. |
Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approach (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models generate inconsistent and sometimes contradictory outputs when presented with a prompt that has equivalent semantics but is expressed differently from the original prompt. |
| Approach: | They propose to refine a Large Language Model (LLM) with prompt-output pairs with equivalent semantics to achieve semantic consistency. |
| Outcome: | The proposed method improves the semantic consistency and task performance of LLMs. |
REBEL: Relation Extraction By End-to-end Language generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to extract relation triplets from text often involve multiple-step pipelines that propagate errors or are limited to a small number of relation types. |
| Approach: | They propose to use autoregressive seq2seq models to simplify Relation Extraction by expressing triplets as a sequence of text and a model that performs end-to-end relation extraction for more than 200 different relation types. |
| Outcome: | The proposed model achieves state-of-the-art on an array of Relation Extraction and Relation Classification benchmarks and achieves top performance in most of them. |
Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge-enhanced pre-trained language models (PLMs) introduce redundant factual knowledge from knowledge bases and require complex modules. |
| Approach: | They propose a knowledge prompting-based PLM framework that incorporates factual knowledge into PLMs. |
| Outcome: | The proposed framework can be flexibly combined with existing mainstream PLMs. |
Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing continual learning approaches suffer from low accuracy and performance fluctuation when the distributions of old and new data are significantly different. |
| Approach: | They propose a hyperparameter-free continual learning model for text data that can stably produce high performance under various environments. |
| Outcome: | The proposed model outperforms the best state-of-the-art method by 20% in average accuracy and each component contributes effectively to overall performance. |
CONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection (2021.findings-acl)
Copied to clipboard
Henry Weld, Guanghao Huang, Jean Lee, Tongshu Zhang, Kunze Wang, Xinghong Guo, Siqu Long, Josiah Poon, Caren Han
| Challenge: | Existing toxic language detection models focus on the single utterance level without deeper understanding of context. |
| Approach: | They propose a dataset for in-game toxic language detection enabling joint intent classification and slot filling analysis, which is the core task of Natural Language Understanding (NLU). |
| Outcome: | The proposed framework handles utterance and token-level patterns, and rich contextual chatting history. |
Varta: A Large-Scale Headline-Generation Dataset for Indic Languages (2023.findings-acl)
Copied to clipboard
| Challenge: | Varta dataset includes more than 41 million pairs of headlines and articles in 14 different Indic languages (and English) |
| Approach: | They present a large-scale multilingual dataset for headline generation in Indic languages. |
| Outcome: | The Varta dataset includes more than 41 million pairs of headlines and articles in 14 different Indic languages (and English) the data can be used to train strong language models that outperform competitive baselines in both NLU and NLG benchmarks. |
Can you map it to English? The Role of Cross-Lingual Alignment in the Multilingual Performance of LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can answer prompts in many languages despite being pre-trained mostly on English text. |
| Approach: | They propose a Discriminative Alignment Index to quantify instance-level alignment across 24 languages other than English and three distinct NLU tasks. |
| Outcome: | The proposed model can perform natural language understanding tasks in 24 languages other than English and three distinct NLU tasks. |
Multi3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue (2023.findings-acl)
Copied to clipboard
| Challenge: | Task-oriented dialogue systems are typically constructed for a single domain or language and do not generalise well beyond this. |
| Approach: | They constructed a multilingual, multi-intent, multi domain dataset to support work on Natural Language Understanding (NLU) in ToD across multiple languages and domains simultaneously. |
| Outcome: | The proposed dataset extends the English-only dataset to include manual translations into a range of high, medium, and low resource languages in two domains (banking and hotels). |
Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have yielded remarkable performance, but objective mismatch issues hinder RLHF learning. |
| Approach: | They propose a Reinforcement Learning framework enhanced with Label-sensitive reward to enhance LLMs' alignment and generation capabilities. |
| Outcome: | The proposed framework improves performance on five diverse models across eight tasks. |
Enhancing Task-oriented Dialogue Systems with Generative Post-processing Networks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work proposes a method to optimize pipelined dialogue systems by fine-tuning modules directly. |
| Approach: | They propose a new post-processing component for natural language generation (NLG) they use dialogue act contribution to evaluate contribution of GenPPN-generated utterances . |
| Outcome: | The proposed method improves the performance of task-oriented dialogue systems by modifying arbitrary modules including non-differentiable ones. |
XNLIeu: a dataset for cross-lingual NLI in Basque (2024.naacl-long)
Copied to clipboard
| Challenge: | XNLI is a popular benchmark used to evaluate cross-lingual Natural Language Understanding (NLU) in languages such as English, Basque and other low-resource languages. |
| Approach: | They expand XNLI to include Basque, a low-resource language that can benefit from transfer-learning approaches. |
| Outcome: | The proposed dataset includes Basque, a low-resource language that can benefit from transfer-learning approaches. |
Not Enough Data to Pre-train Your Language Model? MT to the Rescue! (2023.findings-acl)
Copied to clipboard
| Challenge: | In recent years, transformer-based language models (LMs) have become the default approach for many NLP tasks. |
| Approach: | They compare the performance of transformer-based language models with machine-translated corpora. |
| Outcome: | The proposed model can be improved with real data, but further research is needed. |
Programmable Annotation with Diversed Heuristics and Data Denoising (2022.coling-1)
Copied to clipboard
| Challenge: | Neural natural language generation and understanding models require massive amounts of annotated data to be competitive. |
| Approach: | They propose a data programming framework that can jointly construct labeled data for language generation and understanding tasks by allowing annotators to modify an automatically-inferred alignment rule set between sequence labels and text. |
| Outcome: | The proposed framework generates high-quality data within a 1.48 BLEU and 6.42 slot F1 of 100% human-labeled data with just 100 labeled data samples outperforming benchmark annotation frameworks and other semi-supervised approaches. |
Explainable Slot Type Attentions to Improve Joint Intent Detection and Slot Filling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods analyze and compute features collectively for all slot types, and have no way to explain slot filling model decisions. |
| Approach: | They propose a method that learns to generate additional slot type specific features to improve accuracy and provides explanations for slot filling decisions for the first time in a joint NLU model. |
| Outcome: | The proposed model improves on two widely used datasets and provides an explanation for slot filling decisions for the first time. |
Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing models of event processing do not understand the essentiality of step events towards a goal event. |
| Approach: | They propose to deconstruct a goal event into a discrete representation of finer-grained (step) events, which are not equally important to the goal. |
| Outcome: | The proposed model can understand the essentiality of different step events towards a goal event. |
News Headline Grouping as a Challenging NLU Task (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in Natural Language Understanding (NLU) have seen models outperform human performance on many standard tasks. |
| Approach: | They propose a task of HeadLine Grouping and a dataset consisting of 20,056 pairs of news headlines, each labeled with a binary judgement as to whether the pair belongs within the same group. |
| Outcome: | The proposed model outperforms human models on a task consisting of 20,056 pairs of headlines on HLGD and a dataset with a binary judgement. |
Enhancing Language Representation with Constructional Information for Natural Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in natural language processing focus on acquiring lexico-semantic information. |
| Approach: | They propose a construction grammar which highlights the pairings of form and meaning to enrich language representation. |
| Outcome: | The proposed model is superior to existing models on a variety of NLU tasks. |
VALUE: Understanding Dialect Disparity in NLU (2022.acl-long)
Copied to clipboard
| Challenge: | English Natural Language Understanding systems outperform humans on benchmarks like GLUE and SuperGLUE, but they only use textbook Standard American English (SAE) . fewer studies have considered the effects of dialectal differences on performance . |
| Approach: | They propose a benchmark to evaluate the performance of English natural language understanding systems using a set of lexical and morphosyntactic transformation rules. |
| Outcome: | The proposed model outperforms humans on GLUE and SuperGLUE, but only on standard American English . the proposed model recruits fluent speakers of African American vernacular english to validate each feature transformation . |
No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | idiomatic expressions (IEs) are a non-compositional aspect of a text that makes it difficult for a model to comprehend . general purpose PTLMs are negatively affected by the context, as performance increases with its removal. |
| Approach: | They propose to use idiomatic expressions to infer additional meaning from IEs . they argue that only IE-aware models are suitable for idiom- matic reasoning tasks . |
| Outcome: | The proposed models can reason in the presence of idiomatic expressions, the authors show . they show that general purpose PTLMs are negatively affected by the context . |
PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to build labeled training data from domain-specific data are expensive to obtain. |
| Approach: | They propose a Prompt-based Data Augmentation model which only trains small-scale Soft Promptes in frozen Pre-trained Language Models. |
| Outcome: | The proposed model outperforms several baseline models on four benchmarks and is complementary with unlabeled in-domain data. |
Statistically Profiling Biases in Natural Language Reasoning Datasets and Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate NLP models' weaknesses are limited by “hypothesis-only” tests and CheckLists. |
| Approach: | They propose a lightweight general statistical profiling framework that automatically identifies potential biases in multiple-choice NLU datasets without requiring additional test cases. |
| Outcome: | The proposed framework assesses the extent to which models exploit these biases through black-box testing, confirming prior findings and revealing new insights. |
Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on diagnosing LMs' reasoning abilities in natural language understanding tasks. |
| Approach: | They propose a diagnostic method for first-order logic reasoning with a proposed benchmark, LogicNLI. |
| Outcome: | The proposed method disentangles the target FOL reasoning from commonsense inference and can be used to diagnose LMs from four perspectives: accuracy, robustness, generalization, and interpretability. |
Towards an Automatic Assessment of Crowdsourced Data for NLU (L18-1)
Copied to clipboard
| Challenge: | Recent development of spoken dialog systems aims at allowing a natural input style. |
| Approach: | They investigate how crowdsourced data can be assessed with respect to its naturalness and usefulness by using a word based language model to identify valid data. |
| Outcome: | The proposed methods show that valid data can be identified with the help of a word based language model. |
JGLUE: Japanese General Language Understanding Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | There is no benchmark for Japanese to evaluate and analyze NLU ability from different perspectives. |
| Approach: | They build a Japanese NLU benchmark from scratch without translation to measure general NLU ability in Japanese. |
| Outcome: | a Japanese NLU benchmark is built from scratch without translation to measure general NLU ability in Japanese. |
Tiny Budgets, Big Gains: Parameter Placement Strategy in Parameter Super-Efficient Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods such as LoRA and VeRA use memory-efficient methods to fine-tune large language models. |
| Approach: | They propose a method that uses only 1–5% of the standard LoRA parameters and achieves state-of-the-art performance across a wide range of tasks. |
| Outcome: | The proposed method achieves state-of-the-art performance across a wide range of tasks using only 1–5% of the standard LoRA parameters. |
Imagination-Augmented Natural Language Understanding (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for Natural Language Understanding focus on textual signals, which hinders models from learning efficiently from limited data samples. |
| Approach: | They propose an Imagination-Augmented Cross-modal Encoder to solve natural language understanding tasks from a novel learning perspective. |
| Outcome: | The proposed learning paradigm bridges the gap between human and agent language understanding in both linguistic and perceptual procedures. |
GeoHard: Towards Measuring Class-wise Hardness through Modelling Class Semantics (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in measuring hardness-wise properties of data guide language models in sample selection within low-resource scenarios. |
| Approach: | They propose to use class-wise hardness to measure class-specific properties of data in the semantic embedding space by modeling class geometry in the . semantic embeddining space. |
| Outcome: | The proposed method surpasses instance-level metrics by over 59 percent on Pearson‘s correlation on measuring class-wise hardness. |
Pretraining Language Models Using Translationese (2024.emnlp-main)
Copied to clipboard
| Challenge: | a recent study shows that large language models perform well in low-resource languages . a vast majority of languages don't have comparable data as compared to English . |
| Approach: | They propose to use Translationese as synthetic data for pre-training language models for low-resource languages. |
| Outcome: | The proposed method reduces performance of LMs trained on clean data in Indian languages . the proposed model performs better in English than in other languages, but is not comparable to English. |
BaSCo: An Annotated Basque-Spanish Code-Switching Corpus for Natural Language Understanding (2022.lrec-1)
Copied to clipboard
| Challenge: | Basque-Spanish code-switching is a widespread phenomenon among bilingual speakers in the Basque Country. |
| Approach: | They propose to use annotated utterances to train bilingual chatbots in Basque and Spanish to cover the phenomenon of code-switching. |
| Outcome: | The proposed corpus is the first with annotated linguistic resources encompassing Basque-Spanish code-switching. |
Improving the Efficiency of Visually Augmented Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Autoregressive Language Models lack visual knowledge due to reporting bias in textual corpora. |
| Approach: | They propose to use visual representations obtained from CLIP multimodal system to augment autoregressive language models with visual knowledge. |
| Outcome: | The proposed model outperforms VALM for visual language understanding, natural language understanding and language modeling tasks despite being significantly more efficient and simpler. |
HarfoSokhan: A Comprehensive Parallel Dataset for Transitions between Persian Colloquial and Formal Variations (2026.eacl-long)
Copied to clipboard
Hamid Jahad Sarvestani, Vida Ramezanian, Saee Saadat, Neda Taghizadeh Serajeh, Maryam Sadat Razavi Taheri, Shohreh Kasaei, MohammadAmin Fazli, Ehsaneddin Asgari
| Challenge: | A wide array of NLP/NLU models have been developed for the Persian language but performance drops when applied to the colloquial form of Persian. |
| Approach: | They propose to use a large-scale colloquial to formal Persian parallel dataset to train a GPT2 model that exhibited remarkable proficiency in colloqual to informal text style transfer. |
| Outcome: | The proposed dataset outperforms OpenAI’s GPT-3.5-turbo model and a leading rule-based system in colloquial to formal Persian conversion. |
Want To Reduce Labeling Cost? GPT-3 Can Help (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data annotation is labor-intensive and time-consuming for many NLP tasks. |
| Approach: | They propose to use GPT-3 to train models which are deployed for inference . they propose to combine pseudo labels from GPT3 with human labels . |
| Outcome: | The proposed method can be generalizable to many practical applications. |
On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that mixing up can improve model calibration on image classification tasks, but little is known about using it on natural language understanding (NLU) tasks. |
| Approach: | They propose a mixup strategy for pre-trained language models that improves model calibration further by using the AUM statistic and saliency map. |
| Outcome: | The proposed mixup improves model calibration on natural language understanding tasks while maintaining competitive accuracy. |
LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluation (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is currently the most commonly used PEFT method for fine-tuning models with billions of parameters. |
| Approach: | They propose to use low-rank Adaptation to evaluate LoRA parameter features and then retain LoRA for important layers and the other layers share the same LoRA. |
| Outcome: | The proposed method achieves comparable performance to full fine-tuning and LoRA while retaining 50% of the LoRA parameters on average. |
Does Meta-learning Help mBERT for Few-shot Question Generation in a Cross-lingual Transfer Setting for Indic Languages? (2022.coling-1)
Copied to clipboard
Aniruddha Roy, Rupak Kumar Thakur, Isha Sharma, Ashim Gupta, Amrith Krishna, Sudeshna Sarkar, Pawan Goyal
| Challenge: | Existing approaches to few-shot Question Generation (QG) are limited and require manual annotation. |
| Approach: | They propose to use multilingual BERT to perform few-shot question generation with cross-lingual transfer. |
| Outcome: | The proposed model improves in few-shot QG and human evaluation confirms it. |
Deep Model Compression Also Helps Models Capture Ambiguity (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our method reduces the model size significantly and improves latency. |
| Approach: | They propose a method to capture the degree of relationship between a sample and its candidate classes by deep model compression. |
| Outcome: | The proposed method reduces model size significantly and improves latency. |
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021.naacl-main)
Copied to clipboard
| Challenge: | Evaluation for many natural language understanding (NLU) tasks is broken due to unreliable and biased systems scoring so high on standard benchmarks. |
| Approach: | They argue that current benchmarks fail at four criteria for evaluation . they argue that adversarial data collection does not address the causes of failures . |
| Outcome: | The proposed frameworks fail at four criteria, and adversarial data collection does not address the causes of these failures, the authors argue . restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, reliability with which they are annotated, their size, and the ways they handle social bias. |
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning (2022.emnlp-main)
Copied to clipboard
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao
| Challenge: | Standard fine-tuning of large pre-trained language models requires updating hundreds of millions to billions of parameters and storing a large copy of the PLM weights for every task. |
| Approach: | They propose a parameter-efficient fine-tuning technique where small trainable components are injected into the PLM and updated during fine-uning. |
| Outcome: | The proposed method outperforms SOTA parameter-efficient fine-tuning and full model fine-uning on GLUE development set with RoBERTa-large encoder. |
Self-Training using Rules of Grammar for Few-Shot NLU (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for learning natural language understanding are limited in low-resource settings. |
| Approach: | They propose to use rules of grammar to construct and expand rules of grammatical structure of data without human involvement. |
| Outcome: | The proposed approach outperforms state-of-the-art methods in three benchmark datasets. |
Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel on English NLU tasks, yet struggle to extend their NLU capabilities to underrepresented languages. |
| Approach: | They integrate machine translation models (MT) directly into LLM backbones via sample-efficient self-distillation. |
| Outcome: | The proposed model outperforms translation-test models on 127 low-resource languages. |
InforMask: Unsupervised Informative Masking for Language Model Pretraining (2022.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling is used for pretraining large language models for knowledge-intensive tasks. |
| Approach: | They propose an unsupervised masking strategy that exploits Pointwise Mutual Information to select the most informative tokens to mask. |
| Outcome: | The proposed strategy outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2. |
Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained multilingual language models show significant performance gains for zero-shot cross-lingual model transfer on a wide range of natural language understanding (NLU) tasks. |
| Approach: | They do cross-lingual evaluation using prompt tuning and compare it with fine-tuning . prompt tuning achieves much better cross-linguistic transfer than fine- tuning . |
| Outcome: | The results show that prompt tuning achieves better cross-lingual transfer than fine-tuning across datasets, with only 0.1% to 0.3% tuned parameters. |
On Losses for Modern Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Devlin et al. ( 2018) released a transformer network (BERT) pre-training over two tasks: masked language modelling (MLM) and next sentence prediction (NSP). |
| Approach: | They clarify NSP's effect on BERT pre-training and explore ways to include multiple tasks into pre-train. |
| Outcome: | The proposed framework outperforms BERTBase on the GLUE benchmark using fewer than a quarter of training tokens. |
IDPG: An Instance-Dependent Prompt Generation Method (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing prompt tuning methods use a fixed prompt in each input instance during the model training stage. |
| Approach: | They propose a conditional prompt generation method to generate prompts for each input instance. |
| Outcome: | The proposed method outperforms other prompt tuning methods while tuning fewer parameters. |
Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach (D19-1)
Copied to clipboard
| Challenge: | 0Shot-TC is a challenging NLU problem to which little attention has been paid by the research community. |
| Approach: | They propose to use a standardized evaluation system to classify text snippets without seeing task specific training data. |
| Outcome: | The proposed model is based on a set of standardized evaluations and state-of-the-art baselines. |
End-to-End Slot Alignment and Recognition for Cross-Lingual NLU (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to expand an NLU system to new languages use machine translation with slot label projection from source to translated utterances. |
| Approach: | They propose a novel end-to-end model that learns to align and predict target slot labels jointly for cross-lingual transfer. |
| Outcome: | The proposed model outperforms a simple label projection method on most languages and achieves competitive performance to the more complex, state-of-the-art projection method with only half the training time. |
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020.coling-main)
Copied to clipboard
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, Zhenzhong Lan
| Challenge: | Existing language evaluation benchmarks for English are limited to English . lack of such benchmarks makes it difficult to replicate success in other languages . |
| Approach: | They introduce a large-scale Chinese language understanding evaluation benchmark . the benchmark uses a set of current state-of-the-art pre-trained Chinese models . |
| Outcome: | The first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark is released . the benchmark evaluates models across a wide range of tasks on original Chinese text . existing language evaluation benchmarks are mostly limited to English . |
A Multitask Active Learning Framework for Natural Language Understanding (2020.coling-main)
Copied to clipboard
| Challenge: | Existing studies have shown that relation information between intents and slots can improve the efficiency of active learning algorithms. |
| Approach: | They propose a multitask active learning framework that exploits relation information between sub-tasks provided by a joint model. |
| Outcome: | The proposed framework achieves competitive performance with less training data than baseline methods on all datasets. |
Leveraging LLMs for Bangla Grammar Error Correction: Error Categorization, Synthetic Data, and Model Evaluation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) perform exceedingly well in Natural Language Understanding tasks for many languages including English. |
| Approach: | They propose to use a rule-based noise injection method to create grammatically incorrect sentences . they categorize 12 error classes in Bangla and take a survey of native speakers . |
| Outcome: | The proposed method improves performance of LLMs in Bangla by 3-7 percentage points compared to zero-shot setting . human errors are still superior in error correction, the authors show . |
Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System (2022.lrec-1)
Copied to clipboard
| Challenge: | Contextually aware intelligent agents are often required to understand the users and their surroundings in real-time. |
| Approach: | They propose to build a multimodal dialogue system for children learning basic math concepts using limited datasets. |
| Outcome: | The proposed system improves the Natural Language Understanding (NLU) module of a task-oriented SDS pipeline with limited dataset resources. |
Adversarial NLI: A New Benchmark for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | a new large-scale NLI benchmark dataset is presented to test models on a variety of popular NLIs. |
| Approach: | They propose a large-scale NLI benchmark dataset that is iteratively compared with a human-and-model-in-the-loop procedure. |
| Outcome: | The proposed method can be applied in a never-ending learning scenario, becoming a moving target for NLU, rather than a static benchmark that will quickly saturate. |
Multi-Task Deep Neural Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to learning vector-space representations of text are multitask learning and language model pre-training. |
| Approach: | They propose a multi-task deep neural network (MT-DNN) that leverages cross-task data and incorporates a pre-trained bidirectional transformer language model. |
| Outcome: | The proposed model achieves state-of-the-art on ten NLU tasks and pushes the GLUE benchmark to 82.7% (2.2% absolute improvement) |
MOSAICo: a Multilingual Open-text Semantically Annotated Interlinked Corpus (2024.naacl-long)
Copied to clipboard
Simone Conia, Edoardo Barba, Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Riccardo Orlando, Luigi Procopio, Roberto Navigli
| Challenge: | Existing approaches to integrate semantics into Natural Language Understanding (NLP) systems are cost-effective and environmental impact-related. |
| Approach: | They propose to provide semantically-annotated corpora for four NLU tasks across five languages and to drop the requirement of closed datasets. |
| Outcome: | The proposed model provides hundreds of millions of silver yet high-quality annotations for four NLU tasks across five languages. |
Dual Inference for Improving Language Understanding and Generation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have exploited the duality of the task pairs in machine translation and speech recognition. |
| Approach: | They propose to leverage the duality in the inference stage without retraining whole models. |
| Outcome: | The proposed method is effective in both NLU and NLG tasks, providing the great potential of practical use. |
IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020.findings-emnlp)
Copied to clipboard
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar
| Challenge: | In this paper, we present NLP resources for 11 major Indian languages . distributional representations are the cornerstone of modern NLP, authors say . |
| Approach: | They introduce NLP resources for 11 major Indian languages from two major language families . monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . they also compile a benchmark for Indian language NLU to evaluate their results . |
| Outcome: | The monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . the pre-trained language models are based on the compact ALBERT model . |
MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) can generate coherent text, but they struggle to recognise user intent behind queries. |
| Approach: | They propose a novel approach leveraging multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. |
| Outcome: | The proposed model improves multi-turn conversation understanding by integrating teacher teachers into a student model. |
SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU (2024.naacl-long)
Copied to clipboard
| Challenge: | Task-oriented dialogue (TOD) systems support users in execution of specific, well-defined tasks through natural language interaction. |
| Approach: | They propose a framework for dialog NLU based on instruction tuning and question-answering-based formulation of ID and VE tasks. |
| Outcome: | The proposed framework surpasses existing models in training and cross-domain transfer and significantly outperforms existing large language models in performance and inference efficiency. |
GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (2021.emnlp-main)
Copied to clipboard
| Challenge: | A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. |
| Approach: | They propose an automatic auxiliary task selection method based on gradient calculation in Transformer-based models that improves MT-DNN performance. |
| Outcome: | The proposed method improves MT-DNN performance on 8 natural language understanding (GLUE) tasks, while costing less than AUTOSEM and comparable GPU consumption. |
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text (D19-1)
Copied to clipboard
| Challenge: | Existing datasets for reading comprehension tasks have been used to test the generalization of natural language understanding systems. |
| Approach: | They propose a diagnostic benchmark suite to clarify key issues related to the robustness and systematicity of NLU systems. |
| Outcome: | The proposed benchmark suite clarifies key issues related to the robustness and systematicity of NLU systems. |
Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on end-to-end dialogue models with pre-trained dialogue corpora shows promising performance in the conversational system. |
| Approach: | They propose an end-to-end TOD system with task-optimized adapters which learn independently per task adding only small number of parameters after fixed layers of pre-trained network. |
| Outcome: | The proposed system achieves state-of-the-art performance on the MultiWOZ benchmark compared to existing models. |
IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing MRC datasets in Indonesian are inadequate because of the small size and limited question types. |
| Approach: | They propose to combine automatic and manual unanswerable question generation to minimize the cost of manual dataset construction while maintaining the dataset quality. |
| Outcome: | The proposed dataset significantly improves the performance of Indonesian MRC models, showing a large improvement for unanswerable questions. |
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks. |
| Approach: | They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training. |
| Outcome: | The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague. |
AdaDHP: Fine-Grained Fine-Tuning via Dual Hadamard Product and Adaptive Parameter Selection (2025.acl-long)
Copied to clipboard
| Challenge: | Increasing number of parameters can be challenging under resource-constrained environments. |
| Approach: | They propose a parameter-efficient fine-tuning method with fewer parameters and finer granularity that can adaptively select important parameters for each task. |
| Outcome: | The proposed method can fine-tune important parameters for each task, while maintaining the same weights. |
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models have been widely applied to standard benchmarks due to the limited resources available in a domain. |
| Approach: | They propose a Transformer-based language model called VarMAE for domain-adaptive language understanding that encodes the context of a token into a smooth latent distribution. |
| Outcome: | Experiments on science- and finance-domain NLU tasks show that the proposed model can be efficiently adapted to new domains with limited resources. |
Learning to Rank for Plausible Plausibility (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to encoding sentences using contextualized encoders are inconsistent . |
| Approach: | They propose to use a cross entropy log-loss objective to improve plausibility . they propose a margin-based loss leads to a more plausible model of plausability . |
| Outcome: | The proposed loss is intuitively wrong when applied to plausibility tasks . the proposed loss leads to a more plausible model of plausability . |
ltzGLUE: Luxembourgish General Language Understanding Evaluation (2026.findings-acl)
Copied to clipboard
Alistair Plum, Felicia Körner, Anne-Marie Lutgen, Laura Bernardy, Fred Philippy, Emilia Milano, Nils Rehlinger, Cedric Lothritz, Tharindu Ranasinghe, Barbara Plank, Christoph Purschke
| Challenge: | ltzGLUE is the first official NLU benchmark for Luxembourgish (LTZ) based on the popular GLUE benchmark for English. |
| Approach: | They propose a new natural language understanding (NLU) benchmark for Luxembourgish based on the popular GLUE benchmark for English. |
| Outcome: | The proposed model performs well across many languages and is based on the GLUE benchmark for English. |
bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark (2023.acl-long)
Copied to clipboard
Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev
| Challenge: | bgGLUE is a benchmark for evaluating language models on natural language understanding (NLU) tasks in Bulgarian. |
| Approach: | They propose to use a benchmark to evaluate language models on NLU tasks in Bulgarian. |
| Outcome: | The proposed model performs well on sequence labeling tasks, but there is room for improvement for tasks that require more complex reasoning. |
A Scalable Framework for Learning From Implicit User Feedback to Improve Natural Language Understanding in Large-Scale Conversational AI Systems (2021.emnlp-main)
Copied to clipboard
Sunghyun Park, Han Li, Ameen Patel, Sidharth Mudgal, Sungjin Lee, Young-Bum Kim, Spyros Matsoukas, Ruhi Sarikaya
| Challenge: | Existing methods to improve NLU are laborintensive and expensive. |
| Approach: | They propose a scalable and automatic approach to improving NLU in a large-scale conversational AI system by leveraging implicit user feedback. |
| Outcome: | The proposed framework improves NLU in a large-scale conversational AI system across 10 domains. |
Scaling Laws for BERT in Low-Resource Settings (2023.findings-acl)
Copied to clipboard
| Challenge: | Large language models require huge training corpora, which is unobtainable for most NLP practitioners. |
| Approach: | They propose power-law formulas that relate model size, corpora size and computation power to find the optimal settings in advance given a fixed budget. |
| Outcome: | The proposed models perform better on MLM and NLU tasks on four languages of different linguistic characteristics. |
Does Corpus Quality Really Matter for Low-Resource Languages? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on multilingual pre-training has relied on automatically filtered versions of CommonCrawl. |
| Approach: | They propose to use tailored crawling to identify and scrape websites with high-quality content to improve representation learning in Basque. |
| Outcome: | The proposed corpus, called EusCrawl, has a much higher quality according to native annotators than the Basque portion of popular multilingual corpora like CC100 and mC4. |
Revisiting the Roles of “Text” in Text Games (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work has shown that random text hashes could be complementary rather than contrasting in text games. |
| Approach: | They propose a scheme to extract contextual information into an approximate state hash as extra input for an RNN-based text agent. |
| Outcome: | The proposed scheme achieves competitive performance with state-of-the-art text agents using advanced NLU techniques such as knowledge graph and passage retrieval. |
SciNLI: A Corpus for Natural Language Inference on Scientific Text (2022.acl-long)
Copied to clipboard
| Challenge: | Existing Natural Language Inference (NLI) datasets are not related to scientific text. |
| Approach: | They propose a large dataset for NLI that captures the formality in scientific text and contains 107,412 sentence pairs extracted from scholarly papers on NLP and computational linguistics. |
| Outcome: | The proposed model achieves a Macro F1 score of only 78.18% and an accuracy of 78.23%. |
Parameter-Efficient Tuning Makes a Good Classification Head (2022.emnlp-main)
Copied to clipboard
| Challenge: | In recent years, pretrained models revolutionized the paradigm of natural language understanding . but the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning . |
| Approach: | They propose to append a randomly initialized classification head after the pretrained backbone and finetune the whole model. |
| Outcome: | The proposed classification head can be replaced with the randomly initialized heads for a stable performance gain. |
Debiasing Masks: A New Framework for Shortcut Mitigation in NLU (2022.emnlp-main)
Copied to clipboard
| Challenge: | Debiasing language models from unwanted behaviors in natural language understanding datasets is a topic with increasing interest in the NLP community. |
| Approach: | They propose a method to debiase language models from unwanted behaviors in NLU tasks by identifying pruning masks that can be applied to a finetuned model. |
| Outcome: | The proposed method shows superior performance and performance over standard methods. |
ResLoRA: Identity Residual Mapping in Low-Rank Adaption (2024.findings-acl)
Copied to clipboard
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
| Challenge: | Low-rank adaptation (LoRA) is one of the most popular parameter-efficient fine-tuning methods. |
| Approach: | They propose a low-rank adaptation method that adds residual paths during training and merges them together during inference to achieve better results. |
| Outcome: | The proposed method achieves 2.5x faster convergence speed and improves performance by 14.3% on NLG, NLU, and text-to-image tasks. |
From Evasion to Concealment: Stealthy Knowledge Unlearning for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to unlearning often treat nonsensical responses or template-based refusals as the unlearning target, making the process even more vulnerable to attacks and jailbreaks. |
| Approach: | They propose a method that uses inverted facts to remove the need for auxiliary models or retaining data while avoiding leakage. |
| Outcome: | Evaluated on the ToFU Knowledge Unlearning dataset using Llama2-7B-Chat and Phi-1.5, MEOW outperforms baselines in forgetting quality while preserving model utility. |
Curriculum Learning for Natural Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained language models can be fine tuned to perform NLU tasks in a straightforward manner. |
| Approach: | They propose a pretrain-finetune paradigm for natural language understanding (NLU) they propose 'a cross-trainset' approach that allows users to distinguish easy from difficult examples . |
| Outcome: | The proposed approach achieves significant performance improvements on a wide range of NLU tasks. |
Multi-resolution Annotations for Emoji Prediction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Emojis are able to express various linguistic components, such as emotions, sentiments, events, etc. emojis have the merit of preserving information more densely, compared to words, argues a new study. |
| Approach: | They propose to use passage-level and aspect-level emoji annotations to predict the proper emmojis associated with text. |
| Outcome: | The proposed method is heuristically generated and validated with a pre-trained BERT model. |
Dual Supervised Learning for Natural Language Understanding and Generation (P19-1)
Copied to clipboard
| Challenge: | Natural language understanding and natural language generation are important research topics in the NLP and dialogue fields. |
| Approach: | They propose a dual-supervised learning framework for natural language understanding and generation on top of dual supervised learning. |
| Outcome: | The proposed framework boosts the performance of both tasks simultaneously in the benchmark experiments. |
Language (Re)modelling: Towards Embodied Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | Despite the rapid progress in NLU, current systems lack the rich mental representations that people use for language understanding. |
| Approach: | They propose an approach to representation and learning based on the tenets of embodied cognitive linguistics (ECL) they propose a system architecture along with a roadmap towards realizing this vision. |
| Outcome: | The proposed approach will improve the performance of existing systems and provide a roadmap towards realizing this vision. |
Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models capture semantic and syntactic information, but no study has examined how information loss in input token characters affects their performance. |
| Approach: | They address this gap by pre-training language models using small subsets of token characters. |
| Outcome: | The proposed model retains 90% and 77% of the full-token model in standard NLU benchmarks and probing tasks even under extreme settings. |
CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse paRsing in conversations (2025.findings-acl)
Copied to clipboard
| Challenge: | Discourse parsing datasets based on conversations are restricted to a single domain . a lack of discourse structures in audio-based conversations is a challenge . |
| Approach: | They introduce CoMuMDR: Code-mixed Multi-modal Multi-domain corpus for Discourse parsing in conversations. |
| Outcome: | The proposed corpus is code-mixed in Hindi and English and annotated with nine discourse relations. |
UnNatural Language Inference (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained NLU models understand human-like syntax . however, these models are word order invariant, causing them to assign gold labels to permutations . |
| Approach: | They propose to measure the severity of this issue by examining the properties of particular permutations that lead models to be word order invariant. |
| Outcome: | The proposed model is word order invariant, but it's not human-like syntax. |
SLURP: A Spoken Language Understanding Resource Package (2020.emnlp-main)
Copied to clipboard
| Challenge: | Publicly available datasets for Spoken Language Understanding (SLU) are limited. |
| Approach: | They propose a publicly available SLU resource package that includes a multi-domain dataset in English spanning 18 domains. |
| Outcome: | The proposed dataset is bigger and more diverse than existing datasets. |
tagE: Enabling an Embodied Agent to Understand Human Instructions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for natural language understanding (NLU) are limited due to the inherent ambiguity and incompleteness inherent in natural language. |
| Approach: | They propose a system to extract tasks from natural language instructions and map them to robots' established collection of skills. |
| Outcome: | The proposed system outperforms baseline models in the training and evaluation of a dataset featuring complex instructions. |
SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages (2023.acl-long)
Copied to clipboard
| Challenge: | Prior work on document-level simplification has focused on sentence-level edits, while many desirable edits require document- level context. |
| Approach: | They propose a dataset that reconstructs the document-level editing process from English Wikipedia to paired Simple Wikipedia articles. |
| Outcome: | The proposed dataset reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) pages. |
ORCA: A Challenging Benchmark for Arabic Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Despite efforts to evaluate Arabic NLU, no public benchmark of diverse nature exists . a benchmark targeting Arabic needs to take into account that Arabic is not a single language but a collection of languages and language varieties. |
| Approach: | They propose a publicly available benchmark for Arabic language understanding evaluation dubbed ORCA . it covers diverse Arabic varieties and a wide range of Arabic understanding tasks . |
| Outcome: | The proposed benchmark covers Arabic and multilingual models across seven NLU task clusters. |
Towards Debiasing NLU Models from Unknown Biases (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent proposed debiasing methods rely on the assumption that the types of bias should be known a-priori, which limits their application to many NLU tasks and datasets. |
| Approach: | They propose a framework that prevents models from mainly utilizing biases without knowing them in advance. |
| Outcome: | The proposed framework allows existing methods to retain performance improvement on challenge datasets without specifically targeting biases. |
Inference Annotation of a Chinese Corpus for Opinion Mining (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for opinion mining can accurately predict the writer's attitude in simple explicit sentences. |
| Approach: | They propose to define inference, classify different types and provide an annotation framework to analyze the annotation results. |
| Outcome: | The proposed framework defines inference type, polarity and topic and analyzes the results. |
Mitigate Extrinsic Social Bias in Pre-trained Language Models via Continuous Prompts Adjustment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods of extrinsic bias mitigation rely on manual word lists for sensitive groups . however, these word lists are limited by length and scope, resulting in poor performance. |
| Approach: | They propose a method which generates continuous token lists from the entire vocabulary space and uses them to bridge the gap between outputs and targets in fairness learning process. |
| Outcome: | The proposed method outperforms baseline methods on three NLU tasks. |
A Transformer-based Threshold-Free Framework for Multi-Intent NLU (2022.coling-1)
Copied to clipboard
| Challenge: | Existing models for multi-intent natural language understanding mainly detect multiple intents on threshold settings. |
| Approach: | They propose a transformer-based multi-intent NLU model with multi-task learning that exploits the information of the number of multiple intents in each utterance without additional manual annotations. |
| Outcome: | The proposed model achieves superior results on two public multi-intent datasets. |
Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models are typically optimized for resource-rich languages like English . however, the proprietary nature of these models makes them impractical for many researchers and developers. |
| Approach: | They propose to develop large language models that can follow instructions in Basque . they focus on three key stages: pre-training, instruction tuning, and alignment with human preferences . |
| Outcome: | The proposed models improve natural language understanding (NLU) of the foundational model by 12 points . the results show that the models can follow instructions in Basque with human preferences . |
PCMID: Multi-Intent Detection through Supervised Prototypical Contrastive Learning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to intent detection assume that each utterance represents only a single intent. |
| Approach: | They propose a framework for intent detection that can learn multiple representations of a given user utterance under the context of different intent labels in an optimized semantic space. |
| Outcome: | The proposed framework achieves state-of-the-art on multiple public benchmark datasets and a private real-world dataset for the multi-intent detection task. |
Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs (2022.emnlp-main)
Copied to clipboard
| Challenge: | a framework for language understanding models to track and improve beliefs through intermediate points in text is needed . breakpoint modeling is an efficient and end-to-end learning approach that trains models to train beliefs . understanding the behavior of models remains a formidable challenge for model safety, authors say . |
| Approach: | They propose a framework that trains models to track beliefs through intermediate points in text . their framework allows for efficient and robust learning of this type of model . |
| Outcome: | The proposed model outperforms strong representation learning approaches on a variety of NLU tasks. |
PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs (2023.emnlp-main)
Copied to clipboard
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJeong Choe, David Greene, Chuan He, Rattima Nitisaroj, Anna Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, Zhou Yu
| Challenge: | PRESTO dataset contains 550K contextual multilingual conversations between humans and virtual assistants. |
| Approach: | They propose to use a dataset of 550K contextual multilingual conversations between humans and virtual assistants to study some of the more challenging aspects of parsing realistic conversations. |
| Outcome: | The dataset contains 550K contextual conversations between humans and virtual assistants. |
CLASP: Cross-modal Alignment Using Pre-trained Unimodal Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in speech-text pretraining rely on parallel speech- text data . however, data accessibility is a challenge due to the limited data available. |
| Approach: | They propose a framework for jointly performing speech and text processing without parallel corpora during pre-training but only downstream. |
| Outcome: | The proposed framework extracts distinct representations for speech and text, aligning them effectively in a newly defined space using a multi-level contrastive learning mechanism. |
Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis (2024.findings-acl)
Copied to clipboard
Sitiporn Sae Lim, Can Udomcharoenchaikit, Peerat Limkonchotiwat, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Current NLU models obtain state-of-the-art accuracy on in-distribution benchmarks, but they use annotation bias to make predictions, negatively affecting the models' generalizability. |
| Approach: | They apply causal mediation analysis to gauge how much each component mediates annotation biases and use causal-grounded masking and gradient unlearning to mitigate bias. |
| Outcome: | The proposed methods improve the model's robustness against annotation bias even after employing other training-time debiasing techniques. |
Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages (2023.acl-long)
Copied to clipboard
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, Pratyush Kumar
| Challenge: | Recent advances in Natural Language Understanding are driven by pretrained multilingual models, which can potentially reduce the performance gap between high-resource languages through zero-shot knowledge transfer. |
| Approach: | They propose to create a human-supervised benchmark for Indic languages, IndicXTREME, with nine diverse NLU tasks covering 20 languages. |
| Outcome: | The proposed model improves on the monolingual corpora, IndicCorp, and IndicBERT in Indic languages with 105 evaluation sets across languages and tasks. |
Making Task-Oriented Dialogue Datasets More Natural by Synthetically Generating Indirect User Requests (2025.coling-main)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue benchmarks lack sufficient examples of complex discourse phenomena such as indirectness. |
| Approach: | They propose a set of linguistic criteria and an LLM-based pipeline for generating realistic IURs to test natural language understanding and dialogue state tracking models before deployment in a new domain. |
| Outcome: | The proposed model can handle indirect user requests (IURs) but lacks examples of complex discourse phenomena such as indirectness. |
Creation of a Balanced State-of-the-Art Multilayer Corpus for NLU (L18-1)
Copied to clipboard
Normunds Gruzitis, Lauma Pretkalnina, Baiba Saulite, Laura Rituma, Gunta Nespore-Berzkalne, Arturs Znotins, Peteris Paikens
| Challenge: | Using full stack of language resources, we are creating a balanced text corpus for Latvian. |
| Approach: | They propose to create a syntactically and semantically annotated multilayered corpus for Latvian . they use widely acknowledged and cross-lingual representations for the corpus . |
| Outcome: | The proposed corpus adopts widely recognized and cross-lingual representations for natural language understanding and generation in Latvian. |
Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors (2023.emnlp-main)
Copied to clipboard
| Challenge: | proposed method combines back transcription with fine-grained technique for categorizing speech recognition errors . proposed method relies on the use of synthesized speech in place of audio recording . |
| Approach: | They propose a method for investigating the impact of speech recognition errors on NLU models . they use a back transcription procedure and a fine-grained technique for categorizing errors . |
| Outcome: | The proposed method relies on synthesized speech in place of audio recording to evaluate the model. |
Debias NLU Datasets via Training-free Perturbations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to debiase NLU models capture biased features that are independent of the task but spuriously correlated to labels. |
| Approach: | They propose a framework that conducts training-free perturbations on samples containing biased features to Debias NLU Datasets. |
| Outcome: | The proposed framework shows competitive performance with previous state-of-the-art debiasing strategies. |
Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for data augmentation produce low readability or semantic consistency. |
| Approach: | They propose a framework which augments data through reinforcement learning guided conditional generation. |
| Outcome: | The proposed framework improves F1 performance on three different classification tasks by 8.7% on average when given only 10% of the whole data for training. |
AdapterShare: Task Correlation Modeling with Adapter Differentiation (2022.emnlp-main)
Copied to clipboard
| Challenge: | AdapterShare is an adapter differentiation method to explicitly model the task correlation among multiple tasks. |
| Approach: | They propose an adapter differentiation method to explicitly model the task correlation among multiple tasks. |
| Outcome: | The proposed method achieves 1.90 points improvement on five dialogue understanding tasks and 2.33 points gain on NLU tasks. |
What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions. |
| Approach: | They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels . |
| Outcome: | The proposed dataset examines the distribution of human opinions in NLU evaluation datasets. |
BARThez: a Skilled Pretrained French Sequence-to-Sequence Model (2021.emnlp-main)
Copied to clipboard
| Challenge: | Inductive transfer learning has taken the entire NLU field by storm, with models such as BERT and BART setting new state-of-the-art on countless tasks. |
| Approach: | They introduce a large-scale pretrained seq2seq model for French that is very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. |
| Outcome: | The proposed model outperforms existing models on discriminative and generative tasks on a French summarization dataset. |
Hyperbolic Representations for Prompt Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing techniques to train only continuous prompts while freezing the language model have been developed. |
| Approach: | They propose to use hyperbolic space to model hierarchical relationships between prompts and inputs . they use a Poincaré disk to capture the hierarchic relationship between prompt and input . |
| Outcome: | The proposed approach reduces training time and storage for downstream tasks by reducing training costs. |
IndicXNLI: Evaluating Multilingual Inference for Indian Languages (2022.emnlp-main)
Copied to clipboard
| Challenge: | Indic NLP has made rapid advances in terms of corpora and pre-trained models, but benchmark datasets on standard NLU tasks are limited. |
| Approach: | They propose to use an NLI dataset for 11 Indic languages to test their accuracy. |
| Outcome: | The proposed dataset provides useful insights into the behaviour of pre-trained models for a diverse set of languages. |
Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models rely heavily on idiosyncratic biases of datasets. |
| Approach: | They propose a method which discourages models from exploiting biases while enabling them to receive enough incentive to learn from all the training examples. |
| Outcome: | The proposed method improves on out-of-distribution datasets while maintaining original in-district accuracy. |
Entailment as Robust Self-Learner (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have found that entailment pretraining benefits weakly supervised fine-tuning. |
| Approach: | They propose a prompting strategy that formulates different NLU tasks as contextual entailment and propose an algorithm for better pseudo-labeling quality in self-training. |
| Outcome: | The proposed approach improves the zero-shot adaptation performance on downstream tasks. |
Mitigating Spurious Correlation in Natural Language Understanding with Counterfactual Inference (2022.emnlp-main)
Copied to clipboard
Can Udomcharoenchaikit, Wuttikorn Ponwitayarat, Patomporn Payoungkhamdee, Kanruethai Masuk, Weerayut Buaphet, Ekapol Chuangsuwanich, Sarana Nutanong
| Challenge: | Existing approaches to debias NLU models rely on superficial patterns to produce correct predictions . lexical overlap and annotation artifacts can be used to make shortcuts . |
| Approach: | They propose a causal analysis framework to help debias NLU models by defining causal relationships and utilizing counterfactual inference to mitigate bias. |
| Outcome: | The proposed framework can improve robustness across three NLU tasks while maintaining high in-distribution performance. |
Unified Representation for Non-compositional and Compositional Expressions (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing representations of non-compositional language are based on BART, but they are not as accurate as the state-of-the-art IE representation model, GIEA. |
| Approach: | They propose a language model, PIER+, that builds on BART and can generate semantically meaningful and contextually appropriate representations for English potentially idiomatic expressions. |
| Outcome: | The proposed model achieves 33% higher homogeneity score on embedding clustering than BART, while sacrificing performance on NLU tasks (+/- 1% accuracy) |
Explaining Interactions Between Text Spans (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing highlight-based explanations focus on identifying individual important features or interactions only between adjacent tokens or tuples of tokens. |
| Approach: | They propose a multi-annotator dataset of human span interaction explanations for NLU and FC. |
| Outcome: | The proposed method compares human reasoning processes to those of a fine-tuned large language model. |
When More Data Hurts: A Troubling Quirk in Developing Broad-Coverage Natural Language Understanding Systems (2022.emnlp-main)
Copied to clipboard
Elias Stengel-Eskin, Emmanouil Antonios Platanios, Adam Pauls, Sam Thomson, Hao Fang, Benjamin Van Durme, Jason Eisner, Yu Su
| Challenge: | In natural language understanding systems, users’ evolving needs necessitate the addition of new features over time, indexed by new symbols added to the meaning representation space. |
| Approach: | They propose to use a small set of new symbols to build broad-coverage NLU systems. |
| Outcome: | The proposed model is based on two prototypical NLU tasks: intent recognition and semantic parsing. |
Interventional Training for Out-Of-Distribution Natural Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for NLU training use only known and single confounders, but in many NLU tasks the confounder can be unknown and multifactorial. |
| Approach: | They propose a method that performs multi-granular intervention with identified multifactorial confounders by using a bottom-up automatic intervention method. |
| Outcome: | The proposed method performs multi-granular intervention with identified multifactorial confounders on three NLU tasks, namely, natural language inference, fact verification and paraphrase identification. |
DMLM: Descriptive Masked Language Modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Descriptive Masked Language Modeling (DMLM) is a knowledge-enhanced reading comprehension objective that requires the model to predict the most likely word in a context, being provided with the word’s definition. |
| Approach: | They propose a knowledge-enhanced reading comprehension objective where the model is required to predict the most likely word in a context, being provided with the word’s definition. |
| Outcome: | The proposed model improves on a number of well-established NLU benchmarks and other semantic-focused tasks, e.g., Semantic Role Labeling. |
CycleKQR: Unsupervised Bidirectional Keyword-Question Rewriting (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to query paraphrases are based on encoderdecoder architectures, but they do not support the two important functionalities beyond questions. |
| Approach: | They propose a keyword-question rewriting task to improve query understanding capabilities of NLU systems for all surface forms. |
| Outcome: | Empirically, we show that CycleKQR significantly improves QA performance by rewriting queries into the appropriate form while retaining the original semantic meaning of input queries. |
It’s Not under the Lamppost: Expanding the Reach of Conversational AI (2024.lrec-main)
Copied to clipboard
| Challenge: | Focused probes into the capabilities of language-based assistants easily reveal significant areas of brittleness that demonstrate large gaps in their coverage. |
| Approach: | They propose a process for collecting specific kinds of data to uncover these gaps and an annotation scheme for system responses. |
| Outcome: | The proposed system includes both Conventional and GenAI systems, including ChatGPT and Bard/Gemini. |
Macedon: Minimizing Representation Coding Rate Reduction for Cross-Lingual Natural Language Understanding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual models require limited data to perform cross-linguistic tasks. |
| Approach: | They propose a method to remove language-associated information via minimizing representation coding rate reduction. |
| Outcome: | The proposed model outperforms state-of-the-art models on cross-lingual tasks. |
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)
Copied to clipboard
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
| Challenge: | High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters. |
| Approach: | They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters. |
| Outcome: | The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches. |
CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Story generation and understanding has seen a surge in neurosymbolic work . symbolic methods are expensive and require a lot of time and expertise . |
| Approach: | They use Code-LLMs to bootstrap the use of symbolic methods for story understanding . they show that they can beat current LLM techniques on pre-existing stories with minimal hand engineering . |
| Outcome: | The proposed system beats state-of-the-art structured LLM techniques on pre-existing story understanding tasks with minimal hand engineering. |
The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources (2023.acl-long)
Copied to clipboard
Akshatha Arodi, Martin Pömsl, Kaheer Suleman, Adam Trischler, Alexandra Olteanu, Jackie Chi Kit Cheung
| Challenge: | Existing models that make inferences using information from multiple sources are largely understudied . |
| Approach: | They propose a test suite of coreference resolution subtasks that require reasoning over multiple facts and introduce subtask where knowledge is present only at inference time using fictional knowledge. |
| Outcome: | The proposed subtasks differ in terms of which knowledge sources contain the relevant facts and where knowledge is present only at inference time using fictional knowledge. |
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models excel in natural language understanding (NLU) tasks. |
| Approach: | They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance. |
| Outcome: | The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear. |
Adversarial Knowledge Stimulated Contrastive Prompting for Few-shot Language Learners (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained language models on few-shot Natural Language Understanding (NLU) tasks by employing task-specific prompts. |
| Approach: | They propose a Cloze-driven prompt framework for prompt tuning that implicitly stimulates knowledge from pre-trained language models. |
| Outcome: | The proposed framework outperforms state-of-the-art for prompt-based fine-tuning on few-shot NLU tasks. |
On Degrees of Freedom in Defining and Testing Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are performing increasingly well at various tasks and offering real-world applications. |
| Approach: | They propose to define NLU as an inductive evidence that the test subject understands the language sufficiently well to meet stakeholder objectives. |
| Outcome: | The proposed framework can be used to design credible tests and facilitate scientific communication. |
Data Query Language and Corpus Tools for Slot-Filling and Intent Classification Data (2020.lrec-1)
Copied to clipboard
| Challenge: | Typical machine learning approaches require large amounts of training data . Managing training data can be cumbersome without dedicated tools . |
| Approach: | They propose a toolkit for analyzing slot-filling and intent classification corpora . they propose 'Query Language' for searching such corporan and tools for understanding structure . |
| Outcome: | The proposed toolkit can be used to uncover interesting and surprising insights. |
ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prompt Tuning has been a popular fine-tuning method for large-scale pretrained language models. |
| Approach: | They propose a method that allows all soft prompts to share a set of learnable codebook vectors in each subspace, with each prompt differentiated by a number of adaptive weights. |
| Outcome: | The proposed method achieves superior performance on 17 diverse natural language tasks including natural language understanding (NLU) and question answering (QA) tasks by tuning only 0.3% of parameters of the PLMs. |
Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) is a technique that guides large language models to decompose complex tasks into multi-step reasoning processes. |
| Approach: | They propose a two-step reasoning framework based on prompt tuning to implement step-by-step thinking for MLMs on NLU tasks. |
| Outcome: | The proposed framework outperforms baselines and achieves state-of-the-art performance on two NLU tasks. |
HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current benchmarks for Hebrew Natural Language Processing (NLP) focus mainly on morpho-syntactic tasks, neglecting the semantic dimension of language understanding. |
| Approach: | They propose to use Hebrew machine reading comprehension (MRC) as extractive Question Answering to address this problem. |
| Outcome: | The proposed benchmark features 30,147 question-answer pairs derived from both Hebrew Wikipedia articles and Israeli tech news. |
Speech Act Patterns for Improving Generalizability of Explainable Politeness Detection Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Politeness detection is a task that requires explainability but lacks generalizability . recent approaches for improving explainable models rely on discovering domain-specific word-level features. |
| Approach: | They propose a method for improving the generalizability of explainable politeness models by relying on speech act patterns instead of words. |
| Outcome: | The proposed method improves generalizability of explainable politeness models by relying on speech act patterns instead of words. |
Mitigating Shortcuts in Language Models with Soft Label Encoding (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. |
| Approach: | They propose a framework for debiasing shortcuts and a dummy class to encode shortcuts into a model and use it to generate soft labels. |
| Outcome: | The proposed framework significantly improves out-of-distribution generalization while maintaining satisfactory in-district accuracy. |
Dense Retrieval as Indirect Supervision for Large-space Decision Making (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Dense Decision Retrieval (DDR) is a learning-to-retrieve task for discriminative natural language understanding (NLU) tasks with large label spaces. |
| Approach: | They propose a novel approach to learning large-space discriminative NLU tasks as a learning-to-retrieve task by adopting a dual-encoder architecture that learns to predict by retrieving from a decision thesaurus. |
| Outcome: | The proposed approach outperforms baselines greatly on multi-label classification tasks, 1.17% in F1 score ultra-fine entity typing, and 1.26% in accuracy on three few-shot intent classification tasks on average. |
SADA: Bridging In-Context Learning and Fine-Tuning via State-Aligned Distillation Adapters (2026.acl-long)
Copied to clipboard
| Challenge: | Prompt-based in-context learning and parameter fine-tuning are dominant paradigms for incorporating external information into large language models, but they incur high inference costs or require expensive retraining. |
| Approach: | They propose to convert prompts into temporary adapter weights to bridge this gap by converting prompts to temporary adapters. |
| Outcome: | The proposed model outperforms baselines on long-context language modeling and downstream NLU and summarization benchmarks while significantly reducing memory footprint and latency. |
From Understanding to Generation: An Efficient Shortcut for Evaluating Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Iterative evaluation of large language models during training can be time- and compute-intensive. |
| Approach: | They reformulate generative tasks into computationally cheaper NLU alternatives and test their performance correlation between them. |
| Outcome: | The proposed alternatives reduce evaluation time by 35x compared to NLU benchmarks. |
Prompting Large Language Models for Counterfactual Generation: An Empirical Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made remarkable progress in a wide range of natural language understanding and generation tasks, but their ability to generate counterfactuals has not been examined systematically. |
| Approach: | They propose a framework to evaluate LLMs' ability to generate counterfactuals based on key factors including intrinsic properties and prompt design. |
| Outcome: | The proposed framework examines the strengths and weaknesses of large language models (LLMs) and identifies factors that influence their ability to generate counterfactuals. |
Representation Degeneration Problem in Prompt-based Models for Natural Language Understanding (2024.lrec-main)
Copied to clipboard
| Challenge: | Prompt-based fine-tuning (PF) models have shown improved performance on few-shot natural language understanding benchmarks. |
| Approach: | They propose a framework to alleviate anisotropy in the embedding space by aligning with pre-trained language models' training objective. |
| Outcome: | The proposed method outperforms mainstream methods on many NLU benchmarks. |
Revisiting Data Reconstruction Attacks on Real-world Dataset for Federated Natural Language Understanding (2024.lrec-main)
Copied to clipboard
Zhuo Zhang, Jintao Huang, Xiangjing Hu, Jingyuan Zhang, Yating Zhang, Hui Wang, Yue Yu, Qifan Wang, Lizhen Qu, Zenglin Xu
| Challenge: | Existing DRA methods fail to accurately recover the original text of real-world privacy data. |
| Approach: | They propose to use a real-world privacy dataset to examine the performance of federated learning (FL) methods. |
| Outcome: | The proposed method improves on a real-world privacy dataset and shows that the tokens within a recovery sentence are disordered and intertwined with tokens from other sentences in the same training batch. |
RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in following human instructions and solving NLU tasks. |
| Approach: | They propose to use code style instructions to replace typically natural language instructions to provide more precise instructions and strengthen the robustness of LLMs. |
| Outcome: | The proposed method outperforms natural language models on eight robustness datasets and achieves an improvement of 5.68% in test set accuracy and a reduction of 5.66 points in Attack Success Rate (ASR). |
Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Large pretrained language models (LMs) are commonly adapted via fine-tuning, but full updates are costly at scale. |
| Approach: | They propose a lightweight router that activates an input-dependent subset of LoRA rank directions and turns it into dynamic rank routing. |
| Outcome: | The proposed method improves accuracy–efficiency Pareto frontier versus static-rank LoRA and adaptive-rank baselines, while preserving memory and reducing overhead. |
skLEP: A Slovak General Language Understanding Benchmark (2025.findings-acl)
Copied to clipboard
Marek Suppa, Andrej Ridzik, Daniel Hládek, Tomáš Javůrek, Viktória Ondrejová, Kristína Sásiková, Martin Tamajka, Marian Simko
| Challenge: | skLEP is the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding models. |
| Approach: | They introduce a benchmark specifically designed for evaluating Slovak natural language understanding models. |
| Outcome: | The proposed benchmark covers nine tasks that span token-level, sentence-pair, document-level tasks. |
Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction-fine-tuned large language models (LLMs) under 14B parameters underperform on NLU tasks . we explore a framework to improve the NLU capabilities of LLMs . |
| Approach: | They propose to use Proximal Policy Optimization to improve NLU capabilities . they frame NLU as a reinforcement learning environment and optimize for reward signals . |
| Outcome: | The proposed framework outperforms supervised fine-tuning on GLUE and superGLUE tasks. |
Towards Equitable Natural Language Understanding Systems for Dialectal Cohorts: Debiasing Training Data (2024.lrec-main)
Copied to clipboard
| Challenge: | Prior research has shown that biases exist in these models against certain languages or dialects. |
| Approach: | They propose to use a dialect identification model to obtain targeted training data augmentation for under-represented dialects to debias NLU model for dialectal cohorts in NLU systems. |
| Outcome: | The proposed framework can provide insights on dialect disparity in real-world NLU systems and targeted data argumentation can help narrow the model’s performance gap between standard language speakers and dialect speakers. |
Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained models are limited due to suboptimal activation subspaces. |
| Approach: | They propose a method that leverages tail eigenvectors of model output activations to construct low-rank adapters. |
| Outcome: | The proposed method outperforms existing methods across 16 benchmarks and surpasses full fine-tuning in certain scenarios. |
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to estimate uncertainty use predictive confidence, structural characteristics of representation space, or stochastic variation in model outputs. |
| Approach: | They propose a new uncertainty estimation framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples. |
| Outcome: | The proposed framework outperforms or matches existing methods on several NLU benchmarks and sentiment analysis benchmarks. |
UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages (2026.acl-long)
Copied to clipboard
Pranjal A Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram
| Challenge: | Developing culturally grounded multilingual AI systems is challenging for low-resource languages . synthetic data is underexplored, but its effectiveness in multilingual and multicultural contexts is understudied . |
| Approach: | They propose a top-up synthetic data generation framework grounded in Wikipedia content . they use 9.5M data points across 13 Indian languages and English to generate a high-quality dataset . |
| Outcome: | The proposed model improves on NLG tasks and narrows performance gaps with high-resource languages. |
A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | a novel multi-task learning framework for domain-specific natural language understanding tasks addresses these limitations by combing multiple tasks into a single framework. |
| Approach: | They propose a multi-task learning framework that decomposes the language model into modular skill components and employs a dynamic, learnable skill-combination mechanism to adaptively handle diverse tasks. |
| Outcome: | The proposed framework surpasses conventional multi-task learning approaches in performance. |
SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation (2026.acl-long)
Copied to clipboard
| Challenge: | Slovak embeddings are core infrastructure for semantic search, retrieval-augmented generation (RAG), clustering, and classification. |
| Approach: | They propose a MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language . they use 31 datasets across 7 task types to evaluate the performance of the models . |
| Outcome: | The proposed model achieves competitive performance with proprietary APIs while remaining locally deployable for RAG . the model is based on 31 datasets across 7 task types and is 4 the depth of existing benchmark for Slovak . |