Papers with BERT
ActiveLLM: Large Language Model-Based Active Learning for Textual Few-Shot Scenarios (2026.tacl-1)
Copied to clipboard
| Challenge: | Active learning strategies struggle with a ‘cold-start’ problem, needing substantial initial data to be effective. |
| Approach: | They propose an active learning approach that leverages Large Language Models such as GPT-4, o1, Llama 3, or Mistral Large for selecting instances. |
| Outcome: | The proposed approach outperforms existing methods ADAPET, PERFECT, and SetFit in few-shot scenarios and can be extended to non-few scenarios. |
AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models (D19-3)
Copied to clipboard
| Challenge: | Existing interpretation codebases make it difficult to apply these methods to new models and tasks. |
| Approach: | They propose a framework for interpreting NLP models that provides explanations for specific models. |
| Outcome: | The proposed framework provides interpretation primitives for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. |
Zero-Shot Cross-Lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on improving cross-lingual transferability of NMT model is under-explored. |
| Approach: | They propose a model that leverages a multilingual pretrained encoder to improve cross-lingual transferability. |
| Outcome: | The proposed model outperforms mBART and m2m-100 on a zero-shot cross-lingual transfer task. |
Double Trouble: How to not Explain a Text Classifier’s Decisions Using Counterfactuals Synthesized by Masked Language Models? (2022.aacl-main)
Copied to clipboard
| Challenge: | Input Marginalization (IM) is a method that takes the prediction difference between before-and-after an input feature (here, a token) is removed as its attribution. |
| Approach: | They propose to use a BERT-based method to replace a token with a feature to give more plausible counterfactuals. |
| Outcome: | The proposed method is effective, but the Deletion-BERT metric is biased towards IM, and the results are not convincing. |
Can Monolingual Pretrained Models Help Cross-Lingual Classification? (2020.aacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have shown impressive results for cross-lingual transfer, but due to the constant model capacity, multilingual pre-training usually lags behind the monolingual competitors. |
| Approach: | They propose to transfer the knowledge from monolingual pretrained models to multilingual ones to improve zero-shot cross-lingual classification by using machine translation systems. |
| Outcome: | The proposed methods outperform vanilla multilingual fine-tuning on two cross-lingual classification benchmarks. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
When Choosing Plausible Alternatives, Clever Hans can be Clever (D19-60)
Copied to clipboard
| Challenge: | Pretrained language models have shown large improvements in the commonsense reasoning benchmark COPA, but recent work has identified superficial cues in benchmark datasets which are predictive of the correct answer. |
| Approach: | They propose an extension of COPA that does not suffer from easy-to-exploit single token cues and exploits them. |
| Outcome: | The proposed extension of COPA does not suffer from easy-to-exploit single token cues. |
Applying BERT to Document Retrieval with Birch (D19-3)
Copied to clipboard
| Challenge: | Birch is an open-source document retrieval system that integrates with the Anserini information retrieval toolkit to demonstrate end-to-end search over large document collections. |
| Approach: | They propose to integrate Anserini with a BERT-based document ranking model that provides an end-to-end open-source search engine. |
| Outcome: | The proposed system outperforms existing approaches to document retrieval and question answering on standard newswire and social media test collections. |
Knowledge Enhanced Contextual Word Representations (D19-1)
Copied to clipboard
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith
| Challenge: | Existing methods to embed knowledge bases into large pre-training models do not contain any explicit grounding to real world entities and are difficult to recover factual knowledge. |
| Approach: | They propose a method to embed multiple knowledge bases (KBs) into large pretrained models with a Knowledge Attention and Recontextualization mechanism. |
| Outcome: | The proposed model improves perplexity, ability to recall facts and word sense disambiguation. |
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)
Copied to clipboard
| Challenge: | Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text. |
| Approach: | They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span. |
| Outcome: | The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks. |
A Deep Learning-Based System for PharmaCoNER (D19-57)
Copied to clipboard
Ying Xiong, Yedan Shen, Yuanhang Huang, Shuai Chen, Buzhou Tang, Xiaolong Wang, Qingcai Chen, Jun Yan, Yi Zhou
| Challenge: | Efficient access to mentions of clinical entities is very important for using clinical text. |
| Approach: | They developed a pipeline system based on deep learning methods for this shared task . it achieves a micro-average F1-score of 0.9105 on track 1 and a mini-average LSTM score of 0.8391 on track 2 . |
| Outcome: | The proposed system achieves a micro-average F1-score of 0.9105 on track 1 and a mini-average score of 0.8391 on track 2. |
Simple yet Effective Bridge Reasoning for Open-Domain Multi-Hop Question Answering (D19-58)
Copied to clipboard
| Challenge: | Existing work on open-domain multi-hop question answering relies on off-the-shelf information retrieval techniques to retrieve answer passages. |
| Approach: | They propose a new subproblem for open-domain multi-hop question answering . they aim to recognize the anchor from a set of start passages with a reading comprehension model . |
| Outcome: | The proposed method significantly improves the baseline method on the open-domain hotpotQA benchmark. |
A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsing (2020.tacl-1)
Copied to clipboard
| Challenge: | Chinese word segmentation and dependency parsing suffer from error propagation . a graph-based model can integrate both tasks, but it suffers from performance limitations . |
| Approach: | They propose a graph-based model to integrate Chinese word segmentation and dependency parsing . their model achieves better performance than previous joint models . |
| Outcome: | The proposed model achieves better performance than previous joint models and state-of-the-art results in both Chinese word segmentation and dependency parsing. |
BERT is Not an Interlingua and the Bias of Tokenization (D19-61)
Copied to clipboard
| Challenge: | Cananical Correlation Analysis (CCA) of the internal representations of a pre- trained, multilingual BERT model reveals that the model partitions representations for each language rather than using a common, shared, interlingual space. |
| Approach: | They propose to use a multilingual BERT model to partition representations for each language rather than using a common, shared, interlingual space. |
| Outcome: | The results show that the model partitions representations for each language rather than using a common, shared, interlingual space. |
AdapterHub: A Framework for Adapting Transformers (2020.emnlp-demos)
Copied to clipboard
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, Iryna Gurevych
| Challenge: | AdapterHub framework enables dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
| Approach: | They propose a framework that allows dynamic "stiching-in" of pre-trained adapters for different tasks and languages. |
| Outcome: | The proposed framework allows dynamic “stiching-in” of pre-trained adapters for different tasks and languages. |
Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement (2021.tacl-1)
Copied to clipboard
| Challenge: | RNGTr is a non-recursive Graph-to-Graph Transformer for iterative refinement of graphs . it can improve the accuracy of initial parsers on 13 languages . |
| Approach: | They propose a recursive non-autoregressive Graph-to-Graph Transformer architecture for iterative refinement of arbitrary graphs and apply it to syntactic dependency parsing. |
| Outcome: | The proposed architecture can improve state-of-the-art on 13 languages and the German CoNLL2009 corpus. |
Unsupervised Question Answering for Fact-Checking (D19-66)
Copied to clipboard
| Challenge: | Recent Deep Learning (DL) models have achieved human-level accuracy on natural language tasks such as question-answering, natural language inference, and textual entailment. |
| Approach: | They propose an unsupervised question-answering based approach for a similar task, fact-checking. |
| Outcome: | The proposed approach achieves label accuracy of 80.2% on the development set and 80.25% on the test set. |
Multilingual Language Models Predict Human Reading Behavior (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that cognitively motivated "attention" mechanism in neural models is not a good indicator for relative importance. |
| Approach: | They compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing. |
| Outcome: | The proposed models predict reading time measures on Dutch, English, German, and Russian texts. |
Book QA: Stories of Challenges and Opportunities (D19-58)
Copied to clipboard
| Challenge: | Existing approaches to answer questions based on the full text of books are limited by their unique characteristics. |
| Approach: | They propose a system for answering questions based on the full text of books . they use a memory network to reason and predict an answer, and a novel question generator to improve generalization. |
| Outcome: | The proposed system improves on the recently published NarrativeQA corpus on Who questions . it shows that the proposed system is highly challenging and needs more research . |
Impact of Training Instance Selection on Domain-Specific Entity Extraction using BERT (2022.naacl-srw)
Copied to clipboard
| Challenge: | Named entity recognition (NER) tasks require a large number of training examples and handcrafted features. |
| Approach: | They propose to fine-tune pre-trained language models such as BERT to achieve up to 80% F1 when fine- tuned on only 70 training examples. |
| Outcome: | The proposed model achieves 80% F1 when fine-tuned on only 70 training examples, especially on biomedical domain. |
ABNIRML: Analyzing the Behavior of Neural IR Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc ranking. |
| Approach: | They propose a framework for Analyzing the Behavior of Neural IR ModeLs that includes new types of diagnostic probes that allow us to test several characteristics that are not addressed by previous techniques. |
| Outcome: | The proposed framework tests writing styles, factuality, sensitivity to paraphrasing and word order, and can be used to identify unintended biases. |
Do Multi-hop Readers Dream of Reasoning Chains? (D19-58)
Copied to clipboard
| Challenge: | Existing models for multihop reasoning are limited in their performance . multi-hop reasoning requires the ability to gather information from multiple passages . |
| Approach: | They propose a method that provides the full reasoning chain of multiple passages instead of just one final passage where the answer appears. |
| Outcome: | The proposed model improves on existing models by providing the full reasoning chain of multiple passages instead of just one final passage where the answer appears. |
N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses (2021.acl-short)
Copied to clipboard
| Challenge: | Spoken Language Understanding systems parse spoken utterances into semantic structures like dialog acts and slots. |
| Approach: | They propose to use concatenated N-best ASR alternatives to represent utterances . they propose to employ a simpler utteration representation with no special delimiter . |
| Outcome: | The proposed model outperforms the prior state-of-the-art model on DSTC2 dataset. |
Unification-based Reconstruction of Multi-hop Explanations for Science Questions (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches build explanations considering each question in isolation, but new approach leverages explanatory patterns emerging in scientific explanations. |
| Approach: | They propose a framework for reconstructing multi-hop explanations in science Question Answering . they integrate lexical relevance with the notion of unification power to rank atomic facts . |
| Outcome: | The proposed method achieves results competitive with Transformers, but is faster and scalable to large explanatory corpora. |
The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding (2020.acl-demos)
Copied to clipboard
Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, Jianfeng Gao
| Challenge: | MT-DNN is an open-source natural language understanding toolkit . it allows researchers and developers to train customized deep learning models . |
| Approach: | They present MT-DNN, an open-source natural language understanding toolkit . it is designed to facilitate rapid customization for a broad spectrum of NLU tasks . MT supports multi-task knowledge distillation, which can substantially compress a deep neural model without significant performance drop. |
| Outcome: | The proposed model can significantly compress a large model without significant performance drop. |
How does the pre-training objective affect what large language models learn about linguistic properties? (2022.acl-short)
Copied to clipboard
| Challenge: | Several pre-training objectives have been proposed to pre-train language models . but, to our knowledge, no studies have investigated how different pre- training objectives affect what BERT learns about linguistic properties. |
| Approach: | They propose to use masked language modeling to pre-train language models . they propose to optimize a mangled language modeling objective to learn linguistic information . |
| Outcome: | The proposed objectives improve BERT's learning of linguistic properties compared to non-linguistically motivated objectives. |
Non-Autoregressive Text Generation with Pre-trained Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Autoregressive generation models generate tokens in a left-to-right, token-by-token fashion, resulting in lag in inference. |
| Approach: | They propose to use BERT as the backbone of a non-autoregressive generation model for greatly improved performance. |
| Outcome: | The proposed model outperforms existing non-autoregressive models and achieves competitive performance with many strong autoregressive model. |
BERT-Based Neural Collaborative Filtering and Fixed-Length Contiguous Tokens Explanation (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing models that learn accurate representations of users and items are based on ratings, which oversimplify user preferences and item characteristics. |
| Approach: | They propose a novel, accurate, and explainable recommender model that integrates three key elements: BERT, multilayer perceptron, and maximum subarray problem to derive contextualized review features, model user-item interactions, and generate explanations. |
| Outcome: | The proposed model outperforms state-of-the-art models by an improvement gain of nearly 7% based on the human judges’ assessment . |
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation (2022.emnlp-main)
Copied to clipboard
| Challenge: | despite the importance of question answering, evaluations of QA systems are typically limited by manual annotations . despite this, little progress has been made in QA evaluations based on a single answer . |
| Approach: | They propose to extend over exact match (EM) with predefined rules or token-level F1 measure . they propose to use a BERT matching measure to approximate QA predictions . |
| Outcome: | The proposed model improves AE approximations and more accurately reflects the performance of systems. |
Knowledge extraction from aeronautical messages (NOTAMs) with self-supervised language models for aircraft pilots (2022.naacl-industry)
Copied to clipboard
| Challenge: | During pre-flight briefings, aircraft pilots analyse a long list of NOTAMs . the messages are usually written in the English language, but the phrasing is very special . |
| Approach: | They pretrain language models derived from BERT on circa 1 million unlabeled NOTAMs . they reuse the learnt representations on three downstream tasks valuable for pilots - criticality prediction, named entity recognition and translation into a structured language called Airlang. |
| Outcome: | The proposed language model can be used on criticality prediction, named entity recognition and translation into a structured language called Airlang. |
FASPell: A Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based On DAE-Decoder Paradigm (D19-55)
Copied to clipboard
| Challenge: | Existing spell checkers for Chinese are based on denoising autoencoder and decoder paradigms that require a small amount of data to be effective. |
| Approach: | They propose a Chinese spell checker based on a new paradigm which consists of a denoising autoencoder and a decoder. |
| Outcome: | The proposed spell checker is faster, more Adaptable to simplified and traditional Chinese texts and has a much simpler structure to be as much Powerful in error detection and correction. |
Extractive NarrativeQA with Heuristic Pre-Training (D19-58)
Copied to clipboard
| Challenge: | Automated question answering (QA) from text remains a challenge for humans . a striking gap exists between machine and human performance on NLP tasks . |
| Approach: | They propose a heuristic extractive version of a data set to solve the problem of answer extraction rather than generation. |
| Outcome: | The proposed model outperforms previous models on summary-level QA from full narratives and on the METEOR metric. |
SwahBERT: Language Model of Swahili (2022.naacl-main)
Copied to clipboard
| Challenge: | Social media and Internet forums are valuable sources of citizens’ opinions, which can be analyzed for community development and user behavior analysis. |
| Approach: | They present a pre-training and annotated datasets of Swahili and an emotion classification datasets that are manually annotating by two native Swahils. |
| Outcome: | The proposed model outperforms existing monolingual language model in almost all downstream tasks. |
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding (D19-58)
Copied to clipboard
| Challenge: | In-domain datasets are used to train and validate our model, and other out-of-domain data are used for validation. |
| Approach: | They propose a model which uses cross-task learning with expert representation for the generalization of reading and understanding. |
| Outcome: | The proposed model achieved an average F1 score of 66.1 % in the out-of-domain setting, which is a 4.3 percentage point improvement over the official BERT baseline model. |
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pretraining frameworks do not perform well for all tasks of three main categories, such as natural language understanding (NLU), unconditional generation, and conditional generation. |
| Approach: | They propose a general language model based on autoregressive blank infilling to address this challenge. |
| Outcome: | The proposed model outperforms BERT, T5, and GPT on a wide range of tasks across NLU, conditional and unconditional generation tasks. |
Forged-GAN-BERT: Authorship Attribution for LLM-Generated Forged Novels (2024.eacl-srw)
Copied to clipboard
| Challenge: | generative Large Language Models (LLMs) are capable of producing human-like texts, but they pose challenges related to the authenticity of the text documents. |
| Approach: | They propose a modified GANBERT-based model to improve the classification of forged novels via the Forged Novels Generator and the generator in GAN. |
| Outcome: | The proposed model improves classification of forged novels in two data-augmentation aspects. |
Understanding tables with intermediate pre-training (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Textual entailment is well studied, but is less well studied for table enlargement . a new dataset of millions of examples is used to train the model . |
| Approach: | They adapt a table-based BERT model to recognize entailment from a dataset . they evaluate table pruning techniques as a pre-processing step to improve model efficiency . |
| Outcome: | The proposed model improves training and prediction efficiency at a moderate drop in accuracy. |
CMTA: COVID-19 Misinformation Multilingual Analysis on Twitter (2021.acl-srw)
Copied to clipboard
| Challenge: | myths, sensationalism, rumours and misinformation, generated intentionally or unintentionally, spread rapidly through social networks during the COVID-19 pandemic . evaluation of tweets for recognizing misinformation can create beneficial understanding to review the top quality and also the readability of online information concerning the COV-19. |
| Approach: | They propose a multilingual COVID-19 related tweet analysis method that uses a deep learning model for multilingual tweet misinformation detection and classification. |
| Outcome: | The proposed method outperforms monolingual models in the misinformation detection task and shows that it can be used to improve the quality and readability of online information. |
PG-GSQL: Pointer-Generator Network with Guide Decoding for Cross-Domain Context-Dependent Text-to-SQL Generation (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to text-to-SQL generation depend on interaction history and current utterances. |
| Approach: | They propose an encoder-decoder model based on interaction-level encoder to capture historical information of SQL query and reuse the previous SQL query tokens. |
| Outcome: | The proposed model outperforms the previous state-of-the-art model on the SParC benchmark . it achieves 34.0% question matching accuracy and 19.0% interaction matching accuracy . |
PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models (2020.tacl-1)
Copied to clipboard
| Challenge: | PERL is a representation learning model that uses labeled data from the source domain and unlabeled data not necessarily drawn from the target domain. |
| Approach: | They propose a model that extends contextualized word embedding models with pivot-based fine-tuning to address this bottleneck. |
| Outcome: | The proposed model outperforms strong baselines across 22 sentiment classification domain adaptation setups and improves in-domain model performance. |
Meta-learning Pathologies from Radiology Reports using Variance Aware Prototypical Networks (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for fewshot learning require a large number of in-domain labeled examples for fine tuning. |
| Approach: | They propose to extend the Prototypical Networks for few-shot text classification by replacing Gaussian class prototypes with a regularization term that encourages the examples to be clustered near the appropriate class centroids. |
| Outcome: | The proposed method outperforms baselines on 13 public and 4 internal datasets and detects potential out-of-distribution (OOD) data points during deployment. |
MDERank: A Masked Document Embedding Rank Approach for Unsupervised Keyphrase Extraction (2022.findings-acl)
Copied to clipboard
| Challenge: | Keyphrase extraction (KPE) extracts phrases in a document that provide a concise summary of the core content. |
| Approach: | They propose an unsupervised keyphrase extraction method that ranks candidates by similarity between embeddings of source document and masked document. |
| Outcome: | The proposed method outperforms state-of-the-art methods on six benchmarks . it achieves average 3.53 improvement over the existing method . |
BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have improved performance of text classification tasks, but they still suffer from spurious domain-specific clues. |
| Approach: | They propose a method to augment pretrained language models by generating texts in any desired genre and on any desired topic. |
| Outcome: | The proposed method improves on genre classification tasks while showing no improvement for other topics. |
Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition (2021.tacl-1)
Copied to clipboard
| Challenge: | Name Regularity Bias is a problem in NER models that use contextual information to predict the type of an ambiguous entity. |
| Approach: | They propose a model-agnostic training method that adds learnable adversarial noise to some entity mentions to improve their accuracy. |
| Outcome: | The proposed method outperforms feature-based models on name regularity bias . it adds learnable adversarial noise to some entity mentions, leading to gains . |
Beyond Reptile: Meta-Learned Dot-Product Maximization between Gradients for Improved Single-Task Regularization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve generalization of neural models use a small component of the gradient for maximizing dot-product between batches. |
| Approach: | They propose to use a finite differences first-order algorithm to calculate a gradient from dot-product of gradients and regularize it. |
| Outcome: | The proposed method outperforms previous approaches of Reptile and MAML when used as a regularization technique. |
eMLM: A New Pre-training Objective for Emotion Related Tasks (2021.acl-short)
Copied to clipboard
| Challenge: | Emotion Masked Language Modelling improves the performance of a pretraining language model for emotion detection and sentiment analysis tasks. |
| Approach: | They propose a BERT-based version of Masked Language Modelling that induces emotion into the model. |
| Outcome: | The proposed model improves on emotion detection and sentiment analysis tasks by 1.2% F-1 . the proposed model also shows increased robustness in the test. |
Cross-Lingual Disaster-related Multi-label Tweet Classification with Manifold Mixup (2020.acl-srw)
Copied to clipboard
| Challenge: | Towards this goal, many studies have focused on disaster-related tweet classification. |
| Approach: | They compile a multilingual dataset for multi-label classification of disaster-related tweets . they show that their model generalizes to unseen disasters in the test set . |
| Outcome: | The proposed model generalizes to unseen disasters and improves with Manifold Mixup. |
Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Infrequent names are less similar to initial representations, and are more self-similar, suggesting that models rely on less context-informed representations of uncommon and minority names. |
| Approach: | They use a dataset of U.S. first names with labels based on predominant gender and racial group to examine effect of training corpus frequency on tokenization, contextualization, similarity to initial representation, and bias. |
| Outcome: | The results show that infrequent names are less similar to initial representations and have a Spearman’s rho between frequency and self-similarity as low as .763 . |
Dynamic Masking Rate Schedules for MLM Pretraining (2024.eacl-short)
Copied to clipboard
| Challenge: | Most transformers trained with the Masked Language Modeling objective use a fixed masking rate of 15%, but wettig et al. ( 2022) found that the standard 15% masking rates are suboptimal for a variety of model settings and recommended a higher rate. |
| Approach: | They propose to dynamically schedule the masking rate throughout training to improve GLUE accuracy by 0.46% and 0.25% in BERT-base and BERT large, respectively. |
| Outcome: | The proposed method improves GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT large compared to baselines. |
LECO: Improving Early Exiting via Learned Exits and Comparison-based Exiting Mechanism (2023.acl-srw)
Copied to clipboard
| Challenge: | Recent work on dynamic early exiting has neglected the intermediate exits’ architectural designs. |
| Approach: | They propose a framework for learning exits and COmparison-based early exiting to improve PTMs’ early exit performance. |
| Outcome: | The proposed framework achieves the SOTA performance on multi-exit BERT training and dynamic early exiting on pre-trained models. |
Open Domain Question Answering over Tables via Dense Retrieval (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent advances in open-domain QA focus on retrieving textual passages . a retriever designed to handle tabular context can improve retrieval quality . |
| Approach: | They propose a tabular-based retrieval model that improves retrieval quality over a BERT-based retriever. |
| Outcome: | The proposed retriever improves retrieval quality with mined hard negatives over a BERT-based retriever. |
Sharing Encoder Representations across Languages, Domains and Tasks in Large-Scale Spoken Language Understanding (2023.acl-industry)
Copied to clipboard
Jonathan Hueser, Judith Gaspers, Thomas Gueudre, Chandana Prakash, Jin Cao, Daniil Sorokin, Quynh Do, Nicolas Anastassacos, Tobias Falke, Turan Gojayev
| Challenge: | Larger encoders can improve accuracy for spoken language understanding (SLU) but are difficult to use given the inference latency constraints of online systems. |
| Approach: | They propose to use a larger 170M parameter BERT encoder that shares representations across languages, domains and tasks for SLU. |
| Outcome: | The proposed encoders achieve state-of-the-art performance on numerous NLP tasks. |
oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)
Copied to clipboard
| Challenge: | Recent success of pre-trained language models has spurred widespread interest in their capabilities. |
| Approach: | They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls . |
| Outcome: | The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls. |
OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies on neural networks with pre-trained weights focus on low-dimensional subspace where the embedding vectors computed from input words are located. |
| Approach: | They propose an approach to find and regularize the remainder of the space, referred to as out-of-manifold, which cannot be accessed through the words. |
| Outcome: | The proposed approach is able to fine-tune the out-of-manifold embedding space on text classification benchmarks. |
Classification and Clustering of Arguments with Contextualized Word Embeddings (P19-1)
Copied to clipboard
| Challenge: | Existing methods for argument mining focus on analyzing local argumentation structures, but information-seeking approaches need to be able to deal with heterogeneous sources and topics. |
| Approach: | They propose to use contextualized word embeddings to classify and cluster topic-dependent arguments using a UKP Sentential Argument Mining Corpus and IBM Debater - Evidence Sentences datasets. |
| Outcome: | The proposed method improves state-of-the-art on argument classification and clustering tasks and across multiple datasets. |
Named Entity Recognition for Chinese biomedical patents (2020.coling-main)
Copied to clipboard
| Challenge: | Existing attempts to address NER for Chinese biomedical texts have been limited due to the amount of Chinese biomedicine discoveries being patented. |
| Approach: | They train and evaluate Chinese biomedical patents NER models based on BERT . their model is optimized for Chinese bio-patent data and scored an F1 . |
| Outcome: | The proposed model achieves an F1 score of 0.540.15 for Chinese biomedical patent data. |
Negation Detection in Dutch Spoken Human-Computer Conversations (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing negation detection methods in English are not available. |
| Approach: | They propose to annotate a Dutch dialogue corpus with negation cues and their scopes. |
| Outcome: | The proposed method can detect negation cues and scope in Dutch dialogues with high precision and recall. |
Revisiting and Advancing Chinese Natural Language Understanding with Accelerated Heterogeneous Knowledge Pre-training (2022.emnlp-industry)
Copied to clipboard
Taolin Zhang, Junwei Dong, Jianing Wang, Chengyu Wang, Ang Wang, Yinghui Liu, Jun Huang, Yong Li, Xiaofeng He
| Challenge: | Existing knowledge-enhanced pre-trained language models (KEPLMs) can capture internal knowledge, but can't understand external background knowledge. |
| Approach: | They propose to use Chinese knowledge-enhanced pre-trained language models to improve context-aware representations via learning from structured relations in knowledge bases. |
| Outcome: | Experiments show that Chinese knowledge-enhanced pre-trained language models outperform strong baselines over various benchmark NLP tasks and in different model sizes. |
Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized general natural language preprocessing tasks, but their performance in financial domains is not evaluated comprehensively. |
| Approach: | They propose a framework to evaluate financial language models on financial tasks . they compare performance of auto-encoding language models and ChatGPT . |
| Outcome: | The proposed framework compares the performance of auto-encoding language models and the LLM ChatGPT on financial tasks. |
Liputan6: A Large-scale Indonesian Dataset for Text Summarization (2020.aacl-main)
Copied to clipboard
| Challenge: | Despite having the fourth largest speaker population in the world, 1 Indonesian is under-represented in NLP. |
| Approach: | They propose to use a large-scale Indonesian summarization dataset to test extractive and abstractive summarizing methods. |
| Outcome: | The proposed methods are compared with multilingual and monolingual BERT-based models. |
Disentangling Representations of Text by Masking Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained models such as BERT encode a range of features into monolithic vectors, providing strong predictive accuracy across downstream tasks. |
| Approach: | They explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects. |
| Outcome: | The proposed method disentangles sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
“Will You Find These Shortcuts?” A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared. |
| Approach: | They propose a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. |
| Outcome: | The proposed method is based on partially synthetic data and is compared with lexical shortcuts on a range of datasets and LSTM models. |
A Framework for Flexible Extraction of Clinical Event Contextual Properties from Electronic Health Records (2025.acl-industry)
Copied to clipboard
| Challenge: | EHRs contain vast amounts of valuable clinical data, stored as unstructured text. |
| Approach: | They propose a method that uses existing NER+L methods to classify medical entities at scale using a named entity recognition and linking task. |
| Outcome: | The proposed model outperforms Bi-LSTM in minority class tasks with up to 28% of the time and 32% faster training time. |
Effective Use of Transformer Networks for Entity Tracking (D19-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models for entity-related tasks are not able to handle the nuances of procedural text. |
| Approach: | They propose to use pre-trained transformer networks to track entities in procedural text by restructuring input to focus on a particular entity. |
| Outcome: | The proposed models outperform baseline models on ingredient detection in recipes and QA over scientific processes on two different tasks. |
Potential Idiomatic Expression (PIE)-English: Corpus for Classes of Idioms (2022.lrec-1)
Copied to clipboard
Tosin Adewumi, Roshanak Vadoodi, Aparajita Tripathy, Konstantina Nikolaido, Foteini Liwicki, Marcus Liwicki
| Challenge: | Potential Idiomatic Expression (PIE) dataset for NLP in English contains over 20,100 samples with almost 1,200 cases of idioms from 10 classes (or senses). |
| Approach: | They present a large Potential Idiomatic Expression (PIE) dataset for Natural Language Processing (NLP) in English. |
| Outcome: | The proposed dataset contains over 20,100 samples with almost 1,200 cases of idioms (with their meanings) from 10 classes (or senses). |
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining large (masked) language models over EHR data has yielded consistent performance gains across tasks. |
| Approach: | They propose to use large Transformers to release pretraining models over EHRs . they propose to recover patient names and conditions associated with them . |
| Outcome: | The proposed models recover patient names and conditions associated with patients . the proposed models share the model parameters for use by other researchers . |
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately. |
| Approach: | They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training . |
| Outcome: | Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP. |
Exploring Cross-sentence Contexts for Named Entity Recognition with BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is often addressed as a sequence classification task with each input consisting of one sentence of text. |
| Approach: | They propose a method to combine different predictions from multiple sentences in input samples to increase NER performance. |
| Outcome: | The proposed method improves on the state-of-the-art NER results on English, Dutch, and Finnish and achieves the best reported BERT-based results on German. |
A Self-supervised Joint Training Framework for Document Reranking (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models have been successfully applied to a wide range of tasks . however, the pretraining tasks were based on the context of documents . |
| Approach: | They propose a self-supervised joint training framework with a method called Masked Query Prediction to establish semantic relations between given queries and positive documents. |
| Outcome: | The proposed framework outperforms existing models on document reranking tasks without further pre-training . it uses a self-supervised method to establish semantic relations between given queries and positive documents. |
Towards Improving Adversarial Training of NLP Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for generating NLP adversarial examples involve combinatorial search and expensive sentence encoders for constraining the generated instances. |
| Approach: | They propose to use vanilla adversarial training to train NLP models using a word substitution attack optimized for vanilla adversary training. |
| Outcome: | The proposed approach improves model performance and standard accuracy and can defend against other types of word substitution attacks. |
You May Like This Hotel Because ...: Identifying Evidence for Explainable Recommendations (2020.aacl-main)
Copied to clipboard
| Challenge: | Several studies have addressed explainable recommendations that produce natural language sentences . however, this task cannot explain detailed evidences for each hotel . |
| Approach: | They propose to decompose the process into two subtasks: Evidence Identification and Evidence Explanation. |
| Outcome: | The proposed model can explain evidences in recommending hotels given vague requests . it can find evidence sentences with respect to various vague requests and generate recommendation sentences . |
Robust Question Answering Through Sub-part Alignment (2021.naacl-main)
Copied to clipboard
| Challenge: | Current textual question answering models fail to generalize to out-of-domain settings. |
| Approach: | They propose to decompose question and context into smaller units and align them to find the answer. |
| Outcome: | The proposed model is more robust than the standard BERT QA model on adversarial and out-of-domain datasets. |
Multiˆ2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing open IE systems were based on handcrafted features or fine-grained rules. |
| Approach: | They propose a multi-head argument extraction method using multi-lingual BERT . they use a query, key, and value setting inspired by the Multimodal Transformer . |
| Outcome: | The proposed method outperforms existing sequence-labeling systems on two benchmark datasets and on two languages without training data. |
ConTextING: Granting Document-Wise Contextual Embeddings to Graph Neural Networks for Inductive Text Classification (2022.coling-1)
Copied to clipboard
| Challenge: | Graph neural networks (GNNs) are used to learn document representation from graph structures. |
| Approach: | They propose a unified model with a joint training mechanism to learn from document embeddings and contextual word interactions simultaneously. |
| Outcome: | The proposed model outperforms pure inductive GNNs and BERT-style models . the proposed model also has a joint training mechanism to learn from document embeddings and contextual word interactions simultaneously. |
Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling (2023.emnlp-main)
Copied to clipboard
| Challenge: | asymmetric outliers in transformer language models are a challenge for post-training quantization . we propose a framework for outlier suppression that can be seamlessly migrated into subsequent modules . |
| Approach: | They propose a framework for post-training quantization that includes the channel-wise shifting and scaling for concentration. |
| Outcome: | The proposed framework can be migrated into subsequent modules while maintaining equivalence. |
An Isotropy Analysis in the Multilingual BERT Embedding Space (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored the advantages of multilingual pre-trained models in capturing shared linguistic knowledge. |
| Approach: | They investigate the anisotropic embedding space and outlier dimensions of the multilingual BERT model for two known issues of the monolingual models. |
| Outcome: | The proposed model has no outlier dimension and has highly anisotropic space . the results show that increasing the isotropy of multilingual space can improve its representation power and performance, similar to what had been observed for monolingual CWRs on semantic similarity tasks. |
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that pretrained Masked Language Models are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. |
| Approach: | They propose a contrastive learning technique which turns pretrained MLMs into effective universal lexical and sentence encoders without additional data. |
| Outcome: | The proposed technique can turn MLMs into effective universal lexical and sentence encoders even without additional data. |
A Systematic Search for Compound Semantics in Pretrained BERT Architectures (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing models for noun compounds have been less successful in predicting compositionality than transformers . authors: suboptimal use of encoded information may be a contributing factor . performance of transformer-based models is poor, authors say . |
| Approach: | They propose to use semantic knowledge derived from pretrained BERT to predict compositionality . they find distinct linguistic roles of heads and modifiers are reflected by differences in BERT representations . |
| Outcome: | The proposed model improves on unsupervised implementations of pretrained BERT . empirical properties such as frequency, productivity, and ambiguity affect performance . |
Drop Dropout on Single Epoch Language Model Pretraining (2025.findings-acl)
Copied to clipboard
| Challenge: | Initial dropout was seen as a breakthrough regularization technique that reduced overfitting, yet single-epoch pretraining tasks common to modern LLMs yield minimal overfit. |
| Approach: | They propose to use dropout during single-epoch pretraining to reduce overfitting in language modeling, morpho-syntax, question answering, and MNLI to improve performance. |
| Outcome: | The results show that dropout is not used in large LLMs and improves performance in language modeling, morpho-syntax, question answering, and MNLI. |
Linguistic Knowledge and Transferability of Contextual Representations (N19-1)
Copied to clipboard
| Challenge: | Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence. |
| Approach: | They compare pretrained word representations with 16 diverse probing tasks to examine their transferability. |
| Outcome: | The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds . |
How Much Do Encoder Models Know About Word Senses? (2025.acl-long)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a key task in Natural Language Processing (NLP) however, how well these models inherently disambiguate word senses remains uncertain. |
| Approach: | They evaluate several encoder-only PLMs across WordNet and ODE sense inventories to evaluate their ability to separate word senses without any task-specific fine-tuning. |
| Outcome: | The proposed model outperforms output layer on WordNet and ODE sense inventories by 15 percentage points. |
Autoregressive Score Generation for Multi-trait Essay Scoring (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing holistic approaches to score essays using pre-trained BERT-based models are inefficient, leading to inferior qualities in data-scarce traits. |
| Approach: | They propose an autoregressive prediction of multi-trait scores using pre-trained T5 models. |
| Outcome: | The proposed model shows over 5% improvement in prompts and traits compared to previous models . |
Evaluating the Robustness of Neural Language Models to Input Perturbations (2021.emnlp-main)
Copied to clipboard
| Challenge: | High-performance neural language models have achieved state-of-the-art results on a wide range of NLP tasks, but results for common benchmark datasets often do not reflect model reliability and robustness when applied to noisy, real-world data. |
| Approach: | They propose to implement character-level and word-level perturbation methods to simulate scenarios in which input texts may be slightly noisy or different from the data distribution on which NLP systems were trained. |
| Outcome: | The proposed methods simulate scenarios in which input texts may be slightly noisy or different from the data distribution on which NLP systems were trained. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
GLGR: Question-aware Global-to-Local Graph Reasoning for Multi-party Dialogue Reading Comprehension (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for multi-hop reasoning are lacking for local graph reasoning . existing approaches neglect local semantic structures in utterances . |
| Approach: | They propose a question-aware global-to-local graph reasoning approach that expands the canonical Interlocutor-Utterance graph by introducing a query node. |
| Outcome: | The proposed approach outperforms existing methods on Molweni and FriendsQA. |
Investigating Transferability in Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on deep NLP models has centered on probing, a method that involves training classifiers for different tasks on model representations. |
| Approach: | They propose a method for determining the impact of each pretrained layer on transfer task performance by ablation. |
| Outcome: | The proposed method shows that pretraining models improve performance on downstream tasks . the results highlight the limitations of methods that operate on frozen models or single data samples. |
Improving Zero-Shot Cross-Lingual Transfer Learning via Robust Training (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained multilingual language encoders do not precisely align words and phrases across languages. |
| Approach: | They propose a learning strategy for training robust models by drawing connections between adversarial examples and failure cases of zero-shot cross-lingual transfer. |
| Outcome: | The proposed model can achieve good performance even if representations of different languages are not aligned well. |
Knowledge Augmentation Enhances Token Classification for Recipe Understanding (2026.eacl-long)
Copied to clipboard
| Challenge: | Using entity type-specific and knowledge-augmented token classification, we achieve state-of-the-art (SOTA) results on 5 out of 7 benchmark recipe datasets, significantly outperforming traditional token classification methods. |
| Approach: | They propose an entity type-specific and knowledge-augmented token classification framework to improve encoder models’ performance on recipe texts. |
| Outcome: | The proposed model outperforms traditional token classification methods on 5 out of 7 recipe datasets and is the largest annotated food-related dataset to date. |
Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge (2024.eacl-long)
Copied to clipboard
| Challenge: | Using low-resource languages, multilingual language models (ML-LMs) have been developed to transfer factual knowledge across languages. |
| Approach: | They ask how ML-LMs acquire and represent factual knowledge . they use a multilingual factual information probing dataset to investigate ML . |
| Outcome: | The findings highlight the challenge of maintaining consistency factual knowledge across languages. |
Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on compressing pre-trained language models (PLMs) however, few studies have examined the impact of compression on generalizability and robustness of compressed models for out-of-distribution data. |
| Approach: | They propose to use knowledge distillation and pruning to reduce model generalization and generalization on out-of-distribution data. |
| Outcome: | The proposed compression techniques overfit on shortcut samples and generalize poorly on hard ones. |
Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing financial PLMs are not pretrained on sufficiently diverse financial data, leading to subpar generalization performance. |
| Approach: | They propose to pretrain financial PLMs on financial corpus and train financial models on financial data. |
| Outcome: | The proposed financial language models outperform existing financial PLMs on financial tasks even for unseen corpus groups. |
Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER (D19-1)
Copied to clipboard
| Challenge: | Contextual word embeddings have demonstrated state-of-the-art performance on various NLP tasks. |
| Approach: | They propose to use adversarial learning to improve upon multilingual BERT's zero-resource cross-lingual performance by aligning embeddings of English documents and their translations. |
| Outcome: | The multilingual version of BERT performs surprisingly well in cross-lingual settings, even when only labeled English data is used to finetune the model. |
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Byte-pair encoding (BPE) is a ubiquitous algorithm in the tokenization process of language models but is only based on pre-training data statistics. |
| Approach: | They propose a character-based subword module that learns the subword embedding table in pre-trained language models like BERT. |
| Outcome: | The proposed method significantly improves the performance on the social media linguistic code-switching evaluation (LinCE) benchmark. |
Head-wise Shareable Attention for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. |
| Approach: | They propose two methods that share parameters across attention heads to reduce memory usage and reduce performance drop by using coarse-grained weight sharing rules. |
| Outcome: | The proposed methods reuse pre-trained weights without retraining and then share, denoted as PostShare. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
Czech Dataset for Cross-lingual Subjectivity Classification (2022.lrec-1)
Copied to clipboard
| Challenge: | Using the existing English dataset, we can use the subjectivity classification to test the ability of pre-trained multilingual models to transfer knowledge between languages. |
| Approach: | They propose to use a Czech subjectivity dataset of 10k manually annotated subjective and objective sentences as a cross-lingual benchmark. |
| Outcome: | The proposed dataset is the first subjectivity dataset for the Czech language and also includes 200k automatically labeled sentences. |
Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task (2020.lrec-1)
Copied to clipboard
| Challenge: | In the last two years, significant improvements have occurred in NLP with the development of large language models using contextualized word embeddings based on the Google Transformer architecture. |
| Approach: | They performed experiments on data from the Fake News Challenge stage 1 (FNC-1) they used BERT sentence embeddings as a model feature and BERT, XLNet, and RoBERTa transformers to fine-tune them. |
| Outcome: | The proposed model outperforms the winner's system on class-wise F1 scores and achieves state-of-the-art on the stance detection task. |
Scientific and Creative Analogies in Pretrained Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing analogy datasets focus on a limited set of analogical relations with a high similarity of the two domains between which the analogy holds. |
| Approach: | They propose a dataset that encodes analogy in pretrained language models . they use a system that maps attributes and relational structures across dissimilar domains . |
| Outcome: | The proposed dataset shows that state-of-the-art models achieve low performance on analogy tasks . |
Label Semantics for Few Shot Named Entity Recognition (2022.findings-acl)
Copied to clipboard
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, Dan Roth
| Challenge: | Named entity recognition (NER) is a fundamental natural language understanding task that requires large amounts of high quality annotated in-domain data. |
| Approach: | They propose a neural architecture that leverages the semantic information in the names of the labels to give the model additional signal and enriched priors. |
| Outcome: | The proposed model is especially effective in low resource settings. |
Sentence-Level Resampling for Named Entity Recognition (2022.naacl-main)
Copied to clipboard
| Challenge: | named entity recognition (NER) tasks are often dominated by the majority of non-entity tokens in text . a data imbalance problem is causing the NER models to ignore named entities . |
| Approach: | They propose a set of sentence-level resampling methods to reduce data imbalance . they use a training sentence to compute the importance of each training sentence based on its tokens and entities . |
| Outcome: | The proposed methods outperform sub-sentence-level resampling, data augmentation, and loss functions on multiple corpora. |
DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that learning domain-specific language models are equally important for general-purpose and domain-based learning. |
| Approach: | They propose a domain-oriented learning task that combine the benefits of both general and domain-specific worlds. |
| Outcome: | The proposed task solves the problems in an aspect-based sentiment analysis task. |
Automated Essay Scoring System for Nonnative Japanese Learners (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing systems only provide a holistic score that summarizes the quality of an essay, which provides little feedback for a language learner. |
| Approach: | They developed an automated essay scoring system for Japanese as a second language learners using an essay dataset with annotations for a holistic score and multiple trait scores. |
| Outcome: | The proposed system achieves the highest accuracy in various natural language processing tasks. |
Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders (D19-1)
Copied to clipboard
| Challenge: | Existing models that use ground-truth part-of-speech tags are not always available and have significant weaknesses. |
| Approach: | They propose to use deep inside-outside recursive autoencoders to cluster the learned phrase vectors to induce span labels. |
| Outcome: | The proposed model outperforms ELMo and BERT on two versions of the Wall Street Journal dataset and improves over a previous state-of-the-art system that requires additional human annotations by 5 absolute F1 points (19% relative error reduction). |
Dynamic and Efficient Inference for Text Generation via BERT Family (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve performance of pre-trained language models are limited due to large-scale parameters and the universal autoregressive decoding paradigm. |
| Approach: | They propose a novel fine-tuning method which can make a single pre-trained model support Dynamic and Efficient infERence and achieve an adaptive trade-off between model performance and latency. |
| Outcome: | The proposed method achieves higher BLEU scores than the strong autoregressive Transformer model on translation tasks with 3 12 times speedup and faster inference speed compared with the BART model on four GLGE benchmark tasks. |
A Psycholinguistic Analysis of BERT’s Representations of Compounds (2023.eacl-main)
Copied to clipboard
| Challenge: | 'compound' semantic representations are based on the semantics of constituent words, and are lexical items like any other word. |
| Approach: | They leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD) and semantic transparency (ST). |
| Outcome: | The proposed representations are based on a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD) and semantic transparency (ST). |
Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-document event coreference resolution are prone to learning simple co-occurrences due to the complexity of contexts. |
| Approach: | They propose a collaborative approach to cross-document event coreference resolution that leverages both a universally capable LLM and a task-specific SLM. |
| Outcome: | The proposed approach surpasses the performance of both large and small language models individually, underscoring its effectiveness in diverse scenarios. |
BERTAC: Enhancing Transformer-based Language Models with Adversarially Pretrained Convolutional Neural Networks (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models of NLP are fading away, but new ones are needed to maintain their dominance. |
| Approach: | They propose a method to pretrain a CNN using Wikipedia data and integrate it with standard TLMs. |
| Outcome: | The proposed method outperforms the original ALBERT on GLUE tasks and achieves similar performance to SOTA on open-domain QA tasks. |
Tailoring Vaccine Messaging with Common-Ground Opinions (2024.findings-naacl)
Copied to clipboard
Rickard Stureborg, Sanxing Chen, Roy Xie, Aayushi Patel, Christopher Li, Chloe Zhu, Tingnan Hu, Jun Yang, Bhuwan Dhingra
| Challenge: | Vaccine interventions aim to answer concerns expressed about vaccination. |
| Approach: | They propose a dataset to evaluate how well responses are tailored to a common-ground opinion . they find that GPT-4-Turbo performs significantly better than others . |
| Outcome: | The proposed dataset outperforms fine tuned LLMs on the task of tailoring vaccine responses to common-ground opinions. |
Automatic Data Visualization Generation from Chinese Natural Language Questions (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies on data visualization generation from natural languages have not been conducted on Chinese Text-to-Vis. |
| Approach: | They propose to generate a Chinese text-to-vis dataset using a multilingual encoder and a cross-lingual ability. |
| Outcome: | The proposed dataset is challenging and deserves further research. |
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models require enormous computational resources and long training time. |
| Approach: | They propose an algorithm to reduce inference time and train large NLP models by slimming the self-attention and fully-connected sub-layers inside a transformer. |
| Outcome: | The proposed algorithm achieves comparable performance to standard BERT with 35 45% less training time. |
The impact of lexical and grammatical processing on generating code from natural language (2022.findings-acl)
Copied to clipboard
| Challenge: | Yin and Neubig (2018) identify four key components of importance for natural language to code translation. |
| Approach: | They propose a seq2seq-based architecture that relies on a grammar-based decoder and a lexical substitution component for natural language to code translation. |
| Outcome: | The proposed architecture relies on a grammar-based decoder and a BERT encoder . the proposed architecture is based on lexical substitutions in natural language to code translation . |
Towards Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Chinese word segmentation datasets have ambiguous annotation criteria resulting in multi-grained compounds. |
| Approach: | They propose a domain adaptive segmenter to exploit diverse annotation criteria of datasets . they use bidirectional encoder representations from transformers to introduce open-domain knowledge . |
| Outcome: | The proposed model outperforms the state-of-the-art models on 10 Chinese word datasets with superior efficiency. |
Data Expansion Using WordNet-based Semantic Expansion and Word Disambiguation for Cyberbullying Detection (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to identify cyberbullying from text are limited due to the complexity of the content and the lack of labeled large-scale corpus. |
| Approach: | They propose a data augmentation-based approach that could enhance the automatic detection of cyberbullying in social media texts. |
| Outcome: | The proposed approach overcomes limitations of social media posts with word sense disambiguation and synonymy relation . results show that the proposed approach improves on the existing classifiers with and without data augmentation. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
BanglaAutoKG: Automatic Bangla Knowledge Graph Construction with Semantic Neural Graph Filtering (2024.lrec-main)
Copied to clipboard
| Challenge: | Bangla is underrepresented in KGs due to lack of comprehensive datasets, encoders, NER models, part-of-speech taggers, and lemmatizers. |
| Approach: | Bangla is underrepresented in KGs due to lack of comprehensive datasets, encoders, NER models, part-of-speech taggers, and lemmatizers. authors propose a framework that can automatically construct Bengali KG from any Bangla text. |
| Outcome: | The proposed framework can automatically construct Bengali KGs from any Bangla text. |
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks do not probe professional bias as pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice. |
| Approach: | They propose to use a skew and stereotype metrics to quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. |
| Outcome: | The proposed methods reduce skew and stereotype relative to the unaugmented fine-tuned BERT model. |
Partially-Random Initialization: A Smoking Gun for Binarization Hypothesis of BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained BERT has been used for natural language processing tasks but its performance is limited by memory and computational complexity. |
| Approach: | They propose to use pre-trained BERT to achieve decent accuracy . they propose to combine binary BERT with a randomly-initialized encoder . |
| Outcome: | The proposed model achieves state-of-the-art on GLUE and SQuAD benchmarks. |
On the evolution of syntactic information encoded by BERT’s contextualized representations (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing studies have focused on how linguistic information is encoded in pretrained language models to solve supervised tasks. |
| Approach: | They analyze how the syntax trees are embedded in the geometry of pretrained models for six different tasks, covering all levels of the linguistic structure. |
| Outcome: | The proposed model is able to learn and improve on GLUE and SQUAD, but it lacks the ability to learn the linguistic information required to solve the tasks. |
Diverse Multi-Answer Retrieval with Determinantal Point Processes (2022.coling-1)
Copied to clipboard
| Challenge: | Existing open domain question answering systems provide a single answer to ambiguous questions. |
| Approach: | They propose a re-ranking approach that takes query-passage relevance and passage-passance correlation into account to retrieve passages that are query-relevant and diverse. |
| Outcome: | The proposed method outperforms state-of-the-art on the AmbigQA dataset. |
Knowledge-Empowered Representation Learning for Chinese Medical Reading Comprehension: Task, Model and Resources (2021.findings-acl)
Copied to clipboard
| Challenge: | MRC is a popular task in NLP, aiming to understand a passage and answer the relevant questions. |
| Approach: | They propose a multi-target machine learning task for the medical domain that predicts answers to medical questions and corresponding support sentences from medical information sources simultaneously. |
| Outcome: | The proposed model outperforms baselines by fusing context-aware and knowledge-awful token representations. |
MTRec: Multi-Task Learning over BERT for News Recommendation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing news recommendation methods learn news representations solely based on news titles. Existing methods only utilize title information and neglect other valuable news information such as categories and entities. |
| Approach: | They propose a multi-task method to incorporate multi-field information into BERT, which improves its news encoding capability. |
| Outcome: | Extensive experiments on the MIND news recommendation benchmark show the proposed method is effective. |
Cross-Domain Classification of Moral Values (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to identify moral values in text can be challenging for transferring knowledge between domains. |
| Approach: | They compare a deep learning model with a domain-specific value classifier to find out whether it can transfer knowledge to new domains. |
| Outcome: | The proposed model can generalize and transfer knowledge to novel domains, but introduce catastrophic forgetting. |
Integrating Multimodal Information in Large Pretrained Transformers (2020.acl-main)
Copied to clipboard
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, Ehsan Hoque
| Challenge: | Recent Transformer-based contextual word representations have shown state-of-the-art performance in multiple disciplines within NLP. |
| Approach: | They propose an attachment to BERT and XLNet that allows them to accept multimodal nonverbal data during fine-tuning. |
| Outcome: | The proposed attachment allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. |
Should You Mask 15% in Masked Language Modeling? (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language models (MLMs) traditionally mask 15% of tokens due to the belief that more masking would leave insufficient context to learn good representations. |
| Approach: | They revisit the 15% masking rate of MLMs to examine the role of masking in linguistic training. |
| Outcome: | The proposed masking rate outperforms BERT-large size models on GLUE and SQUAD while maintaining 95% accuracy. |
Progressive Class Semantic Matching for Semi-supervised Text Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent semi-supervised learning methods have achieved impressive performance . semi-controlled learning can be used to reduce the annotation cost of text classifiers . |
| Approach: | They propose a semi-supervised learning process that builds a standard K-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). |
| Outcome: | The proposed method improves baselines and overall is more stable. |
Slovene SuperGLUE Benchmark: Translation and Evaluation (2022.lrec-1)
Copied to clipboard
| Challenge: | a new benchmarking suite for natural language processing (NLP) is proposed to measure progress in the area of natural language understanding. |
| Approach: | They propose to use machine translation to translate a superGLUE benchmark into Slovene . they propose to combine monolingual, cross-lingual, and multilingual models . |
| Outcome: | The proposed model is superior to multilingual models but lags behind the best English models. |
Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel retrofitting method to induce emotion aspects into pre-trained language models is proposed . the models are computationally less expensive and open, but do not capture affective aspects of human communication well. |
| Approach: | They propose a retrofitting method to induce emotion aspects into pre-trained language models . they retrofit text fragments exhibiting similar emotions into pretrained networks . |
| Outcome: | The proposed method produces emotion-aware text representations for sentiment analysis and sarcasm detection tasks. |
Domain Adaptation for Arabic Cross-Domain and Cross-Dialect Sentiment Analysis from Contextualized Word Embedding (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies have classified dialectal Arabic into more fine-grained levels, including countries and cities. |
| Approach: | They propose to use Arabic domains to transfer knowledge from labeled source domains into unlabeled target domains by transferring the learned knowledge from a labele . |
| Outcome: | The proposed method outperforms other domain adaptation methods and improves performance by 20.8% over the zero-shot transfer learning from BERT. |
Deep Natural Language Feature Learning for Interpretable Prediction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Using a small transformer language model, we can break down a complex task into a set of intermediary easier sub-tasks. |
| Approach: | They propose a method to break down a main task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. |
| Outcome: | The proposed method breaks down a complex task into a set of easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. |
Detecting Syntactic Change with Pre-trained Transformer Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a fine-tuned BERT model can distinguish between text from the early 1800s and late 1900s . we use it to identify specific instances of syntactic change and specific words for which a new part of speech was introduced. |
| Approach: | They propose to use a BERT-based model to find syntactic differences between English of the early 1800s and that of the late 1900s. |
| Outcome: | The proposed model can distinguish between English of the early 1800s and that of the late 1900s using only syntactic information. |
Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models are weak and take advantage of failures and errors in datasets to improve performance. |
| Approach: | They evaluate three Transformer-based models in Natural Language Inference and Question Answering tasks to see if they are more robust or have the same flaws as their predecessors. |
| Outcome: | The proposed models outperform recurrent neural network models to stress tests on both NLI and QA tasks. |
Unsupervised Extractive Summarization-Based Representations for Accurate and Explainable Collaborative Filtering (2021.acl-long)
Copied to clipboard
| Challenge: | Existing extractive summarization-based collaborative filtering models learn accurate representations of users and items based on user-given numeric ratings, but employing them is an oversimplification of user preferences and item characteristics. |
| Approach: | They propose to use BERT, K-Means embedding clustering, and multilayer perceptron to learn sentence embeddations, representation-explanations, and user-item interactions to create extractive summaries. |
| Outcome: | The proposed model improves rating prediction accuracy and user/item explainability. |
PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction (2021.acl-long)
Copied to clipboard
| Challenge: | Chinese spelling correction (CSC) is a task to detect and correct spelling errors in texts. |
| Approach: | They propose a Pre-trained masked Language model with Misspelled knowledgE (PLOME) which jointly learns how to understand language and correct spelling errors. |
| Outcome: | The proposed model outperforms state-of-the-art methods on widely used benchmarks and achieves superior performance against existing models. |
Grid Tagging Scheme for Aspect-oriented Fine-grained Opinion Extraction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect-oriented Fine-grained Opinion Extraction (AFOE) aims to extract aspect terms and opinion terms from review text in the form of opinion pairs or opinion triplets. |
| Approach: | They propose a grid-based AFOE tagging scheme to address the task in an end-to-end fashion only with one unified grid tracking task. |
| Outcome: | The proposed tagging scheme outperforms baselines and achieves state-of-the-art performance. |
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline (2022.naacl-main)
Copied to clipboard
Xiangyang Liu, Tianxiang Sun, Junliang He, Jiawen Wu, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, Xipeng Qiu
| Challenge: | Rather than pursuing the reachless SOTA accuracy, researchers are focusing on model efficiency and usability. |
| Approach: | They propose an evaluation and a public leaderboard for efficient NLP models that depicts the Pareto Frontier for various language understanding tasks. |
| Outcome: | The proposed model outperforms or performs on par with SOTA compressed and early exiting models. |
GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the early exiting of natural language processing (NLP) are notoriously gigantic and slow in both training and inference. |
| Approach: | They propose a framework for improving the early exiting of BERT by asking each exit to distill knowledge from each other. |
| Outcome: | The proposed framework outperforms the state-of-the-art (SOTA) BERT early exiting methods on the GLUE benchmark. |
BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover’s Distance (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have been proposed and applied to many NLP tasks, yielding state-of-the-art performance, but high storage and computational costs obstruct them to be effectively deployed on resource-constrained devices and real-time applications. |
| Approach: | They propose a BERT distillation method which allows each intermediate student layer to learn from any intermediate teacher layers. |
| Outcome: | The proposed method can learn from different teacher layers adaptively for different NLP tasks. |
CLiMP: A Benchmark for Chinese Language Model Evaluation (2021.eacl-main)
Copied to clipboard
| Challenge: | Linguistically informed analyses of language models (LMs) contribute to understanding and improvement of such models. |
| Approach: | They introduce a corpus of Chinese linguistic minimal pairs (CLiMP) to investigate what knowledge Chinese LMs acquire. |
| Outcome: | The proposed corpus of Chinese linguistic minimal pairs (CLiMP) covers 9 major Chinese linguist phenomena. |
Don’t Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on building a meta-learner from input text but ignore abundant semantic information beneath class labels. |
| Approach: | They propose a framework to make full use of label semantics in few-shot text classification systems. |
| Outcome: | The proposed framework can be plugged into the existing few-shot text classification system. |
Universal Dependencies According to BERT: Both More Specific and More General (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that individual BERT heads encode particular dependency relation types, but they do not match one-to-one. |
| Approach: | They propose a method for relation identification and syntactic tree construction that can be applied with minimal supervision and generalizes well across languages. |
| Outcome: | The proposed method produces significantly more consistent dependency trees than previous work and can be applied with only a minimal amount of supervision and generalizes well across languages. |
schuBERT: Optimizing Elements of BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer based models have achieved state-of-the-art performance for many natural language processing tasks including machine translation, question-answering tasks and semantic role labeling. |
| Approach: | They propose to reduce the number of parameters of BERT to obtain a much efficient light model. |
| Outcome: | The proposed model achieves 6.6% higher average accuracy on GLUE and SQuAD datasets than the previous model with three encoder layers while having the same number of parameters. |
Simple but Challenging: Natural Language Inference Models Fail on Simple Sentences (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Natural language inference (NLI) tasks are difficult to perform on large datasets . a small number of simple sentences can improve model performance, authors say . |
| Approach: | They propose to use syntactically simple sentences to test the inference ability of NLI models. |
| Outcome: | The proposed set of simple sentences shows that the models fine-tuned on MNLI and SNLI perform poorly on Simple Pair. |
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (D19-1)
Copied to clipboard
| Challenge: | Existing models that can handle cross-lingual tasks with limited or no training data are insensitive to different languages. |
| Approach: | They propose to use Unicoder to train models in one language and apply it to other languages. |
| Outcome: | Experiments show that Unicoder learns the mappings among different languages from more perspectives. |
Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)
Copied to clipboard
| Challenge: | Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data. |
| Approach: | They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain. |
| Outcome: | The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning. |
LEGAL-BERT: The Muppets straight out of Law School (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing guidelines for pre-training and fine-tuning do not always generalize well in the legal domain. |
| Approach: | They propose to use BERT out of the box, adapt it by additional pre-training on domain-specific corpora, and pre-train it from scratch on domains. |
| Outcome: | The proposed strategies are: use the original BERT out of the box, adapt it by additional pre-training on domain-specific corpora, and pre-train it from scratch on domain specific corpors. |
Token Dropping for Efficient BERT Pretraining (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks. |
| Approach: | They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead. |
| Outcome: | The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks. |
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on societal bias in NLP focuses on race and gender . linguistic background is a unique attribute that has been largely ignored in the field . |
| Approach: | They examine linguistic background to craft plausible adversarial examples that expose biases in popular NLP models. |
| Outcome: | The proposed model improves robustness without sacrificing performance on clean data. |
Attention Can Reflect Syntactic Structure (If You Let It) (2021.eacl-main)
Copied to clipboard
| Challenge: | a recent study has attempted to decode linguistic structure from the Transformer . but, much of the work focused on English, a language with rigid word order and a lack of inflectional morphology. |
| Approach: | They propose to fine-tune a feature encoder for BERT to learn linguistic structure from its multi-head attention mechanism. |
| Outcome: | The proposed model can decode full trees above baseline accuracy from single attention heads across languages. |
Exploiting Global and Local Hierarchies for Hierarchical Text Classification (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods encode label hierarchy in a global view, which makes them hard to exploit hierarchical information. |
| Approach: | They propose to leverage label hierarchy in multi-label text classification by encoding label hierarchy as a static hierarchical structure containing all labels. |
| Outcome: | The proposed method achieves significant improvement on three benchmark datasets compared with the state-of-the-art method HGCLR. |
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focused on attention mechanisms to model complex interactions in visual dialog . a new framework for visual dialog is based on pretrained BERT language models . |
| Approach: | They propose a framework for a vision-dialog Transformer that leverages pretrained BERT language models for Visual Dialog tasks. |
| Outcome: | The proposed framework achieves the top position on the visual dialog leaderboard without pretraining on external vision-language data. |
DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence (2023.eacl-main)
Copied to clipboard
| Challenge: | DiscoScore is a parametrized discourse metric that uses BERT to model discourse coherence . it is weak when operated at system level, and is therefore not reliable in a way to spot improvements . |
| Approach: | They propose a parametrized discourse metric which uses BERT to model discourse coherence from different perspectives. |
| Outcome: | The proposed model outperforms existing models on document-level machine translation and summarization. |
Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training (2024.lrec-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have been successful in addressing word boundaries in Chinese sequence labeling tasks, but they rarely consider boundary information explicitly. |
| Approach: | They propose a method to integrate unsupervised boundary information into Chinese BERT's pre-training objectives and a supervised boundary-aware PLM. |
| Outcome: | The proposed model outperforms the vanilla version on Chinese sequence labeling tasks and in broader Chinese natural language understanding tasks. |
CogBERT: Cognition-Guided Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods fine-tune pre-trained models on cognitive data, ignoring the semantic gap between texts and cognitive signals. |
| Approach: | They propose a framework that can induce fine-grained cognitive features from cognitive data and incorporate them into pre-trained language models by adaptively adjusting the weight of cognitive features for different NLP tasks. |
| Outcome: | The proposed framework can induce fine-grained cognitive features from cognitive data and incorporate them into BERT by adaptively adjusting weight of cognitive features for different NLP tasks. |
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression (2020.coling-main)
Copied to clipboard
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, Jing Bai
| Challenge: | Existing models that use knowledge distillation are memory-intensive and latency-prohibitive . Existing solutions that use this knowledge distilling framework are expensive . |
| Approach: | They propose a solution that uses weight pruning, matrix factorization and knowledge distillation to learn a smaller model. |
| Outcome: | The proposed model reduces the training overheads by an order of magnitude on public datasets while preserving state-of-the-art accuracy. |
Jump To Hyperspace: Comparing Euclidean and Hyperbolic Loss Functions for Hierarchical Multi-Label Text Classification (2025.coling-main)
Copied to clipboard
| Challenge: | Hierarchical Multi-Label Text Classification (HMTC) is a challenging machine learning task . a recent study evaluated the effectiveness of Euclidean and hyperbolic loss functions on HMTC . |
| Approach: | They evaluate label-aware and contrastive losses in the Euclidean and hyperbolic space . they find contrastive loss functions are less effective when deployed in the hyperbolical space compared to non-hyperbolic ones . |
| Outcome: | The proposed model improves on four commonly used HMTC datasets. |
SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check (2021.emnlp-main)
Copied to clipboard
| Challenge: | Chinese Spelling Check is a nontrivial task because of the nature of ideographic language. |
| Approach: | They propose a pretrained model with graph-based extra features that captures erroneous patterns . they use a graph neural network to introduce radical and pinyin information as visual and phonetic features. |
| Outcome: | The proposed model can show competitive performance on OCR datasets where most errors are not covered by existing confusion set. |
Emotion Classification by Jointly Learning to Lexiconize and Classify (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to identify emotions in short text are limited and lack coverage and inaccuracies when applied to informal short text. |
| Approach: | They propose a novel emotional network to jointly learn sentence emotions and construct emotion lexicons which are dynamically adapted to a given context. |
| Outcome: | The proposed model outperforms several approaches proposed in previous studies and achieves new state-of-the-art on the benchmark Twitter dataset. |
A Simple and Effective Approach to Automatic Post-Editing with Transfer Learning (P19-1)
Copied to clipboard
| Challenge: | Existing APE systems generate artificial triplets of source sentences, machine translation outputs and human post-edits. |
| Approach: | They propose to use human post-edits to refine black-box machine translation (MT) models by fine-tuning pre-trained BERT models on both encoder and decoder of an APE system. |
| Outcome: | The proposed method improves on a dataset of 23K sentences on x86 GPUs. |
Sentence Matching with Syntax- and Semantics-Aware BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Sentence matching aims to determine the special relationship between two sentences. |
| Approach: | They propose to integrate syntactic and semantic information into BERT with sentence matching by using an implicit integration method that is less sensitive to the output structure information. |
| Outcome: | The proposed method achieves state-of-the-art or competitive performance on several sentence matching datasets. |
Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on LMs' internal vector representations has demonstrated the existence of syntactic information in those vectors, but has not described how LM models use this information. |
| Approach: | They propose a causal intervention paradigm to show that some linguistic features are represented in a linear format. |
| Outcome: | The proposed model can generate verb conjugations 91% correct before and 85% incorrect after the intervention. |
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (N19-1)
Copied to clipboard
| Challenge: | In this paper we build a reading comprehension dataset of yes/no questions that are naturally occurring . they often query for complex, non-factoid information, and require difficult entailment-like inference to solve. |
| Approach: | They build a reading comprehension dataset of yes/no questions that are naturally occurring . they find they are unexpectedly challenging and require difficult inferences to solve . |
| Outcome: | The proposed method achieves 80.4% accuracy compared to 90% accuracy of human annotators and 62% majority-baseline. |
RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent named entity recognition models have great performance on many conventional benchmarks, but it is not reliable in realistic applications. |
| Approach: | They propose a method to create natural adversarial examples using Wikidata and pre-trained language models. |
| Outcome: | The proposed method produces natural adversarial examples with a shifted distribution from training data. |
Literature Retrieval for Precision Medicine with Neural Matching and Faceted Summarization (2020.findings-emnlp)
Copied to clipboard
| Challenge: | IR for precision medicine often involves looking for multiple pieces of evidence that characterize a patient case. |
| Approach: | They propose a document reranking approach that combines neural query-document matching and text summarization toward such retrieval scenarios. |
| Outcome: | The proposed approach achieves state-of-the-art performance on NIST's TREC-PM track dataset. |
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) capture a diverse range of linguistic and factual knowledge without the use of finetuning. |
| Approach: | They combine BERT with a traditional information retrieval step and a kNN search to improve the recall of facts encountered during training. |
| Outcome: | The proposed approach outperforms baselines on cloze-style QA by large margins without further training. |
Probing for idiomaticity in vector space models (2021.eacl-main)
Copied to clipboard
| Challenge: | Contextualised word representation models are used to represent idiomaticity in language. |
| Approach: | They propose probing measures to assess if some of the expected linguistic properties of noun compounds are readily available in some standard and widely used representations. |
| Outcome: | The proposed models show that idiomaticity is not yet accurately represented by contextualised models. |
On the Robustness of Language Encoders against Grammatical Errors (2020.acl-main)
Copied to clipboard
| Challenge: | Pre-trained language encoders are effective in facilitating downstream natural language processing tasks, but they often assume training and test corpora are clean and it is unclear how the models behave when confronted with noisy input. |
| Approach: | They conduct adversarial attacks to simulate grammatical errors on clean text data. |
| Outcome: | The proposed model performs better when confronted with natural grammatical errors than when faced with noisy input. |
Mining Knowledge for Natural Language Inference from Wikipedia Categories (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Accurate lexical entailment (LE) and natural language inference (NLI) tasks require expensive annotations. |
| Approach: | They propose to pretrain Wikipedia categories for lexical entailment and natural language inference by pretraining them on WikiNLI and transferring them to other knowledge bases. |
| Outcome: | The proposed model can improve strong baselines such as BERT and RoBERTa by pretraining on WikiNLI and transferring the models on downstream tasks. |
How is BERT surprised? Layerwise detection of linguistic anomalies (2021.acl-long)
Copied to clipboard
| Challenge: | a number of studies have shown that transformer-based language models detect when a word is anomalous in context, but likelihood scores do not tell the cause of the anomaly. |
| Approach: | They propose to use Gaussian models for density estimation at intermediate layers of three language models to evaluate grammaticality. |
| Outcome: | The proposed method on BLiMP shows that language models employ different mechanisms to detect different types of linguistic anomalies. |
Extracting a Knowledge Base of COVID-19 Events from Social Media (2022.coling-1)
Copied to clipboard
| Challenge: | a flood of COVID-19 related information has appeared on social media since December 2019 . this includes reports on public figures who have tested positive/negative for the virus . |
| Approach: | They construct a corpus of 10,000 tweets with annotated public reports of five COVID-19 events, using slot-filling questions to fill in slots. |
| Outcome: | The proposed method can be quickly applied to develop knowledge bases for new domains in response to emerging crises, including natural disasters or future disease outbreaks. |
Differential Privacy for Text Analytics via Natural Text Sanitization (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing text sanitization mechanisms provide low utility, as cursed by the high-dimensional text representation. |
| Approach: | They propose to use sanitized texts to samaritize training data . they propose to retrain and fine-tune the senitization-aware language model . |
| Outcome: | The proposed approach enables privacypreserving natural language processing over the BERT language model with promising utility. |
BERT-like Models for Slavic Morpheme Segmentation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing morpheme segmentation algorithms for Slavic languages have been improved but performance is still low for words with roots not present in training data. |
| Approach: | They propose to fine-tune BERT-like models for morpheme segmentation using data from Belarusian, Czech, and Russian to account for word semantics. |
| Outcome: | The proposed models outperform all previous approaches in Czech and Russian, with word-level accuracy of 92.5-95.1%. |
Multilingual Constituency Parsing with Self-Attention and Pre-Training (P19-1)
Copied to clipboard
| Challenge: | a range of pre-training conditions can be used for constituency parsing, but large model sizes make it expensive to train separate models for each language. |
| Approach: | They compare the benefits of no pre-training, fastText, ELMo, and BERT for English . they also find that pre- training is beneficial across all 11 languages tested . |
| Outcome: | The proposed model outperforms fastText, ELMo, and BERT for English . but large model sizes make it expensive to train separate models for each language . |
CxGBERT: BERT meets Construction Grammar (2020.coling-main)
Copied to clipboard
| Challenge: | lexico-semantic elements capture a large amount of linguistic information, but they do not capture all information contained in text. |
| Approach: | They propose to use BERT to train a model that uses a deep bidirectional transformer to capture a significant amount of lexico-semantic information. |
| Outcome: | The proposed model captures lexico-semantic information, but it is redundantly encoded in lexical information. |
The Topic Confusion Task: A Novel Evaluation Scenario for Authorship Attribution (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Autorship attribution is the problem of identifying the most plausible author of an anonymous text from a set of candidate authors. |
| Approach: | They propose a topic confusion task where they switch the author-topic configuration between training and testing sets and propose attribution errors that are caused by the topic shift and by the features’ inability to capture the writing styles. |
| Outcome: | The proposed task combines author-topic configuration with other features to lower topic confusion and higher attribution accuracy. |
Effective Attention Sheds Light On Interpretability (2021.findings-acl)
Copied to clipboard
| Challenge: | Using a subset of the GLUE tasks and BERT, we compare the two attention matrices and show that their interpretations differ. |
| Approach: | They propose to use visualizing effective attention to interpret a transformer's behavior since it is more pertinent to the model output by design. |
| Outcome: | The proposed method is more relevant to the model output by design than visualizing attention weights. |
SciBERT: A Pretrained Language Model for Scientific Text (D19-1)
Copied to clipboard
| Challenge: | SciBERT is a pretrained language model based on BERT to improve performance on scientific NLP tasks. |
| Approach: | They propose a pretrained language model based on BERT to improve NLP performance . they evaluate on sequence tagging, sentence classification and dependency parsing . |
| Outcome: | The proposed model improves on sequence tagging, sentence classification and dependency parsing tasks with datasets from a variety of scientific domains. |
Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence (2021.findings-acl)
Copied to clipboard
| Challenge: | a new corpus-level evaluation approach for event extraction is needed in social science applications . human annotations are often required to extract the actions of political actors and actors . a novel corpus evaluation approach can guide creation of similar social science-oriented resources . |
| Approach: | They propose a corpus-based approach to event extraction that integrates corpus evaluation with real-world social science . they use human annotations to read and label every document for mentions of police activity events . |
| Outcome: | The proposed method can guide creation of similar social-science-oriented resources in the future. |
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021.emnlp-main)
Copied to clipboard
| Challenge: | Similarity measures are a vital tool for understanding how language models represent and process language. |
| Approach: | They propose to use cosine similarity and Euclidean distance to understand how words cluster in semantic space. |
| Outcome: | The proposed measures show that rogue dimensions dominate similarity measures and reveal representational quality. |
Curation of Benchmark Templates for Measuring Gender Bias in Named Entity Recognition Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) models are susceptible to gender bias . benchmark datasets are curated specifically for a given NLP task . |
| Approach: | They propose to filter out benchmark templates with a higher probability of detecting gender bias in NER models. |
| Outcome: | The proposed method is based on masked token prediction and tested in English and german using the corresponding fine-tuned BERT base model. |
MLMLM: Link Prediction with Mean Likelihood Masked Language Model (2021.findings-acl)
Copied to clipboard
| Challenge: | Knowledge Bases (KBs) are easy to query, verifiable, and interpretable. however, they scale with man-hours and high-quality data. |
| Approach: | They propose to commit the knowledge embedded in MLMs to a KB, making it interpretable . they propose to use a mean likelihood Masked Language Model to compare the likelihood of generating different entities to perform link prediction in a tractable manner. |
| Outcome: | The proposed approach compares the likelihood of generating different entities to perform link prediction in a tractable manner. |
PRIDE: Predicting Relationships in Conversations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for extracting interpersonal relationships from dialogues are limited to end-to-end learning. |
| Approach: | They propose a neural multi-label classifier that infers relationships from dialogues by external knowledge about speaker features and conversation style. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on large-scale datasets with directed relationships of conversation participants. |
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. |
| Approach: | They propose an infusion approach that partitions a large knowledge graph into smaller sub-graphs and infuses their specific knowledge into various BERT models using lightweight adapters. |
| Outcome: | The proposed approach improves the underlying BERTs and achieves new SOTA performance on six downstream tasks. |
Personalized Transformer for Explainable Recommendation (2021.acl-long)
Copied to clipboard
| Challenge: | Recent years have witnessed the successful application of natural language generation. |
| Approach: | They propose a model that uses user and item IDs to predict the words in the target explanation to make personalized Transformer. |
| Outcome: | The proposed model outperforms BERT on the explainable recommendation task in terms of effectiveness and efficiency. |
BERT for Monolingual and Cross-Lingual Reverse Dictionary (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to find the proper word for a word are based on the surface form of words, but they cannot extract the semantic meaning. |
| Approach: | They propose a method to make BERT generate the target word for this task . cross-lingual reverse dictionary is the task to find the proper target word . |
| Outcome: | The proposed method can generate the target word for cross-lingual reverse dictionary task even without the parallel corpus. |
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |
Analyzing Redundancy in Pretrained Transformer Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that transformer-based deep NLP models are over-parameterized and do not require all the representational power lent by the rich architectural choices during inference. |
| Approach: | They define a notion of Redundancy and propose a feature-based transfer learning procedure which maintains 97% performance while using at-most 10% of the original neurons. |
| Outcome: | The proposed model maintains 97% performance while using 10% of the original neurons. |
A Label-Aware BERT Attention Network for Zero-Shot Multi-Intent Detection in Spoken Language Understanding (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for speech-based dialogs have found the inadequacy of relying on simple classification techniques to accomplish the automation task. |
| Approach: | They propose a Label-Aware BERT Attention Network (LABAN) for zero-shot multi-intent detection by encoding input utterances with BERT and building a label embedded space by considering embedded semantics in intent labels. |
| Outcome: | The proposed approach can detect many unseen intent labels correctly on a few/zero-shot setting, and achieves state-of-the-art performance on five multi-intent datasets in normal cases. |
ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence (2022.naacl-main)
Copied to clipboard
Yibo Hu, MohammadSaleh Hosseini, Erick Skorupa Parolin, Javier Osorio, Latifur Khan, Patrick Brandt, Vito D’Orazio
| Challenge: | Traditionally, researchers used manual coding to track conflict processes worldwide, but the high costs and slow pace of domain experts make it difficult and costly to monitor complex and rapidly changing conflicts. |
| Approach: | They propose a domain-specific pre-trained language model for conflict and political violence that can be used to train a language model from scratch and continue training. |
| Outcome: | The proposed model outperforms BERT in conflict research. |
Combining Unsupervised Pre-training and Annotator Rationales to Improve Low-shot Text Classification (D19-1)
Copied to clipboard
| Challenge: | supervised learning models perform poorly at low-shot tasks for which little labeled data is available for training. |
| Approach: | They propose to combine a bag-of-words embedding approach and a context-aware method to improve low-shot text classification. |
| Outcome: | The proposed method improves low-shot text classification with pre-training and rationales . the simple bag-of-words approach is the clear top performer when there are few training instances or less . |
ConvTextTM: An Explainable Convolutional Tsetlin Machine Framework for Text Classification (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent advances in natural language processing (NLP) have reshaped the industry . complexity of such models makes them a “black box” and can cause ethical concerns . |
| Approach: | They propose a convolutional TM architecture that breaks down text into a sequence of fragments . they propose to use a tokenization scheme to bind the tokens to the text fragments. |
| Outcome: | The proposed architecture improves on a set of text fragments and eliminates the need for a corpus-specific vocabulary. |
Evaluating Methods for Extraction of Aspect Terms in Opinion Texts in Portuguese - the Challenges of Implicit Aspects (2022.lrec-1)
Copied to clipboard
| Challenge: | In aspect-based sentiment analysis, the implicit mention of aspects is difficult to identify and may require world knowledge to do so. |
| Approach: | They evaluate frequency-based, hybrid, and machine learning methods to extract aspect terms from opinionated texts in Portuguese. |
| Outcome: | The proposed methods show that they are more efficient and more efficient than previous methods. |
SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task. |
| Approach: | They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines . |
| Outcome: | The proposed model improves sample-efficiency and performance over strong baselines. |
DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering (2020.acl-main)
Copied to clipboard
| Challenge: | DeFormer is a transformer-based QA model that uses input-wide self-attention at all layers, causing them to be slow and memory-intensive. |
| Approach: | They propose a transformer which substitutes the full self-attention with question-wide and passage-wide self- attentions in the lower layers. |
| Outcome: | The proposed model can be used to speed up QA by over 4.3x and with simple distillation-based losses they incur only a 1% drop in accuracy. |
Embeddings models for Buddhist Sanskrit (2022.lrec-1)
Copied to clipboard
| Challenge: | Despite extensive scholarly endeavors, much uncertainty still surrounds this body of literature, especially regarding matters of chronology, authorship, compositional history. |
| Approach: | They propose a corpus of Buddhist texts, a general corpus and word similarity and word analogy datasets for embeddings models. |
| Outcome: | The proposed models perform better on semantic similarity and word analogy tasks than on contextual models. |
Sustainable Modular Debiasing of Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing debiasing methods modify all of the PLM parameters, which is costly and leads to (catastrophic) forgetting of useful language knowledge. |
| Approach: | They propose a modular debiasing approach based on dedicated adapters that inject adapter modules into the original PLM layers and update only the adapters. |
| Outcome: | The proposed approach is based on dedicated adapters and retains fairness even after large-scale training. |
How does the brain process syntactic structure while listening? (2023.findings-acl)
Copied to clipboard
| Challenge: | Syntactic parsing is the task of assigning a syntactical structure to a sentence. |
| Approach: | They investigate the predictive power of the brain encoding models in three settings: individual performance of constituency and dependency parsing based embedding methods, relative effectiveness of each of the syntactic parsers, and relative importance of syntaktic information versus semantic information using BERT embeddngs. |
| Outcome: | The proposed methods explain variance in brain regions, especially in listening tasks, and the relative predictive power of the various syntax parsers is unexplored. |
Exploring BERT’s Sensitivity to Lexical Cues using Tests from Semantic Priming (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using English lexical stimuli, we find that BERT models show "priming" predicting a word with greater probability when the context includes a related word versus an unrelated one. |
| Approach: | They analyze a pre-trained BERT model with tests informed by semantic priming . they find that BERT too shows "priming" predicting a word with greater probability when context includes a related word versus an unrelated one. |
| Outcome: | The proposed model shows a tendency to be distracted by related prime words as context becomes more informative, and lower probability of related words. |
MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering (2020.findings-emnlp)
Copied to clipboard
| Challenge: | MMFT-BERT is a multimodal fusion transformer that decomposes input modalities into different BERT instances with similar architectures, but variable weights. |
| Approach: | They propose a multimodal fusion transformer with BERT encodings to solve Visual Question Answering (VQA) . |
| Outcome: | The proposed method achieves SOTA results on the TVQA dataset and TVQA-Visual, an isolated diagnostic subset of TVQA, which strictly requires the knowledge of visual (V) modality based on a human annotator’s judgment. |
A Structural Probe for Finding Syntax in Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Approach: | They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Outcome: | The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry. |
Detecting Conceptual Abstraction in LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | a novel approach to detecting noun abstraction within a large language model is proposed . a first step towards the explainability of conceptual abstraction in LLMs is shown . |
| Approach: | They propose a method to detect noun abstraction within a large language model . they instantiate taxonomic relationships and analyze attention matrices produced by BERT . |
| Outcome: | The proposed approach can detect hypernymy in a large language model . the results are a first step towards the explainability of conceptual abstraction in LLMs . |
On the Cross-lingual Transferability of Monolingual Representations (2020.acl-main)
Copied to clipboard
| Challenge: | State-of-the-art unsupervised multilingual models generalize in zero-shot cross-lingual setting . generalization ability attributed to shared subword vocabulary and joint training across multiple languages . |
| Approach: | They propose an approach that transfers a monolingual model to new languages at the lexical level. |
| Outcome: | The proposed approach is competitive with multilingual BERT on cross-lingual classification benchmarks and on a new cross-linguistic question answering dataset. |
Neural Legal Judgment Prediction in English (P19-1)
Copied to clipboard
| Challenge: | Recent work on legal judgment prediction has focused on Chinese, but only feature-based models have been considered in English. |
| Approach: | They propose a hierarchical version of BERT which bypasses BERT’s length limitation. |
| Outcome: | The proposed model outperforms existing models in binary violation classification, multi-label classification and case importance prediction. |
ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained text encoders can perform many NLP tasks with less resource. |
| Approach: | They propose a BERT-based Chinese text encoder enhanced by n-gram representations . they show reasonable performance when ZEN is trained on a small corpus . |
| Outcome: | The proposed encoder incorporates the comprehensive information of both the character sequence and words or phrases it contains. |
How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work probing pre-trained language models for downstream tasks is difficult to explain . a growing body of research is devoted to understanding what linguistic properties these language models have acquired. |
| Approach: | They propose a procedure and analysis method that takes a hypothesis of how a transformer-based model might encode a linguistic phenomenon and tests its validity. |
| Outcome: | The proposed method tests a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. |
On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining. |
| Approach: | They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss. |
| Outcome: | The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures. |
ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Causal chain reasoning models suffer from two main transitive problems: threshold effect and scene drift. |
| Approach: | They propose a framework that uses exogenous variables to represent causal pairs and estimates the threshold and scene contradictions using structural causal recurrent neural networks. |
| Outcome: | The proposed framework outperforms baselines on Chinese and English CCR datasets. |
Adapting Monolingual Models: Data can be Scarce when Language Similarity is High (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained language models are the dominant approach for solving many tasks in natural language processing. |
| Approach: | They propose to retrain the lexical layers of four BERT-based models using data from two low-resource target languages while the Transformer layers are independently finetuned on a POS-tagging task in the model's source language. |
| Outcome: | The proposed method achieves high performance for both target and target languages with high similarity. |
Incorporating medical knowledge in BERT for clinical relation extraction (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are used for diverse NLP tasks such as Information Extraction, Sentiment Analysis and Question/Answering. |
| Approach: | They propose to add medical knowledge to pre-trained language models to facilitate clinical relation extraction using a large text corpus. |
| Outcome: | The proposed model outperforms the state-of-the-art systems on the benchmark i2b2/VA 2010 clinical relation extraction dataset. |
Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events (2020.emnlp-main)
Copied to clipboard
Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, Yaser Al-Onaizan
| Challenge: | Existing models for temporal ordering of events rely on pretrained representations, transfer and multitask learning, and self-training techniques. |
| Approach: | They propose a neural architecture and a set of training methods for ordering events by predicting temporal relations by pre-training models. |
| Outcome: | The proposed models can predict temporal relations between two pairs of events within a span of text and identify temporal relationships between them. |
Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | BRIDGE is a powerful sequential architecture for cross-modal semantic parsing . BRidege captures cross-modal dependencies between natural language questions and relational databases . |
| Approach: | They propose a sequential architecture that captures cross-modal dependencies between questions and relational databases in cross-DB semantic parsing. |
| Outcome: | The proposed architecture performs well on the well-studied Spider benchmark (65.5% dev, 59.2% test). |
How transfer learning impacts linguistic knowledge in deep NLP models? (2021.findings-acl)
Copied to clipboard
| Challenge: | Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. |
| Approach: | They propose to fine-tune pre-trained models towards downstream NLP tasks to capture linguistic knowledge. |
| Outcome: | The proposed model is adapted to GLUE tasks and retains linguistic information in the network while forgetting it. |
Sentiment Analysis for Emotional Speech Synthesis in a News Dialogue System (2020.coling-main)
Copied to clipboard
| Challenge: | In smart speakers and conversational robots, the demand for expressive speech synthesis has increased. |
| Approach: | They propose to annotate a news dataset with emotion labels for each sentence and to evaluate its effectiveness using the constructed dataset. |
| Outcome: | The proposed method improves the performance of the proposed model by preferentially annotating news articles with low confidence in the human-in-the-loop machine learning framework. |
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing domain-adaptive pre-training (DAPT) models tend to forget the general knowledge acquired by general PLMs, leading to catastrophic forgetting and sub-optimal performance. |
| Approach: | They propose a framework which augments the domain-specific PLM by a memory built from the frozen general PLM without losing the general knowledge. |
| Outcome: | The proposed framework augments the domain-specific PLM by a memory built from the frozen general PLM without losing the general knowledge. |
Code and Named Entity Recognition in StackOverflow (2020.acl-main)
Copied to clipboard
| Challenge: | StackOverflow has 15 million programming related questions written by 8.5 million users . however, there is a lack of fundamental NLP resources and techniques for identifying software-related named entities within natural language sentences. |
| Approach: | They propose a named entity recognition corpus for the computer programming domain with 15,372 sentences annotated with 20 fine-grained entity types. |
| Outcome: | The proposed model improves on 152 million sentences from StackOverflow and achieves 79.10 F-1 score for code and named entity recognition. |
Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Prior work identifies a linear gender subspace and removes gender information by eliminating the subspace. |
| Approach: | They propose to use DensRay to obtain interpretable dense subspaces by applying it to attention heads and layers of BERT. |
| Outcome: | The proposed method performs on-par with prior approaches, but is more robust and preserves language model performance better. |
A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters (2021.acl-long)
Copied to clipboard
| Challenge: | Few-shot crosslingual transfer outperforms zero-shot with pretrained encoders like multilingual BERT. |
| Approach: | They conduct an experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. |
| Outcome: | The proposed model outperforms state-of-the-art approaches on lexical features and a full model finetuning approach outperformed several state- of-the art approaches. |
Explore Unsupervised Structures in Pretrained Models for Relation Extraction (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Syntactic trees are widely used in relation extraction (RE) but they are not stable on different text domains and a pre-defined grammar may not fit the target relation schema. |
| Approach: | They propose to use unsupervised structures to extract relation extraction models . they also conduct detailed analyses on their abilities of adapting new RE domains . |
| Outcome: | The proposed models obtain competitive (even the best) performance scores on benchmark RE datasets. |
Transformers in the loop: Polarity in neural models of language (2022.acl-long)
Copied to clipboard
| Challenge: | Recent Transformer-based language representation models (LRMs) show impressive results on practical text analysis tasks, but do they have access to complex linguistic notions? |
| Approach: | They propose to use polarity as a case study to compare metrics derived from language models to human judgments obtained in psycholinguistic experiments. |
| Outcome: | The proposed model is more accurate than linguistic theory predictions for polarity, and allows us to use language models to discover new insights into natural language grammar beyond existing linguistic theories. |
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text (D19-1)
Copied to clipboard
| Challenge: | Existing datasets for reading comprehension tasks have been used to test the generalization of natural language understanding systems. |
| Approach: | They propose a diagnostic benchmark suite to clarify key issues related to the robustness and systematicity of NLU systems. |
| Outcome: | The proposed benchmark suite clarifies key issues related to the robustness and systematicity of NLU systems. |
Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have shown strong potential in various downstream tasks. |
| Approach: | They propose to model adversarial attack task as a sequential decision-making problem where the whole attack process is sequential with two decision- making problems, i.e., word finder and word substitution. |
| Outcome: | The proposed approach achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT. |
AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information. |
| Approach: | They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model. |
| Outcome: | The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods. |
Enhancing Structure-aware Encoder with Extremely Limited Data for Graph-based Dependency Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Dependency parsing is an important natural language processing task which analyzes the syntactic structure of an input sentence. |
| Approach: | They propose a structure-aware encoder pre-trained on auto-parsed data to improve dependency parsing . they propose combining gold dependency trees with existing parsers to improve parser performance . |
| Outcome: | The proposed approach outperforms baselines under different parsers and dependency standards under different parameters and model architectures. |
Finding Universal Grammatical Relations in Multilingual BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work has found that multilingual masked language models learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. |
| Approach: | They propose an unsupervised method to find syntactic tree distances in languages other than English and that these subspaces are approximately shared across languages. |
| Outcome: | The proposed method shows that mBERT learns representations of syntactic dependency labels, in the form of clusters, which largely agree with the Universal Dependencies taxonomy. |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
SkipBERT: Efficient Inference with Shallow Layer Skipping (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models have significant demands in computation and inference time, limiting their use in resource-constrained or latencysensitive applications. |
| Approach: | They propose to encode text chunks into independent representations and skip computation of shallow layers to accelerate inference. |
| Outcome: | The proposed approach can reduce latency by 65% without sacrificing performance. |
Parameter-Efficient Tuning Makes a Good Classification Head (2022.emnlp-main)
Copied to clipboard
| Challenge: | In recent years, pretrained models revolutionized the paradigm of natural language understanding . but the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning . |
| Approach: | They propose to append a randomly initialized classification head after the pretrained backbone and finetune the whole model. |
| Outcome: | The proposed classification head can be replaced with the randomly initialized heads for a stable performance gain. |
Span-level Aspect-based Sentiment Analysis via Table Filling (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to analyze aspect-based sentiment analysis focus on word-level dependencies between aspect and opinion expressions. |
| Approach: | They propose a span-level ABSA model which considers consistency of multi-word opinion expressions at the span- level. |
| Outcome: | The proposed model can be used to identify the sentiment polarity of a given aspect . it is based on a table filling method and a regularizer to guarantee consistency . |
Cluster & Tune: Boost Cold Start Performance in Text Classification (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to fine-tune pre-trained models for text classification are poor in practice. |
| Approach: | They propose to add an intermediate unsupervised classification task between pre-training and fine-tuning phases to boost performance of pre-trained models. |
| Outcome: | The proposed method improves performance on topical classification tasks when labeled data is scarce. |
Bit-Flip Error Resilience in LLMs: A Comprehensive Analysis and Defense Framework (2025.emnlp-main)
Copied to clipboard
Yuhang Chen, Zhen Tan, Ajay Kumar Jaiswal, Huaizhi Qu, Xinyu Zhao, Qi Lin, Yu Cheng, Andrew Kwong, Zhichao Cao, Tianlong Chen
| Challenge: | Bit-flip errors (BFEs) are hardware faults where individual bits in memory or processing units are unintentionally flipped. |
| Approach: | They propose a novel defense strategy to mitigate bit-flip errors (BFEs) they propose bfe protection and a self-correction mechanism to minimize performance degradation . |
| Outcome: | The proposed defense strategy minimizes performance degradation while significantly improving robustness against BFEs. |
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing models cannot capture numeracy, but they can be useful for complex reasoning tasks. |
| Approach: | They investigate numerical reasoning capabilities of a question-answering model . they probe token embedding methods on synthetic list maximum, number decoding, and addition tasks. |
| Outcome: | The proposed model excels on questions that require numerical reasoning, i.e., it already captures numeracy. |
Are ELECTRA’s Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | ELECTRA's sentence embeddings are poorer than BERT's, resulting in a significant drop in performance for semantic textual similarity (STS). |
| Approach: | They propose to use a truncated model fine-tuning method to repair the embeddings by reducing the number of parameters and producing smaller embeddables. |
| Outcome: | The proposed method improves Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS Benchmark. |
Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption (2025.acl-long)
Copied to clipboard
| Challenge: | a new privacy-preserving language model, Powerformer, is designed to reduce computation overhead while maintaining model performance. |
| Approach: | They propose an efficient homomorphic encryption-based privacy-preserving language model . it incorporates three key techniques to optimize encrypted computations . |
| Outcome: | The proposed model achieves 45% reduction in computation time compared to state-of-the-art models . authors say the model preserves data privacy and AI capabilities in MLaaS environments . |
LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language model pretraining methods do not capture dependencies or knowledge that span across documents. |
| Approach: | They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs . |
| Outcome: | The proposed method outperforms existing methods on downstream tasks across two domains. |
Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT (2020.lrec-1)
Copied to clipboard
| Challenge: | Massive digital data processing can endanger personal data privacy . anonymisation involves removing or replacing sensitive information from data . |
| Approach: | They propose to use a BERT-based sequence labelling model to conduct an experiment on clinical datasets in Spanish. |
| Outcome: | The proposed model outperforms existing models on clinical datasets in Spanish and shows that it is highly competitive with other models. |
It is a Bird Therefore it is a Robin: On BERT’s Internal Consistency Between Hypernym Knowledge and Logical Words (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing models for lexical knowledge of NLP systems only use collocational cues for questions . lexicon of these systems is limited to content words and logical words. |
| Approach: | They propose a method to test the understanding of the hypernymy relationship by measuring its antisymmetry according to the models. |
| Outcome: | The proposed model shows that it is weak on the semantic tasks of the Masked Language Modeling task. |
Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. |
| Approach: | They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian. |
| Outcome: | The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps. |
Beyond Distribution: Investigating Language Models’ Understanding of Sino-Korean Morphemes (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Transformer-based language models can learn compositional morphology of SK morphemes . morphological models trained on Hangul text can learn SK, but performance is based on frequency of words . |
| Approach: | They investigate whether Transformer-based language models can learn compositional morphology of Sino-Korean morphemes. |
| Outcome: | The proposed models learn the compositional morphology of SK morphemes from real and fake pairs. |
HateCheckHIn: Evaluating Hindi Hate Speech Detection Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Hate speech detection models are evaluated on a held-out test data, but they are incapable of identifying weaknesses. |
| Approach: | They propose to use multilingual hate speech detection models to evaluate their performance on social media conversation. |
| Outcome: | The proposed model can detect hate speech in multiple languages using a real-world conversation on social media. |
MS-LaTTE: A Dataset of Where and When To-do Tasks are Completed (2022.lrec-1)
Copied to clipboard
| Challenge: | Tasks are a fundamental unit of work in the daily lives of people, who are increasingly using digital means to keep track of, organize, triage, and act on them. |
| Approach: | They compile and release a large-scale dataset that captures location and time for tasks and a BERT-fine-tuned model that predicts task co-occurrence. |
| Outcome: | The proposed framework captures location and time, and predicts task co-occurrence with a BERT fine-tuned model outperforming baselines. |
Coreferential Reasoning Learning for Language Representation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing language representation models cannot explicitly handle coreference, which is essential to the coherent understanding of the whole discourse. |
| Approach: | They propose a language representation model that captures coreferential relations in context. |
| Outcome: | The proposed model can achieve significant improvements on downstream NLP tasks while maintaining comparable performance to baseline models on other common NLP task. |
EnsyNet: A Dataset for Encouragement and Sympathy Detection (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on emotional support detection focus on the presence or absence of emotional support, while the available datasets are limited or scarce in terms of size. |
| Approach: | They propose to use a dataset of 6,500 sentences annotated with encouragement and sympathy to train BERT-based classifiers on this dataset and apply their best BERT model to two large scale experiments. |
| Outcome: | The proposed model improves the emotional state of users while the lack of emotional support negatively impacts patients’ emotional state. |
Merge and Label: A Novel Neural Network Architecture for Nested NER (P19-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is one of the best studied tasks in natural language processing. |
| Approach: | They propose a neural network architecture that merges tokens and/or entities into nested entities and labels them independently. |
| Outcome: | The proposed approach achieves state-of-the-art F1 of 74.6 and improves with contextual embeddings to 82.4. |
Fair Evaluation in Concept Normalization: a Large-scale Comparative Analysis for BERT-based Models (2020.coling-main)
Copied to clipboard
| Challenge: | a large number of biomedical entity mentions are retrieved from different ontologies, requiring non-syntactic interpretation. |
| Approach: | They propose to use bidirectional encoder representations from transformers to link biomedical entities across three domains for a task called medical concept normalization. |
| Outcome: | The proposed neural architectures are efficient for linking biomedical entities across domains and corpora. |
A Model of the Language Process (2026.acl-long)
Copied to clipboard
| Challenge: | Language is a process that changes over time as new vocabulary emerges, word meanings shift, and narratives progress. |
| Approach: | They introduce a BERT style transformer encoder that models language by jointly learning to predict document contents and classify document publication dates. |
| Outcome: | The proposed model can predict document contents and classify document publication dates and accurately detects changes in word meanings. |
Event Causality Recognition Exploiting Multiple Annotators’ Judgments and Background Knowledge (D19-1)
Copied to clipboard
| Challenge: | Existing methods for recognizing event causality written in web texts ignore each annotator's independent judgments, but we exploit each anorator''s judgments to predict the majority vote labels. |
| Approach: | They propose to grasp each annotator's policy by training multiple classifiers that predict the labels given by a single annotators and combine the outputs to predict the final labels determined by majority vote. |
| Outcome: | The proposed methods grasp each annotator's policy and combine the outputs to predict the final labels determined by majority vote. |
The Right Tool for the Job: Matching Model and Instance Complexities (2020.acl-main)
Copied to clipboard
| Challenge: | a large increase in the size of NLP models can increase production costs and reduce adoption on real-time devices. |
| Approach: | They propose a modification to contextual representation fine-tuning which allows for an early exit from neural network calculations for simple instances and late exit for hard instances. |
| Outcome: | The proposed method produces models which are up to five times faster than the state of the art while preserving their accuracy. |
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (P19-1)
Copied to clipboard
| Challenge: | Existing methods to train multi-task neural networks outperform or even match their single-task counterparts are difficult to implement. |
| Approach: | They propose a method that uses knowledge distillation to train multi-task neural networks that outperform or even match their single-task counterparts. |
| Outcome: | The proposed method outperforms or matches single-task neural networks on the GLUE benchmark. |
BERT Knows Punta Cana is not just beautiful, it’s gorgeous: Ranking Scalar Adjectives with Contextualised Representations (2020.emnlp-main)
Copied to clipboard
| Challenge: | Adjectives describe positive properties of nouns but with different intensity. |
| Approach: | They propose a BERT-based approach to intensity detection for scalar adjectives by generating vectors directly from contextualised representations. |
| Outcome: | The proposed model outperforms static embeddings and previous models with dedicated resources on an Indirect Question Answering task. |
Harvesting and Refining Question-Answer Pairs for Unsupervised QA (2020.acl-main)
Copied to clipboard
| Challenge: | Recent research attempts to extend unsupervised question answering to settings with few or no labeled data available. |
| Approach: | They propose two approaches to improve unsupervised question answering . first, they harvest lexically and syntactically divergent Wikipedia questions to automatically construct a corpus of question-answer pairs . second, they take advantage of the QA model to extract more appropriate answers . |
| Outcome: | The proposed approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models. |
Probing for the Usage of Grammatical Number (2022.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can be used to perform a wide array of NLP tasks, but their encoding is still a mystery. |
| Approach: | They propose a usage-based probing setup to find an encoding that the model actually uses, and propose 'a use-based approach' they propose to use a behavioral task to remove the linguistic property, and to identify which encodes are used to transfer information from a noun to its head verb. |
| Outcome: | The proposed encodings are based on a behavioral task which cannot be solved without the linguistic property. |
Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification (2020.lrec-1)
Copied to clipboard
| Challenge: | Aspect-Target Sentiment Classification (ATSC) is a subtask of Aspect Based Sentimence Analysis (ABSA) . recent deep transfer-learning methods have been applied successfully to a myriad of NLP tasks. |
| Approach: | They propose to use a self-supervised domain-specific BERT language model to exploit ATSC . they also perform cross-domain evaluation to explore the real-world robustness of their models . |
| Outcome: | The proposed model outperforms baseline models on the SemEval 2014 task 4 restaurants dataset. |
#HowYouTagTweets: Learning User Hashtagging Preferences via Personalized Topic Attention (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods based on latent topics cannot capture user interests and thus can't be used to predict how likely a user will post with a hashtag. |
| Approach: | They propose a personalized topic attention model that captures salient contents to personalize hashtag contexts by predicting how likely a user will post with a hashtag. |
| Outcome: | The proposed model significantly outperforms the state-of-the-art recommendation approach without exploiting latent topics. |
SUM-QE: a BERT-based Summary Quality Estimation Model (D19-1)
Copied to clipboard
| Challenge: | SUM-QE is a quality estimation model for summarization that captures linguistic qualities that traditional evaluation metrics fail to capture. |
| Approach: | They propose a new quality estimation model based on BERT that addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation without comparison with human ratings. |
| Outcome: | The proposed model outperforms existing models on linguistic quality aspects that are only indirectly captured by content-based summarization evaluations without comparison with human ratings. |
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Neural architecture search (NAS) uses weight-sharing supernets to generate diverse subnetworks without retraining. |
| Approach: | They propose a weight-sharing supernet that leverages mixture-of-experts to enhance supernet model expressiveness with minimal training overhead. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) performance in NAS for fast machine translation models, surpassing NAS-BERT and AutoDistil across various model sizes. |
FrameEOL: Semantic Frame Induction using Causal Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. |
| Approach: | They propose a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label . |
| Outcome: | The proposed method outperforms existing methods on English and Japanese datasets. |
ParlVote: A Corpus for Sentiment Analysis of Political Debates (2020.lrec-1)
Copied to clipboard
| Challenge: | Debate transcripts from the UK Parliament contain information about the positions taken by politicians towards important topics, but are difficult for humans to process manually. |
| Approach: | They propose to use a linear classifier and a transformer word embedding model to classify sentiment polarity in debate speeches to evaluate sentiment analysis systems for the political domain. |
| Outcome: | The proposed method performs better on the largest dataset and is more robust to other datasets. |
Aggregating Bidirectional Encoder Representations Using MatchLSTM for Sequence Matching (D19-1)
Copied to clipboard
| Challenge: | Recent work on text sequence matching tasks uses task specific supervised datasets, which are always limited to the amount due to the cost of annotation. |
| Approach: | They propose an aggregation method to combine Bidirectional Encoder Representations from Transformer (BERT) with a MatchLSTM layer for Sequence Matching. |
| Outcome: | The proposed model improves on two publicly available datasets, WikiQA and SNLI. |
Transition-based Semantic Dependency Parsing with Pointer Networks (2020.acl-main)
Copied to clipboard
| Challenge: | Existing dependency parsers cannot be directly applied, so they need to be adaptable to deal with the absence of singlehead and connectedness constraints. |
| Approach: | They propose a transition system that produces labelled directed acyclic graphs and performs semantic dependency parsing with Pointer Networks. |
| Outcome: | The proposed system outperforms graph-based models and outperformed existing models on a harder NLP problem. |
tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection (2020.acl-main)
Copied to clipboard
| Challenge: | Recent pretrained contextual representations such as ELMo and BERT have led to impressive performance gains across a variety of NLP tasks, including semantic similarity detection. |
| Approach: | They propose a topic-informed BERT-based architecture for pairwise semantic similarity detection that adds topic information to pretrained contextual representations such as BERT. |
| Outcome: | The proposed model outperforms existing models on a variety of English language datasets and is highly performant. |
PatchBERT: Just-in-Time, Out-of-Vocabulary Patching (2020.emnlp-main)
Copied to clipboard
| Challenge: | a pre-trained language model with low OOV can improve performance for transfer learning . a vocabulary surrogate can provide performance boosts with no additional computation cost . |
| Approach: | They propose multiple methods to mitigate OOV during downstream task fine-tuning . they demonstrate that vocabulary surrogates can provide performance boosts with no additional computation cost . |
| Outcome: | The proposed methods improve performance with the same parameter count when combined with fine-tuning. |
Exploring Graph Representations of Logical Forms for Language Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | Graph-based formal-logical distributional semantics models are more data-efficient than textual counterparts. |
| Approach: | They propose a pretrained language model over graph representations of logical forms as a proof-of-concept. |
| Outcome: | The proposed model outperforms textual, transformer LMs on downstream tasks . the model is likely to scale with additional parameters and pretraining data . |
Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have demonstrated state-of-the-art predictive performance when adapted into a range of language understanding tasks. |
| Approach: | They propose to use salient information extracted a priori from training data to complement the task-specific information learned by the model during fine-tuning on a downstream task. |
| Outcome: | The proposed model can provide more faithful explanations across four different feature attribution methods compared to vanilla BERT. |
In Plain Sight: Media Bias Through the Lens of Factual Reporting (D19-1)
Copied to clipboard
| Challenge: | lexical bias stems from content realization, or how things are said, but other forms of bias stem from content selection and organization. |
| Approach: | They use a dataset to analyze news articles annotated with 1,727 bias spans to investigate informational bias. |
| Outcome: | The proposed model shows that informational bias appears more frequently than lexical bias. |
Understanding Advertisements with BERT (2020.acl-main)
Copied to clipboard
| Challenge: | Recent results have shown that the embedded scene-text in the image holds a vital cue for this task. |
| Approach: | They propose to use the embedded scene-text as a cue for a sentence-pair classification task based on CVPR 2018 challenge dataset on advertisement understanding to rank valid and negatively sampled invalid interpretations of an image. |
| Outcome: | The proposed model achieves 89.69% accuracy, an improvement of 4.7% on the previous model. |
Generating Multiple-choice Questions for Medical Question Answering with Distractors and Cue-masking (2024.lrec-main)
Copied to clipboard
| Challenge: | Medical multiple-choice question answering (MCQA) requires high accuracy to be useful in practice. |
| Approach: | They propose to focus masked language modeling on disease name prediction when using medical encyclopedic paragraphs as input. |
| Outcome: | The proposed model outperforms the masked language model on disease name prediction and masks the cues to the answers. |
Distilling Knowledge Learned in BERT for Text Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models such as BERT have revolutionized the state of the art in many language understanding tasks. |
| Approach: | They propose a conditional masked language modeling approach to fine tune BERT on target generation tasks by imposing global sequence-level supervision on conventional Seq2Seq models. |
| Outcome: | The proposed model outperforms strong Transformer baselines on multiple language generation tasks such as machine translation and text summarization. |
Attention-Enhancing Backdoor Attacks Against BERT-based Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing textual backdoor attacks focus on generating stealthy triggers or modifying model weights. |
| Approach: | They propose a Trojan Attention Loss (TAL) which enhances the Trojan behavior by directly manipulating attention patterns. |
| Outcome: | The proposed method improves the effectiveness of the backdoor attacks on different backbone models and tasks. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
KOLD: Korean Offensive Language Dataset (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent directions for offensive language detection focus on English and do not transfer well to other languages because of cultural and linguistic differences. |
| Approach: | They present a Korean offensive language dataset annotated with offensive language comments . they use the comments as training data for Korean BERT and RoBERTa models . |
| Outcome: | The proposed model improves offensiveness detection, target classification, and span detection while having room for improvement for target group classification and span prediction. |
Cross-lingual Emotion Detection (2022.lrec-1)
Copied to clipboard
| Challenge: | Emotion detection is a useful tool for understanding human behavior, but constructing annotated datasets to train models can be expensive. |
| Approach: | They propose to use English as the source language with Arabic and Spanish as target languages to train models for emotion detection in a target language. |
| Outcome: | The proposed approaches surpass state-of-the-art models in Arabic and Spanish by 4% and 5% respectively. |
The MERSA Dataset and a Transformer-Based Approach for Speech Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models for speech emotion recognition lack a comprehensive dataset to design accurate models. |
| Approach: | They propose to use a multimodal dataset to build a model that integrates pre-trained wav2vec 2.0 and BERT to learn hidden representations from fused representations of speech and text. |
| Outcome: | The proposed model predicts emotions on dimensions of arousal, valence, and dominance . it achieved competitive results on the MSP-PODCAST dataset . |
Minority Positive Sampling for Switching Points - an Anecdote for the Code-Mixing Language Modeling (2020.lrec-1)
Copied to clipboard
| Challenge: | Multilingual people code-mix using English phonetic typing and insertion of anglicisms in their native language. |
| Approach: | They propose to use minority positive sampling to selectively induce more sample to achieve better performance. |
| Outcome: | The proposed model performs better than other models, but switching points are the main challenge . |
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition (2020.acl-main)
Copied to clipboard
| Challenge: | Natural language inference (NLI) is an increasingly important task for natural language understanding . however, the ability of NLI models to make pragmatic inferences remains understudied . |
| Approach: | They use semi-automatically generated sentence pairs to evaluate whether NLI models make pragmatic inferences. |
| Outcome: | The proposed model trains on multiNLI and shows that it learns to draw pragmatic inferences. |
Syntax-driven Approach for Semantic Role Labeling (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies focus on auto-generated syntactic knowledge to enhance semantic role labeling . experimental results show that map memories can enhance SRL . |
| Approach: | They propose to map memories to enhance semantic role labeling by encoding auto-generated syntactic knowledge from off-the-shelf toolkits. |
| Outcome: | The proposed model outperforms baselines and achieves state-of-the-art results on two English benchmark datasets. |
Revisiting Higher-Order Dependency Parsers (2020.acl-main)
Copied to clipboard
| Challenge: | Neural encoders have allowed dependency parsers to shift from higher-order structured models to simpler first-order ones, making decoding faster and still achieving better accuracy than non-neural parser. |
| Approach: | They found that neural parsers may benefit from higher-order features when employing a powerful pre-trained encoder, such as BERT. |
| Outcome: | Using a pre-trained encoder, we found that higher-order models are more accurate on full sentence parses and match of modifier lists. |
AUTOSUMM: Automatic Model Creation for Text Summarization (2021.emnlp-main)
Copied to clipboard
Sharmila Reddy Nangi, Atharv Tyagi, Jay Mundra, Sagnik Mukherjee, Raj Snehal, Niyati Chhaya, Aparna Garimella
| Challenge: | Recent efforts to develop deep learning models for text generation tasks are challenging for non-experts. |
| Approach: | They propose methods to automatically create deep learning models for extractive and abstractive summarization tasks using large language models. |
| Outcome: | The proposed methods achieve near state-of-the-art performance on a range of datasets. |
A Curious Case of Searching for the Correlation between Training Data and Adversarial Robustness of Transformer Textual Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that fine-tuned textual transformer models are vulnerable to adversarial text perturbations. |
| Approach: | They extract 13 different features representing a wide range of input fine-tuning corpora properties and use them to predict adversarial robustness of the fine- tuned models. |
| Outcome: | The proposed framework can be used as an additional tool for robustness evaluation since it saves 30x-193x runtime compared to the traditional technique and can be easily used under adversarial training. |
Investigating the Saliency of Sentiment Expressions in Aspect-Based Sentiment Analysis (2023.findings-acl)
Copied to clipboard
| Challenge: | In a set of masking experiments, we examine the extent to which the tokens identified as salient by LIME and a gradient-based method are being used by the classifier. |
| Approach: | They use a BERT BASE model to mask the sentiments of an English dataset and find that both methods produce faithful rationales. |
| Outcome: | The proposed classifier outperforms both the gradient-based and black-box saliency methods on the SemEval 2016 english dataset. |
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)
Copied to clipboard
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
| Challenge: | High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters. |
| Approach: | They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters. |
| Outcome: | The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches. |
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies on compression of pretrained language models usually use preserved accuracy as the metric for evaluation. |
| Approach: | They propose two new metrics that measure how closely a compressed model mimics the original model. |
| Outcome: | The proposed metrics measure how closely a compressed model (i.e., student) mimics the original model (e.g., teacher). |
Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models (2025.findings-acl)
Copied to clipboard
Ahmed Elshabrawy, Thanh-Nhi Nguyen, Yeeun Kang, Lihan Feng, Annant Jain, Faadil Abdullah Shaikh, Jonibek Mansurov, Mohamed Fazli Mohamed Imam, Jesus-German Ortiz-Barajas, Rendi Chevi, Alham Fikri Aji
| Challenge: | Large Language Models excel in zero-shot and few-shot tasks, but their architecture makes them difficult to use. |
| Approach: | They adapt Large Language Models (LLMs) for zero-shot generalization using Statement Tuning . they find encoders can achieve zero- shot cross-lingual generalization . |
| Outcome: | The proposed model generalizes well across languages while being more efficient. |
Exploring Large Language Models for Classical Philology (2023.acl-long)
Copied to clipboard
| Challenge: | Recent advances in NLP have led to the creation of powerful language models for many languages including Ancient Greek and Latin. |
| Approach: | They propose to use encoder-only and encoder decoder architectures to create four models for Ancient Greek that vary along two dimensions for tasks of interest for Classical languages. |
| Outcome: | The proposed models improve on existing models of Ancient Greek and Latin and provide a large pre-training corpus for Ancient Greek to support the creation of a larger, comparable model zoo for Classical Philology. |
Language-Agnostic Bias Detection in Language Models with Bias Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) contain strong social biases, which are difficult to quantify because current methods focusing on fill-the-mask objectives are sensitive to slight changes in input. |
| Approach: | They propose a bias probing technique called LABDet to evaluate social bias in pretrained language models with a language-agnostic method. |
| Outcome: | The proposed method “surfaces” nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection. |
FinEntity: Entity-level Sentiment Classification for Financial Texts (2023.emnlp-main)
Copied to clipboard
| Challenge: | FinEntity annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. |
| Approach: | They introduce an entity-level sentiment classification dataset called FinEntity that annotates financial entity spans and their sentiment in financial news. |
| Outcome: | The proposed dataset annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. |
Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for implicit hate speech detection do not have significant advantage over cross-entropy loss-based learning. |
| Approach: | They propose a label-aware hard negative sampling strategy that encourages the model to learn detailed features from hard negative samples instead of random batch. |
| Outcome: | The proposed models outperform existing models for implicit hate speech detection both in- and cross-datasets. |
Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages (2023.findings-emnlp)
Copied to clipboard
Elena Ruzzetti, Federico Ranaldi, Felicia Logozzo, Michele Mastromattei, Leonardo Ranaldi, Fabio Zanzotto
| Challenge: | a recent study examined how models for typologically similar languages encode structural information. |
| Approach: | They propose to layer-wise compare transformers for typologically similar languages to observe similarities . they use a domain adaptation on semantically equivalent texts to measure similarity . |
| Outcome: | The proposed model outperforms all other models on unseen sentences . the proposed model is based on a typologically similar language . |
Annotation Sensitivity: Training Data Collection Methods Affect Model Performance (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using an annotation instrument, the design of the annotation instrument and the instructions given to annotators can impact training data. |
| Approach: | They investigate the impact of an annotation instrument on training data . they collect hate speech and offensive language annotations in a tweet corpus . |
| Outcome: | The proposed model performs better on holdout conditions than on the standard model. |
Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. |
| Approach: | They compare performance of financial BERT-like models to their fully fine-tuned counterparts by using parameter-efficient tuning methods. |
| Outcome: | The proposed approaches match full fine-tuning performance on common NLP tasks, but are less studied in finance. |
BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models like BERT deteriorate in the face of dialect variation or noise. |
| Approach: | They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. |
| Outcome: | The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts. |
Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods modify attention mechanism to be bidirectional, undermining LLMs’ ability to extract semantic information acquired during pre-training. |
| Approach: | They propose a general-purpose embedding model that pre-encodes input text into a single Contextual token and then prepends it to the LLM's input sequence. |
| Outcome: | The proposed model improves performance of decoder-only large language models without altering their architectures or introducing significant computational overhead. |
New Evaluation Methodology for Qualitatively Comparing Classification Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Text Classification is one of the most common tasks in Natural Language Processing. |
| Approach: | They propose a method for performing qualitative assessment over multiple classification models using a fine-tuned BERT and Logistic Regression evaluation methodology. |
| Outcome: | The proposed evaluation methodology outperforms the baseline model in linguistic clustering and Sentiment Analysis. |
Maximizing the Effectiveness of Larger BERT Models for Compression (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for capturing large BERT models as teachers do not fully exploit the potential advantages of larger teachers. |
| Approach: | They propose a method that leverages a pretrained teacher model to guide the training of a lightweight student model to enhance knowledge transfer. |
| Outcome: | The proposed method enhances knowledge transfer by leveraging a pretrained teacher model to guide the training of a lightweight student model. |
Projective Methods for Mitigating Gender Bias in Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Mitigating gender bias in NLP has a long history tied to debiasing static word embeddings. |
| Approach: | They propose a masked language modelling task where content is developed around known social stereotypes and a projective debiasing method is used to reduce bias. |
| Outcome: | The proposed methods reduce intrinsic bias and mitigat observed bias in a downstream setting, but the two outcomes are not necessarily correlated. |
Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Structured pruning is an effective technique for compressing pre-trained language models (PLMs), but it requires retraining, leading to additional computational overhead. |
| Approach: | They propose a task-specific pruning framework that prunes redundant modules of pre-trained language models before fine-tuning them. |
| Outcome: | The proposed pruning framework achieves higher performance on GLUE, SQUAD, WikiText-2, Wik-103, and PTB datasets while reducing the time required for fine-tuning. |
BiasWipe: Mitigating Unintended Bias in Text Classifiers through Model Interpretability (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate unintended bias in social media platforms are re-training and adding extra parameters to the model. |
| Approach: | They propose a technique to mitigate unintended bias in language models by pruning the neuron weights responsible for univ bias. |
| Outcome: | The proposed technique achieves fairness by pruning the neuron weights responsible for unintended bias without loss of original performance. |
MedAdapter: Efficient Test-Time Adaptation of Large Language Models Towards Medical Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have improved generation and reasoning capabilities compared to traditional BERT-sized models due to massive number of parameters and extensive pre-training on vast textual corpora. |
| Approach: | They propose a unified post-hoc adapter for test-time adaptation of large language models . they propose to fine-tune only a small BERT-sized adapter to rank candidate LLMs . |
| Outcome: | The proposed adapter improves performance on four biomedical tasks without requiring computational resources or sharing data with third parties. |
What Do Transformers Know about Government? (2024.lrec-main)
Copied to clipboard
Jue Hou, Anisia Katinskaia, Lari Kotilainen, Sathianpong Trangcasanchai, Anh-Duc Vu, Roman Yangarber
| Challenge: | Currently, data is lacking for the research community working on grammatical constructions, and government in particular. |
| Approach: | They use transformer language models to study how government relations are encoded . they use morphologically rich languages to train a classifier capable of discovering new types of government . |
| Outcome: | The proposed classifiers can learn new types of government, the authors show . they find that the classifier can learn government relations in two languages . |
Exploring Large Language Models for Detecting Mental Disorders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Detecting mental disorders and patient emotions through text analysis and machine learning is of increasing interest to researchers over the past decade. |
| Approach: | They compare the performance of traditional machine learning methods and encoder-based models on Russian-language datasets to those of large language models. |
| Outcome: | The proposed models outperform traditional methods on small and noisy datasets, but can perform comparable to language models when trained on patients with clinically confirmed depression. |