Papers with pre-trained
Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models (2023.acl-srw)
Copied to clipboard
| Challenge: | Existing datasets and analyses focusing on English do not address the need for resources in other languages. |
| Approach: | They propose a Japanese NLI benchmark focused on temporal inference . they use a set of temporal patterns to generate diverse examples . |
| Outcome: | The proposed model can perform fine-grained analysis in Japanese and English. |
Relevance, Diversity, and Exclusivity: Designing Keyword-augmentation Strategy for Zero-shot Classifiers (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods incorporate semantically similar keywords related to class names, but the properties of effective keywords remain unclear. |
| Approach: | They propose a method for acquiring keywords that satisfy these properties without additional knowledge bases or data. |
| Outcome: | The proposed method outperforms existing methods in fully zero-shot and generalized zero- shot settings. |
Lacuna Reconstruction: Self-Supervised Pre-Training for Low-Resource Historical Document Transcription (2022.findings-naacl)
Copied to clipboard
| Challenge: | Document transcription models are limited by extremely varied style and content across domains. |
| Approach: | They propose a self-supervised approach for learning rich visual representations for both handwritten and printed historical document transcription using a heterogeneous set of handwritten Islamicate manuscript images and early modern English printed documents. |
| Outcome: | The proposed model improves on a supervised model with as few as 30 line image transcriptions on two languages with a single line of image training. |
Zuo Zhuan Ancient Chinese Dataset for Word Sense Disambiguation (2022.naacl-srw)
Copied to clipboard
| Challenge: | Word Sense Disambiguation (WSD) is a core task in natural language processing . ancient Chinese has rarely been used in WSD tasks due to lack of a dataset . |
| Approach: | They annotate ancient Chinese text Zuo Zhuan using a copyright-free dictionary . they apply a method to find the most appropriate sense in a context using k-NN . |
| Outcome: | The proposed dataset will be available on GitHub. |
Stateful Memory-Augmented Transformers for Efficient Dialogue Modeling (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing Transformers models are computationally expensive for long context inputs. |
| Approach: | They propose a transformer that can interchange information between memory states and context . they evaluate the efficiency of their model on three dialogue datasets and two language datasets . |
| Outcome: | The proposed model is compatible with existing transformer models and can preserve dialogue history information. |
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for summarizing dialogues lack in taking into account the structure of dialogues and rely heavily on labeled data. |
| Approach: | They propose a pre-trained encoder-decoder model for summarizing dialogues in any new domain. |
| Outcome: | The proposed model outperforms existing methods on six datasets and shows ROUGE scores in zero-shot and few-shot settings. |
A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products (2022.coling-1)
Copied to clipboard
| Challenge: | Existing pre-trained language models lack medicinal product knowledge for product vertical search. |
| Approach: | They propose a biomedical knowledge enhanced pre-trained language model for medicinal product vertical search using ELECTRA’s replaced token detection (RTD) pre-training. |
| Outcome: | The proposed model improves query-title relevance, query intent classification, and named entity recognition in query. |
DivHSK: Diverse Headline Generation using Self-Attention based Keyword Selection (2023.findings-acl)
Copied to clipboard
| Challenge: | Diverse headline generation is an NLP task where the goal is to generate multiple headlines that are true to the content of the article but are different among themselves. |
| Approach: | They propose a novel model that generates multiple diverse headlines by using a pre-trained encoder and a cluster-specific keyword set. |
| Outcome: | The proposed model outperforms existing literature and their strong baselines and emerges as a state-of-the-art model. |
Pre-trained Language Model Based Active Learning for Sentence Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Existing active learning approaches for natural language processing ignore the characteristics of natural language. |
| Approach: | They propose a pre-trained language model based active learning approach for sentence matching that provides linguistic criteria to measure instances and help select more effective instances for annotation. |
| Outcome: | The proposed approach can achieve greater accuracy with fewer labeled training instances. |
Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation Approach (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have achieved competitive performance with limited labeled data for many NLP tasks. |
| Approach: | They propose a prompt-based data selection method for pre-trained language models fine-tuning under cold-start scenarios. |
| Outcome: | The proposed method outperforms the strongest cold-start data selection baselines on six text classification datasets with 128 labels. |
AttentionRank: Unsupervised Keyphrase Extraction using Self and Cross Attentions (2021.emnlp-main)
Copied to clipboard
| Challenge: | Keyword or keyphrase extraction is to identify words or phrases presenting the main topics of a document. |
| Approach: | They propose a hybrid attention model to identify keyphrases from a document in an unsupervised manner. |
| Outcome: | The proposed model is effective and robust on long and short documents. |
Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune pre-trained models to downstream tasks are limited by labeled examples. |
| Approach: | They propose to apply post-training on unlabeled task data before fine-tuning by contrastive learning that considers either token-level or sequence-level similarity. |
| Outcome: | Empirical results show that contrastive masked language modeling surpasses other methods in few-shot settings without the need for data augmentation. |
Multi-Task Learning for Knowledge Graph Completion with Pre-trained Language Models (2020.coling-main)
Copied to clipboard
| Challenge: | Existing knowledge graph completion methods are lacking in ranking metrics such as Hits@k . despite the high performance, the proposed method is still behind state-of-the-art models. |
| Approach: | They propose a multi-task learning method that integrates relational and relevance ranking tasks with target link prediction to improve ranking performance. |
| Outcome: | The proposed method improves ranking performance but still behind state-of-the-art models in Hits@k and Mean Rank metrics. |
Non-compositional Expression Generation and its Continual Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent work shows that pre-trained language models are limited in their ability to generate non-compositional expressions. |
| Approach: | They propose a mask-infilling task to examine non-compositional expressions in English . they compare large pre-trained language models and continual learning methods . |
| Outcome: | The proposed task aims to investigate the ability of pre-trained language models to generate non-compositional expressions in English and their continual learning. |
Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-trained language models address information confusion in position encoding and model representations. |
| Approach: | They propose two techniques to improve pre-trained language models by decoupling directions and auxiliary regularizers. |
| Outcome: | The proposed techniques can improve pre-trained language models on GLUE benchmarks. |
Definition Modelling for Appropriate Specificity (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing definition generation techniques have faced various problems such as the out-of-vocabulary problem and over/under-specificity problems. |
| Approach: | They propose to leverage a pre-trained encoder-decoder model and introduce a re-ranking mechanism to model specificity in definitions. |
| Outcome: | The proposed method significantly outperforms the state-of-the-art method on standard evaluation datasets and shows that it addresses the over/under-specificity problems. |
Clues Before Answers: Generation-Enhanced Multiple-Choice QA (2022.naacl-main)
Copied to clipboard
| Challenge: | Multiple-choice question answering (MCQA) uses text-to-text framework . but, there is an under-utilization of the decoder and knowledge that can be decoded . |
| Approach: | They propose a generative multiple-choice question answering model which generates a clue from the question and leverages it to enhance a reader for MCQA. |
| Outcome: | The proposed model outperforms text-to-text models on multiple MCQA datasets. |
Scientific Paper Extractive Summarization Enhanced by Citation Graphs (2022.emnlp-main)
Copied to clipboard
| Challenge: | citation graphs can be used to extract scientific papers under different conditions. |
| Approach: | They propose a multi-granularity unsupervised summarization model that fine tunes a pre-trained encoder model on the citation graph by link prediction tasks. |
| Outcome: | The proposed model outperforms baseline models on a public benchmark dataset. |
Exploring Robust Overfitting for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent literature has revealed their vulnerability to crafted adversarial examples on a wide range of NLP tasks. |
| Approach: | They propose to combine regularization methods with adversarial training to mitigate robust overfitting for pre-trained language models by scaling the model's loss. |
| Outcome: | The proposed methods mitigate robust overfitting upon three top adversarial training methods and further promote adversarially robustness. |
On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that mixing up can improve model calibration on image classification tasks, but little is known about using it on natural language understanding (NLU) tasks. |
| Approach: | They propose a mixup strategy for pre-trained language models that improves model calibration further by using the AUM statistic and saliency map. |
| Outcome: | The proposed mixup improves model calibration on natural language understanding tasks while maintaining competitive accuracy. |
Pre-training Universal Language Representation (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the cutting-edge representation learning, most language models focus on specific levels of linguistic units. |
| Approach: | They propose a training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by an algorithm for pre-trained language models. |
| Outcome: | The proposed model achieves highest accuracy on analogy tasks in different language levels and significantly improves performance on downstream tasks. |
Order-Agnostic Data Augmentation for Few-Shot Named Entity Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing DA methods for named entity recognition (NER) are costly and labor-intensive to acquire, necessitating innovative approaches to data scarcity. |
| Approach: | They propose an order-agnostic data augmentation solution that exploits the order-based property in the training phase of sequence-to-sequence NER methods for data augmented. |
| Outcome: | The proposed method significantly enhances the few-shot capabilities of pre-trained language models in low-resource settings. |
Prototypical Verbalizer for Prompt-based Few-shot Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Prompt-based tuning for pre-trained language models has shown its effectiveness in few-shot learning. |
| Approach: | They propose a prototypical verbalizer which learns prototype vectors as verbalizes by contrastive learning. |
| Outcome: | The proposed verbalizer outperforms existing verbalizing methods on topic classification and entity typing tasks. |
Pruning Pre-trained Language Models with Principled Importance and Self-regularization (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models often contain a vast amount of parameters, posing nontrivial requirements for storage and computation. |
| Approach: | They propose a pruning method where model prediction is regularized by the latest checkpoint with increasing sparsity throughout pruning. |
| Outcome: | The proposed approach is effective at sparsity levels, and can be applied to natural language understanding, question answering, and data-to-text generation tasks. |
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022.emnlp-main)
Copied to clipboard
Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova
| Challenge: | Pre-trained language models struggle on out-of-distribution compositional generalization . recent work shows considerable improvements on many NLP tasks from model scaling . |
| Approach: | They evaluate encoder-decoder models up to 11B parameters and decoder-only models up 540B parameters . they compare scaling curves for fine-tuning, prompt tuning, and in-context learning methods . |
| Outcome: | The proposed scaling methods improve compositional generalization on many tasks . fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional . larger models are better at modeling the syntax of the output space, the study finds . |
Once Upon a Time in Graph: Relative-Time Pretraining for Complex Temporal Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on strengthening the knowledge-time association between text and time-stamps, but this is insufficient for downstream tasks. |
| Approach: | They propose a model that explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. |
| Outcome: | The proposed model outperforms baseline T5 on multiple temporal question answering datasets . it is especially good at modeling long-range complex temporal dependencies, the authors say . |
Does Listener Gaze in Face-to-Face Interaction Follow the Entropy Rate Constancy Principle: An Empirical Study (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies have shown that nonverbal behaviours are rich in communicative functions, such as gaze, head movements, and speech-accompanying manual gestures. |
| Approach: | They train a transformer-based neural sequence model to process gaze data extracted from video-recorded conversations and compute its information density. |
| Outcome: | The proposed model computes listeners’ gaze behaviour and the information density of speech using a pre-trained language model. |
IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing reference-free automatic grammatical error correction methods do not correlate with human evaluation. |
| Approach: | They propose a reference-free automatic grammatical error correction evaluation method with enhanced gramma-ed capabilities. |
| Outcome: | The proposed method achieves highest correlation with human evaluations on a meta-evaluation dataset. |
Text Filtering Classifiers for Medium-Resource Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | linguistic and NLP researchers use web-crawled corpora to filter low-quality texts . early Transformer-based language models were typically pre-trained on curated corporum . |
| Approach: | They compare the effectiveness of various text filtering classifiers on Icelandic, Estonian and Basque texts . they use a perplexity-based classifier and a self-supervised classifier trained on TQ-IS to discern between documents from curated and web-crawled corpora. |
| Outcome: | The proposed classifiers achieve F1 scores of 94.48%, 99.01% and 93.40% on the Icelandic, Estonian and Basque datasets. |