SPANBERT

Papers with SpanBERT

SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)

Copied to clipboard

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy

Challenge:	Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text.
Approach:	They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span.
Outcome:	The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks.

Euphemistic Phrase Detection by Masked Language Model (2021.findings-emnlp)

Copied to clipboard

Wanzheng Zhu, Suma Bhat

Language Models

Challenge:	euphemisms are ordinary-sounding words with a secret meaning that are used to conceal information . a primary motive of their use on social media is to evade content moderation efforts .
Approach:	They propose to use social media to detect euphemisms without human effort . they first perform phrase mining on a raw text corpus to extract quality phrases . then they use word embedding similarities to select a set of euphoristic phrase candidates .
Outcome:	The proposed algorithm shows 20-50% higher detection accuracies than baselines.

Relation Classification with Entity Type Restriction (2021.findings-acl)

Copied to clipboard

Shengfei Lyu, Huanhuan Chen

Information Extraction & Text Mining

Challenge:	Existing methods regard all relations as candidate relations for the two entities, which leads to inappropriate relations being candidate relations.
Approach:	They propose a paradigm which exploits entity types to restrict candidate relations by mutual restrictions.
Outcome:	The proposed paradigm improves GCN and SpanBERT on a standard dataset by 6.9 and 4.4 F1 points.

On the Robustness of Reading Comprehension Models to Entity Renaming (2022.naacl-main)

Copied to clipboard

Jun Yan, Yang Xiao, Sagnik Mukherjee, Bill Yuchen Lin, Robin Jia, Xiang Ren

Language Models

Challenge:	SpanBERT model is more robust than RoBERTa, despite having similar accuracy on unperturbed test data.
Approach:	They propose a pipeline to replace entity names with names from a variety of sources.
Outcome:	The proposed model performs worse when entities are renamed, the authors show . SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa .

Domain-Adaptive Pretraining Methods for Dialogue Understanding (2021.acl-short)

Copied to clipboard

Han Wu, Kun Xu, Linfeng Song, Lifeng Jin, Haisong Zhang, Linqi Song

Language Models

Challenge:	Recent advances in pretraining methods have achieved promising results on NLP tasks . however, it is unclear which pretraining objective is the most effective for each downstream task .
Approach:	They evaluate the effectiveness of domain-adaptive pretraining objectives on downstream tasks . they use open-domain data to pretrain language models like BERT and SpanBERT .
Outcome:	The proposed model improves on two dialogue understanding tasks with domain-adaptive pretraining objectives.

Exploiting Rich Textual User-Product Context for Improving Personalized Sentiment Analysis (2023.findings-acl)

Copied to clipboard

Chenyang Lyu, Linyi Yang, Yue Zhang, Yvette Graham, Jennifer Foster

Sentiment Analysis

Challenge:	Typical approaches do not exploit the potential of historical reviews or do not make full use of user/product associations.
Approach:	They propose to use historical reviews to initialize user and product representations and incorporate textual associations via a user-product cross-context module.
Outcome:	The proposed method outperforms existing state-of-the-art models on IMDb, Yelp and Longformer benchmarks.

BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection (2021.eacl-main)

Copied to clipboard

Beatrice Portelli, Edoardo Lenzi, Emmanuele Chersoni, Giuseppe Serra, Enrico Santus

Language Models

Challenge:	Pretrained transformer-based models are a common choice for identifying drug events from social media texts.
Approach:	They propose to compare transformer-based models with in-domain language pretraining to find out which one is better at ADE detection.
Outcome:	The proposed models outperform SpanBERT and PubMedBERT on two benchmarks.

How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks (2021.findings-emnlp)

Copied to clipboard

Hoang Van, Zheng Tang, Mihai Surdeanu

Language Models

Challenge:	Recent studies have focused on rule-based and neural sequence-to-sequence (seq2sequ) TS is a technique that reduces text complexity for human consumption.
Approach:	They evaluate two possible uses of neural TS: simplifying input texts at prediction time and augmenting training data to provide machines with additional information during training.
Outcome:	The proposed approach improves performance on two datasets.

Multi-level Contrastive Learning for Script-based Character Understanding (2023.emnlp-main)

Copied to clipboard

Dawei Li, Hengyuan Zhang, Yanran Li, Shiping Yang

Semantic Text Processing

Challenge:	Scripts are written text for plays, movies, or broadcasts.
Approach:	They propose a multi-level contrastive learning framework to capture characters’ global information in a fine-grained manner.
Outcome:	The proposed framework improves on three character understanding sub-tasks by a considerable margin.

PairSpanBERT: An Enhanced Language Model for Bridging Resolution (2023.acl-long)

Copied to clipboard

Hideo Kobayashi, Yufang Hou, Vincent Ng

Semantic Text Processing

Challenge:	bridging resolution is crucial for machine comprehension of discourse entities for various downstream applications.
Approach:	They propose a SpanBERT-based pre-trained model specialized for bridging resolution.
Outcome:	The proposed model achieves the best results on three evaluation datasets for bridging resolution despite the noise inherent in the automatically generated data .

Word-Level Coreference Resolution (2021.emnlp-main)

Copied to clipboard

Vladimir Dobrovolskii

Information Extraction & Text Mining

Challenge:	Recent coreference resolution models rely heavily on span representations to find coreference links between word spans.
Approach:	They propose to consider coreference links between individual words rather than word spans and reconstruct the word span.
Outcome:	The proposed model outperforms existing models on the OntoNotes benchmark while being highly efficient.

Revealing the Myth of Higher-Order Inference in Coreference Resolution (2020.emnlp-main)

Copied to clipboard

Liyan Xu, Jinho D. Choi

Information Extraction & Text Mining

Challenge:	Adapted coreference resolution models have only marginally improved performance over representation learning.
Approach:	They implement an end-to-end coreference system and four HOI approaches to analyze the impact of higher-order inference on coreference resolution.
Outcome:	The proposed model shows that the impact of higher-order inference (HOI) on coreference resolution is negative to marginal, providing a new perspective on the task.

Misery Loves Complexity: Exploring Linguistic Complexity in the Context of Emotion Detection (2023.findings-emnlp)

Copied to clipboard

Pranaydeep Singh, Luna De Bruyne, Orphée De Clercq, Els Lefever

Semantic Text Processing

Challenge:	a negative emotion is a cognitive bias that affects how we express thoughts and opinions online . a recent study shows that negative words generate more engagement and clicks than positive ones .
Approach:	They propose to use readability and linguistic complexity metrics to better understand emotions . they propose to fine-tune three state-of-the-art transformers to detect emotions based on a dataset .
Outcome:	The proposed model fails to predict emotions on complex texts, the authors show . they also show that more advanced models fail to predict complex texts .