ELMO

How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (D19-1)

Challenge:	Embeddings have been a key topic of interest in NLP for the past decade . a quick warm-up introduction to NLP and why it is important to have a semantic comprehension of texts .
Approach:	This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and then move to other types of embeddable vectors .
Outcome:	This tutorial will provide a high-level synthesis of the main embedding techniques in NLP . it will start with word embedds and move to other types of embeddable representations .

Copied to clipboard

Kawin Ethayarajh

Semantic Specialization of Distributional Word Vectors (D19-2)

Challenge:	Existing word embeddings were static, requiring all senses of a polysemous word to share the same representation.
Approach:	They found that the contextualized representations of all words are not isotropic in any layer of the contextualizing model.
Outcome:	The results show that the representations of all words are not isotropic in any layer of the contextualizing model.

Copied to clipboard

Goran Glavaś, Edoardo Maria Ponti, Ivan Vulić

Low-Resource NLP

Challenge:	Distributional word vectors conflate various paradigmatic and syntagmatic lexico-semantic relations.
Approach:	This tutorial provides an overview of specialization methods for distributional word vectors . a common solution is to include external lexico-semantic knowledge in a reshaped vector space .
Outcome:	This paper provides an overview of specialization methods for distributional word vectors . the most recent developments include a new method for asymmetric relations in Euclidean .

The Amazing World of Neural Language Generation (2020.emnlp-tutorials)

Copied to clipboard

Yangfeng Ji, Antoine Bosselut, Thomas Wolf, Asli Celikyilmaz

GRUBERT: A GRU-Based Method to Fuse BERT Hidden Layers for Twitter Sentiment Analysis (2020.aacl-srw)

Challenge:	Recent years have seen a paradigm shift in neural text generation due to advances in deep contextual language modeling and transfer learning.
Approach:	They will discuss how and why NLG models succeed/fail at generating coherent text.
Outcome:	This paper will discuss how and why these models succeed/fail at generating coherent text, and provide insights on several applications.

Copied to clipboard

Leo Horne, Matthias Matti, Pouya Pourjafar, Zuowen Wang

Neural Token Representations and Negation and Speculation Scope Detection in Biomedical and General Domain Text (D19-62)

Challenge:	GRUBERT learns to map the different BERT hidden layers to fused embeddings . aims to achieve high accuracy on Twitter sentiment analysis task .
Approach:	They propose a GRU-based architecture that learns to map BERT hidden layers to fused embeddings to capture tweets' full extent.
Outcome:	The proposed method outperforms well-known embeddings and heuristics on Twitter sentiment analysis.

Copied to clipboard

Elena Sergeeva, Henghui Zhu, Amir Tahmasebi, Peter Szolovits

NSIT@NLP4IF-2019: Propaganda Detection from News Articles using Transfer Learning (D19-50)

Challenge:	Existing evidence for improved performance on natural language tasks is unclear to what degree the learned token representations capture and encode highlevel morphological/syntactic knowledge about the usage of a given token in a sentence.
Approach:	They propose to use context-aware token representations to capture morphological/syntactic knowledge about the usage of a given word/token in a sentence.
Outcome:	The proposed representations capture and encode high-level morphological/syntactic knowledge about the usage of a given token in a sentence.

Copied to clipboard

Kartik Aggarwal, Anubhav Sadana

Reevaluating Argument Component Extraction in Low Resource Settings (D19-61)

Challenge:	In this paper, we describe our approach and system description for NLP4IF 2019 Workshop: Shared Task on Fine-Grained Propaganda Detection.
Approach:	They propose to use document Embeddings and LSTM to detect whether a sentence contains a propagandistic agenda.
Outcome:	The proposed approach ranked 21st in the NLP4IF 2019 Workshop: Shared Task on Fine-Grained Propaganda Detection.

Copied to clipboard

Anirudh Joshi, Timothy Baldwin, Richard Sinnott, Cecile Paris

Embedding Strategies for Specialized Domains: Application to Clinical Entity Recognition (P19-2)

Challenge:	Argument component extraction is a challenging and complex high-level semantic extraction task.
Approach:	They propose to use character-level, GloVe, ELMo, and BERT encodings to compare arguments extracted using standard BiLSTM-CRF encoders.
Outcome:	The proposed approaches perform better than baselines in higher-level semantic extraction tasks and suggest future improvements.

Copied to clipboard

Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Pierre Zweigenbaum

Classification and Clustering of Arguments with Contextualized Word Embeddings (P19-1)

Challenge:	Off-the-shelf word embeddings tend to perform poorly on texts from specialized domains such as clinical reports.
Approach:	They combine off-the-shelf contextual embeddings with static word2vec embedders trained on a small in-domain corpus built from task data to reach and sometimes outperform representations learned from a large corpus in the medical domain.
Outcome:	The proposed embedding strategies outperform representations learned from a large corpus in the medical domain.

Copied to clipboard

Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, Iryna Gurevych

Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations (D19-1)

Challenge:	Existing methods for argument mining focus on analyzing local argumentation structures, but information-seeking approaches need to be able to deal with heterogeneous sources and topics.
Approach:	They propose to use contextualized word embeddings to classify and cluster topic-dependent arguments using a UKP Sentential Argument Mining Corpus and IBM Debater - Evidence Sentences datasets.
Outcome:	The proposed method improves state-of-the-art on argument classification and clustering tasks and across multiple datasets.

Copied to clipboard

Mingda Chen, Zewei Chu, Kevin Gimpel

On Measuring Social Biases in Sentence Encoders (N19-1)

Challenge:	Prior work on pretrained sentence embeddings and benchmarks focused on the capabilities of stand-alone sentences.
Approach:	They propose a test suite of tasks to evaluate whether sentence representations include broader context information.
Outcome:	The proposed training objectives help to encode different aspects of information in document structures.

Copied to clipboard

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, Rachel Rudinger

Gender Bias in Contextualized Word Embeddings (N19-1)

Challenge:	Word embeddings such as word2vec and GloVe exhibit human-like implicit biases based on gender, race, and other social constructs.
Approach:	They propose a simple generaliza test to measure bias in word embeddings by comparing two sets of target-concept words to two sets .
Outcome:	The proposed test shows that word2vec and word2Ve exhibit human-like implicit biases based on gender, race, and other social constructs.

Copied to clipboard

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, Kai-Wei Chang

MTSI-BERT: A Session-aware Knowledge-based Conversational Agent (2020.lrec-1)

Challenge:	Existing studies show that training word embeddings in large corpora could lead to encoding societal biases present in these human-produced data.
Approach:	They conduct several intrinsic analyses to quantify, analyze and mitigate gender bias exhibited in ELMo’s contextualized word vectors.
Outcome:	The proposed method mitigates gender bias on WinoBias probing corpus and demonstrates that it can be implemented in other systems.

Copied to clipboard

Matteo Antonio Senese, Giuseppe Rizzo, Mauro Dragoni, Maurizio Morisio

Natural Language Interfaces

Challenge:	Several models have been published achieving promising results in all the major linguistic tasks.
Approach:	They propose to exploit a BERT-based model to handle multi-turn conversations . they propose to use PuffBot to monitor asthma patients .
Outcome:	The proposed model can handle multi-turn conversations, a type of conversations that differs from single-turn by the presence of multiple related interactions.

Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution (2020.coling-main)

Copied to clipboard

Nikolay Arefyev, Boris Sheludko, Alexander Podolskiy, Alexander Panchenko

Linguistic Knowledge and Transferability of Contextual Representations (N19-1)

Challenge:	Lexical substitution is a powerful technology used in various NLP applications . it generates plausible words that can replace a given word in a textual context .
Approach:	They propose to use a large-scale comparative study to compare lexical substitution methods . they compare existing and new methods using word sense induction datasets .
Outcome:	The proposed methods improve competitive results by incorporating information about the target word into the models.

Copied to clipboard

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, Noah A. Smith

Retrofitting Contextualized Word Embeddings with Paraphrases (D19-1)

Challenge:	Recent work has explored contextual word representations, which assign each word a vector that is a function of the entire input sequence.
Approach:	They compare pretrained word representations with 16 diverse probing tasks to examine their transferability.
Outcome:	The pretrained representations are successful across a diverse set of NLP tasks . the models are competitive with state-of-the-art models but fail on fine-grained tasks requiring fine-granular knowledge, the study finds .

Copied to clipboard

Weijia Shi, Muhao Chen, Pei Zhou, Kai-Wei Chang

Evaluating the Robustness of Neural Language Models to Input Perturbations (2021.emnlp-main)

Challenge:	Contextualized word embeddings can be useful for downstream applications, but they can be over-sensitive to contexts.
Approach:	They propose a method to retrofit contextualized word embeddings with paraphrases to minimize the variance of word representations on paraphrased contexts.
Outcome:	The proposed method improves on sentence classification and inference tasks.

Copied to clipboard

Milad Moradi, Matthias Samwald

Robustness in NLP

Challenge:	High-performance neural language models have achieved state-of-the-art results on a wide range of NLP tasks, but results for common benchmark datasets often do not reflect model reliability and robustness when applied to noisy, real-world data.
Approach:	They propose to implement character-level and word-level perturbation methods to simulate scenarios in which input texts may be slightly noisy or different from the data distribution on which NLP systems were trained.
Outcome:	The proposed methods simulate scenarios in which input texts may be slightly noisy or different from the data distribution on which NLP systems were trained.

Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER (D19-1)

Copied to clipboard

Phillip Keung, Yichao Lu, Vikas Bhardwaj

DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis (2020.findings-emnlp)

Challenge:	Contextual word embeddings have demonstrated state-of-the-art performance on various NLP tasks.
Approach:	They propose to use adversarial learning to improve upon multilingual BERT's zero-resource cross-lingual performance by aligning embeddings of English documents and their translations.
Outcome:	The multilingual version of BERT performs surprisingly well in cross-lingual settings, even when only labeled English data is used to finetune the model.

Copied to clipboard

Hu Xu, Bing Liu, Lei Shu, Philip Yu

Sentiment Analysis

Challenge:	Recent studies show that learning domain-specific language models are equally important for general-purpose and domain-based learning.
Approach:	They propose a domain-oriented learning task that combine the benefits of both general and domain-specific worlds.
Outcome:	The proposed task solves the problems in an aspect-based sentiment analysis task.

A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages (2020.acl-main)

Copied to clipboard

Pedro Javier Ortiz Suárez, Laurent Romary, Benoît Sagot

Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders (D19-1)

Challenge:	a recent trend in neural NLP has been the introduction of feature-based and fine-tuning methods . we train monolingual contextualized word embeddings for five mid-resource languages .
Approach:	They use common Crawl corpus to train monolingual contextualized word embeddings . they compare performance of OSCAR-based and Wikipedia-based embeddables on part-of-speech tasks .
Outcome:	The results show that OSCAR-based and Wikipedia-based embeddings perform better than Wikipedia-style embedders on part-of-speech tagging and parsing tasks.

Copied to clipboard

Andrew Drozdov, Patrick Verga, Yi-Pei Chen, Mohit Iyyer, Andrew McCallum

Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation (N19-1)

Challenge:	Existing models that use ground-truth part-of-speech tags are not always available and have significant weaknesses.
Approach:	They propose to use deep inside-outside recursive autoencoders to cluster the learned phrase vectors to induce span labels.
Outcome:	The proposed model outperforms ELMo and BERT on two versions of the Wall Street Journal dataset and improves over a previous state-of-the-art system that requires additional human annotations by 5 absolute F1 points (19% relative error reduction).

Copied to clipboard

Sebastian Gehrmann, Steven Layne, Franck Dernoncourt

Challenge:	a study of long documents that do not include short sections in their titles shows that they improve comprehension and speed .
Approach:	They propose to extract the most salient sentence and apply deletion-based compression to generate section titles in low-resource environments.
Outcome:	The proposed approach outperforms other methods in low-resource environments while outperforming other approaches.

Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021.eacl-main)

Copied to clipboard

Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, Pasquale Minervini

Evaluating Neural Model Robustness for Machine Comprehension (2021.eacl-main)

Challenge:	Existing benchmarks do not probe professional bias as pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice.
Approach:	They propose to use a skew and stereotype metrics to quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task.
Outcome:	The proposed methods reduce skew and stereotype relative to the unaugmented fine-tuned BERT model.

Copied to clipboard

Winston Wu, Dustin Arendt, Svitlana Volkova

Semi-Supervised Sequence Modeling with Cross-View Training (D18-1)

Challenge:	evaluating model robustness to adversarial attacks can provide deeper understanding of how deep neural networks work and what kind of linguistic information is actually captured by neural networks.
Approach:	They propose a method for strategic sentence-level perturbations to evaluate model robustness to adversarial attacks using character and word perturbations.
Outcome:	The proposed model improves model performance during adversarial attacks by using ensembles and predicts errors in adversarials.

Copied to clipboard

Kevin Clark, Minh-Thang Luong, Christopher D. Manning, Quoc Le

Low-Resource NLP

Challenge:	Unsupervised representation learning algorithms such as word2vec and ELMo only learn from task-specific labeled data during the main training phase.
Approach:	They propose a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data.
Outcome:	The proposed algorithm improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data.

LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation (2020.lrec-1)

Copied to clipboard

Gustavo Aguilar, Sudipta Kar, Thamar Solorio

Code-Switching

Challenge:	Recent trends in NLP research have raised an interest in linguistic code-switching . however, many of these approaches are limited to a few language pairs and a specific domain .
Approach:	They propose a centralized benchmark for Linguistic Code-switching Evaluation that combines eleven corpora covering four different code-switch languages and four tasks.
Outcome:	The proposed benchmark provides a centralized benchmark and compares with other benchmarks in real-time.

Improving Compositional Generalization in Semantic Parsing (2020.findings-emnlp)

Copied to clipboard

Inbar Oren, Jonathan Herzig, Nitish Gupta, Matt Gardner, Jonathan Berant

Adversarial Domain Adaptation for Machine Reading Comprehension (D19-1)

Challenge:	Generalization of models to out-of-distribution data has sparked substantial interest . compositional generalization is the ability to systematically generalize to test examples composed of components seen during training .
Approach:	They propose to extend compositional generalization in semantic parsing by using contextual representations and training attention to agree with pre-computed token alignments.
Outcome:	The proposed extensions improve compositional generalization on OOD compositions.

Copied to clipboard

Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Hongning Wang

Responsible & Trustworthy NLP

Challenge:	Existing models for machine reading comprehension rely on large amounts of human-annotated in-domain data.
Approach:	They propose an unsupervised domain adaptation framework for Machine Reading Comprehension where the source domain has a large amount of labeled data, while only unlabeled passages are available in the target domain.
Outcome:	The proposed framework can be generalizable to different MRC models and datasets and can be extended to semi-supervised learning.

Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension (N19-1)

Copied to clipboard

Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, Jianfeng Gao

Designing and Interpreting Probes with Control Tasks (D19-1)

Challenge:	Existing models for Machine Reading Comprehension (MRC) are small, compared to their size, and there are many studies on using pre-trained word embeddings and back-translation approaches to improve model generalization.
Approach:	They propose a multi-task learning framework to learn a machine reading comprehension model that can be applied to a wide range of MRC tasks in different domains.
Outcome:	The proposed model can be applied to a wide range of MRC tasks in different domains.

Copied to clipboard

John Hewitt, Percy Liang

Syntactic Text Processing

Challenge:	Existing studies on supervised models to predict properties from representations have shown high accuracy on a range of linguistic tasks.
Approach:	They propose control tasks which associate word types with random outputs to complement linguistic tasks by construction . they find that popular probes on ELMo representations are not selective .
Outcome:	The proposed tasks associate word types with random outputs to complement linguistic tasks.

Specializing Word Embeddings (for Parsing) by Information Bottleneck (D19-1)

Copied to clipboard

Xiang Lisa Li, Jason Eisner

Probing for idiomaticity in vector space models (2021.eacl-main)

Challenge:	Pre-trained word embeddings contain rich syntactic and semantic information.
Approach:	They propose a variational information bottleneck method to nonlinearly compress word embeddings . they compress each word embedded to either a discrete tag or a continuous vector .
Outcome:	The proposed method can be extended to other syntactic or semantic tasks.

Copied to clipboard

Marcos Garcia, Tiago Kramer Vieira, Carolina Scarton, Marco Idiart, Aline Villavicencio

On the Robustness of Language Encoders against Grammatical Errors (2020.acl-main)

Challenge:	Contextualised word representation models are used to represent idiomaticity in language.
Approach:	They propose probing measures to assess if some of the expected linguistic properties of noun compounds are readily available in some standard and widely used representations.
Outcome:	The proposed models show that idiomaticity is not yet accurately represented by contextualised models.

Copied to clipboard

Fan Yin, Quanyu Long, Tao Meng, Kai-Wei Chang

Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks (P19-1)

Challenge:	Pre-trained language encoders are effective in facilitating downstream natural language processing tasks, but they often assume training and test corpora are clean and it is unclear how the models behave when confronted with noisy input.
Approach:	They conduct adversarial attacks to simulate grammatical errors on clean text data.
Outcome:	The proposed model performs better when confronted with natural grammatical errors than when faced with noisy input.

Copied to clipboard

Shikhar Vashishth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, Partha Talukdar

Multilingual Constituency Parsing with Self-Attention and Pre-Training (P19-1)

Challenge:	Existing word embedding methods utilize sequential context of a word to learn its embeddment, but such methods result in an explosion of the vocabulary size.
Approach:	They propose a flexible Graph Convolution based method for learning word embeddings that utilizes the dependency context of a word without increasing the vocabulary size.
Outcome:	The proposed model outperforms existing methods on intrinsic and extrinsic tasks and provides an advantage when used with ELMo.

Copied to clipboard

Nikita Kitaev, Steven Cao, Dan Klein

Improving Disfluency Detection by Self-Training a Self-Attentive Model (2020.acl-main)

Challenge:	a range of pre-training conditions can be used for constituency parsing, but large model sizes make it expensive to train separate models for each language.
Approach:	They compare the benefits of no pre-training, fastText, ELMo, and BERT for English . they also find that pre- training is beneficial across all 11 languages tested .
Outcome:	The proposed model outperforms fastText, ELMo, and BERT for English . but large model sizes make it expensive to train separate models for each language .

Copied to clipboard

Paria Jamshid Lou, Mark Johnson

Facilitating Corpus Usage: Making Icelandic Corpora More Accessible for Researchers and Language Users (2020.lrec-1)

Challenge:	Existing self-attentive parsers using contextualized word embeddings produce state-of-the-art results in joint parsing and disfluency detection.
Approach:	They propose to use contextualized word embeddings to train a neural model using unlabeled data to train parsers.
Outcome:	The proposed method produces state-of-the-art results in parsing and disfluency detection in speech transcripts.

Copied to clipboard

Steinþór Steingrímsson, Starkaður Barkarson, Gunnar Thor Örnólfsson

A Structural Probe for Finding Syntax in Word Representations (N19-1)

Challenge:	Gigaword corpus is a large text corpus used in natural language processing . large corpora are needed to achieve better performance in the field of NLP .
Approach:	They propose a set of tools to facilitate the use of the Icelandic Gigaword Corpus . they provide n-grams based on the corpus, and a variety of pre-trained word embeddings models .
Outcome:	The proposed tools facilitate the use of the Icelandic Gigaword corpus in the field of Natural Language Processing and other fields.

Copied to clipboard

John Hewitt, Christopher D. Manning

Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings (2020.acl-main)

Challenge:	Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space.
Approach:	They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space.
Outcome:	The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry.

Copied to clipboard

Rishi Bommasani, Kelly Davis, Claire Cardie

Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (D19-1)

Challenge:	Contextualized representations have become the default for downstream NLP applications.
Approach:	They propose a method for converting from contextualized representations to static lookup-table embeddings and apply it to 5 popular pretrained models and 9 sets of pretrained weights.
Outcome:	The proposed methods show that pooling over many contexts significantly improves representational quality under intrinsic evaluation.

Copied to clipboard

Xiaochuang Han, Jacob Eisenstein

Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020.acl-main)

Challenge:	Contextualized word embeddings are becoming a ubiquitous component of natural language processing.
Approach:	They propose a domain-adaptive fine-tuning approach to pretrain on unlabeled text . they test this approach on sequence labeling in two challenging domains .
Outcome:	The proposed approach improves on sequence labeling in two domains: Early Modern English and Twitter.

Copied to clipboard

Josef Klafka, Allyson Ettinger

Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events (2020.emnlp-main)

Challenge:	a suite of probing tasks test contextual embeddings for encoding of information about surrounding words . authors: little is known about what information embeddables encode about the context words encode . a recent study shows that contextual embeds can be powerful for many tasks .
Approach:	They propose probing tasks that enable fine-grained testing of contextual embeddings . they examine popular contextual encoders and find that each encodes contextual information across tokens a little different .
Outcome:	The proposed probing tasks show that word embeddings encode information about words . the tests show that the encoded information is encoded across tokens with near-perfect recoverability .

Copied to clipboard

Miguel Ballesteros, Rishita Anubhai, Shuai Wang, Nima Pourdamghani, Yogarshi Vyas, Jie Ma, Parminder Bhatia, Kathleen McKeown, Yaser Al-Onaizan

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (P19-1)

Challenge:	Existing models for temporal ordering of events rely on pretrained representations, transfer and multitask learning, and self-training techniques.
Approach:	They propose a neural architecture and a set of training methods for ordering events by predicting temporal relations by pre-training models.
Outcome:	The proposed models can predict temporal relations between two pairs of events within a span of text and identify temporal relationships between them.

Copied to clipboard

Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman

Classifier Probes May Just Learn from Linear Context Features (2020.coling-main)

Challenge:	State-of-the-art models in natural language processing (NLP) often incorporate sentence encoder functions which generate a sequence of vectors intended to represent the in-context meaning of each word in an input text.
Approach:	They conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks as alternatives and complements to language modeling.
Outcome:	The proposed model can be used to train sentences on language modeling tasks.

Copied to clipboard

Jenny Kunz, Marco Kuhlmann

On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)

Challenge:	Current probing methods can help to better estimate the complexity of learning, but not build a foundation for speculations about the nature of the linguistic structure encoded in the learned representations.
Approach:	They propose to use token embeddings to test whether probing tasks contain linguistic structure . they argue that current probing methods do not provide enough information to support this hypothesis .
Outcome:	The proposed method can be scrutinized and proves that representations encode linguistic structure even without additional linguistic structures.

Copied to clipboard

Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro

Towards Debiasing Sentence Representations (2020.acl-main)

Challenge:	Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work .
Approach:	They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data .
Outcome:	The proposed models can achieve competitive or better performance than BERT under comparable conditions.

Copied to clipboard

Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, Louis-Philippe Morency

Responsible & Trustworthy NLP

Challenge:	Recent work has shown word-level embeddings reflect and propagate social biases present in training corpora.
Approach:	They propose a method to debias word embeddings to reduce biases at sentence level . they hope their work will inspire future research on characterizing and removing biase .
Outcome:	The proposed method reduces biases and preserves performance on downstream tasks such as sentiment analysis and natural language understanding.

Revisiting the Importance of Encoding Logic Rules in Sentiment Classification (D18-1)

Copied to clipboard

Kalpesh Krishna, Preethi Jyothi, Mohit Iyyer

Information Retrieval

Challenge:	Neural models that explicitly encode word order, syntax and semantic features are unequipped to deal with complex syntactic structures that affect sentiment, such as contrastive conjunctions.
Approach:	They propose to use contextualized ELMo embeddings instead of logic rules to compare models on syntactically complex sentences.
Outcome:	The proposed models outperform baseline models even on sentences with ambiguous sentiment labels.

Relating Simple Sentence Representations in Deep Neural Networks and the Brain (P19-1)

Copied to clipboard

Sharmistha Jat, Hao Tang, Partha Talukdar, Tom Mitchell

Neural Architectures for Nested NER through Linearization (P19-1)

Challenge:	Existing deep learning models for natural language processing are not fully studied.
Approach:	They investigate whether deep recurrent models learn sentences against those encoded by the brain and whether there is any correspondence between hidden layers of these models and brain regions when processing sentences.
Outcome:	The proposed models can be used to synthesize brain data and improve subsequent stimuli decoding accuracy.

Copied to clipboard

Jana Straková, Milan Straka, Jan Hajic

Named Entity Recognition

Challenge:	a nested named entity recognition (NER) is a set of entities that can overlap and be labeled with more than one label.
Approach:	They propose two neural network architectures for nested named entity recognition . they propose to model nesting entities as multilabels and predict a sequence-to-sequence problem .
Outcome:	The proposed methods outperform the state-of-the-art on four corpora . the proposed models also improve on the recently published contextual embeddings .

Towards Semi-Supervised Learning for Deep Semantic Role Labeling (D18-1)

Copied to clipboard

Sanket Vaibhav Mehta, Jay Yoon Lee, Jaime Carbonell

Linguistically-Informed Self-Attention for Semantic Role Labeling (D18-1)

Challenge:	Existing methods for semantic role labeling require an immense amount of semantic-role corpora and are therefore not suitable for low-resource languages or domains.
Approach:	They propose a semi-supervised method that outperforms the state-of-the-art on SRL . method explicitly enforcs syntactic constraints by augmenting the training objective with a syntastic-inconsistency loss component.
Outcome:	The proposed method outperforms the state-of-the-art on limited SRL training corpora on CoNLL-2012 English section.

Copied to clipboard

Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, Andrew McCallum

Sean Trott, Benjamin Bergen

Challenge:	Existing models of semantic role labeling use no explicit linguistic features. prior work has shown that syntax trees can dramatically improve SRL decoding.
Approach:	They propose a neural network model that incorporates syntax using only raw tokens . they show that LISA out-performs the state-of-the-art with contextually-encoded word representations a 1.0 F1 on newswire and 2.0 F1 in out-of domain text .
Outcome:	The proposed model outperforms the state-of-the-art model with word embeddings and predicted predicates.

Copied to clipboard

Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020.emnlp-main)

Challenge:	lexical ambiguity is a problem for NLP, but few tasks evaluate its impact on human intuitions.
Approach:	They propose to use contextualized word embeddings to evaluate word meaning . they use a dataset of human relatedness judgments and human estimates of sense dominance .
Outcome:	The proposed model matches human intuitions with contextualized embeddings on 112 ambiguous words in context with 672 sentence pairs.

Copied to clipboard

Julian Michael, Jan A. Botha, Ian Tenney

Is Language Modeling Enough? Evaluating Effective Embedding Combinations (2020.lrec-1)

Challenge:	Recent work on model analysis indicates that they may learn a lot about linguistic structure, including part of speech, syntax, word sense, and more.
Approach:	They introduce latent subclass learning, a modification to classifier-based probing that induces a latent categorization (or ontology) of the probe’s inputs.
Outcome:	The proposed model induces a latent categorization (or ontology) of the probe’s inputs without access to fine-grained gold labels.

Copied to clipboard

Rudolf Schneider, Tom Oberhauser, Paul Grundmann, Felix Alexander Gers, Alexander Loeser, Steffen Staab

BERT for Coreference Resolution: Baselines and Analysis (D19-1)

Challenge:	specialized embeddings are not available for tasks like entity linking or paragraph classification.
Approach:	They evaluate whether universal embeddings can be complemented by specialized embeddables.
Outcome:	The proposed embeddings outperform state-of-the-art embeddables without any fine-tuning.

Copied to clipboard

Mandar Joshi, Omer Levy, Luke Zettlemoyer, Daniel Weld

What Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition (D19-1)

Challenge:	Recent BERT-based models have reported dramatic gains on multiple semantic benchmarks including question-answering, natural language inference, and named entity recognition.
Approach:	They apply BERT to coreference resolution, achieving a new state of the art on the GAP and OntoNotes benchmarks.
Outcome:	A qualitative analysis of model predictions shows that BERT-large is better at distinguishing between related but distinct entities, but there is room for improvement in modeling document-level context, conversations, and mention paraphrasing.

Copied to clipboard

Ting-Yun Chang, Yun-Nung Chen

AIA-BDE: A Corpus of FAQs in Portuguese and their Variations (2020.lrec-1)

Challenge:	Contextualized word embeddings have boosted many NLP tasks compared with static word embeds.
Approach:	They propose a framework that can explain word meanings given contextualized word embeddings for better interpretation.
Outcome:	The proposed framework can explain word meanings given contextualized word embeddings for better interpretation.

Copied to clipboard

Hugo Gonçalo Oliveira, João Ferreira, José Santos, Pedro Fialho, Ricardo Rodrigues, Luisa Coheur, Ana Alves

Information Retrieval

Challenge:	a corpus of 380 domain-oriented FAQs in Portuguese is presented . paraphrases or entailed questions are created manually, by humans, or automatically, with Google Translate.
Approach:	They present a corpus of 380 domain-oriented FAQs in Portuguese and their variations, i.e., paraphrases or entailed questions, created manually, by humans, or automatically, with Google Translate.
Outcome:	The proposed system outperforms other systems in the domain of question-answering . it performs well when matching variations with their original questions .

Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task (2020.lrec-1)

Copied to clipboard

Md Tahmid Rahman Laskar, Jimmy Xiangji Huang, Enamul Hoque

From English to Code-Switching: Transfer Learning with Strong Morphological Clues (2020.acl-main)

Challenge:	Word embeddings that consider context have attracted great attention for natural language processing tasks in recent years.
Approach:	They propose two different approaches to integrate contextualized word embeddings with transformer encoders for sentence similarity modeling.
Outcome:	The proposed model outperforms the feature-based approach on six datasets.

Copied to clipboard

Gustavo Aguilar, Thamar Solorio

Syntactic Text Processing

Challenge:	Linguistic code-switching (CS) is an understudied area in natural language processing . lack of resources and annotated data makes it difficult to strive for progress in CS-related tasks.
Approach:	They propose a method to adapt monolingual models to code-switched text in various tasks . they transfer English knowledge from a pre-trained ELMo model to different code-paired languages .
Outcome:	The proposed method outperforms multilingual BERT and homologous CS-unaware models and provides state-of-the-art in CS tasks.

Improving Neural Metaphor Detection with Visual Datasets (2020.lrec-1)

Copied to clipboard

Gitit Kehat, James Pustejovsky

Visual Data in NLP

Challenge:	a new method for metaphor detection uses text from visual datasets to identify words . a metaphor is a complex interaction between two terms, creating an "implicationcomplex"
Approach:	They propose a technique for sampling text from visual datasets to create a visibility word embedding.
Outcome:	The proposed method improves on previous approaches that use more complex neural networks and richer linguistic features for verb classification.

One Classifier for All Ambiguous Words: Overcoming Data Sparsity by Utilizing Sense Correlations Across Words (2020.lrec-1)

Copied to clipboard

Prafulla Kumar Choubey, Ruihong Huang

Clinical Concept Linking with Contextualized Neural Representations (2020.acl-main)

Challenge:	Existing word-specific classifiers lack the ability to generalize across words and require limited sense-annotated data for every word.
Approach:	They propose to learn a single model that derives sense representations and enforces congruence between a word instance and its right sense by using both sense-annotated data and lexical resources.
Outcome:	Empirical evaluation shows the proposed model outperforms classifier-based models by 1.7%, 2.5% and 3.8% in F1-score on GloVe, ELMo and BERT word embeddings respectively.

Copied to clipboard

Elliot Schumacher, Andriy Mulyar, Mark Dredze

Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021.emnlp-main)

Challenge:	Entity linking systems rely on three sources of information: 1) similarity between mention string and entity name; 2) similarity of context of document to entity; 3) broader information about knowledge base; 4) contextual information; 5) semantic information; and 6) semantic information.
Approach:	They propose an approach to linking medical concepts to a medical concept ontology that leverages recent work in contextualized neural models.
Outcome:	The proposed approach outperforms a baseline approach and provides better initialization for the ranker.

Copied to clipboard

Pierre Colombo, Guillaume Staerman, Chloé Clavel, Pablo Piantanida