Transactions of the Association for Computational Linguistics, Volume 8

What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)

Challenge:	Abstract meaning representation (AMR)-to-text generation is challenging task for natural language processing.
Approach:	They propose a graph-to-sequence model that directly encodes AMR graphs and learns node representations.
Outcome:	The proposed model outperforms the current state-of-the-art neural approach by 1.5 BLEU points on LDC2015E86 and 4.8 BLUE points on the LDC2017T10 and achieves new state- of-the art performance.

Copied to clipboard

Allyson Ettinger

Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? (2020.tacl-1)

Challenge:	Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process.
Approach:	They propose diagnostics that ask questions about information used by language models for generating predictions in context.
Outcome:	The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction.

Copied to clipboard

Sorami Hisamoto, Matt Post, Kevin Duh

Robustness in NLP

Challenge:	Data privacy is an important issue for “machine learning as a service” providers.
Approach:	They propose an attack on membership inference attacks using a sequence-to-sequence model and a machine translation dataset to investigate the feasibility of a privacy attack.
Outcome:	The proposed model can infer sentence-level membership from the output of the model, but it is difficult to infer it.

SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)

Copied to clipboard

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy

A Graph-based Model for Joint Chinese Word Segmentation and Dependency Parsing (2020.tacl-1)

Challenge:	Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text.
Approach:	They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span.
Outcome:	The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks.

Copied to clipboard

Hang Yan, Xipeng Qiu, Xuanjing Huang

A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation (2020.tacl-1)

Challenge:	Chinese word segmentation and dependency parsing suffer from error propagation . a graph-based model can integrate both tasks, but it suffers from performance limitations .
Approach:	They propose a graph-based model to integrate Chinese word segmentation and dependency parsing . their model achieves better performance than previous joint models .
Outcome:	The proposed model achieves better performance than previous joint models and state-of-the-art results in both Chinese word segmentation and dependency parsing.

Copied to clipboard

Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, Minlie Huang

Improving Candidate Generation for Low-resource Cross-lingual Entity Linking (2020.tacl-1)

Challenge:	Existing models for story generation suffer from repetition, logic conflicts, and lack of long-range coherence .
Approach:	They propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories by multi-task learning.
Outcome:	The proposed model can generate more reasonable stories than state-of-the-art models, compared with existing models, showing that it can capture useful semantic and syntactic features.

Copied to clipboard

Shuyan Zhou, Shruti Rijhwani, John Wieting, Jaime Carbonell, Graham Neubig

Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks (2020.tacl-1)

Challenge:	Existing approaches to cross-lingual entity linking (XEL) do not extend well to low-resource languages with few Wikipedia pages.
Approach:	They propose to improve the model by combining Wikipedia references with a list of plausible candidate entities.
Outcome:	The proposed method yields 16.9% in Top-30 gold candidate recall compared with state-of-the-art models.

Copied to clipboard

R. Thomas McCoy, Robert Frank, Tal Linzen

Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension (2020.tacl-1)

Challenge:	Inductive biases can arise from any aspect of the model architecture, study finds . we investigate which architectural factors affect how models generalize .
Approach:	They investigate which architectural factors affect generalization behavior of neural network models . they use English question formation and English tense reinflection as test cases .
Outcome:	The findings suggest that human-like generalization requires architectural syntactic structure.

Copied to clipboard

Kai Sun, Dian Yu, Dong Yu, Claire Cardie

Reasoning

Challenge:	''Language is, at best, a means of directing others to construct similar-thoughts from their own prior knowledge,'' says K. S. Adams and Bruce.
Approach:	They present a free-form multiple-choice Chinese machine reading Comprehension dataset (C3) containing 13,369 documents and their associated 19,577 multiple-CHOice free- form questions.
Outcome:	The proposed model outperforms human models on linguistic, domain-specific, and general world knowledge problems.

Theoretical Limitations of Self-Attention in Neural Sequence Models (2020.tacl-1)

Copied to clipboard

Michael Hahn

Target-Guided Structured Attention Network for Target-Dependent Sentiment Analysis (2020.tacl-1)

Challenge:	Existing work suggests that the computational capabilities of self-attention to model hierarchical structures are limited.
Approach:	They investigate the computational power of self-attention to model formal languages . they show strong theoretical limitations of self attention to model periodic finite-state languages unless the number of layers or heads increases with input length.
Outcome:	The proposed models can model periodic finite-state languages, nor hierarchical structure unless the number of layers or heads increases with input length.

Copied to clipboard

Ji Zhang, Chengyao Chen, Pengfei Liu, Chao He, Cane Wing-Ki Leung

Sentiment Analysis

Challenge:	TDSA aims to classify the sentiment of a text towards a given target.
Approach:	They propose a novel Target-Guided Structured Attention Network (TG-SAN) which captures target-related contexts for TDSA in a fine-to-coarse manner.
Outcome:	The proposed network outperforms the state-of-the-art in terms of accuracy and Marco-F1 on three benchmarks with three major findings.

Break It Down: A Question Understanding Benchmark (2020.tacl-1)

Copied to clipboard

Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant

Question Answering

Challenge:	Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer.
Approach:	They introduce a Question Decomposition Meaning Representation (QDMR) for questions . they demonstrate that QDMRs can be annotated at scale using a hotpotQA dataset .
Outcome:	The proposed model outperforms several natural baselines in the open-domain question answering hotpotQA dataset and can be deterministically converted to a pseudo-SQL formal language.

Acoustic-Prosodic and Lexical Cues to Deception and Trust: Deciphering How People Detect Lies (2020.tacl-1)

Copied to clipboard

Xi (Leslie) Chen, Sarah Ita Levitan, Michelle Levine, Marko Mandic, Julia Hirschberg

Challenge:	LieCatcher collects ratings of perceived deception using corpus of deceptive and truthful interviews . acoustic-prosodic and linguistic characteristics of language trusted and mistrusted are not reliable cues .
Approach:	They used a game framework to collect ratings of perceived deception using deceptive and truthful interviews to understand how perception aligns with reality.
Outcome:	The proposed framework detects deception using a corpus of deceptive and truthful interviews.

Unsupervised Discourse Constituency Parsing Using Viterbi EM (2020.tacl-1)

Copied to clipboard

Noriki Nishida, Hideki Nakayama

Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models (2020.tacl-1)

Challenge:	Existing studies on unsupervised discourse parsing have shown that it is expensive, time-consuming, and sometimes highly ambiguous.
Approach:	They propose an unsupervised parsing algorithm using Viterbi EM with a margin-based criterion and initialization methods for Viterbia training of discourse constituents based on prior knowledge of text structures.
Outcome:	The proposed method outperforms fully supervised parsers in terms of performance and learning of discourse constituents.

Copied to clipboard

Vesna G. Djokic, Jean Maillard, Luana Bulat, Ekaterina Shutova

Representation Learning

Challenge:	Existing research has focused on applying semantic models to decode brain activity associated with the meaning of individual words.
Approach:	They evaluate a range of semantic models to capture metaphor processing in the brain . they found that compositional models and word embeddings capture differences in the processing of literal and metaphoric sentences .
Outcome:	The proposed models capture differences in the processing of literal and metaphoric sentences, providing support for the idea that the literal meaning is not fully accessible during familiar metaphor comprehension.

Machine Learning–Driven Language Assessment (2020.tacl-1)

Copied to clipboard

Burr Settles, Geoffrey T. LaFlair, Masato Hagiwara

Challenge:	Language proficiency tests are cumbersome to create and maintain, and items may be copied and leaked or simply used too often.
Approach:	They propose a method that uses machine learning and natural language processing to induce proficiency scales and linguistic models to estimate item difficulty directly for computer-adaptive testing.
Outcome:	The proposed method produces scores that are reliable and reliable while generating item banks large enough to satisfy security requirements.

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks (2020.tacl-1)

Copied to clipboard

Sascha Rothe, Shashi Narayan, Aliaksei Severyn

CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset (2020.tacl-1)

Challenge:	Unsupervised pre-training of large neural models has revolutionized Natural Language Processing.
Approach:	They propose to use pre-trained checkpoints for Sequence Generation to initialize a Transformer-based sequence-to-sequence model that is compatible with these checkpoint.
Outcome:	The proposed model is compatible with pre-trained BERT, GPT-2, and RoBERTa checkpoints and achieves state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentance Fusion.

Copied to clipboard

Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu, Minlie Huang

Natural Language Interfaces

Challenge:	Despite the significant contributions to the community, there is still a gap between existing dialogue corpora and real-life human dialogue data.
Approach:	They propose to develop Chinese cross-domain wizard-of-oz task-oriented dataset CrossWOZ with rich annotations of dialogue states and dialogue acts on both user and system sides.
Outcome:	The proposed dataset contains 6K dialogue sessions and 102K utterances for 5 domains, including hotel, restaurant, attraction, metro, and taxi.

How Furiously Can Colorless Green Ideas Sleep? Sentence Acceptability in Context (2020.tacl-1)

Copied to clipboard

Jey Han Lau, Carlos Armendariz, Shalom Lappin, Matthew Purver, Chang Shu

Challenge:	a recent study shows that context affects our perception of sentence acceptability, but few studies investigate how it affects language models.
Approach:	They compare acceptability ratings of sentences judged in isolation with a relevant context and with an irrelevant context.
Outcome:	The proposed model achieves state-of-the-art for unsupervised acceptability prediction.

Learning Lexical Subspaces in a Distributional Vector Space (2020.tacl-1)

Copied to clipboard

Kushal Arora, Aishik Chakraborty, Jackie C. K. Cheung

Representation Learning

Challenge:	Existing word embeddings that can cluster distributionally related words are weak, but they can be used to cluster words that might not be semantically similar.
Approach:	They propose a framework that injects lexical-semantic relations into distributional word embeddings by defining subspaces of the distributional vector space in which a lexically related relation should hold.
Outcome:	The proposed framework outperforms existing systems on relatedness and hypernymy tasks while being competitive on word similarity tasks.

Syntax-Guided Controlled Generation of Paraphrases (2020.tacl-1)

Copied to clipboard

Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, Partha Talukdar

Better Document-Level Machine Translation with Bayes’ Rule (2020.tacl-1)

Challenge:	Recent work has explored the incorporation of complex syntactic-guidance as constraints in the task of controlled text generation.
Approach:	They propose an end-to-end framework for controlled paraphrase generation that incorporates complex syntactic-guidance constraints into the task.
Outcome:	The proposed framework generates syntax-conforming sentences while not compromising on relevance.

Copied to clipboard

Lei Yu, Laurent Sartran, Wojciech Stokowiec, Wang Ling, Lingpeng Kong, Phil Blunsom, Chris Dyer

Hierarchical Mapping for Crosslingual Word Embedding Alignment (2020.tacl-1)

Challenge:	Existing document translation models are based on autoregressive language models, but they are not able to be learned from monolingual documents.
Approach:	They propose to use Bayes' rule to create document translation models that can be learned from only parallel sentences and monolingual documents.
Outcome:	The proposed model outperforms existing document translation approaches and is based on a novel left-to-right beam-search algorithm.

Copied to clipboard

Ion Madrazo Azpiazu, Maria Soledad Pera

Representation Learning

Challenge:	Existing strategies that map word embeddings into a crosslingual space are biased towards the choice of the pivot language.
Approach:	They propose to map any two languages into a different middle space by learning mappings across languages in a hierarchical way.
Outcome:	The proposed strategy significantly improves vocabulary induction scores in all existing benchmarks and in a new non-English–centered benchmark.

BLiMP: The Benchmark of Linguistic Minimal Pairs for English (2020.tacl-1)

Copied to clipboard

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, Samuel R. Bowman

Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems (2020.tacl-1)

Challenge:	Recent studies have examined how linguistic knowledge of language models (LMs) varies across English phenomena.
Approach:	They propose a benchmark to evaluate linguistic knowledge of language models on major grammatical phenomena in English.
Outcome:	The proposed benchmark evaluates the linguistic knowledge of language models on major grammatical phenomena in English.

Copied to clipboard

Xuan Zhang, Kevin Duh

Consistent Unsupervised Estimators for Anchored PCFGs (2020.tacl-1)

Challenge:	Optimal versus suboptimal hyperparameters can lead to dramatic swings in system performance.
Approach:	They propose to use a library of pre-trained models for fast, low cost HPO experimentation and to propose metrics for evaluating HPO methods on NMT.
Outcome:	The proposed method uses a library of pre-trained models for fast, low cost experimentation on neural machine translation (NMT) .

Copied to clipboard

Alexander Clark, Nathanaël Fijalkow

How Can We Know What Language Models Know? (2020.tacl-1)

Challenge:	a novel approach for learning probabilistic context-free grammars from strings is proposed . strong learning means that there can be many structurally different PCFGs that define the same distribution over strings.
Approach:	They propose an algorithm that is a consistent estimator for a class of PCFGs that are anchored . they show that if the grammar is anchored, the parameters can be directly related to distributional properties of the anchoring strings.
Outcome:	The proposed algorithm is consistent for a large class of probabilistic context-free grammars . it shows that the proposed algorithm has good finite sample behavior .

Copied to clipboard

Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig

Topic Modeling in Embedding Spaces (2020.tacl-1)

Challenge:	Recent work examines knowledge contained in language models by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”.
Approach:	They propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts.
Outcome:	The proposed methods improve accuracy from 31.1% to 39.6% on the LAMA benchmark for extracting relational knowledge from LMs.

Copied to clipboard

Adji B. Dieng, Francisco J. R. Ruiz, David M. Blei

Explainability & Interpretability in NLP

Challenge:	Existing topic models fail to learn interpretable topics when working with large and heavy-tailed vocabularies.
Approach:	They propose an embedded topic model that integrates word embeddings with a categorical distribution that is the natural parameter between the word’s embeddment and an embeddement of its assigned topic.
Outcome:	The embedded topic model outperforms existing topic models in terms of topic quality and predictive performance.

TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020.tacl-1)

Copied to clipboard

Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, Jennimaria Palomaki

Natural Language Interfaces

Challenge:	Existing models for multilingual modeling are based on a set of typological features that are used to express meaning in languages such as English.
Approach:	They present a question-answer-typed question-referenced dataset that covers 11 typologically diverse languages with 204K question-and-answered pairs.
Outcome:	The proposed dataset covers 11 typologically diverse languages with 204K question-answer pairs.

A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings (2020.tacl-1)

Copied to clipboard

Lixing Zhu, Yulan He, Deyu Zhou

Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings (2020.tacl-1)

Challenge:	Experimental results show that the proposed model outperforms word-level embedding methods in word similarity evaluation and word sense disambiguation.
Approach:	They propose a generative model that explores local and global context for joint learning topics and topic-specific word embeddings.
Outcome:	The proposed model outperforms word-level embedding methods in word similarity evaluation and word sense disambiguation.

Copied to clipboard

Vaibhav Kumar, Tenzin Singhay Bhotia, Vaibhav Kumar, Tanmoy Chakraborty

PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models (2020.tacl-1)

Challenge:	Existing methods for debiasing word embeddings lack gender-based debiases . Existing approaches only reduce gender-related proximity biases by at least 42.02% .
Approach:	They propose a gender debiasing methodology that eliminates bias in word vectors and alters spatial distribution of neighboring vectors, achieving a bias-free setting while maintaining minimal semantic offset.
Outcome:	The proposed method outperforms the state-of-the-art in reducing proximity bias by at least 42.02% and reduces direct bias, adding minimal semantic disturbance, and achieves the best performance in a downstream application task.

Copied to clipboard

Eyal Ben-David, Carmel Rabinovitz, Roi Reichart

Juri Opitz, Letitia Parcalabescu, Anette Frank

Challenge:	PERL is a representation learning model that uses labeled data from the source domain and unlabeled data not necessarily drawn from the target domain.
Approach:	They propose a model that extends contextualized word embedding models with pivot-based fine-tuning to address this bottleneck.
Outcome:	The proposed model outperforms strong baselines across 22 sentiment classification domain adaptation setups and improves in-domain model performance.

Copied to clipboard

Unsupervised Quality Estimation for Neural Machine Translation (2020.tacl-1)

Challenge:	Abstract Meaning Representation (AMR) graphs are rooted, acyclic, directed, and edge-labeled.
Approach:	They propose a canonical Smatch metric that aligns variables of two graphs and assesses triple matches.
Outcome:	The proposed metric is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria.

Copied to clipboard

Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia

Task-Oriented Dialogue as Dataflow Synthesis (2020.tacl-1)

Challenge:	Existing approaches require large amounts of expert annotated data, computation, and time for training.
Approach:	They propose an unsupervised approach to QE where no training is required . they use a dataset that enables work on both black-box and glass-box approaches .
Outcome:	The proposed approach rivals state-of-the-art supervised QE models in terms of correlation with human judgments of quality.

Copied to clipboard

Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, Alexander Zotov

Multimodality

Challenge:	Existing approaches to task-oriented dialogue represent dialogue state as a dataflow graph . microsoft's SMCalFlow dataset features complex dialogues about events, weather, places, and people .
Approach:	They propose a dataflow graph-based dialogue agent that maps each user utterance to a program that extends this graph.
Outcome:	The proposed framework improves representability and predictability in natural dialogues . it uses dataflow graphs and metacomputation to map user intents to a program .

What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge (2020.tacl-1)

Copied to clipboard

Kyle Richardson, Ashish Sabharwal

Question Answering

Challenge:	Existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept.
Approach:	They propose a method for automatically building probe datasets from expert knowledge sources, allowing systematic control and a comprehensive evaluation.
Outcome:	The proposed model is predisposed to recognize certain types of structural linguistic knowledge, but performance degrades even with a slight increase in the number of “hops” in the underlying taxonomic hierarchy.

Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020.tacl-1)

Copied to clipboard

Leonardo F. R. Ribeiro, Yue Zhang, Claire Gardent, Iryna Gurevych

Structured Data in NLP

Challenge:	Recent graph-to-text models generate text from graph data using global or local aggregation . global node encoding allows explicit communication between two distant nodes, but fails to capture long-range relationships.
Approach:	They propose to combine global and local aggregation to learn node representations . they propose to use global and locally encoding to learn contextualized node embeddings based on graph data .
Outcome:	The proposed models outperform state-of-the-art models on two graph-to-text datasets by 18.01 and 63.69 points.

Nested Named Entity Recognition via Second-best Sequence Learning and Decoding (2020.tacl-1)

Copied to clipboard

Takashi Shibuya, Eduard Hovy

Information Extraction & Text Mining

Challenge:	Named entity recognition (NER) is the task of identifying text spans associated with proper names and classifying them according to their semantic class such as person or organization.
Approach:	They propose a method that treats the tag sequence for nested entities as the second best path within the span of their parent entity.
Outcome:	The proposed method achieves F1-scores of 85.82%, 84.34%, and 77.36% on ACE-2004, ACE 2005, and GENIA datasets.

An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020.tacl-1)

Copied to clipboard

Lifu Tu, Garima Lalwani, Spandana Gella, He He

Data Weighted Training Strategies for Grammatical Error Correction (2020.tacl-1)

Challenge:	Recent work shows that pre-trained language models perform poorly on challenging datasets where spurious correlations do not hold.
Approach:	They propose to use multi-task learning to improve generalization from minority examples . they propose to combine MTL with auxiliary tasks to improve performance .
Outcome:	The proposed model generalizes from minority examples without hurting in-distribution performance.

Copied to clipboard

Jared Lichtarge, Chris Alberti, Shankar Kumar

The Return of Lexical Dependencies: Neural Lexicalized PCFGs (2020.tacl-1)

Challenge:	Recent advances in the task of Grammatical Error Correction (GEC) have been driven by addressing data sparsity, both through new methods for generating large and noisy pretraining data and through the publication of small and higher-quality finetuning data in the BEA-2019 shared task.
Approach:	They propose to incorporate delta-log-perplexity, a type of example scoring, into a training schedule for Grammatical Error Correction (GEC) they perform experiments that shed light on the function and applicability of delta- log-perplicity.
Outcome:	The proposed methods incorporate delta-log-perplexity, a type of example scoring, into a training schedule for the task.

Copied to clipboard

Hao Zhu, Yonatan Bisk, Graham Neubig

Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020.tacl-1)

Challenge:	Existing approaches to grammar induction focus on discovering constituents or dependencies.
Approach:	They propose to model lexical dependencies using context free grammars instead of lexicals . they show that this unified framework induces both constituents and dependencies .
Outcome:	The proposed model overcomes sparsity problems and induces constituents and dependencies better than the current methods.

Copied to clipboard

Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, Pontus Stenetorp

Sketch-Driven Regular Expression Generation from Natural Language and Examples (2020.tacl-1)

Challenge:	Innovations in annotation methodologies have been a catalyst for Reading Comprehension (RC) datasets and models.
Approach:	They propose to use a model-in-the-annotation-loop approach to train adversarial models in three different settings to explore reproducibility of the adversarial effect, transfer from data collected with varying model- in-the loop strengths, and generalization to data collected without a modeling model.
Outcome:	The proposed approach achieves 39.9F1 on questions it cannot answer when trained on SQUAD, but lower than when trained using RoBERTa itself (41.0F1).

Copied to clipboard

Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, Greg Durrett

Consistent Transcription and Translation of Speech (2020.tacl-1)

Challenge:	Recent systems for converting natural language descriptions into regexes have achieved some success, but typically deal with short, formulaic text and can only produce simple regexe.
Approach:	They propose a framework for regex synthesis in a context where both natural language and examples are available.
Outcome:	The proposed framework achieves state-of-the-art on two prior datasets and a real-world dataset, which existing neural systems completely fail on.

Copied to clipboard

Matthias Sperber, Hendra Setiawan, Christian Gollan, Udhyakumar Nallasamy, Matthias Paulik

Multimodality

Challenge:	Existing models that translate without transcribing focus on translation quality, while transcription receives less emphasis.
Approach:	They propose a method to evaluate consistency and compare different approaches . they propose 'coupled inference' models that feature a coupled inference procedure can achieve strong consistency.
Outcome:	The proposed model is poorly suited to the joint transcription/translation task, but is strong enough to train for consistency.

Synthesizing Parallel Data of User-Generated Texts with Zero-Shot Neural Machine Translation (2020.tacl-1)

Copied to clipboard

Benjamin Marie, Atsushi Fujita

Multilingual Denoising Pre-training for Neural Machine Translation (2020.tacl-1)

Challenge:	Neural machine translation systems are usually trained on clean parallel data, but the quality of translations is poor when translating noisy texts.
Approach:	They synthesize parallel data of UGT and exploit monolingual data to generate translations . they propose to use monolingual parallel data to train or adapt NMT systems .
Outcome:	The proposed approach improves the translation quality of noisy texts while making them more robust.

Copied to clipboard

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer

Multilinguality

Challenge:	Existing approaches to pre-train models focus on only English corpora, but this is not common in machine translation.
Approach:	They propose a sequence-to-sequence denoising auto-encoder pre-trained on monolingual corpora . they show that it produces significant performance gains across MT tasks .
Outcome:	The proposed model can achieve significant performance gains across a wide variety of MT tasks.

oLMpics-On What Language Model Pre-training Captures (2020.tacl-1)

Copied to clipboard

Alon Talmor, Yanai Elazar, Yoav Goldberg, Jonathan Berant

Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization (2020.tacl-1)

Challenge:	Recent success of pre-trained language models has spurred widespread interest in their capabilities.
Approach:	They propose an evaluation protocol that includes zero-shot evaluation and no fine-tuning . they propose to compare the learning curve of a fine- tuned LM to the learning of multiple controls .
Outcome:	The proposed evaluation protocol compares the learning curve of a fine-tuned LM to the learning of multiple controls.

Copied to clipboard

Edwin Simpson, Yang Gao, Iryna Gurevych

Summarization

Challenge:	Existing methods that focus on learning a ranking across the whole candidate space are lacking user or task-specific training data.
Approach:	They propose an interactive ranking approach that actively selects pairs of candidates, from which the user selects the best.
Outcome:	The proposed approach outperforms existing methods in community question answering and extractive multidocument summarization and is an effective reward function for reinforcement learning.

Syntactic Structure Distillation Pretraining for Bidirectional Encoders (2020.tacl-1)

Copied to clipboard

Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, Phil Blunsom

Clara Meister, Tim Vieira, Ryan Cotterell

Challenge:	Textual representation learners trained on large amounts of data have been successful on downstream tasks.
Approach:	They propose a knowledge distillation strategy for injecting syntactic biases into BERT pretraining by distilling the approximate marginal distribution over words in context from the syntaktic LM.
Outcome:	The proposed method reduces relative error by 2–21% on a diverse set of structured prediction tasks.

Best-First Beam Search (2020.tacl-1)

Copied to clipboard

Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretraining (2020.tacl-1)

Challenge:	Currently, beam search is the default for decoding structured predictors . however, little work has been done to speed up beam search itself .
Approach:	They propose a beam search algorithm that prunes the scoring function to a monotonic sequence length, which allows for safe pruning of hypotheses that cannot be in the final set of hypothecies.
Outcome:	The proposed method can be implemented up to 10x faster in practice.

Copied to clipboard

Ananya B. Sai, Akash Kumar Mohankumar, Siddhartha Arora, Mitesh M. Khapra

Unsupervised Bitext Mining and Translation via Self-Trained Contextual Embeddings (2020.tacl-1)

Challenge:	Existing models for dialog evaluation are trained using a single relevant response and multiple random negatives.
Approach:	They propose a dataset to test whether model-based dialog evaluation metrics can be used to train models . they propose n-gram based metrics and embedding based ones to be used for model-driven evaluation .
Outcome:	The proposed model outperforms existing models on a reddit dataset on relevant responses and adversarial responses.

Copied to clipboard

Phillip Keung, Julian Salazar, Yichao Lu, Noah A. Smith

A Primer in BERTology: What We Know About How BERT Works (2020.tacl-1)

Challenge:	Existing methods to extract parallel sentences from unaligned text yield surprisingly good results.
Approach:	They propose an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text using multilingual BERT to create source and target sentence embeddings for nearest-neighbor search and adapt the model via self-training.
Outcome:	The proposed method outperforms existing methods and outperformed previous state-of-the-art methods by boosting translation performance by up to 3.5 BLEU on the WMT’14 French-English and WMT'16 German-English tasks.

Copied to clipboard

Anna Rogers, Olga Kovaleva, Anna Rumshisky