Transactions of the Association for Computational Linguistics, Volume 10
Word Acquisition in Neural Language Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Language models acquire individual words during training, based on unigram token frequencies, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. |
| Approach: | They examine how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 words on the MacArthur-Bates Communicative Development Inventory. |
| Outcome: | The models follow consistent patterns during training for both unidirectional and bidirectional models, and for both LSTM and Transformer architectures. |
Decomposing and Recomposing Event Structure (2022.tacl-1)
Copied to clipboard
| Challenge: | Using annotated sentences and document-level UDS graphs, we induce an event structure classification with semantic role, entity, and event-event relation classifications. |
| Approach: | They propose to use Universal Decompositional Semantics (UDS) graphs to induce event structure classification . they augment existing annotations with inferential properties capturing fine-grained aspects of temporal and aspectual structure of events. |
| Outcome: | The proposed model is the largest annotation of event structure and (partial) event coreference to date. |
FeTaQA: Free-form Table Question Answering (2022.tacl-1)
Copied to clipboard
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, Dragomir Radev
| Challenge: | Existing table-based question answering datasets lack advanced information-based questions that require reasoning and integration of information pieces retrieved from structured knowledge sources. |
| Approach: | They propose a dataset with 10K Wikipedia-based table, question, free-form answer, supporting table cells pairs that can be used to generate an answer. |
| Outcome: | The proposed dataset has 10K Wikipedia-based table, question, free-form answer, supporting table cells pairs. |
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2022.tacl-1)
Copied to clipboard
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, Mofetoluwa Adeyemi
| Challenge: | Lower-resource corpora have systematic issues, including mislabeled or nonstandard/ambiguous language codes. |
| Approach: | They manually audit the quality of 205 language-specific corpora released with five major public datasets. |
| Outcome: | The results show that lower-resource corpora have systematic issues even for non-proficient speakers. |
Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation (2022.tacl-1)
Copied to clipboard
| Challenge: | End-to-end neural models have replaced the traditional pipeline and require an explicit tokenization step. |
| Approach: | They propose a neural encoder that operates directly on character sequences without explicit tokenization or vocabulary and a pre-training strategy that optionally uses subwords as a soft inductive bias. |
| Outcome: | The proposed model outperforms a comparable mBert model on a multilingual benchmark by 5.7 F1 on the TyDi QA benchmark. |
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations (2022.tacl-1)
Copied to clipboard
| Challenge: | Annotators may systematically disagree with one another, reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. |
| Approach: | They propose to combine multi-annotator models with multi-task based approaches to resolve disagreements between annotations and derive single ground truth labels. |
| Outcome: | The proposed model outperforms majority voting and averaging methods and estimates uncertainty in predictions. |
Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent efforts to create challenge benchmarks that test the abilities of natural language understanding models have largely depended on human annotations. |
| Approach: | They propose a framework for automatic reasoning-oriented perturbation of question-answer pairs that decomposes a question into reasoning steps required to answer it and generates new question-anchor pairs. |
| Outcome: | The proposed framework generates evaluation sets for reading comprehension benchmarks and generates examples without human intervention. |
Out-of-Domain Discourse Dependency Parsing via Bootstrapping: An Empirical Analysis on Its Effectiveness and Limitation (2022.tacl-1)
Copied to clipboard
| Challenge: | Discourse parsing accuracy degrades significantly on out-of-domain text. |
| Approach: | They propose to use bootstrapping methods to adapt modern discourse dependency parsers to out-of-domain text without additional human supervision. |
| Outcome: | The proposed methods are significantly and consistently effective for unsupervised domain adaptation of discourse dependency parsing, but the low coverage of accurately predicted pseudo labels is a bottleneck for further improvement. |
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2022.tacl-1)
Copied to clipboard
Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, Mitesh Shantadevi Khapra
| Challenge: | We present Samanantar, the largest publicly available parallel corpora collection for Indic languages . based on existing corporative, there has been limited benefit for resource-poor languages despite the lack of parallel corporals and monolingual corporata. |
| Approach: | They compile 12.4 million sentence pairs from existing corpora and mine 37.4 million from the Web. |
| Outcome: | The proposed model outperforms existing models and benchmarks on public datasets. |
SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent studies have shown that even state-of-the-art pre-trained language models can generate inconsistent summaries in more than 70% of all cases. |
| Approach: | They propose a method that enables NLI models to be used for inconsistency detection by segmenting documents into sentence units and aggregating scores between pairs of sentences. |
| Outcome: | The proposed method achieves state-of-the-art accuracy of 74.4% on six large inconsistency detection datasets. |
A Survey on Automated Fact-Checking (2022.tacl-1)
Copied to clipboard
| Challenge: | Fact-checking is an essential task in journalism due to the speed with which information and misinformation can spread in the media ecosystem. |
| Approach: | They propose to use natural language processing to automate fact-checking by identifying common concepts and defining definitions. |
| Outcome: | The proposed method can predict the veracity of claims using natural language processing, machine learning, and databases. |
Predicting Document Coverage for Relation Extraction (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for predicting document coverage for relation extraction (RE) are limited in their predictive power. |
| Approach: | They propose a task of predicting the coverage of a text document for relation extraction . they analyze a dataset of 31,366 diverse documents for 520 entities . |
| Outcome: | The proposed model achieves an F1 score of up to 46% on two use cases. |
ABNIRML: Analyzing the Behavior of Neural IR Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc ranking. |
| Approach: | They propose a framework for Analyzing the Behavior of Neural IR ModeLs that includes new types of diagnostic probes that allow us to test several characteristics that are not addressed by previous techniques. |
| Outcome: | The proposed framework tests writing styles, factuality, sensitivity to paraphrasing and word order, and can be used to identify unintended biases. |
Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | a neural network model for natural language inference (NLI) is proposed. |
| Approach: | They propose a neuro-symbolic natural logic framework based on reinforcement learning with introspective revision that rewards specific reasoning paths through policy gradients. |
| Outcome: | The proposed model shows superior capability in monotonicity inference, generalization, and interpretability compared with previous models on the existing datasets. |
Time-Aware Language Models as Temporal Knowledge Bases (2022.tacl-1)
Copied to clipboard
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, William W. Cohen
| Challenge: | Existing language models are trained on snapshots of data collected at a specific moment in time. |
| Approach: | They propose a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time. |
| Outcome: | The proposed method improves memorization of seen facts and calibration on unseen facts from future time periods. |
Multilingual Autoregressive Entity Linking (2022.tacl-1)
Copied to clipboard
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, Fabio Petroni
| Challenge: | mGENRE is a sequence-to-sequence system for multilingual entity linking . mGenRE is used to solve language-specific mentions to a multilingual Knowledge Base . |
| Approach: | They propose a sequence-to-sequence system for multilingual entity linking . they match language-specific mentions against a multilingual Knowledge Base (KB) mGENRE is a sequential system that predicts the name of the target entity token-by-token . |
| Outcome: | The proposed system improves on three popular MEL benchmarks and shows improvements in accuracy. |
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2022.tacl-1)
Copied to clipboard
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel
| Challenge: | a number of pre-trained language models use sequences of tokens corresponding to word units . token-free models that operate directly on raw text have many advantages . |
| Approach: | They propose a standard Transformer architecture that can be used to process byte sequences . they also characterize trade-offs in terms of parameter count, training FLOPs, and inference speed . |
| Outcome: | The proposed model is more robust to noise and more robust on spelling and pronunciation tasks. |
Designing an Automatic Agent for Repeated Language–based Persuasion Games (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing work on persuasion games assumes communication with stylized messages that do not consist of real-world natural language. |
| Approach: | They propose to use a repeated sender-decision maker game to persuade a receiver to accept a deal by sending one of several possible natural language reviews to the expert. |
| Outcome: | The proposed expert is superior to baselines and adaptable to different decision makers and potential proposed deals. |
Towards General Natural Language Understanding with Probabilistic Worldbuilding (2022.tacl-1)
Copied to clipboard
| Challenge: | Probabilistic worldbuilding model is a Bayesian model of semantic parsing and reasoning . large-scale language models are domain-general, despite training on text from virtually every domain . |
| Approach: | They propose a Bayesian probabilistic worldbuilding model that parses and abduces sentences . they use a dataset to test their method against heuristics and to generate a probability model . |
| Outcome: | The proposed model outperforms baselines on two out-of-domain question-answering datasets. |
A Multi-Level Optimization Framework for End-to-End Text Augmentation (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for text augmentation perform data augmentation and downstream tasks separately. |
| Approach: | They propose a framework to perform text augmentation and the downstream task end-to-end. |
| Outcome: | The proposed framework performs text augmentation and the downstream task end-to-end on a text classification dataset. |
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)
Copied to clipboard
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen
| Challenge: | Existing methods to explain predictions by highlighting salient features are often unstated. |
| Approach: | They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. |
| Outcome: | The proposed framework allows principled, automatic, model-agnostic evaluation of attributions. |
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has improved extraction accuracy by incorporating elementary layout information, for example, each token’s 2D position on the page, into language model pretraining. |
| Approach: | They propose a method that explicitly models VIsual LAyout (VILA) groups, that is, text lines or text blocks, to further improve extraction accuracy. |
| Outcome: | The proposed methods show that inserting special tokens denoting layout group boundaries can lead to a 1.9% Macro F1 improvement in token classification. |
Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation (2022.tacl-1)
Copied to clipboard
| Challenge: | morphological segmentation is a common method of evaluation for multilingual tasks . authors often examine models with one data set that is representative of all possible data . |
| Approach: | They compare three broad classes of models with different parameterizations using morphological segmentation as the test case. |
| Outcome: | The results show that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data sets size. |
PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains (2022.tacl-1)
Copied to clipboard
| Challenge: | Domain Adaptation (DA) algorithms suffer degradation when applied to out-of-distribution examples. |
| Approach: | They propose an example-based autoregressive Prompt learning algorithm for on-the-fly Any-Domain Adaptation . the algorithm is trained to generate a unique prompt that maps the test example to a semantic space . |
| Outcome: | The proposed model outperforms baselines in 14 multi-source adaptation scenarios. |
LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language processing focus on understanding or generating short texts . lack of standardized benchmarks makes it difficult to assess and compare models . |
| Approach: | They propose a story-centric benchmark for Chinese long text modeling that aggregates two understanding tasks and two generation tasks. |
| Outcome: | The proposed model outperforms similar-sized models on understanding and generation tasks. |
Czech Grammar Error Correction with a Large and Diverse Corpus (2022.tacl-1)
Copied to clipboard
| Challenge: | a large and diverse corpus of Czech grammar error correction corpora is available for other languages . despite efforts to mitigate the notorious shortage of national GEC-annotated corpors, the lack of adequate data is even more acute in languages other than English. |
| Approach: | They propose to annotate a large and diverse Czech corpus for grammar error correction . they compare several Czech GEC systems and meta-evaluate common GEC metrics against human judgments on data. |
| Outcome: | The proposed corpus is annotated for grammar error correction (GEC) in Czech. |
TopiOCQA: Open-domain Conversational Question Answering with Topic Switching (2022.tacl-1)
Copied to clipboard
| Challenge: | Current datasets for conversational question answering do not contain topic switches . people often engage in information-seeking conversations to discover new knowledge . |
| Approach: | They propose an open-domain conversational dataset with topic switches based on Wikipedia. |
| Outcome: | The proposed dataset achieves an F1 of 55.8, falling short of human performance by 14.2 points, indicating the difficulty of the dataset. |
A Neighborhood Framework for Resource-Lean Content Flagging (2022.tacl-1)
Copied to clipboard
Sheikh Muhammad Sarwar, Dimitrina Zlatkova, Momchil Hardalov, Yoan Dinkov, Isabelle Augenstein, Preslav Nakov
| Challenge: | Existing approaches to cross-lingual content flagging with limited target language data are lacking in many languages. |
| Approach: | They propose a framework for cross-lingual content flagging with limited target- language data based on a nearest-neighbor architecture and a transformer representation in all its components. |
| Outcome: | The proposed framework outperforms previous work in terms of predictive performance on eight languages from two different datasets. |
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2022.tacl-1)
Copied to clipboard
| Challenge: | Current approaches to cross-modal retrieval process text and visual input jointly . current approaches are pretrained from scratch and suffer from huge retrieval latency and inefficiency issues . |
| Approach: | They propose a cooperative retrieve-and-rerank framework that turns pretrained text-image multi-modal models into efficient retrieval models. |
| Outcome: | The proposed framework improves retrieval performance over current approaches . it uses twin networks to encode all items of a corpus and a cross-encoder component for a more nuanced ranking . |
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2022.tacl-1)
Copied to clipboard
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, Angela Fan
| Challenge: | a lack of good evaluation benchmarks hinders progress in low-resource and multilingual machine translation . despite advances in translation quality for a handful of languages, many low-source languages are not even supported by most popular translation engines. |
| Approach: | They propose a high-quality evaluation benchmark for machine translation using 3001 sentences from Wikipedia . they aim to improve evaluation of models on long tail of low-resource languages . |
| Outcome: | The proposed evaluation benchmarks are based on 3001 sentences extracted from Wikipedia . the results show that the models can be used to evaluate multilingual systems . |
♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing multihop reasoning benchmarks are largely solvable via shortcuts . a bottom–up approach allows us to create a multihop QA dataset that requires proper multihop thinking. |
| Approach: | They propose a bottom–up approach that selects composable pairs of single-hop questions that are connected and adds stringent filters to the construction process. |
| Outcome: | The proposed approach creates a multihop question answering dataset with 25K 2–4 hop questions. |
Relational Memory-Augmented Language Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing language models rely on word correlation and are difficult to interpret . existing models often lack explicit representations for such information . |
| Approach: | They propose a memory-augmented approach to condition autoregressive language models on knowledge graphs. |
| Outcome: | The proposed model improves perplexity and bits per character in an autoregressive language model . it is complementary to token-based memory and enables causal interventions . |
Sentence Similarity Based on Contexts (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods to measure sentence similarity face limited dataset size and training-test gap . existing methods lack large-scale labeled datasets with labeles that are labor-intensive and expensive . |
| Approach: | They propose a framework that measures sentence similarity by comparing probabilities of generating two sentences given the same context. |
| Outcome: | The proposed framework achieves significant performance boosts over baselines under supervised and unsupervised settings. |
It’s not Rocket Science: Interpreting Figurative Language in Narratives (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing text representations by design rely on compositionality, while figurative language is often non-compositional. |
| Approach: | They propose to use a pre-trained language model to interpret figurative language types to adopt human strategies for interpreting figurativ language types: inferring meaning from context and relying on constituent words’ literal meanings. |
| Outcome: | The proposed models perform significantly worse than humans on discriminative and generative tasks, bridging the gap from human performance. |
Ultra-fine Entity Typing with Indirect Supervision from Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods for ultra-fine entity typing fail to capture type semantics because of the large number of types and the scarcity of data per type. |
| Approach: | They propose a method that formulates entity typing as a natural language inference problem . they use indirect supervision from NLI to infer type information as textual hypotheses . |
| Outcome: | The proposed method achieves state-of-the-art performance on the ultra-fine entity typing task with limited training data. |
Document Summarization with Latent Queries (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing benchmarks for query-focused summarization are small for training large neural models. |
| Approach: | They propose a unified modeling framework for query-focused summarization . they model queries as discrete latent variables over document tokens . |
| Outcome: | The proposed framework outperforms strong comparison systems across benchmarks, query types, document settings, and target domains. |
End-to-end Argument Mining with Cross-corpora Multi-task Learning (2022.tacl-1)
Copied to clipboard
| Challenge: | Argument(ation) mining is a task of identifying argument structure from text . lack of training data makes it difficult to train models based on limited data sets. |
| Approach: | They propose an end-to-end cross-corpus argument mining method that uses auxiliary argument mining corpora to train models. |
| Outcome: | The proposed method outperforms models trained on a single corpus on arguments on arguments in argument mining tasks. |
Is My Model Using the Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing models that claim to reason about evidence should avoid spurious patterns . tabular inputs are well-suited for the study—they admit systematic probes . |
| Approach: | They propose to use tabular data to test whether models can reason about evidence . they show that a RoBERTa-based model fails to reason on the following counts . |
| Outcome: | The proposed model fails to reason on tabular data on the following counts . the model is over-sensitive to annotation artifacts and ignores relevant parts of the evidence . |
Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression (2022.tacl-1)
Copied to clipboard
| Challenge: | State-of-the-art classification and regression models are often not well calibrated and can be inaccurate. |
| Approach: | They quantify calibration of pre- trained language models for text regression . they apply uncertainty estimates to augment training data in low-resource domains . |
| Outcome: | The proposed model calibrations improve performance and generalizability in low-resource settings. |
Data-to-text Generation with Variational Sequential Planning (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent advances in data-to-text generation have greatly facilitated the task of generating textual output from non-linguistic input. |
| Approach: | They propose a neural model enhanced with a planning component responsible for organizing high-level information in a coherent and meaningful way. |
| Outcome: | The proposed model outperforms baseline models and is sample-efficient in the face of limited training data. |
True Few-Shot Learning with Prompts—A Real-World Perspective (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work has cast doubt on the effectiveness of prompt-based approaches at few-shot learning in a “true” few- shot setting. |
| Approach: | They propose a method that combines textual instructions with example-based finetuning to give prompt-based learning a powerful method for few-shot text classification. |
| Outcome: | The proposed method performs well in a few-shot setting without a dev set and is able to handle multiple prompts. |
Heterogeneous Supervised Topic Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Researchers in the social sciences are interested in the relationship between text and an outcome of interest. |
| Approach: | They develop a probabilistic approach to text analysis and prediction using a joint model of text and outcomes to find heterogeneous patterns. |
| Outcome: | The proposed model outperforms other methods on eight datasets and consistently outperformed other models. |
Fact Checking with Insufficient Evidence (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing work on how to automate fact checking relies on information obtained from external sources. |
| Approach: | They propose a fluency-preserving method for omitting information from the evidence at the constituent and sentence level and a diagnostic dataset for FC with omitted evidence. |
| Outcome: | The proposed method improves evidence sufficiency prediction by 17.8 F1 score and 2.6 F1 scores. |
Text-based NP Enrichment (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing NLP tasks and benchmarks do not cover all NP-mediated relations . we aim to enrich each NP in a text with all the preposition-mediated relationships that hold between it and other NPs in the text. |
| Approach: | They propose a task to enrich NPs with preposition-mediated relations that hold between them . they build a large-scale dataset and analyze the data to test the task . |
| Outcome: | The proposed task is based on a large-scale dataset and fine-tuned language models. |
Minimum Description Length Recurrent Neural Networks (2022.tacl-1)
Copied to clipboard
| Challenge: | We show that neural networks that optimize a minimum description length score master memory challenges and perform addition with 100% accuracy. |
| Approach: | They train neural networks to optimize a Minimum Description Length score . they show that they master tasks involving memory challenges and perform addition . |
| Outcome: | The proposed models master languages and perform addition with 100% accuracy . they show that they can generalize from small training corpora and large training corpus . |
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity (2022.tacl-1)
Copied to clipboard
| Challenge: | averaging hard attention is used to recognize formal languages that UHAT and GUHAT cannot recognize. |
| Approach: | They analyze three formal Transformer encoders that differ in the form of their self-attention mechanism . they find that UHAT and GUHAT Transformers can only recognize formal languages in AC0 . |
| Outcome: | The proposed models can recognize languages that UHAT and GUHAT cannot . the proposed models are based on the DYCK and PARITY languages . |
High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics (2022.tacl-1)
Copied to clipboard
| Challenge: | Neural machine translations are ranked below human translations in professional evaluations . |
| Approach: | They apply minimum bayes risk decoding to optimize different metrics of translation quality . they show that model estimates and translation quality only vaguely correlate . |
| Outcome: | The proposed method improves human translations with different models and metric. |
Generate, Annotate, and Learn: NLP with Synthetic Text (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing methods to generate unlabeled text are difficult to find. |
| Approach: | They propose a general framework called "generate, annotate, and learn" to take advantage of synthetic text within knowledge distillation, self-training, and few-shot learning applications. |
| Outcome: | The proposed framework achieves state-of-the-art knowledge distillation results for 6-layer transformers on the GLUE leaderboard. |
Saturated Transformers are Constant-Depth Threshold Circuits (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent work shows that transformers with hard attention are limited in power, but hard attention is a strong assumption. |
| Approach: | They propose a generalization of hard attention that captures attention patterns in transformers with saturated attention. |
| Outcome: | The proposed generalization of hard attention more closely captures the attention patterns learnable in practical transformers. |
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration (2022.tacl-1)
Copied to clipboard
| Challenge: | Neural generative open-domain english-language dialogue agents are currently unsuitable for applications other than entertainement and research. |
| Approach: | They propose to incorporate metacognitive features into the training of a controllable generation model to improve likelihood of correctness. |
| Outcome: | The proposed model improves likelihood of correctness by incorporating metacognitive features into the training of a controllable generation model. |
A Survey of Text Games for Reinforcement Learning Informed by Natural Language (2022.tacl-1)
Copied to clipboard
| Challenge: | Interactive Fiction Games (text games) are a problem type that require natural language to solve complex tasks. |
| Approach: | They propose to use interactive fiction games as a testing environment to test the new Reinforcement Learning solutions using natural language. |
| Outcome: | The proposed solutions are based on the proposed interactive fiction games and the generated environments. |
Dependency Parsing with Backtracking using Deep Reinforcement Learning (2022.tacl-1)
Copied to clipboard
| Challenge: | Greedy algorithms for NLP such as transition-based parsing are prone to error propagation. |
| Approach: | They propose to allow transition-based parsing to backtrack in cases where evidence contradicts the current solution. |
| Outcome: | The proposed behavior can be implemented on POS tagging and dependency parsing . it shows that backtracking is an effective means to fight error propagation . |
Temporal Effects on Pre-trained Models for Language Processing Tasks (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study shows that language models can be improved as time passes . a number of approaches to solving language tasks have evolved rapidly without a model . |
| Approach: | They examine temporal effects on model performance on downstream language tasks . they also examine the efficacy of two approaches for temporal domain adaptation without human annotations . |
| Outcome: | The proposed methods improve self-labeling and named entity recognition on new data. |
Learning English with Peppa Pig (2022.tacl-1)
Copied to clipboard
| Challenge: | Current approaches to model or simulate the acquisition of spoken language via grounding in perception are not generalizable to real-life situations that humans or adaptive artificial agents experience. |
| Approach: | They propose to use a dataset based on the children’s cartoon Peppa Pig to train a bi-modal architecture that learns aspects of the visual semantics of spoken language. |
| Outcome: | The proposed model learns to represent speech and visual data in a joint vector space. |
Compositional Generalization in Multilingual Semantic Parsing over Wikidata (2022.tacl-1)
Copied to clipboard
| Challenge: | Semantic parsers are mostly designed for and evaluated on English resources, such as CFQ. |
| Approach: | They propose a method for creating a multilingual, parallel question-query dataset . they analyze compositional generalization of parsers in Hebrew, Kannada, Chinese, and English . |
| Outcome: | The proposed method analyzes compositional generalization of parsers in Hebrew, Kannada, Chinese, and English. |
Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks (2022.tacl-1)
Copied to clipboard
| Challenge: | Natural language understanding (NLU) has made massive progress driven by large benchmarks, but a long tail of infrequent phenomena is underrepresented. |
| Approach: | They conceptualize the long tail using macro-level dimensions and perform a meta-analysis of 100 representative papers on transfer learning for NLU. |
| Outcome: | The results highlight avenues for future research in transfer learning for the long tail . authors suggest that the results may be useful for future studies . |
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces (2022.tacl-1)
Copied to clipboard
| Challenge: | Pretrained embeddings based on the Transformer architecture have taken the NLP community by storm . a novel decomposition of Transformer output embeddables is demonstrated . |
| Approach: | They propose to decompose Transformer output embeddings into a sum of vector factors . they show multi-head attentions and feed-forwards are not equally useful in downstream applications . |
| Outcome: | The proposed method outperforms recurrent architectures on a wide variety of tasks. |
On Decoding Strategies for Neural Text Generators (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study suggests that decoding strategies may be more important than the model architecture itself when generating text from probabilistic models. |
| Approach: | They propose to measure changes in attributes of generated text as a function of decoding strategy and task using human and automatic evaluation. |
| Outcome: | The proposed study shows that decoding strategies do not always transfer across tasks . authors show that the differences in attributes are not always consistent across tasks, they say . |
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2022.tacl-1)
Copied to clipboard
| Challenge: | Recent fact verification systems rely on neural network classifiers for veracity prediction, which lack explainability. |
| Approach: | They propose a model that generates natural logic-based inferences as proofs using lexical mutations between spans in the claim and the evidence retrieved. |
| Outcome: | The proposed model has highest label accuracy and second best score in the FEVER leaderboard. |
Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations (2022.tacl-1)
Copied to clipboard
| Challenge: | a rich literature has emerged in the last few years addressing these questions, including whether specific LMs have acquired specific linguistic constructions. |
| Approach: | They introduce a novel metric and release Prime-LM, a large corpus where they control for various linguistic factors that interact with priming strength. |
| Outcome: | The proposed model can learn abstract structural information independent of the structure of a sentence and is able to perform tasks that require natural language understanding skills. |
DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexicon (2022.tacl-1)
Copied to clipboard
Robin Algayres, Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoît Sagot, Emmanuel Dupoux
| Challenge: | Existing nonparametric models for text segmentation use a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Approach: | They propose a Bayesian nonparametric model that uses a Dirichlet process to jointly segment sentences and build a lexicon of word types. |
| Outcome: | The proposed model improves on the Zero Resource Speech Benchmark 2017 and can learn semantic and syntactic representations as assessed by a new spoken word embedding benchmark. |
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2022.tacl-1)
Copied to clipboard
| Challenge: | Knowledge-grounded dialogue systems powered by large language models often generate responses that, while fluent, are not attributable to a relevant source of information. |
| Approach: | They propose to evaluate the validity of 12k dialogue turns generated by neural dialogue systems trained on three knowledge-grounded dialogue corpora and to use them to analyze eight evaluation metrics. |
| Outcome: | The proposed evaluation metrics rely on spurious correlations, do not reliably distinguish attributable abstractive responses from unattributable ones, and perform substantially worse when the knowledge source is longer. |
Modeling Non-Cooperative Dialogue: Theoretical and Empirical Insights (2022.tacl-1)
Copied to clipboard
| Challenge: | a robust dialogue agent cannot assume a cooperative conversational counterpart when deployed in the wild. |
| Approach: | They propose a theoretical model for identifying non-cooperative interlocutors . they use reinforcement learning to implement multiple communication strategies . |
| Outcome: | The proposed model is validated by using reinforcement learning to implement multiple communication strategies. |
Diff-Explainer: Differentiable Convex Optimization for Explainable Multi-hop Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing explainable multi-hop inference models are regarded as black-boxes due to their ability to transfer linguistic and semantic information to downstream tasks, posing concerns about interpretability and transparency of their predictions. |
| Approach: | They propose a hybrid framework that integrates explicit constraints with neural architectures through differentiable convex optimization to answer and explain multi-hop questions in natural language. |
| Outcome: | The proposed framework improves performance on scientific and commonsense QA tasks while still providing structured explanations in support of its predictions. |
Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions (2022.tacl-1)
Copied to clipboard
| Challenge: | Prior work has identified deficiencies in their contextualized representation stemming from the underlying compositional paradigm of representation. |
| Approach: | They propose to use an adapter as a lightweight non-compositional language expert trained on idiomatic sentences to build idiomity into BART. |
| Outcome: | The proposed approach improves idiomaticity over baselines and up to 25% higher sequence accuracy on idiom processing tasks. |
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2022.tacl-1)
Copied to clipboard
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, Diyi Yang
| Challenge: | causality has not had the same importance in natural language processing, says aaron e. smith . he says research on causality in NLP remains scattered across domains without unified definitions . |
| Approach: | They propose to consolidate research on causality in NLP across academic areas . they explore potential uses of causal inference to improve robustness, fairness, interpretability . |
| Outcome: | The proposed method is a unified overview of causal inference for the NLP community. |
Learning Fair Representations via Rate-Distortion Maximization (2022.tacl-1)
Copied to clipboard
| Challenge: | Empirical evaluations show that FaRM debiases representations with or without a target task at hand. |
| Approach: | They propose a method that makes representations belonging to the same protected attribute class uncorrelated, using the rate-distortion function. |
| Outcome: | Empirical results show that the proposed technique achieves state-of-the-art performance on several datasets and leaks significantly less protected attribute information against an attack by a non-linear probing network. |
Robust Dialogue State Tracking with Weak Supervision and Sparse Data (2022.tacl-1)
Copied to clipboard
Michael Heck, Nurul Lubis, Carel van Niekerk, Shutong Feng, Christian Geishauser, Hsien-Chin Lin, Milica Gašić
| Challenge: | Generalizing dialogue state tracking (DST) to new data and domains is especially challenging due to the strong reliance on abundant and fine-grained supervision during training. |
| Approach: | They propose a training strategy to build extractive DST models without the need for fine-grained manual span labels. |
| Outcome: | The proposed model improves robustness against sample sparsity, new concepts, and topics, leading to state-of-the-art performance on a range of benchmarks. |
Unit Testing for Concepts in Neural Networks (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing theories of language and cognition hold that these representations are structured in a compositional way and that the meanings of composite concepts (''gray car'') are inherited predictably from the meaning of the parts. |
| Approach: | They propose to test models for determining whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. |
| Outcome: | The proposed models succeed on tests of groundedness, modularity, and reusability of concepts, but important questions about causality remain open. |
Multi-task Active Learning for Pre-trained Transformer-based Models (2022.tacl-1)
Copied to clipboard
| Challenge: | Multi-task learning requires annotating the same text with multiple annotation schemes, which can be costly and laborious. |
| Approach: | They propose to use multi-task active learning paradigm to optimize annotation processes by iteratively selecting unlabeled examples whose annotation is most valuable for the NLP model. |
| Outcome: | The proposed model minimizes annotation efforts for multi-task NLP models by iterating on the most valuable examples. |
Template-based Abstractive Microblog Opinion Summarization (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing work on Twitter uses extractive summarization to filter through information, but this approach often includes incomplete or redundant information. |
| Approach: | They propose to use Twitter data to generate 3100 gold-standard opinion summaries. |
| Outcome: | The proposed method outperforms previous work on extractive summarization models and fine-tunes to improve performance. |
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation (2022.tacl-1)
Copied to clipboard
| Challenge: | Large pretrained language models are often domain- or task-adapted via finetuning or prompting. |
| Approach: | They propose to use domain-adaptive pretraining to prepare large pretrained language models for domain- or task-adaptation by learning to learn the difference between general and adapted PLMs. |
| Outcome: | Experiments on few-shot dialogue completion, low-resource abstractive summarization, and multi-domain language modeling show improvements in adaptation time and performance over finetuning or preparation via domain-adaptive pretraining. |
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
Neuron-level Interpretation of Deep NLP Models: A Survey (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing work on deep neural networks has focused on representation analysis, but recent work focused on analyzing neurons within these models. |
| Approach: | They propose to analyze neural networks to uncover linguistic concepts captured by the network . they propose to use a granular approach to analyze neurons within these models . |
| Outcome: | The proposed method combines methods to discover and understand neurons in a network with evaluation methods. |
A Survey on Cross-Lingual Summarization (2022.tacl-1)
Copied to clipboard
| Challenge: | Cross-lingual summarization is a task of generating a summary in one language for a given document in a different language. |
| Approach: | They present a systematic review of the literature on cross-lingual summarization . they summarize previous efforts and compare them with each other . |
| Outcome: | The proposed approach is compared with previous approaches and summarizes them to provide a deeper analysis. |
An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing approaches to learning data representations using contrastive learning perform data augmentation and contrastive training separately. |
| Approach: | They propose a framework that performs data augmentation and contrastive learning end-to-end . they propose to combine data augmented with text encoders to optimize for contrastive training . |
| Outcome: | Experiments on GLUE and Gururangan datasets show the proposed framework is effective in NLP. |
Draw Me a Flower: Processing and Grounding Abstraction in Natural Language (2022.tacl-1)
Copied to clipboard
| Challenge: | Abstraction is a core tenet of human cognition and communication. yet, interpreting and grounding abstraction expressed in natural language (NL) has not been systematically studied in NLP. |
| Approach: | They propose a 2D instruction-following game that elicits abstract instructions from 4k natural language instructions. |
| Outcome: | The proposed method elicits 4k natural language instructions rich with diverse types of abstractions and assesses neural models. |
Investigating Reasons for Disagreement in Natural Language Inference (2022.tacl-1)
Copied to clipboard
| Challenge: | Several disagreements in natural language inference (NLI) annotation are due to uncertainty in the sentence meaning, others to annotator biases and task artifacts. |
| Approach: | They propose a 4-way classification approach and a multilabel classification approach for detecting disagreements in natural language inference annotations. |
| Outcome: | The proposed model is more expressive and gives better recall of possible interpretations in the data. |
The Emergence of Argument Structure in Artificial Languages (2022.tacl-1)
Copied to clipboard
| Challenge: | a new study shows that natural languages are shaped by cognitive and sociocultural factors. |
| Approach: | They propose a setup where agents talk about a variable number of entities that can be partially observed by the listener. |
| Outcome: | The proposed setup shows that awareness of object structure yields a more natural sentence organization. |
Scientia Potentia Est—On the Role of Knowledge in Computational Argumentation (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing research on argumentation models does not provide a systematic overview of the types of knowledge required in CA tasks. |
| Approach: | They propose a taxonomy of the types of knowledge required in CA tasks . authors propose exploitation of these knowledge types for four main research areas . |
| Outcome: | The proposed taxonomy proposes a systematic overview of the types of knowledge required in CA tasks. |
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)
Copied to clipboard
| Challenge: | a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions. |
| Approach: | They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions. |
| Outcome: | The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics. |
Explainable Abuse Detection as Intent Classification and Slot Filling (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing models learn what abuse is from labeled examples and base their predictions on spurious cues. |
| Approach: | They propose a machine-friendly representation of the policy that moderators wish to enforce by breaking it down into intents and slots. |
| Outcome: | The proposed model breaks down a social media policy into intents and slots and provides a rationale for its decisions. |
Morphology Without Borders: Clause-Level Morphology (2022.tacl-1)
Copied to clipboard
| Challenge: | Morphological tasks use large multi-lingual datasets that organize words into inflection tables . lack of a clear linguistic and operational definition of what is a word impairs universality of tasks . |
| Approach: | They propose to view morphology as a clause-level phenomenon, rather than word-level . they propose to use a dataset for clause- level morphological tasks in 4 different languages . |
| Outcome: | The proposed dataset for clause-level morphology covers 4 typologically different languages: English, German, Turkish, and Hebrew. |
FaithDial: A Faithful Benchmark for Information-Seeking Dialogue (2022.tacl-1)
Copied to clipboard
| Challenge: | a new benchmark for hallucination-free dialogues is based on knowledge-based conversational models that generate unsupported utterances . a recent study shows that models that are trustworthy generate unverifiable or factually incorrect statements . |
| Approach: | They propose a data-centric solution to edit hallucinated responses in the Wizard of Wikipedia benchmark. |
| Outcome: | The proposed model improves on the Wizard of Wikipedia benchmark while maintaining engaging conversations. |