Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models

Challenge:	Existing pre-trained language models that ignore the logical structures underlying natural language text often lack the ability to capture and encode key logical information in the input sequences.
Approach:	They propose to construct logic-aware input embeddings for transformer language models through logic detection, logic mapping and hierarchical logical projections and then develop a new modeling paradigm that can upgrade existing transformer language model into logical transformers to boost their performance.
Outcome:	The proposed model can achieve superior performance on four important and challenging tasks.

Similar Papers

Overview

Challenge

Approach

Outcome

Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)

Copied to clipboard

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush

Language Models

Challenge:	Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community.
Approach:	they propose an open-source library that aims to open up advances in machine learning to the wider community.
Outcome:	Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community.

Identifying the limits of transformers when performing model-checking with natural language (2023.eacl-main)

Copied to clipboard

Tharindu Madusanka, Riza Batista-navarro, Ian Pratt-hartmann

Language Models

Challenge:	Recent studies have focused on transformer models’ ability to perform reasoning on text, but the above question has not been adequately answered.
Approach:	They investigated the problem of model-checking with natural language to determine whether transformers can comprehend logical semantics in natural language.
Outcome:	The proposed model-checking problem is suited to address this issue but is untouched in natural language inference research.

Can Language Models Learn Embeddings of Propositional Logic Assertions? (2024.lrec-main)

Copied to clipboard

Nurul Fajrin Ariyani, Zied Bouraoui, Richard Booth, Steven Schockaert

Semantic Text Processing

Challenge:	Existing methods for automating reasoning can no longer be used for natural language tasks.
Approach:	They propose to use transformer-based language models to reason about knowledge expressed in natural language rather than using LMs to perform reasoning directly.
Outcome:	The proposed approach is feasible to some extent, but lacks robustness.

Can Transformers Reason in Fragments of Natural Language? (2022.emnlp-main)

Copied to clipboard

Viktor Schlegel, Kamen Pavlov, Ian Pratt-Hartmann

Language Models

Challenge:	Recent work on natural language inference has identified two strands of research .
Approach:	They investigate whether neural networks have acquired logical principles from natural language . they use transformer-based models to detect valid inferences in controlled fragments of natural language.
Outcome:	The proposed model overfits to superficial patterns in the data rather than acquiring the logical principles governing reasoning in natural language fragments.

Learning Semantic Structure through First-Order-Logic Translation (2024.findings-emnlp)

Copied to clipboard

Akshay Chaturvedi, Nicholas Asher

Language Models

Challenge:	a recent study shows that transformer-based language models can confuse which predicates apply to which objects . a this is a crucial building block of semantic structure, but if an LM mixes up which objects have which property, it makes errors in reasoning .
Approach:	They propose to use transformer-based language models to learn predicate argument structure from simple sentences.
Outcome:	The proposed model can learn predicate argument structure from simple sentences.

Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences (2022.coling-1)

Copied to clipboard

Jad Kabbara, Jackie Chi Kit Cheung

Semantic Text Processing

Challenge:	Presuppositions are assumptions that are taken for granted by an utterance.
Approach:	They propose to use heuristics to create alternative "contrastive" test cases . they also analyze samples from ImpPres datasets to better understand their predictions .
Outcome:	The proposed model performs better on the ImpPres dataset than on the other datasets.

Developmental Negation Processing in Transformer Language Models (2022.acl-short)

Copied to clipboard

Antonio Laverghetta Jr., John Licato

Language Models

Challenge:	Negation is an important construct in language for reasoning over the truth of propositions, garnering interest from philosophy (Horn, 1989) and psycholinguistics (Zwaan, 2012).
Approach:	They propose to frame a natural language inference task as a problem and examine how well transformers can process negation categories.
Outcome:	The proposed models perform better on certain categories, suggesting clear differences in how they are processed.

Explaining How Transformers Use Context to Build Predictions (2023.acl-long)

Copied to clipboard

Javier Ferrando, Gerard I. Gállego, Ioannis Tsiamas, Marta R. Costa-jussà

Explainability & Interpretability in NLP

Challenge:	Existing methods for analyzing input attributions for a model's prediction are unclear how prior words affect the model' s decision throughout the layers.
Approach:	They propose a procedure to analyze models for language generation using the Transformer and a comparison of their results with evidence of the linguistic phenomena.
Outcome:	The proposed method consistently aligns better than gradient-based and perturbation-based baselines and generates human-like source-target alignments for building predictions.

Document Structure in Long Document Transformers (2024.eacl-long)

Copied to clipboard

Jan Buchmann, Max Eichler, Jan-Micha Bodensohn, Ilia Kuznetsov, Iryna Gurevych

Language Models

Challenge:	Existing long-document Transformers do not learn representations of document structure during pretraining.
Approach:	They propose to use long-document Transformers to acquire an internal representation of document structure during pre-training and evaluate the effects of structure infusion on QASPER and Evidence Inference.
Outcome:	The proposed models acquire implicit understanding of document structure during pre-training, which can be enhanced by structure infusion, leading to improved end-task performance.

Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale (2022.tacl-1)

Copied to clipboard

Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom, Chris Dyer

Language Models

Challenge:	a novel class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers and recursive syntactic compositions.
Approach:	They introduce Transformer Grammars, a class of Transformer language models that combine expressive power and recursive syntactic compositions.
Outcome:	The proposed model outperforms strong baselines on sentence-level language modeling perplexity and syntax-sensitive language evaluation metrics.