| Challenge: | Existing approaches to reasoning using transformers are limiting, resulting in inconsistent results in arithmetic and QA benchmarks. |
| Approach: | They propose a novel approach that utilizes probabilistic logical rules as constraints in the fine-tuning phase without relying on them in the inference stage. |
| Outcome: | The proposed approach improves the transformer-based language model’s intrinsic reasoning and makes their probabilistic logical reasoning process more explicit and explainable. |
Similar Papers
Can Language Models Learn Embeddings of Propositional Logic Assertions? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for automating reasoning can no longer be used for natural language tasks. |
| Approach: | They propose to use transformer-based language models to reason about knowledge expressed in natural language rather than using LMs to perform reasoning directly. |
| Outcome: | The proposed approach is feasible to some extent, but lacks robustness. |
Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models (2023.findings-acl)
Copied to clipboard
Borui Wang, Qiuyuan Huang, Budhaditya Deb, Aaron Halfaker, Liqun Shao, Daniel McDuff, Ahmed Hassan Awadallah, Dragomir Radev, Jianfeng Gao
| Challenge: | Existing pre-trained language models that ignore the logical structures underlying natural language text often lack the ability to capture and encode key logical information in the input sequences. |
| Approach: | They propose to construct logic-aware input embeddings for transformer language models through logic detection, logic mapping and hierarchical logical projections and then develop a new modeling paradigm that can upgrade existing transformer language model into logical transformers to boost their performance. |
| Outcome: | The proposed model can achieve superior performance on four important and challenging tasks. |
Identifying the limits of transformers when performing model-checking with natural language (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on transformer models’ ability to perform reasoning on text, but the above question has not been adequately answered. |
| Approach: | They investigated the problem of model-checking with natural language to determine whether transformers can comprehend logical semantics in natural language. |
| Outcome: | The proposed model-checking problem is suited to address this issue but is untouched in natural language inference research. |
Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Syntactic structures were deemed essential in natural language processing . but since the deep learning revolution, NLP has been dominated by neural models that do not consider syntactical structures in their design. |
| Approach: | They propose a model that models latent representations of words in a sentence . they use a conditional random field to model latent and dependency arcs . |
| Outcome: | The proposed model performs competitively to transformers on small to medium sized datasets. |
Can Transformers Reason in Fragments of Natural Language? (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work on natural language inference has identified two strands of research . |
| Approach: | They investigate whether neural networks have acquired logical principles from natural language . they use transformer-based models to detect valid inferences in controlled fragments of natural language. |
| Outcome: | The proposed model overfits to superficial patterns in the data rather than acquiring the logical principles governing reasoning in natural language fragments. |
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are limited in their ability to capture and use common-sense knowledge. |
| Approach: | They propose to teach PLMs how to reason with soft Horn rules by leveraging logical rules to learn how to predict precise probabilities. |
| Outcome: | The proposed model performs well on logical rules that were unseen at training. |
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing techniques to fine-tune pre-trained language models on downstream tasks are inadequate. |
| Approach: | They propose a technique to perturb hidden Transformers representations by enhancing generalization of hidden representations from different layers. |
| Outcome: | The proposed technique outperforms vanilla fine-tuning and enhances generalization of hidden representations from different layers. |
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis (2022.findings-emnlp)
Copied to clipboard
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, Louis-Philippe Morency
| Challenge: | Pre-trained language models (PLMs) have gained increasing popularity due to compelling prediction performance in diverse natural language processing tasks. |
| Approach: | They compare three popular options for encoding and Temp Scaling for PLMs . they recommend using Temp Loss as uncertainty quantifier and Focal Loss for fine-tuning . |
| Outcome: | Using pre-trained language models, we compare three options on NLP classification tasks and domain shift. |
FaiRR: Faithful and Robust Deductive Reasoning over Natural Language (2022.acl-long)
Copied to clipboard
| Challenge: | Currently, black-box models generate both the proof graph and intermediate inferences within the same model and thus may be unfaithful. |
| Approach: | They propose a transformer-based model that can perform deductive reasoning on a logical rulebase containing rules and statements written in natural language. |
| Outcome: | The proposed model is robust to language perturbations and faster at inference than previous models on existing reasoning datasets. |
Transformer versus LSTM Language Models trained on Uncertain ASR Hypotheses in Limited Data Scenarios (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies show that domain-specific LMs can only rely on limited in-domain speech data . a qualitative analysis reveals that Transformer LM can predict less frequent words . |
| Approach: | They propose a method to train Transformer LMs on ASR confusion networks . they find they are better at exploiting alternate uncertain ASR hypotheses . |
| Outcome: | The proposed method reduces perplexity by 3-6% on AMI scenarios but performs similar to LSTM LMs on Verbmobil conversational corpus. |