IIT-KGP at COIN 2019: Using pre-trained Language Models for modeling Machine Comprehension

Challenge:	Using pre-trained language models, we can model machine comprehension using commonsense reasoning.
Approach:	They propose a machine comprehension model that leverages pre-trained language models over commonsense knowledge bases.
Outcome:	The proposed model improves on baseline models and other commonsense knowledge bases.

Similar Papers

Overview

Challenge

Approach

Outcome

Pingan Smart Health and SJTU at COIN - Shared Task: utilizing Pre-trained Language Models and Common-sense Knowledge in Machine Reading Tasks (D19-60)

Copied to clipboard

Xiepeng Li, Zhexi Zhang, Wei Zhu, Zheng Li, Yuan Ni, Peng Gao, Junchi Yan, Guotong Xie

Semantic Text Processing

Challenge:	Existing approaches to represent knowledge in the low-dimensional space are to leverage large-scale unsupervised text corpus to train fixed or contextual representations.
Approach:	They propose to leverage large-scale unsupervised text corpus to train fixed or contextual language representations and to express knowledge into a knowledge graph (KG) they incorporate distributional representations of a KG onto the representations from pre-trained language models, via simply concatenation or multi-head attention.
Outcome:	The proposed models outperform the other models on the COIN: COmmonsense INference in Natural Language Processing (COIN) Workshop datasets.

KARNA at COIN Shared Task 1: Bidirectional Encoder Representations from Transformers with relational knowledge for machine comprehension with common sense (D19-60)

Copied to clipboard

Yash Jain, Chinmay Singh

Semantic Text Processing

Challenge:	Using Bidirectional Encoder Representations from Transformers(BERT) and external relational knowledge from ConceptNet, we are able to achieve an accuracy of 73.3 % on the official test data.
Approach:	They propose a model that uses Bidirectional Encoder Representations from Transformers and ConceptNet to tackle the problem of commonsense inference in natural language processing.
Outcome:	The proposed model achieves 73.3 % accuracy on the official test data.

BLCU-NLP at COIN-Shared Task1: Stagewise Fine-tuning BERT for Commonsense Inference in Everyday Narrations (D19-60)

Copied to clipboard

Chunhua Liu, Dong Yu

Reasoning

Challenge:	Experimental results show that our system achieves significant improvements over the baseline systems with 84.2% accuracy on the official test dataset.
Approach:	They propose a system to inject more external knowledge into everyday narrations . they use a pre-trained BERT model to fine-tune on a machine reading comprehension dataset .
Outcome:	The proposed system achieves significant improvements over baseline systems with 84.2% accuracy on the official test dataset.

Commonsense Inference in Natural Language Processing (COIN) - Shared Task Report (D19-60)

Copied to clipboard

Simon Ostermann, Sheng Zhang, Michael Roth, Peter Clark

Reasoning

Challenge:	The workshop on Commonsense Inference in NLP (COIN) evaluated text understanding systems' ability to draw inferences about facts that are not mentioned in the text, but that are assumed to be common ground.
Approach:	They propose to use commonsense knowledge to evaluate systems' ability to answer questions/queries about a text.
Outcome:	The proposed tasks evaluated systems in two contexts: Commonsense Inference and Commonsensible Inference.

Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (P19-1)

Copied to clipboard

An Yang, Quan Wang, Jing Liu, Kai Liu, Yajuan Lyu, Hua Wu, Qiaoqiao She, Sujian Li

Semantic Text Processing

Challenge:	Recent results show pre-trained language models (LMs) can improve machine reading comprehension (MRC) Experimental results indicate that KT-NET offers significant and consistent improvements over BERT .
Approach:	They propose a method that leverages external knowledge bases to improve machine reading comprehension (MRC) KT-NET employs an attention mechanism to select desired knowledge from KBs and fuses selected knowledge with BERT to enable context- and knowledge-aware predictions.
Outcome:	The proposed model outperforms baseline models on ReCoRD and SQuAD1.1 benchmarks and ranks 1st on the ReCoDR and SQUAD1.1 leaderboards.

REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training (2021.findings-acl)

Copied to clipboard

Fangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji, Feng-Lin Li, Liqiang Nie

Reasoning

Challenge:	Pre-trained language models have achieved great success on Machine Reading Comprehension (MRC) however, the poor support in evidence extraction hinders them from further advancing MRC.
Approach:	They propose a REtrieval-based pre-training approach that strengthens evidence extraction during pre-training by inherited downstream MRC tasks.
Outcome:	The proposed approach strengthens evidence extraction during pre-training, which is further inherited by downstream tasks.

AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization (2021.findings-acl)

Copied to clipboard

Xinsong Zhang, Pengshuai Li, Hang Li

Semantic Text Processing

Challenge:	Pre-trained language models such as BERT have shown great power in natural language understanding . fine-grained tokenizations have advantages and disadvantages for learning of pre-tried models .
Approach:	They propose a pretrained language model based on both fine-grained and coarse-grain tokenizations . they propose to use both tokenization techniques to learn pre-trained models .
Outcome:	The proposed model outperforms BERT on benchmark datasets for Chinese and English . it can perform better with the same computational cost as BERT, the authors show .

Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge (2023.acl-long)

Copied to clipboard

Xuebo Liu, Yutong Wang, Derek F. Wong, Runzhe Zhan, Liangxuan Yu, Min Zhang

Reasoning

Challenge:	CR is the ability to understand and navigate the world using basic knowledge and understanding shared by most people.
Approach:	They propose to incorporate pretrained knowledge into NMT models and use them as robust testbeds for investigating CR in NMT.
Outcome:	The proposed method improves the training of NMT models with high CR abilities and provides accurate evaluation metrics.

MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge (L18-1)

Copied to clipboard

Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, Manfred Pinkal

Reasoning

Challenge:	Various approaches for script knowledge extraction and processing have been proposed in recent years.
Approach:	They propose a dataset to evaluate natural language understanding approaches based on commonsense knowledge.
Outcome:	The proposed dataset provides test cases for the broader natural language understanding community.

Life after BERT: What do Other Muppets Understand about Language? (2022.acl-long)

Copied to clipboard

Vladislav Lialin, Kevin Zhao, Namrata Shivagunde, Anna Rumshisky

Language Models

Challenge:	Existing pre-trained transformer analysis studies focus on one or two model families at a time, overlooking the variability of the architecture and pre-training objectives.
Approach:	They utilize oLMpics bench- mark and psycholinguistic probing datasets for a diverse set of 29 models including T5, BART, and ALBERT.
Outcome:	The proposed model fails to resolve compositional questions in a zero-shot fashion, suggesting that pre-training objectives are not predictive of a model’s linguistic capabilities.