Papers with SciBERT
INDUS: Effective and Efficient Language Models for Scientific Applications (2024.emnlp-industry)
Copied to clipboard
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain Costes, Sergi Blanco-Cuaresma, Kelly Lockhart, Thomas Allen, Felix Grezes, Megan Ansdell, Alberto Accomazzi, Yousef El-Kurdi, Davis Wertheimer, Birgit Pfitzmann, Cesar Berrospi Ramis, Michele Dolfi, Rafael Lima, Panagiotis Vagenas, S. Mukkavilli, Peter Staar, Sanaz Vahidinia, Ryan McGranaghan, Tsengdar Lee
| Challenge: | Large language models trained on general domain corpora showed remarkable results on natural language processing tasks. |
| Approach: | They develop a suite of large language models trained on general domain corpora that address NLP tasks and smaller versions of them created using knowledge distillation. |
| Outcome: | The proposed models outperform general-purpose and domain-specific encoders on new and existing tasks and in industrial settings. |
CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Scientific document understanding is challenging due to the highly domain specific nature of scientific language. |
| Approach: | They propose a large, contextualized, rigorously cleaned labelled dataset for cite-worthiness detection built from extracted scientific documents. |
| Outcome: | The proposed model improves on a paragraphlevel contextualized sentence labelling model based on Longformer . the model shows a 5 F1 point improvement over SciBERT which considers only individual sentences . |
Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to augment pre-trained language models with disease knowledge are lacking. |
| Approach: | They propose a method to augment BERT-like pre-trained language models with disease knowledge. |
| Outcome: | The proposed method improves on a suite of BERT models over three tasks. |
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. |
| Approach: | They propose an infusion approach that partitions a large knowledge graph into smaller sub-graphs and infuses their specific knowledge into various BERT models using lightweight adapters. |
| Outcome: | The proposed approach improves the underlying BERTs and achieves new SOTA performance on six downstream tasks. |
CitRet: A Hybrid Model for Cited Text Span Retrieval (2022.coling-1)
Copied to clipboard
| Challenge: | Current methods for citing text span retrieval (CTSR) rely on pre-trained off-the-shelf deep learning models like SciBERT. |
| Approach: | They propose a hybrid model for cited text span retrieval that leverages unique semantic and syntactic structural characteristics of scientific documents. |
| Outcome: | The proposed model improves state-of-the-art by 15% on the CLSciSumm shared tasks. |
Scientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Scientific keyphrase identification and classification is challenging due to lack of labeled data for training deep neural networks. |
| Approach: | They propose to use pre-trained language models BERT and SciBERT to train scientific keyphrase identification and classification using intermediate task transfer learning. |
| Outcome: | The proposed model achieves competitive performance in scientific keyphrase identification and classification compared to both previous studies and strong baselines. |
Aspect-based Document Similarity for Research Papers (2020.coling-main)
Copied to clipboard
| Challenge: | Traditional document similarity measures do not consider in what aspects two documents are similar. |
| Approach: | They extend document similarity with aspect information by performing a pairwise document classification task. |
| Outcome: | The proposed approach is best performing on 172,073 research paper pairs from the ACL Anthology and CORD-19 corpus. |
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification (2024.findings-emnlp)
Copied to clipboard
Zeren Shui, Petros Karypis, Daniel Karls, Mingjian Wen, Saurav Manchanda, Ellad Tadmor, George Karypis
| Challenge: | Prior research has shown that pretrained language models (PLMs) can achieve state-of-the-art performance on CIC benchmarks. |
| Approach: | They propose a multi-task learning framework that fine-tunes pretrained language models on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals. |
| Outcome: | The proposed framework outperforms current state-of-the-art models on small datasets while aligning with the best-performing model on a large dataset. |
SciDMT: A Large-Scale Corpus for Detecting Scientific Mentions (2024.lrec-main)
Copied to clipboard
| Challenge: | SciDMT is an enhanced and expanded corpus for scientific mention detection . existing corpora are limited by their small volume and entity linking capabilities . |
| Approach: | They propose to enhance SciDMT, an annotated scientific corpus for scientific mention detection. |
| Outcome: | The proposed corpus is the largest for scientific entity mention detection . it is based on deep learning architectures like SciBERT and GPT-3.5 . |