Papers by Iz Beltagy
SciBERT: A Pretrained Language Model for Scientific Text (D19-1)
Copied to clipboard
| Challenge: | SciBERT is a pretrained language model based on BERT to improve performance on scientific NLP tasks. |
| Approach: | They propose a pretrained language model based on BERT to improve NLP performance . they evaluate on sequence tagging, sentence classification and dependency parsing . |
| Outcome: | The proposed model improves on sequence tagging, sentence classification and dependency parsing tasks with datasets from a variety of scientific domains. |
CDLM: Cross-Document Language Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks. |
| Approach: | They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective. |
| Outcome: | The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks. |
TESS: Text-to-Text Self-Conditioned Simplex Diffusion (2024.eacl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan
| Challenge: | Existing models for diffusion generation are expensive and discrete, resulting in a large number of diffusion steps to generate text. |
| Approach: | They propose a text diffusion model that is fully non-autoregressive and employs a new form of self-conditioning and applies the diffusion process on the logit simplex space rather than the learned embedding space. |
| Outcome: | The proposed model outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. |
What Language Model to Train if You Have One Million GPU Hours? (2022.findings-emnlp)
Copied to clipboard
Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, Iz Beltagy
| Challenge: | Recent years have seen the advent of large language models characterized by emergent capabilities arising from sheer scale alone. |
| Approach: | They propose to use a multilingual model to compare performance to the English-only model by ablation at the billion-parameter scale. |
| Outcome: | The proposed model is based on a multilingual model and its performance against the English-only model. |
Few-Shot Self-Rationalization with Natural Language Prompts (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing models that generate free-text explanations for tasks are limited by human-written explanations. |
| Approach: | They propose to use a standardized collection of natural language prompts to create a model that generates free-text explanations for tasks. |
| Outcome: | The proposed model can predict task labels and generate free-text explanations for predictions . plausibility of human explanations is 76%, while human explanation is 51% . |
Construction of the Literature Graph in Semantic Scholar (N18-3)
Copied to clipboard
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Lu Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, Oren Etzioni
| Challenge: | Fig. 1 summarizes a scalable system for organizing published scientific literature into a heterogeneous graph . authors describe methods used to enable semantic features in www.semanticscholar.org . |
| Approach: | They describe a scalable system for organizing published scientific literature into a heterogeneous graph to facilitate algorithmic manipulation and discovery. |
| Outcome: | The proposed system can be deployed on a scalable platform and report empirical results for each task. |
Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for zero-shot learning are based on in-context training, but performance drops when no demonstrations are available. |
| Approach: | They propose a new method that constructs pseudo-demonstrations for a given test input using a raw text corpus and applies techniques to reduce copying. |
| Outcome: | The proposed method outperforms previous zero-shot methods on nine classification datasets and is on par with in-context learning with labeled training data in the few-shot setting. |
SciREX: A Challenge Dataset for Document-Level Information Extraction (2020.acl-main)
Copied to clipboard
| Challenge: | Conventional datasets and methods for information extraction focus on within-sentence relations from general Newswire text. |
| Approach: | They propose a document-level IE dataset that integrates automatic and human annotations to annotate entities and document- level N-ary relation identification from scientific articles. |
| Outcome: | The proposed dataset extends state-of-the-art IE models to document-level IE. |
MSˆ2: Multi-Document Summarization of Medical Studies (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing datasets for multi-document summarization (MDS) are either in the general domain, such as WikiSum, or very small such as DUC 1 or TAC 2011 . Existing systems for summarizing biomedical literature take 1-2 years to complete . |
| Approach: | They propose to use a multi-document summarization system based on BART to assess the quality of the summarized biomedical literature. |
| Outcome: | The proposed system has high summarization quality, but significant work remains to achieve it. |
PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pretrained models require domain-specific additional information to be effective. |
| Approach: | They propose a pre-trained model for multi-document representation with a focus on summarization that uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. |
| Outcome: | PRIMERA outperforms current state-of-the-art models on most datasets with large margins . PRImerA uses efficient encoder-decoder transformers to simplify processing of concatenated input documents. |
FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained models are capable of few-shot in-context learning (ICL) however, concatenated demonstrations are often excessively long and require additional computation. |
| Approach: | They propose to apply fusion-in-decoder (FiD) models to perform few-shot in-context learning (ICL) they propose to use concatenation-based, early-fusion, intermediate- and late-fusion methods to improve efficiency . |
| Outcome: | The proposed methods outperform concatenation-based models on 11 held-out tasks. |
Beyond Paragraphs: NLP for Long Sequences (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we will introduce document-level representation learning techniques . document-based learning is challenging due to the limited sequence length of many models . |
| Approach: | They will provide an overview of established long sequence NLP techniques and discuss memory-saving methods that are key to processing long sequences. |
| Outcome: | The tutorial will introduce the latest and ongoing techniques for document-level representation learning. |
Combining Distant and Direct Supervision for Neural Relation Extraction (N19-1)
Copied to clipboard
| Challenge: | Existing methods to train relation extraction with distant supervision use noisy labels and implicitly assumes that all the KB facts are mentioned in the text. |
| Approach: | They propose to combine distant supervision data with additional directly-supervised data to train relation extraction models by using sigmoidal attention weights with max pooling. |
| Outcome: | The proposed method achieves state-of-the-art on the widely used FB-NYT dataset. |
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020.acl-main)
Copied to clipboard
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, Noah A. Smith
| Challenge: | Language models prerained on text from a wide variety of sources form the foundation of today’s NLP. |
| Approach: | They propose to tailor a pretrained model to the domain of a target task by using domain-adaptive pretraining in-domain. |
| Outcome: | The proposed model can be tailored to the domain of a target task and perform well under both high- and low-resource settings. |
SPECTER: Document-level Representation Learning using Citation-informed Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Recent Transformer language models do not leverage information on inter-document relatedness, which limits their document-level representation power. |
| Approach: | They propose a method to generate document-level embeddings using citation graphs. |
| Outcome: | The proposed method outperforms baselines on document-level tasks. |
Zero- and Few-Shot NLP with Pretrained Language Models (2022.acl-tutorials)
Copied to clipboard
| Challenge: | a tutorial aims to introduce NLP researchers to the latest techniques for learning from little-to-no data . aims at bringing interested researchers up to speed about the latest and ongoing techniques . |
| Approach: | They aim to introduce techniques for learning from little-to-no data using pretrained language models. |
| Outcome: | This tutorial aims to bring interested NLP researchers up to speed about recent techniques . it will cover methods from manual engineering, better inference algorithms to better tuning methods . |
Transparency Helps Reveal When Language Models Learn Meaning (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing language models are trained to optimize unsupervised objectives on text . despite their centrality, current models do not represent natural language semantics well . |
| Approach: | They show that autoregressive and masked language models learn to emulate semantic relations between expressions when context-dependent . they argue that a learner that has access to all Java code can never learn execution . |
| Outcome: | a new study shows that language models fail to represent natural language semantics well . the authors show that the model learning fails when denotations are changed to be context-dependent . |
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing information-seeking question answering datasets do not perform well on answering these questions . existing models that do well on other QA tasks do not do well answering these tasks . |
| Approach: | They present a dataset of 5049 questions over 1585 NLP papers . they use a question-seeking QA model that seeks information in the full text . |
| Outcome: | The proposed dataset underperforms existing models on other QA tasks by 27 F1 points . the focus is on document-grounded, information-seeking QA . |
SciFact-Open: Towards open-domain scientific claim verification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Current scientific claim verification systems can achieve very strong performance on limited contexts, in some cases approaching human agreement. |
| Approach: | They propose to pool and annotate top predictions from four state-of-the-art scientific claim verification models to evaluate their performance against large corpora. |
| Outcome: | The proposed system performs well on a corpus of 500K scientific abstracts. |
Continued Pretraining for Better Zero- and Few-Shot Promptability (2022.emnlp-main)
Copied to clipboard
Zhaofeng Wu, Robert L Logan IV, Pete Walsh, Akshita Bhagia, Dirk Groeneveld, Sameer Singh, Iz Beltagy
| Challenge: | Recent language model prompting methods can achieve high accuracy in zero- and few-shot settings while requiring few to no learned task-specific parameters. |
| Approach: | They propose to use a dedicated pretraining stage to improve promptability in zero-shot settings and few-shot tuning. |
| Outcome: | The proposed method improves promptability in zero- and few-shot settings, while the existing method yields subpar performance. |
MultiVerS: Improving scientific claim verification with weak supervision and full-document context (2022.findings-naacl)
Copied to clipboard
| Challenge: | a new approach to scientific claim verification uses a document-level fact-checking label to label scientific documents . a multitask approach combines a shared encoding of the claim and document context . |
| Approach: | They propose a system which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. |
| Outcome: | The proposed approach outperforms baselines on three scientific claim verification datasets . it can learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales based on the datasets. |
Pretrained Language Models for Sequential Sentence Classification (D19-1)
Copied to clipboard
| Challenge: | Recent successful models for document-level understanding have used hierarchical encoding and CRFs to capture dependencies between subsequent labels. |
| Approach: | They propose a pretrained language model that captures contextual dependencies without hierarchical encoding nor a CRF. |
| Outcome: | The proposed model captures contextual dependencies without hierarchical encoding nor a CRF on four datasets, including a new dataset of structured scientific abstracts. |