MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling (2023.acl-long)
Copied to clipboard
| Challenge: | Using publicly available materials science text data, we construct a benchmark for evaluating the performance of natural language processing (NLP) models on materials science texts. |
| Approach: | They propose a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. |
| Outcome: | The proposed model outperforms BERT-based models on scientific text and a model pretrained on materials science journals. |
Similar Papers
Rethinking NLP for Chemistry: A Critical Look at the USPTO Benchmark (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Natural Language Processing (NLP) has revolutionized computer-aided synthesis planning by reframing chemical synthesis prediction as a sequence-to-sequence modeling problem over molecular string representations like SMILES. |
| Approach: | They propose to reframe chemical synthesis prediction as a sequence-to-sequence modeling problem over molecular string representations like SMILES. |
| Outcome: | The proposed framework yields impressive benchmark scores on the USPTO dataset, a large corpus of reactions extracted from US patents. |
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research (2023.acl-long)
Copied to clipboard
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, Adhiguna Kuncoro
| Challenge: | Despite rapid recent progress, current research practices conflate different sources of model improvement without conducting proper ablation studies and principled comparisons . authors conclude with recommendations for how to encourage and incentivize this line of work . |
| Approach: | They critique current research practices in the field of language model pre-training . they examine the success of language models pre-trained on large amounts of data . |
| Outcome: | The proposed models can achieve competitive or better performance than BERT under comparable conditions. |
Dive into Deep Learning for Natural Language Processing (D19-2)
Copied to clipboard
| Challenge: | GluonNLP is a powerful new toolkit that automates the most laborious aspects of deep learning for NLP. |
| Approach: | This hands-on tutorial demonstrates how to scale unsupervised pre-training techniques with Apache MXNet and GluonNLP. |
| Outcome: | This hands-on tutorial examines the challenges of scaling these models and algorithms effectively with Apache MXNet and GluonNLP. |
Efficient Methods for Natural Language Processing: A Survey (2023.tacl-1)
Copied to clipboard
Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
| Challenge: | Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data, but using only scale to improve performance means resource consumption also grows. |
| Approach: | They propose to use data, time, storage, or energy to improve model performance. |
| Outcome: | The proposed methods and findings provide guidance for conducting NLP under limited resources and point towards promising research directions for developing more efficient methods. |
End-to-End Construction of NLP Knowledge Graph (2021.findings-acl)
Copied to clipboard
| Challenge: | a new schema for NLP knowledge about tasks, datasets and metrics is proposed. |
| Approach: | They propose a new schema that represents knowledge about tasks, datasets and metrics in the NLP domain. |
| Outcome: | The proposed framework can be automatically built into scientific leaderboards . the proposed system achieves reasonable results for all relation types on this small-scale graph . |
INDUS: Effective and Efficient Language Models for Scientific Applications (2024.emnlp-industry)
Copied to clipboard
Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasubramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain Costes, Sergi Blanco-Cuaresma, Kelly Lockhart, Thomas Allen, Felix Grezes, Megan Ansdell, Alberto Accomazzi, Yousef El-Kurdi, Davis Wertheimer, Birgit Pfitzmann, Cesar Berrospi Ramis, Michele Dolfi, Rafael Lima, Panagiotis Vagenas, S. Mukkavilli, Peter Staar, Sanaz Vahidinia, Ryan McGranaghan, Tsengdar Lee
| Challenge: | Large language models trained on general domain corpora showed remarkable results on natural language processing tasks. |
| Approach: | They develop a suite of large language models trained on general domain corpora that address NLP tasks and smaller versions of them created using knowledge distillation. |
| Outcome: | The proposed models outperform general-purpose and domain-specific encoders on new and existing tasks and in industrial settings. |
MEGA: Multilingual Evaluation of Generative AI (2023.emnlp-main)
Copied to clipboard
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, Sunayana Sitaram
| Challenge: | Large Large Models (LLMs) have shown impressive performance on many natural language processing tasks such as language understanding, reasoning, and language generation. |
| Approach: | They present a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field. |
| Outcome: | The proposed framework evaluates generative models on 16 NLP datasets across 70 typologically diverse languages and compares them to state-of-the-art non-autoregressive models. |
NLPre: A Revised Approach towards Language-centric Benchmarking of Natural Language Preprocessing Systems (2024.lrec-main)
Copied to clipboard
| Challenge: | GLUE benchmarking system enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Approach: | They propose a language-centric benchmarking system that enables ongoing evaluation of multiple NLPre tools while credibly tracking their performance. |
| Outcome: | The proposed system is configured for Polish and integrated with the thoroughly assembled NLPre-PL benchmark. |
Language Models as Knowledge Bases? (D19-1)
Copied to clipboard
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller
| Challenge: | Recent advances in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. |
| Approach: | They present a method for pretraining language models on large textual corpora . they find that they can store relational knowledge and answer queries structured as "fill-in-the-blank" queries. |
| Outcome: | The proposed language models can recall factual knowledge without fine-tuning without fine tuning . the proposed models can answer queries structured as "fill-in-the-blank" cloze statements . |
IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020.findings-emnlp)
Copied to clipboard
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, Pratyush Kumar
| Challenge: | In this paper, we present NLP resources for 11 major Indian languages . distributional representations are the cornerstone of modern NLP, authors say . |
| Approach: | They introduce NLP resources for 11 major Indian languages from two major language families . monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . they also compile a benchmark for Indian language NLU to evaluate their results . |
| Outcome: | The monolingual corpora contains 8.8 billion tokens across all 11 languages and Indian English . the pre-trained language models are based on the compact ALBERT model . |