Training Text-to-Molecule Models with Context-Aware Tokenization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Text-to-molecule models have shown great potential across chemical applications . however, they rely on atom-level tokenizations, which limiting the ability of models to capture global structural context within molecules. |
| Approach: | They propose a text-to-molecule model that uses substructure-level tokenizations to model global connectivity. |
| Outcome: | The proposed model outperforms state-of-the-art models using only 2% of training tokens. |
Similar Papers
Bridging the Gap Between Molecule and Textual Descriptions via Substructure-aware Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models lack the ability to learn fine-grained alignments between molecules and their descriptions. |
| Approach: | They propose a molecule–text learning framework based on substructure-aware alignments that augments original molecule-description pairs with additional alignment signals derived from molecular substructures and chemical phrases. |
| Outcome: | The proposed framework outperforms state-of-the-art models on a wide range of molecular benchmarks. |
BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning (2024.findings-acl)
Copied to clipboard
Qizhi Pei, Lijun Wu, Kaiyuan Gao, Xiaozhuan Liang, Yin Fang, Jinhua Zhu, Shufang Xie, Tao Qin, Rui Yan
| Challenge: | BioT5+ is an extension of the BioT5, but lacked a nuanced understanding of molecular structures. |
| Approach: | They propose a new bio-entity modeling framework, BioT5+, which integrates IUPAC names and molecule data. |
| Outcome: | The proposed model bridges the gap between molecular representations and textual descriptions and improves the grounded reasoning of bio-text and bio-sequences. |
Translation between Molecules and Natural Language (2022.emnlp-main)
Copied to clipboard
| Challenge: | MolT5 pretrains models on unlabeled natural language text and molecule strings . bringing a new drug to market can cost over a billion dollars and take over ten years . |
| Approach: | They propose a self-supervised learning framework for pretraining models on unlabeled natural language text and molecule strings. |
| Outcome: | The proposed framework pretrains models on unlabeled natural language text and molecule strings, and it generates high quality outputs. |
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations (2023.emnlp-main)
Copied to clipboard
| Challenge: | et al., 2022) argue that the current models for drug discovery lack the ability to integrate molecules, proteins, and natural language. |
| Approach: | They propose a framework that integrates biological knowledge with chemical knowledge and natural language associations. |
| Outcome: | The proposed framework shows superior performance across a wide range of tasks. |
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2022.tacl-1)
Copied to clipboard
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel
| Challenge: | a number of pre-trained language models use sequences of tokens corresponding to word units . token-free models that operate directly on raw text have many advantages . |
| Approach: | They propose a standard Transformer architecture that can be used to process byte sequences . they also characterize trade-offs in terms of parameter count, training FLOPs, and inference speed . |
| Outcome: | The proposed model is more robust to noise and more robust on spelling and pronunciation tasks. |
Categorizing Semantic Representations for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Modern neural machine translation models suffer limitation in compositional generalization, resulting in weakened translation performance on unseen compounds. |
| Approach: | They propose to introduce categorization to the contextualized representations to improve generalization by reducing sparsity and overfitting. |
| Outcome: | The proposed method reduces compositional generalization error rates by 24% on a dedicated MT dataset. |
ReactXT: Understanding Molecular “Reaction-ship” via Reaction-Contextualized Molecule-Text Pretraining (2024.findings-acl)
Copied to clipboard
Zhiyuan Liu, Yaorui Shi, An Zhang, Sihang Li, Enzhi Zhang, Xiang Wang, Kenji Kawaguchi, Tat-Seng Chua
| Challenge: | Molecular-text modeling is an emerging research field that aims to facilitate molecule-relevant tasks with a textual interface and textual knowledge. |
| Approach: | They propose a new method for reaction-text modeling that uses three types of input contexts to incrementally pretrain LMs. |
| Outcome: | The proposed method improves experimental procedure prediction and molecule captioning and offers competitive results in retrosynthesis. |
CrystalICL: Enabling In-Context Learning for Crystal Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for crystal generation are limited to zero-shot scenarios and are unable to benefit from few-shot situations. |
| Approach: | They propose a model designed for few-shot crystal generation that exploits in-context learning by capturing structure-property relationships from limited data. |
| Outcome: | The proposed model reduces complexity of modeling crystal symmetry in LLMs and exploits ICL by capturing structure-property relationships from limited data. |
Lost in Translation: Chemical Language Models and the Misunderstanding of Molecule Structures (2024.findings-emnlp)
Copied to clipboard
Veronika Ganeeva, Andrey Sakhovskiy, Kuzma Khrabrov, Andrey Savchenko, Artur Kadurin, Elena Tutubalina
| Challenge: | chemistry and natural language processing (NLP) have advanced drug discovery. |
| Approach: | They propose a framework for assessment of Chemistry LMs of different natures that relies on augmentations that preserve an underlying chemical. |
| Outcome: | The proposed framework relies on augmentations that preserve an underlying chemical, such as kekulization and cycle replacements. |
Learn Your Tokens: Word-Pooled Tokenization for Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Language models typically tokenize text into subwords, using a deterministic, hand-engineered heuristic of combining characters into longer surface-level strings such as ‘ing’ or whole words. |
| Approach: | They propose a 'learn your tokens' scheme which pooles bytes/characters into word representations and decodes individual characters/bytes per word in parallel. |
| Outcome: | The proposed tokenizer outperforms subword models and byte/character models over the word boundary and outperformed on rare words by a factor of 30! |