Papers by Sampo Pyysalo
CRAFT Shared Tasks 2019 Overview — Integrated Structure, Semantics, and Coreference (D19-57)
Copied to clipboard
William Baumgartner, Michael Bada, Sampo Pyysalo, Manuel R. Ciosici, Negacy Hailu, Harrison Pielke-Lombardo, Michael Regan, Lawrence Hunter
| Challenge: | CRAFT corpus provides a unique foundation for integrating natural language processing (NLP) tasks involving structure, semantics, and coreference. |
| Approach: | They propose to use the CRAFT corpus to evaluate three fundamental language processing tasks over full-text biomedical articles. |
| Outcome: | The CRAFT corpus provides a unique foundation for integrating natural language processing tasks involving structure, semantics, and coreference. |
Exploring Cross-sentence Contexts for Named Entity Recognition with BERT (2020.coling-main)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is often addressed as a sequence classification task with each input consisting of one sentence of text. |
| Approach: | They propose a method to combine different predictions from multiple sentences in input samples to increase NER performance. |
| Outcome: | The proposed method improves on the state-of-the-art NER results on English, Dutch, and Finnish and achieves the best reported BERT-based results on German. |
A New Massive Multilingual Dataset for High-Performance Language Technologies (2024.lrec-main)
Copied to clipboard
Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen, Jörg Tiedemann
| Challenge: | a new massive multilingual dataset is available for language modeling and machine translation training. |
| Approach: | They present a massive multilingual dataset using web crawls from the Internet Archive and CommonCrawl . they use open-source software tools and high-performance computing to acquire, manage and process large corpora . |
| Outcome: | The HPLT language resources is a massive multilingual dataset . it includes monolingual and bilingual corpora extracted from CommonCrawl and the Internet Archive . the results are published online at the journal journal cense4 . |
Silver Syntax Pre-training for Cross-Domain Relation Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Relation Extraction (RE) is the task of extracting structured knowledge from unstructured text. |
| Approach: | They exploit the affinity between syntactic structure and semantic RE to obtain low-cost pre-training data. |
| Outcome: | The proposed model outperforms baseline models in five out of six cross-domain setups without additional annotated data. |
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code (2025.coling-industry)
Copied to clipboard
Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Barbosa Junior, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Nour Moustafa-Fahmy, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Hiep Nguyen, Sampo Pyysalo
| Challenge: | Pretrained language models are integral part of AI applications, but their high computational cost limits accessibility. |
| Approach: | They evaluate Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
| Outcome: | The proposed model outperforms existing models on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers (2021.eacl-srw)
Copied to clipboard
Liina Repo, Valtteri Skantsi, Samuel Rönnqvist, Saara Hellström, Miika Oinonen, Anna Salmela, Douglas Biber, Jesse Egbert, Sampo Pyysalo, Veronika Laippala
| Challenge: | Existing studies on register classification for web documents have limited results due to skewed datasets and low performance. |
| Approach: | They propose two new register-annotated corpora for French and Swedish . they show that deep pre-trained language models perform strongly in these languages . |
| Outcome: | The proposed models outperform existing models in English and Finnish and can match or surpass existing models. |
An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT) (2025.acl-long)
Copied to clipboard
Laurie Burchell, Ona De Gibert Bonet, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, Amanda Myntti, Dayyán O’Brien, Stephan Oepen, Proyag Pal, Jousia Piha, Sampo Pyysalo, Gema Ramírez-Sánchez, David Samuel, Pavel Stepachev, Jörg Tiedemann, Dušan Variš, Tereza Vojtěchová, Jaume Zaragoza-Bernabeu
| Challenge: | a large number of textual data is needed to train state-of-the-art large language models. |
| Approach: | They propose a collection of monolingual and parallel corpora from the Internet Archive . they document the entire data pipeline and release the code to reproduce it . |
| Outcome: | The proposed collection of monolingual and parallel corpora is based on the HPLT v2 dataset . it includes 8T tokens covering 193 languages and 380M sentence pairs covering 51 languages . |
Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection (2020.lrec-1)
Copied to clipboard
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, Daniel Zeman
| Challenge: | Universal Dependencies is an open community effort to create cross-linguistically consistent treebank annotation for many languages. |
| Approach: | They describe version 2 of the universal guidelines and discuss major changes from UD v1 to UD 2 . they propose a morphological layer, a syntactic layer and a word segmentation layer . |
| Outcome: | The proposed treebanks are available for 90 languages and have been updated to meet the needs of multilingual parsers and researchers. |
Building Question-Answer Data Using Web Register Identification (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in web register (genre) identification have created a shortage of QA datasets for English and Finnish. |
| Approach: | They propose a machine learning-based method for extracting QA pairs from web-scale data using XLM-R and a multilingual CORE web register corpus . they then develop a NER-style token classifier to identify the QA text spans within these documents. |
| Outcome: | The proposed method is adaptable to any language given the availability of language models and extensive web data, but it is limited to English and Finnish. |
A Broad-coverage Corpus for Finnish Named Entity Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a fundamental task in natural language processing (NLP). |
| Approach: | They propose to annotate Finnish named entity names using a new corpus built on the Universal Dependencies corpus. |
| Outcome: | The new annotation identifies over 10,000 mentions and maintains compatibility with a previously released single-domain corpus for Finnish NER. |
The birth of Romanian BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale pretrained language models are available in high-resource languages, in particular English, or as multilingual models that compromise performance on individual languages for coverage. |
| Approach: | They propose to use a Romanian transformer-based language model to pretrained a large text corpus to evaluate the model. |
| Outcome: | The proposed model is open-source and can be used in production. |
FinGPT: Large Generative Models for a Small Language (2023.emnlp-main)
Copied to clipboard
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, Sampo Pyysalo
| Challenge: | Neural language models excel in many tasks in NLP but are limited to smaller languages. |
| Approach: | They propose two approaches to pretrain large language models for Finnish . they train seven monolingual models from scratch and use Finnish as pretraining data . |
| Outcome: | The proposed model is based on a dataset of Finnish web crawls, news, social media and eBooks. |
Neural Dependency Parsing of Biomedical Text: TurkuNLP entry in the CRAFT Structural Annotation Task (D19-57)
Copied to clipboard
| Challenge: | Syntactic analysis (parsing) is a fundamental task in natural language processing (NLP). |
| Approach: | They propose to use the Turku neural parser to adapt it to the biomedical domain . they evaluated custom word embeddings, combination with other in-domain resources . |
| Outcome: | The proposed approach achieved a labeled attachment score of 89.7%, the best among task participants. |
Biomedical Named Entity Recognition with Multilingual BERT (D19-57)
Copied to clipboard
| Challenge: | a multilingual model is not specifically tailored to either the language nor the application domain. |
| Approach: | They propose a CRF-based baseline approach and multilingual BERT to the task . they achieve an F-score of 88% on the development data and 87% on the test set with BERT . |
| Outcome: | The proposed model achieves an F score of 88% on the development data and 87% on the test set with BERT. |