PIRB: A Comprehensive Benchmark of Polish Dense and Hybrid Text Retrieval Methods (2024.lrec-main)
Copied to clipboard
| Challenge: | PIRB is a framework for text information retrieval in Polish . existing and new datasets are evaluated to evaluate the performance of 41 models . |
| Approach: | They propose a framework for 41 text information retrieval tasks in Polish . they evaluate over 20 dense and sparse retrieval models and build sparser-dense hybrid retrievers . |
| Outcome: | The proposed framework outperforms the best available methods in 41 tasks for Polish . the proposed models outperformed the best solutions available to date . |
Similar Papers
Evaluation of Transfer Learning for Polish with a Text-to-Text Model (2022.lrec-1)
Copied to clipboard
Aleksandra Chrabrowa, Łukasz Dragan, Karol Grzegorczyk, Dariusz Kajtoch, Mikołaj Koszowski, Robert Mroczkowski, Piotr Rybak
| Challenge: | Recent years have brought significant progress in natural language understanding (NLU) and natural language generation (NLG). |
| Approach: | They propose a benchmark for assessing the quality of text-to-text models for Polish . they evaluate the performance of plT5, mT5, Polish BART, and Polish GPT-2 . |
| Outcome: | The proposed model can be fine-tuned on various NLP tasks with a single training objective. |
BEIR-PL: Zero Shot Information Retrieval Benchmark for the Polish Language (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing multilingual evaluation benchmarks focus on IR in the Polish language, but the Polish is a relatively new field due to the limited availability of Polish datasets. |
| Approach: | They propose to establish large-scale resources for IR in the Polish language and translate them into a new benchmark which includes 13 datasets. |
| Outcome: | The proposed benchmarks are based on 13 open IR datasets in Polish and are a pioneering development in this area. |
KLEJ: Comprehensive Benchmark for Polish Language Understanding (2020.acl-main)
Copied to clipboard
| Challenge: | Recent introduction of robust, general-purpose models for fine-tuning has enabled improvements in general natural language understanding (NLU) but such benchmarks are only available for a handful of languages. |
| Approach: | They propose a multi-task benchmark for the Polish language understanding with an online leaderboard . they also propose GLUE, a task for named entity recognition and sentiment analysis . |
| Outcome: | The proposed model performs best on three out of nine tasks in the Polish language . the proposed model is also used in an e-commerce domain to analyze the sentiments of users . |
Developing PUGG for Polish: A Modern Approach to KBQA, MRC, and IR Dataset Construction (2024.findings-acl)
Copied to clipboard
Albert Sawczyn, Katsiaryna Viarenich, Konrad Wojtasik, Aleksandra Domogała, Marcin Oleksy, Maciej Piasecki, Tomasz Kajdanowicz
| Challenge: | Existing KBQA datasets are outdated and inefficient in human labor, and assisting tools like Large Language Models (LLM) are not utilized to reduce the workload. |
| Approach: | They propose a semi-automated question answering task that uses structured knowledge graphs to answer extensive knowledge-intensive questions. |
| Outcome: | The proposed approach includes KBQA, MRC, and Information Retrieval tasks for low-resource languages. |
Evaluation of Sentence Representations in Polish (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for learning sentence representations have been limited in low-resource languages such as Polish . |
| Approach: | They propose two new Polish datasets for evaluating sentence embeddings and evaluate eight different methods including Polish and multilingual models. |
| Outcome: | The proposed methods show strengths and weaknesses in Polish and multilingual models. |
PL-MTEB: Polish Massive Text Embedding Benchmark (2026.findings-acl)
Copied to clipboard
| Challenge: | Text embeddings are used in many NLP tasks, including document clustering, semantic search, question answering, and classification. |
| Approach: | They introduce the Polish Massive Text Embedding Benchmark (PL-MTEB) it is a comprehensive benchmark for text embeddings in the Polish language. |
| Outcome: | The proposed model is based on 30 different NLP tasks in the Polish language. |
Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multilingual and cross-lingual retrieval are lacking in low-resource, morphologically rich languages such as Amharic. |
| Approach: | They propose to train Amharic-specific dense retrieval models based on pre-trained Amharican BERT and RoBERTa backbones. |
| Outcome: | The proposed model achieves 17.6% improvement in MRR@10 and 9.86% gain in Recall@10 over the strongest multilingual baseline, Arctic Embed 2.0. |
Silver Retriever: Advancing Neural Passage Retrieval for Polish Question Answering (2024.lrec-main)
Copied to clipboard
| Challenge: | lexical approaches to find passages have outperformed lexicals due to their superior performance . however, for some languages, such as Polish, few models are available . a recent study shows that neural retrievers are more efficient and efficient than lexica. |
| Approach: | They present a neural retriever for Polish trained on a diverse collection of manual and weakly labeled datasets. |
| Outcome: | The proposed model outperforms lexical retrieval models in Polish on three retrieval tasks. |
PolQA: Polish Question Answering Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent proposed systems for open-domain question answering (OpenQA) require large amounts of training data to achieve state-of-the-art performance. |
| Approach: | They propose an efficient annotation strategy that increases passage retrieval accuracy@10 by 10.55 p.p. while reducing the annotation cost by 82%. |
| Outcome: | The proposed approach increases passage retrieval accuracy @10 by 10.55 p.p. while reducing the annotation cost by 82%. |
PLLuM-Align: Polish Preference Dataset for Large Language Model Alignment (2025.emnlp-main)
Copied to clipboard
Karolina Seweryn, Anna Kołos, Agnieszka Karlińska, Katarzyna Lorenc, Katarzyna Dziewulska, Maciej Chrabaszcz, Aleksandra Krasnodebska, Paula Betscher, Zofia Cieślińska, Katarzyna Kowol, Julia Moska, Dawid Motyka, Paweł Walkowiak, Bartosz Żuk, Arkadiusz Janz
| Challenge: | Large language models generate preferred responses while avoiding harmful or inappropriate outputs, despite their ability to generate cross-language transferability. |
| Approach: | They introduce the first Polish preference dataset PLLuM-Align, created entirely through human annotation to reflect Polish language and cultural nuances. |
| Outcome: | The proposed dataset lays the groundwork for more aligned Polish LLMs and contributes to the broader goal of multilingual alignment in underrepresented languages. |