Papers by Artem Shelmanov

Efficient Hallucination Detection in Automatic Code Generation (2026.findings-acl)

Challenge:	Large language models generate fluent responses to user queries, but they are also susceptible to misuse in journalism, education, and academia.
Approach:	They propose a large-scale benchmark for machine-generated text detection that is a multi-generator, multi-domain, and multi-lingual corpus.
Outcome:	The proposed system can detect machine-generated text and pinpoint misuse . the proposed system is based on a large-scale benchmark dataset .

Copied to clipboard

Georgii Andriushchenko, Roman Garaev, Lyudmila Rvanova, Artem Shelmanov, Vladimir V. Ivanov

ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning (2026.acl-demo)

Challenge:	Large language models produce source code that appears correct and well-formed, but includes hallucinated elements that cause downstream test failures.
Approach:	They develop a transformer-based detector that uses LLM internal representations to identify hallucinations.
Outcome:	The proposed detector outperforms existing methods and unsupervised methods in the code generation domain.

Copied to clipboard

Vladislav Smirnov, Quang-Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Mrinmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, Artem Shelmanov

Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models (2025.naacl-long)

Challenge:	Existing TTC scaling strategies and reasoning scorers are fragmented and evaluated under inconsistent protocols.
Approach:	They propose a framework for seamless test-time compute scaling of large language model reasoning . they use a modular Python library to implement state-of-the-art scaling strategy and scorer families .
Outcome:	The proposed framework evaluates performance and computational efficiency on mathematical and coding tasks.

Copied to clipboard

Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov

How Certain is Your Transformer? (2021.eacl-main)

Challenge:	Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs).
Approach:	They propose to use a well-established method for text generation to extract token embeddings from multiple layers of LLMs and compute MD scores for each token.
Outcome:	The proposed method improves on existing methods and provides accurate and computationally efficient uncertainty scores for sequence-level selective generation and claim-level fact-checking tasks.

Copied to clipboard

Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, Maxim Panov

M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection (2024.acl-long)

Challenge:	Obtaining reliable uncertainty estimations for such neural networks (NNs) is challenging due to the huge number of parameters in these deep learning models.
Approach:	They propose to use Monte Carlo dropout to estimate uncertainty for Transformer-based models and construct inexpensive estimates using Determinantal Point Processes.
Outcome:	The proposed estimates improve the quality of detection of error-prone instances.

Copied to clipboard

Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov

Hybrid Uncertainty Quantification for Selective Text Classification in Ambiguous Tasks (2023.acl-long)

Challenge:	Large Language Models (LLMs) have brought an unprecedented surge in machine-generated text (MGT) societal implications are posed by their potential misuse and lack of training data.
Approach:	They propose a benchmark to detect machine-generated text in multiple languages . they use multi-domain and multi-generator corpus to identify which model generated the text .
Outcome:	The proposed benchmark compares a multilingual, multi-domain and multi-generator corpus of MGTs with human-generated content.

Copied to clipboard

Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, Artem Shelmanov

LM-Polygraph: Uncertainty Estimation for Language Models (2023.emnlp-demo)

Challenge:	Existing methods for text classification tasks are inherently ambiguous and can cause errors.
Approach:	They propose a method that combines epistemic and aleatoric uncertainty to estimate toxicity detection errors.
Outcome:	The proposed method outperforms existing methods for toxicity detection and other ambiguous text classification tasks.

Copied to clipboard

Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov

Uncertainty Estimation of Transformer Predictions for Misclassification Detection (2022.acl-long)

Challenge:	Large language models often "hallucinate" i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements.
Approach:	They propose a framework with implementations of state-of-the-art UE methods for LLMs with unified program interfaces in Python.
Outcome:	The proposed framework implements state-of-the-art UE methods for LLMs with unified program interfaces in Python and an extendable benchmark for consistent evaluation by researchers.

Copied to clipboard

Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, Leonid Zhukov

Uncertainty Quantification for Large Language Models (2025.acl-tutorials)

Challenge:	Uncertainty estimation (UE) of model predictions is crucial step for a variety of tasks such as active learning, misclassification detection, adversarial attack detection, etc.
Approach:	They propose to modify UE methods for Transformer models for misclassification detection in named entity recognition and text classification tasks to improve model expressiveness and computational performance.
Outcome:	The proposed methods outperform computationally intensive methods on misclassification detection tasks and are based on a large dataset of simulated datasets.

Copied to clipboard

Artem Shelmanov, Maxim Panov, Roman Vashurin, Artem Vazhentsev, Ekaterina Fadeeva, Timothy Baldwin

Efficient Out-of-Domain Detection for Sequence to Sequence Models (2023.findings-acl)

Challenge:	Large language models (LLMs) produce hallucinations, which undermine user trust and reliability.
Approach:	This tutorial offers the first systematic introduction to uncertainty quantification (UQ) for LLMs in text generation tasks.
Outcome:	The proposed framework provides tools for communicating the reliability of a model answer.

Copied to clipboard

Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexander Panchenko, Artem Shelmanov

Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models (2025.emnlp-main)

Challenge:	Sequence-to-sequence (seq2sequ) models are a ubiquitous tool for text generation but they are not suitable for many other tasks.
Approach:	They propose to use UE techniques to identify out-of-domain (OOD) inputs where the model is susceptible to errors.
Outcome:	The proposed methods outperform heavyweight ensembles on the task of OOD detection.

Copied to clipboard

Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Gleb Kuzmin, Ivan Lazichny, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov, Artem Shelmanov

A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs (2025.emnlp-main)

Challenge:	Uncertainty quantification (UQ) is a promising approach for detecting hallucinations and low-quality outputs of Large Language Models (LLMs).
Approach:	They propose to learn conditional dependency between autoregressive LLM generation steps from attention-based features and a two-staged training procedure to incorporate recurrent features.
Outcome:	The proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.

Copied to clipboard

Artem Shelmanov, Ekaterina Fadeeva, Akim Tsvigun, Ivan Tsvigun, Zhuohan Xie, Igor Kiselev, Nico Daheim, Caiqi Zhang, Artem Vazhentsev, Mrinmaya Sachan, Preslav Nakov, Timothy Baldwin

Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI (2026.acl-long)

Challenge:	Uncertainty quantification (UQ) is a framework for assessing the reliability of model outputs.
Approach:	They introduce pre-trained UQ heads for LLMs that are highly robust and generalized to languages they were not explicitly trained on.
Outcome:	The pre-trained heads significantly improve their ability to capture uncertainty compared to unsupervised methods.

Copied to clipboard

Yuxia Wang, Rui Xing, Jonibek Mansurov, Giovanni Puccetti, Zhuohan Xie, Minh Ngoc Ta, Jiahui Geng, Jinyan Su, Mervat Abassy, Saadeldine Eletter, Kareem Elozeiri, Nurkhan Laiyk, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Alexander Aziz, Ryuto Koike, Masahiro Kaneko, Artem Shelmanov, Ekaterina Artemova, Vladislav Mikhailov, Akim Tsvigun, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, Preslav Nakov

Semantic Text Processing

Challenge:	Prior studies have shown that distinguishing text generated by Large Language Models from human-written text is challenging for humans and often no better than random guessing.
Approach:	They conduct extensive case study to determine the upper bound of human detection accuracy.
Outcome:	The findings challenge previous conclusions on human detection accuracy across languages and domains.

LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection (2024.emnlp-demo)

Copied to clipboard

Mervat Abassy, Kareem Elozeiri, Alexander Aziz, Minh Ta, Raj Tomar, Bimarsha Adhikari, Saad Ahmed, Yuxia Wang, Osama Mohammed Afzal, Zhuohan Xie, Jonibek Mansurov, Ekaterina Artemova, Vladislav Mikhailov, Rui Xing, Jiahui Geng, Hasan Iqbal, Zain Mujahid, Tarek Mahmoud, Akim Tsvigun, Alham Aji, Artem Shelmanov, Nizar Habash, Iryna Gurevych, Preslav Nakov

Challenge:	a large number of machine-generated texts are often hard to distinguish between human-written and machine-generated text . this raises concerns about potential misuse, especially within educational and academic domains .
Approach:	They propose a system that can detect whether a text is human-written or machine-generated . they use a fine-grained classification schema to identify the use of machine-generated text .
Outcome:	The proposed system can distinguish between human-written and machine-generated text . it can detect attempts to obfuscate the fact that a text was machine- generated .

Medical Crossing: a Cross-lingual Evaluation of Clinical Entity Linking (2022.lrec-1)

Copied to clipboard

Anton Alekseev, Zulfat Miftahutdinov, Elena Tutubalina, Artem Shelmanov, Vladimir Ivanov, Vladimir Kokh, Alexander Nesterov, Manvel Avetisian, Andrei Chertok, Sergey Nikolenko

Cross-Lingual Transfer

Challenge:	Existing approaches to medical entity linking are limited in terms of data volume and languages.
Approach:	They propose to use clinical reports, clinical guidelines, and medical research papers to evaluate cross-lingual medical entity linking.
Outcome:	The proposed model outperforms existing models on clinical reports, clinical guidelines, and medical research papers.

Entity Linking over Nested Named Entities for Russian (2022.lrec-1)

Copied to clipboard

Natalia Loukachevitch, Pavel Braslavski, Vladimir Ivanov, Tatiana Batura, Suresh Manandhar, Artem Shelmanov, Elena Tutubalina

Knowledge Representation

Challenge:	Entity linking is a popular NLP task, where a system needs to link a named entity to a concept in a knowledge base such as Wikidata.
Approach:	They describe the main design principles behind entity linking annotation in the recently released Russian NEREL dataset for information extraction.
Outcome:	The NEREL dataset is the largest Russian dataset annotated with entities and relations.

Active Learning for Abstractive Text Summarization (2022.findings-emnlp)

Copied to clipboard

Akim Tsvigun, Ivan Lysenko, Danila Sedashov, Ivan Lazichny, Eldar Damirov, Vladimir Karlov, Artemy Belousov, Leonid Sanochkin, Maxim Panov, Alexander Panchenko, Mikhail Burtsev, Artem Shelmanov

Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)

Challenge:	Abstractive text summarization (ATS) requires a long document and short summaries.
Approach:	They propose a query strategy for AL in abstractive text summarization that uses uncertainty estimation to reduce model performance.
Outcome:	The proposed query strategy improves ROUGE and consistency scores for annotated datasets . it also increases the performance of the model, compared to passive annotation.

Copied to clipboard

Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov

How to Compare Things Properly? A Study of Argument Relevance in Comparative Question Answering (2025.acl-long)

Challenge:	Large language models are notorious for producing erroneous claims in their output.
Approach:	They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use.
Outcome:	The proposed method can fact-check the atomic claims in the output of large language models.

Copied to clipboard

Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Artem Shelmanov, Chris Biemann

Natural Language Interfaces

Challenge:	Comparative Question Answering (CQA) is a task that involves processing information and diverse viewpoints.
Approach:	They construct a dataset of arguments annotated with their relevance and use it to answer comparative questions.
Outcome:	The proposed dataset contains arguments annotated with their relevance and enables precise traceability and faithfulness.

Active Learning for Sequence Tagging with Deep Pre-trained Models and Bayesian Uncertainty Estimates (2021.eacl-main)

Copied to clipboard

Artem Shelmanov, Dmitri Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, Alexander Panchenko

Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval-Augmented Generation (2026.findings-acl)

Challenge:	Annotating training data for sequence tagging of texts is usually very time-consuming . active learning can help to reduce the amount of annotation required to train a good model by multiple times .
Approach:	They are the first to thoroughly investigate active learning and transfer learning for natural language processing . they propose to combine active learning with active learning to improve model acquisition .
Outcome:	The proposed combination of active learning and Bayesian uncertainty estimation improves performance and reduces obstacles for applying it in practice.

Copied to clipboard

Ekaterina Fadeeva, Aleksandr Rubashevskii, Dzianis Piatrashyn, Roman Vashurin, Shehzaad Dhuliawala, Artem Shelmanov, Timothy Baldwin, Preslav Nakov, Mrinmaya Sachan, Maxim Panov

NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis (2021.emnlp-main)

Challenge:	Existing approaches to mitigating hallucinations conflate factuality with faithfulness to the retrieved evidence, incorrectly labeling factually correct statements as hallucinos . Existing methods to mitigate hallucinics rely on a lack of training data coverage, input ambiguity, and architectural constraints.
Approach:	They propose a method for hallucination detection in Large Language Models enhanced with knowledge retrieval based on faithfulness to the retrieved context.
Outcome:	The proposed method outperforms unsupervised UQ baselines, RAG-specific methods, and supervised classifiers across multiple tasks and LLMs.

Copied to clipboard

Nikolay Arefyev, Dmitrii Kharchev, Artem Shelmanov

Beyond Memorization: Extending Reasoning Depth with Recurrence, Memory and Test-Time Compute Scaling (2026.findings-acl)

Challenge:	Pre-training Masked Language Models (MLMs) on massive datasets is expensive, but it is performed for each domain or task individually and is resource-demanding.
Approach:	They propose a method for more efficient adaptation that focuses on predicting words with large weights of the Naive Bayes classifier trained for the task at hand.
Outcome:	The proposed method improves sentiment analysis by focusing on predicting words with large weights of the Naive Bayes classifier trained for the task at hand.

Copied to clipboard

Ivan Rodkin, Daniil Orel, Konstantin Smirnov, Arman Bolatov, Bilal Elbouardi, Besher Hassan, Yuri Kuratov, Aydar Bulatov, Preslav Nakov, Timothy Baldwin, Artem Shelmanov, Mikhail Burtsev

Inference-Time Selective Debiasing to Enhance Fairness in Text Classification Models (2025.naacl-short)

Challenge:	Reasoning is a core capability of large language models, yet how multi-step reasoning is learned and executed remains unclear.
Approach:	They evaluate how large language models learn multi-step reasoning without memorization . they find that most neural architectures trained from scratch can learn rule inference .
Outcome:	The proposed framework fails to solve a natural-language proxy task with high accuracy.

Copied to clipboard

Gleb Kuzmin, Neemesh Yadav, Ivan Smirnov, Timothy Baldwin, Artem Shelmanov

ATGen: A Framework for Active Text Generation (2025.acl-demo)

Challenge:	Several studies have investigated and promoted fairness, and a variety of definitions have been proposed to address this problem.
Approach:	They propose a selective debiasing method that removes bias from model predictions instead of discarding them at inference time.
Outcome:	The proposed method achieves better results than standard uncertainty quantification methods on text classification datasets with encoder-based classification models.

Copied to clipboard

Akim Tsvigun, Daniil Vasilev, Ivan Tsvigun, Ivan Lysenko, Talgat Bektleuov, Aleksandr Medvedev, Uliana Vinogradova, Nikita Severin, Mikhail Mozikov, Andrey Savchenko, Ilya Makarov, Grigorev Rostislav, Ramil Kuleev, Fedor Zhdanov, Artem Shelmanov

ALToolbox: A Set of Tools for Active Learning Annotation of Natural Language Texts (2022.emnlp-demos)

Challenge:	Despite the surging popularity of natural language generation tasks, the application of active learning (AL) to NLG has been limited.
Approach:	They propose a framework that bridges AL with text generation tasks and provides a unified platform for smooth implementation and benchmarking of novel AL strategies tailored to NLG tasks.
Outcome:	The proposed framework simplifies AL-empowered annotation in NLG tasks using both human annotators and automatic annotation agents based on large language models (LLMs).

Copied to clipboard

Akim Tsvigun, Leonid Sanochkin, Daniil Larionov, Gleb Kuzmin, Artem Vazhentsev, Ivan Lazichny, Nikita Khromov, Danil Kireev, Aleksandr Rubashevskii, Olga Shahmatova, Dmitry V. Dylov, Igor Galitskiy, Artem Shelmanov

Low-Resource NLP

Challenge:	Currently, the framework supports text classification, sequence tagging, and seq2seq tasks.
Approach:	They propose an open-source framework for active learning annotation in natural language processing that provides an easy-to-deploy GUI annotation tool directly in the Jupyter IDE.
Outcome:	The proposed framework reduces computational overhead and duration of AL iterations and increases annotated data reusability.

Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models (2026.acl-long)

Copied to clipboard

Jingwei Ni, Ekaterina Fadeeva, Tianyi Wu, Mubashara Akhtar, Jiaheng Zhang, Elliott Ash, Markus Leippold, Timothy Baldwin, See-Kiong Ng, Artem Shelmanov, Mrinmaya Sachan

RuCCoN: Clinical Concept Normalization in Russian (2022.findings-acl)

Challenge:	Existing verification approaches, such as Process Reward Models, are computationally expensive and limited to specific domains.
Approach:	They propose a transformer-based probe that uses internal states of frozen LLMs to estimate credibility of reasoning steps during generation.
Outcome:	The proposed probes match or exceed PRMs that are up to 810 larger.

Copied to clipboard

Alexandr Nesterov, Galina Zubkova, Zulfat Miftahutdinov, Vladimir Kokh, Elena Tutubalina, Artem Shelmanov, Anton Alekseev, Manvel Avetisian, Andrey Chertok, Sergey Nikolenko

Challenge:	a new dataset for clinical concept normalization in Russian is available for download . ehrs contains over 16,028 entity mentions manually linked to over 2,409 unique concepts .
Approach:	They present a dataset for clinical concept normalization in Russian manually annotated by medical professionals.
Outcome:	The proposed dataset contains 16,028 entity mentions manually linked to over 2,409 unique concepts from the Russian language part of the UMLS ontology.

Towards Computationally Feasible Deep Active Learning (2022.findings-naacl)

Copied to clipboard

Akim Tsvigun, Artem Shelmanov, Gleb Kuzmin, Leonid Sanochkin, Daniil Larionov, Gleb Gusev, Manvel Avetisian, Leonid Zhukov