Papers by Sergey Petrakov
LM-Polygraph: Uncertainty Estimation for Language Models (2023.emnlp-demo)
Copied to clipboard
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Large language models often "hallucinate" i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements. |
| Approach: | They propose a framework with implementations of state-of-the-art UE methods for LLMs with unified program interfaces in Python. |
| Outcome: | The proposed framework implements state-of-the-art UE methods for LLMs with unified program interfaces in Python and an extendable benchmark for consistent evaluation by researchers. |
Efficient Out-of-Domain Detection for Sequence to Sequence Models (2023.findings-acl)
Copied to clipboard
Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexander Panchenko, Artem Shelmanov
| Challenge: | Sequence-to-sequence (seq2sequ) models are a ubiquitous tool for text generation but they are not suitable for many other tasks. |
| Approach: | They propose to use UE techniques to identify out-of-domain (OOD) inputs where the model is susceptible to errors. |
| Outcome: | The proposed methods outperform heavyweight ensembles on the task of OOD detection. |
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification (2024.findings-acl)
Copied to clipboard
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
| Challenge: | Large language models are notorious for producing erroneous claims in their output. |
| Approach: | They propose a fact-checking and hallucination detection pipeline based on token-level uncertainty quantification that removes the impact of uncertainty about what claim to generate on the current step and what surface form to use. |
| Outcome: | The proposed method can fact-check the atomic claims in the output of large language models. |