Papers by Dmitry Ustalov
Reliable, Reproducible, and Really Fast Leaderboards with Evalica (2025.coling-demos)
Copied to clipboard
| Challenge: | Using open-source evaluation tools, we create reliable and reproducible model leaderboards with human and machine feedback. |
| Approach: | They propose an open-source evaluation toolkit that facilitates the creation of reliable and reproducible model leaderboards. |
| Outcome: | The evaluation tool facilitates the creation of reliable and reproducible model leaderboards. |
Confidence and Stability of Global and Pairwise Scores in NLP Evaluation (2025.acl-srw)
Copied to clipboard
| Challenge: | Modern natural language processing benchmarks are often represented as pairwise comparison leaderboards, such as LMSYS Arena. |
| Approach: | They investigate the strengths and weaknesses of global scores and pairwise comparisons to aid decision-making in selecting appropriate model evaluation strategies. |
| Outcome: | The proposed method underestimates strong models with rare errors or low confidence, while relying on global scores can be more effective. |
Improving Hypernymy Extraction with Distributional Semantic Classes (L18-1)
Copied to clipboard
| Challenge: | Existing methods for extracting hypernyms focus on the acquisition of binary hypernies . |
| Approach: | They propose a distributionally-induced semantic class for extracting hypernyms . they also use distributional semantics to induce sense-aware semantic classes . |
| Outcome: | The proposed method improves the quality of the hypernymy extraction in terms of precision and recall. |
Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on crowdsourcing for efficient data annotation will introduce crowdsourcing and provide an overview of the technology. |
| Approach: | This tutorial will introduce users to efficient data annotation via crowdsourcing marketplaces. |
| Outcome: | This tutorial will introduce users to the use of crowdsourcing for data annotation. |
Unsupervised Semantic Frame Induction using Triclustering (P18-2)
Copied to clipboard
| Challenge: | Recent work on frame-semantics has enabled the development of wide-coverage frame parsers using supervised learning. |
| Approach: | They propose to use dependency triples to perform unsupervised frame induction on a Web-scale corpus. |
| Outcome: | The proposed approach performs state-of-the-art on a FrameNet-derived dataset and performs on par with competitive methods on . verb class clustering task. |
TextGraphs 2019 Shared Task on Multi-Hop Inference for Explanation Regeneration (D19-53)
Copied to clipboard
| Challenge: | Detailed extended analyses of all submitted systems showed large relative improvements in accessing the most challenging multi-hop inference problems, while absolute performance remains low. |
| Approach: | The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to regenerate detailed gold explanations for elementary science questions by selecting facts from a knowledge base of semi-structured tables. |
| Outcome: | The top-performing system achieved a mean average precision of 0.56 . the task combines facts from a knowledge base and supervised training data . |
An Unsupervised Word Sense Disambiguation System for Under-Resourced Languages (L18-1)
Copied to clipboard
Dmitry Ustalov, Denis Teslenko, Alexander Panchenko, Mikhail Chernoskutov, Chris Biemann, Simone Paolo Ponzetto
| Challenge: | Existing systems for word sense disambiguation are limited to the Russian language and lack of resources to address the problem. |
| Approach: | They propose an unsupervised system for word sense disambiguation that uses a traditional vector space model to estimate the most similar word sense corresponding to its context. |
| Outcome: | The proposed system outperforms the sparse mode on all datasets according to the adjusted Rand index. |
Word Sense Disambiguation for 158 Languages using Word Embeddings Only (2020.lrec-1)
Copied to clipboard
Varvara Logacheva, Denis Teslenko, Artem Shelmanov, Steffen Remus, Dmitry Ustalov, Andrey Kutuzov, Ekaterina Artemova, Chris Biemann, Simone Paolo Ponzetto, Alexander Panchenko
| Challenge: | Existing methods of disambiguation of word senses are based on knowledge bases, taxonomies, and other externally built resources. |
| Approach: | They propose a method that takes a pre-trained word embedding model and induces a fully-fledged word sense inventory for 158 languages. |
| Outcome: | The proposed model is based on a pre-trained word embedding model and induces a fully-fledged word sense inventory in 158 languages. |