Papers by Ilia Kuznetsov
Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) have brought advances in text generation, but their potential for enhancing classification tasks remains underexplored. |
| Approach: | They propose a framework for thoroughly investigating fine-tuning LLMs for classification . they instantiate this framework in edit intent classification (EIC) a challenging and underexplored classification task. |
| Outcome: | The proposed framework is applied to edit intent classification (EIC) The proposed methods are generalizable on five further classification tasks. |
M2QA: Multi-domain Multilingual Question Answering (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Language varies along several axes, most importantly, language instance and domain . lack of evaluation datasets prevents transfer of NLP systems to non-dominant languages . |
| Approach: | They propose a multi-domain multilingual question answering benchmark to explore cross-lingual cross-domain performance of fine-tuned models and state-of-the-art LLMs. |
| Outcome: | The proposed benchmark compared 13,500 SQuAD 2.0-style question-answer instances in German, Turkish, and Chinese for the domains of product reviews, news, and creative writing. |
CiteBench: A Benchmark for Scientific Citation Text Generation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on citation text generation are based upon widely diverging task definitions, making it hard to study this task systematically. |
| Approach: | They propose a benchmark for citation text generation that unifies multiple datasets and enables standardized evaluation of citation texts across task designs and domains. |
| Outcome: | The proposed benchmark examines the performance of multiple strong baselines and enables standardized evaluation of citation text generation models across task designs and domains. |
Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision (2024.acl-long)
Copied to clipboard
| Challenge: | a framework for collaborative document revision is lacking for empirical analysis and NLP. |
| Approach: | They propose a framework for joint analysis of collaborative document revision that instantiates a corpus of aligned scientific paper revisions manually labeled according to their action and intent. |
| Outcome: | The proposed framework provides first empirical insights into collaborative document revision in the academic domain and assesses its capabilities. |
From Text to Lexicon: Bridging the Gap between Word Embeddings and Lexical Resources (C18-1)
Copied to clipboard
| Challenge: | Distributional word representations are omnipresent in modern NLP. |
| Approach: | They propose to combine lemmatization and part of speech (POS) typing to improve word embedding performance. |
| Outcome: | The proposed methods improve word embedding performance on verbs and verbs. |
A matter of framing: The impact of linguistic formalism on probing results (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextualized encoders have had a major impact on the field of natural language processing. |
| Approach: | They conduct an in-depth cross-formalism layer probing study in role semantics to investigate the linguistic knowledge implicitly learned by pre-trained contextualized encoders. |
| Outcome: | The proposed model outperforms pre-trained models on a range of downstream tasks. |
ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links (2026.eacl-long)
Copied to clipboard
| Challenge: | Using retrieval models and LLMs achieves a 73% approval rate for suggested links, more than doubling the acceptance of strong retrievers alone. |
| Approach: | They propose a domain-agnostic framework for bootstrapping sentence-level cross-document links from scratch and apply it to large-scale human-in-the-loop annotation of natural text pairs. |
| Outcome: | The proposed framework generates semi-synthetic datasets and uses them to benchmark and shortlist the best-performing methods and applies them in large-scale human-in-the-loop annotation of natural text pairs. |
Does My Rebuttal Matter? Insights from a Major NLP Conference (N19-1)
Copied to clipboard
| Challenge: | Peer review is a core element of the scientific process, but few studies have evaluated its properties empirically. |
| Approach: | They propose to use peer review to assess the effectiveness of rebuttal phase in NLP conferences. |
| Outcome: | The proposed task predicts after-rebuttal scores from initial reviews and author responses. |
NLPeer: A Unified Resource for the Computational Study of Peer Review (2023.acl-long)
Copied to clipboard
| Challenge: | Existing studies of peer review for scholarly publications lack datasets and multi-domain corpora to support this complex process. |
| Approach: | They propose to use NLPeer to build a multi-domain corpus of more than 5k papers and 11k review reports from five different venues to support reviewers. |
| Outcome: | The proposed datasets and analysis of three review assistance tasks include a guided skimming task. |
Document Structure in Long Document Transformers (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing long-document Transformers do not learn representations of document structure during pretraining. |
| Approach: | They propose to use long-document Transformers to acquire an internal representation of document structure during pre-training and evaluate the effects of structure infusion on QASPER and Evidence Inference. |
| Outcome: | The proposed models acquire implicit understanding of document structure during pre-training, which can be enhanced by structure infusion, leading to improved end-task performance. |
Yes-Yes-Yes: Proactive Data Collection for ACL Rolling Review and Beyond (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to data collection are under-resourced and can be difficult to implement in the peer review domain. |
| Approach: | They propose a donation-based peer review data collection workflow that takes into account ethical, legal and confidentiality-related aspects of data collection into account. |
| Outcome: | The proposed workflows are based on a donation-based peer review platform and show that the datasets are larger than the current workflows. |
STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work treats critical text assessment as a black box problem, limiting interpretability and human-AI collaboration. |
| Approach: | They propose a framework to model critical text assessment as an explicit, step-wise reasoning process. |
| Outcome: | The proposed framework breaks down assessment into a graph of interconnected reasoning steps drawing on causality theory. |
Systematic Task Exploration with LLMs: A Study in Citation Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) provide unprecedented flexibility in defining and executing complex, creative natural language generation tasks. |
| Approach: | They propose a framework that consists of input manipulation, reference data, and output measurement to explore citation text generation. |
| Outcome: | The proposed framework explores citation text generation, a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric and has not yet been tackled within the LLM paradigm. |
CARE: Collaborative AI-Assisted Reading Environment (2023.acl-demo)
Copied to clipboard
| Challenge: | Recent years have seen impressive progress in AI-assisted writing, yet the developments in AI assisted reading are lacking. |
| Approach: | They propose an open integrated platform for the study of inline commentary and reading. |
| Outcome: | The proposed platform is used in a scholarly peer review study and invites the community to build upon it. |
An Inclusive Notion of Text (2023.acl-long)
Copied to clipboard
| Challenge: | despite its central role, the notion of text in natural language processing is vague, authors argue . a conceptual framework for capturing text differences is lacking, authors say . authors propose a two-tier taxonomy of linguistic and non-linguistic elements available in textual sources . |
| Approach: | They propose a taxonomy of linguistic and non-linguistic elements available in textual sources and can be used in NLP modeling. |
| Outcome: | The proposed taxonomy examines the production and transformation of textual data . it outlines key desiderata and challenges of the emerging inclusive approach to text in NLP . |
Identifying Aspects in Peer Reviews (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to peer review are limited in how they identify aspects . a growing volume of peer review submissions is straining the process . |
| Approach: | They propose a data-driven schema for deriving aspects from peer reviews . they propose augmented peer reviews and show how it can be used for community-level review analysis. |
| Outcome: | The proposed approach can be used to support peer review, but lacks formal definition of aspect . it also shows that the choice of aspects can impact downstream applications . |