A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution (2023.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art natural language processing models have been shown to achieve remarkable performance in ‘closed-world’ settings where all the labels in the evaluation set are known at training time. |
| Approach: | They propose a multi-stage task to evaluate a system's performance on pipelined novelty ‘detection’ and ‘accommodation’ tasks. |
| Outcome: | The proposed model performs poorly in ‘closed-world’ settings where all the labels in the evaluation set are known at training time. |
Similar Papers
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback (2026.eacl-long)
Copied to clipboard
| Challenge: | Novelty assessment is a central yet understudied aspect of peer review . manuscript submissions double roughly every 15 years, and individual reviewers now complete an average of 14 reviews per year. |
| Approach: | They propose a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages: content extraction, retrieval and synthesis of related work, and structured comparison for evidence-based assessment. |
| Outcome: | The proposed approach outperforms existing LLM-based baselines on 182 ICLR 2025 submissions with human-annotated reviewer novelty assessments. |
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for evaluating novelty have been proposed, but there is no systematic evaluation of their ability to generate novelty evaluations. |
| Approach: | They propose a benchmark to evaluate large language models’ ability to generate novelty evaluations in support of human peer review. |
| Outcome: | The proposed framework evaluates the quality of LLM-generated novelty evaluations under different prompting strategies. |
TAP-DLND 1.0 : A Corpus for Document Level Novelty Detection (L18-1)
Copied to clipboard
| Challenge: | Detecting novelty of an entire document is an AI frontier problem . present state-of-the-art text matching techniques are unable to process such redundancy. |
| Approach: | They propose a document-level novelty detection resource that can be used to benchmark techniques . they crawl news documents across several domains and use it to find out whether they contain new information . |
| Outcome: | The proposed dataset is compared with a standard system for document novelty detection . the proposed system can detect elements that have not appeared before, or new or original . |
NLP-ADBench: NLP Anomaly Detection Benchmark (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Anomaly detection (AD) is an important machine learning task, but its effectiveness in detecting harmful content, phishing attempts, and spam reviews is limited. |
| Approach: | They introduce NLP-ADBench, the most comprehensive NLP anomaly detection benchmark to date . it includes eight curated datasets and 19 state-of-the-art algorithms . |
| Outcome: | The NLP-ADBench benchmark includes 19 state-of-the-art methods and 8 curated datasets . no single model dominates across all datasets, indicating need for automated model selection . |
AttributionBench: How Hard is Automatic Attribution Evaluation? (2024.findings-acl)
Copied to clipboard
| Challenge: | generative search engines enhance the reliability of large language model responses by providing cited evidence. |
| Approach: | They propose to use a benchmark to evaluate whether a large language model supports the generated responses or not . |
| Outcome: | The proposed benchmark shows that even a fine-tuned GPT-3.5 only achieves around 80% macro-F1 under a binary classification formulation. |
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking (2024.eacl-demo)
Copied to clipboard
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali
| Challenge: | Recent development and success of Large Language Models necessitate evaluation of their performance across diverse NLP tasks in different languages. |
| Approach: | They propose a framework that can be customized to evaluate LLMs for any NLP task, regardless of language. |
| Outcome: | The LLMeBench framework can be customized to evaluate LLMs for any NLP task, regardless of language. |
Evaluating Research Novelty Detection: Counterfactual Approaches (D19-53)
Copied to clipboard
| Challenge: | Despite its importance, this direction of research has not been explored as much. |
| Approach: | They propose to use counterfactual simulations to evaluate paper novelty detection models . they ask models to differentiate papers at time t and counterf actual paper from future time . |
| Outcome: | The proposed models can be compared against a set of papers with a given date and with different annotations. |
TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in generative language models have enabled machines to generate realistic texts. |
| Approach: | They propose a benchmark environment to test the 'Turing Test' problem for neural text generation methods. |
| Outcome: | The proposed benchmark environment is based on 200K human- or machine-generated samples across 20 labels Human, GPT-1, GTP-2_small, GTT-2_medium, GPG-2_large, GGT-2_PyTorch, GGP-3, GROVER_base, griover_large and GRover_mega. |
Semantic Novelty Detection and Characterization in Factual Text Involving Named Entities (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic-based novelty detection methods do not perform semantic reasoning involving relations between named entities in text and their background knowledge. |
| Approach: | They propose a model to detect whether a text is novel or not . they propose to use a factual text to characterize novelty. |
| Outcome: | The proposed model outperforms 10 baselines by large margins on the novelty detection task. |
Automatic Evaluation of Attribution by Large Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Generative large language models (LLMs) incorporate external references to generate and support claims. however, evaluating the attribution remains an open problem. |
| Approach: | They investigate automatic evaluation of attribution given by large language models . they define different types of attributed errors and then explore two approaches . |
| Outcome: | The proposed methods highlight promising signals and challenges. |