| Challenge: | Impact assessment is an evolving area of research that aims at measuring and predicting the potential effects of projects or programs. |
| Approach: | They propose a framework for automatically assessing the impact of scientific research by identifying pertinent sections in project reports that indicate potential impacts. |
| Outcome: | The proposed method achieves accuracy scores up to 0.81 and is generalizable to scientific research from different domains and languages. |
Similar Papers
Beyond Citations: Corpus-based Methods for Detecting the Impact of Research Outcomes on Society (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for assessing the impact of research are ineffective for identifying impact beyond academia and text-based indicators beyond those that capture attention. |
| Approach: | They propose a deductive and inductive approach to categorize research impact categories using a corpus-based approach . they use a combination of deductive methods and machine learning to infer impact categories from project reports. |
| Outcome: | The proposed method predicts deductively and inductively derived impact categories with 76.39% accuracy and 78.81% accuracy. |
SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior work on scientific impact prediction has focused on citation counts and its variants, leaving limited evaluation of models’ capability to reason about other dimensions. |
| Approach: | They propose a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields. |
| Outcome: | The proposed model outperforms larger models and close-source models in a wide range of fields and measures of scientific impact across 19 fields. |
A Multi-level Annotated Corpus of Scientific Papers for Scientific Document Summarization and Cross-document Relation Discovery (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent studies have proposed to take advantage of the scientific paper's citation network to approach literature summarization. |
| Approach: | They propose to annotate related work sections, cite papers and sentences using machine readable data and an additional layer of papers citing the references. |
| Outcome: | The proposed corpus expands the existing data-set of related work sections and cites the papers cited in the related work section. |
Evaluating Scholarly Impact: Towards Content-Aware Bibliometrics (2021.emnlp-main)
Copied to clipboard
| Challenge: | Scientific, engineering, and technological (SET) innovations drive many positive advances in our modern economy, society, and life. |
| Approach: | They propose a new metric that uses the content of the paper as a source of distant-supervision to quantify how much the cited-node informs the citing-n node. |
| Outcome: | The proposed method achieves up to 103% improvement over the second-best method. |
Annotating Research Infrastructure in Scientific Papers: An NLP-driven Approach (2023.acl-industry)
Copied to clipboard
Seyed Amin Tabatabaei, Georgios Cheirmpos, Marius Doornenbal, Alberto Zigoni, Veronique Moore, Georgios Tsatsaronis
| Challenge: | a pipeline is used to identify, extract and link research infrastructure used in scientific publications. |
| Approach: | They propose a natural language processing pipeline for the identification, extraction and linking of Research Infrastructure (RI) used in scientific publications. |
| Outcome: | The proposed pipeline can be used to identify, extract and link research infrastructure used in scientific publications. |
Automating Qualitative Data Analysis with Large Language Models (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing methods for qualitative data analysis are far from resembling a human's analysis outcome. |
| Approach: | They propose a method based on Large Language Models to tackle automated coding and make it as close as possible to the results of human researchers. |
| Outcome: | The proposed method is based on large language models and can be as close as possible to the results of human researchers. |
Impact of Evaluation Methodologies on Code Summarization (2022.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation methodologies for code summarization tasks do not consider timestamps of code and comments. |
| Approach: | They propose a time-segmented evaluation methodology for code summarization that considers timestamps of code and comments during evaluation. |
| Outcome: | The proposed evaluation methodology compares with other evaluation methodologies that have been widely used. |
In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | citation counts are a shallow view that fails to capture how a paper has influenced subsequent work. |
| Approach: | They propose a task to generate nuanced, expressive, and time-aware impact summaries . they analyze fine-grained confirmatory and correction citation intents to generate summary . |
| Outcome: | The proposed task shows moderate to strong human correlation on subjective metrics such as insightfulness. |
Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction (P19-1)
Copied to clipboard
| Challenge: | Recent years have witnessed a significant increase in laboratory-based evaluation benchmarks in many scientific disciplines. |
| Approach: | They propose to use NLP datasets to extract task, dataset, metric and score from NLP papers to build automatic leaderboards. |
| Outcome: | The proposed model outperforms baselines in the NLP domain by a large margin. |
Datasets for Scientific Literature Understanding: A Survey (2026.findings-acl)
Copied to clipboard
| Challenge: | Empowering machines to understand scientific literature is crucial for accelerating scientific discovery and advancing the AI for Science paradigm. |
| Approach: | They propose a systematic taxonomy that organizes resources spanning structural understanding, text understanding, multimodal understanding and pre-training/instruction fine-tuning. |
| Outcome: | The proposed taxonomy organizes resources spanning structural understanding, text understanding, multimodal understanding and pre-training/instruction fine-tuning. |