Transitioning from benchmarks to a real-world case of information-seeking in Scientific Publications (2023.findings-acl)
Copied to clipboard
| Challenge: | Scientific publications are one of the primary means by which researchers disseminate their findings and discoveries to the community, but the amount of information to go through can easily become daunting and challenging. |
| Approach: | They propose a zero-shot text search case for information seeking in scientific papers that uses a biomedical IR benchmark and an industrial case of vitamin B's impact on health. |
| Outcome: | The proposed model is based on a zero-shot text search case in scientific publications . it shows that benchmarks are not faithfully representative of the task at hand . |
Similar Papers
Mind the Gap between the Application Track and the Real World (2023.acl-short)
Copied to clipboard
| Challenge: | Recent advances in NLP have led to a rise in inter-disciplinary and application-oriented research. |
| Approach: | They examine the relationship between motivations described in NLP papers and models and evaluations which comprise the proposed solution. |
| Outcome: | The proposed solution improves educational dialog understanding system when used in a realistic classroom environment. |
Targeting the Benchmark: On Methodology in Current Natural Language Processing Research (2021.acl-short)
Copied to clipboard
| Challenge: | a language benchmark is a task devised that is restricted enough to be managable with current methods, but is deemed challenging enough to serve as a benchmark. |
| Approach: | They propose to use a language task as a benchmark and a baseline model to argue it is challenging enough to be a good one. |
| Outcome: | The proposed language benchmarks are based on a dataset and a language task . the proposed benchmarks can be used to measure progress towards the goal of the research . |
Uncovering Limitations of Large Language Models in Information Seeking from Tables (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks for Table Information Seeking (TabIS) are lacking in reliable evaluation. |
| Approach: | They propose a benchmark to evaluate the table information seeking abilities of large language models . they use a single-choice question format instead of a text-based evaluation . |
| Outcome: | The proposed benchmark is more reliable than existing models and is available online. |
Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers (2026.findings-acl)
Copied to clipboard
Qingcheng Zeng, Yuheng Lu, Zeqi Zhou, Heli Qi, Puxuan Yu, Fuheng Zhao, Hitomi Yanaka, Weihao Xuan, Naoto Yokoya
| Challenge: | a new study examines the performance of code-switching IR in monolingual contexts . code-witching is a pervasive linguistic phenomenon in global communication . |
| Approach: | They propose a benchmark to evaluate code-switching IR in monolingual contexts . they propose CS-MTEB, which measures performance declines of up to 27% . |
| Outcome: | The proposed benchmark shows that code-switching performance is degraded by 27% . the proposed benchmark is based on a dataset of mixed-language queries . |
SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing search methods for COVID-19 are not based on scientific data, but use a neural re-ranking model pre-trained on scientific text. |
| Approach: | They propose a zero-shot ranking algorithm that adapts to COVID-related scientific literature . they use a neural re-ranking model pre-trained on scientific text and filters the target document . |
| Outcome: | The proposed algorithm outperforms models on the TREC COVID Round 1 leaderboard . it outperformed models that do not rely on TREC-COVID data . |
AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Effective zero-shot dense retrieval in the medical domain remains difficult due to the scarcity of relevance-labeled data. |
| Approach: | They propose a framework that leverages large language models to generate hypothetical documents . they also propose 'CMIRB' to provide a rigorous evaluation suite . |
| Outcome: | The proposed framework outperforms HyDE in retrieval accuracy and generalization . it leverages large language models to generate hypothetical documents conditioned on a query . |
NLP-KG: A System for Exploratory Search of Scientific Literature in Natural Language Processing (2024.acl-demos)
Copied to clipboard
| Challenge: | Existing systems for scientific literature search are typically tailored to keyword-based lookup searches, limiting possibilities for exploration. |
| Approach: | They propose a feature-rich system that supports the exploration of research literature in unfamiliar natural language processing fields. |
| Outcome: | The proposed system supports the exploration of research literature in unfamiliar natural language processing fields. |
Issues with Entailment-based Zero-shot Text Classification (2021.acl-short)
Copied to clipboard
| Challenge: | Pre-trained BERT models with no fine-tuning can yield competitive performance against BERT fine- tuned for NLI. |
| Approach: | They propose to use any target label into a sentence of hypothesis and verify whether it could be entailed by the input. |
| Outcome: | The proposed models perform better than models fine-tuned for BERT, but the results are in general negative. |
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing information-seeking question answering datasets do not perform well on answering these questions . existing models that do well on other QA tasks do not do well answering these tasks . |
| Approach: | They present a dataset of 5049 questions over 1585 NLP papers . they use a question-seeking QA model that seeks information in the full text . |
| Outcome: | The proposed dataset underperforms existing models on other QA tasks by 27 F1 points . the focus is on document-grounded, information-seeking QA . |
Do Text-to-Vis Benchmarks Test Real Use of Visualisations? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for visualisations are limited and do not reflect real-world use. |
| Approach: | They analysed visualisation code from Python, R, Javascript, and Vega to find similarities and differences between real-world and benchmark datasets. |
| Outcome: | The results show that benchmark datasets do not test the same distribution of chart types, attributes, and actions as real-world examples. |