Papers by Nachshon Cohen
Extremely efficient online query encoding for dense retrieval (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing dense retrieval systems use the same model architecture for encoding both passages and queries, even though queries are much shorter and simpler than passages. |
| Approach: | They propose a small efficient RNN query encoder that can reduce latency by 12 with only a minor decrease in quality. |
| Outcome: | The proposed solution reduces latency by up to 12 while achieving 35.5 MRR@10 score. |
WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing summarization datasets are limited in their ability to evaluate output . a human evaluation is necessary to understand and improve summarizing systems . |
| Approach: | They propose a dataset based on how-to articles and coherent paragraph summaries written in plain language. |
| Outcome: | The proposed dataset makes human evaluation easier and more effective . the authors compare the proposed dataset to existing ones on PubMed and the literature. |
Multi Document Summarization Evaluation in the Presence of Damaging Content (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics evaluate a summary based on relevance and consistency with the source documents. |
| Approach: | They propose to measure the ability of MDS systems to handle damaging documents in their input set by lexical similarity and language model likelihood. |
| Outcome: | The proposed metrics show that they can summarize a set of documents without damaging content. |
SDR: Efficient Neural Re-ranking using Succinct Document Representation (2022.acl-long)
Copied to clipboard
| Challenge: | BERT based ranking models have been successful on various information retrieval tasks, but they are prone to storage and network fetching latency. |
| Approach: | They propose a late-interaction architecture that allows pre-computation of intermediate document representations, thus reducing latency. |
| Outcome: | The proposed model achieves 4x–11.6x higher compression rates on the MSMARCO passage re-reranking task compared to existing methods. |
ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots (2025.naacl-short)
Copied to clipboard
Shani Goren, Oren Kalinsky, Tomer Stav, Yuri Rapoport, Yaron Fairstein, Ram Yazdi, Nachshon Cohen, Alexander Libov, Guy Kushilevitz
| Challenge: | Large Language Models (LLMs) have revolutionized many NLP applications. |
| Approach: | They propose an autocomplete evaluation framework for LLM-based chatbot interactions that includes a formal definition of the task and suitable metrics. |
| Outcome: | The proposed framework evaluates 11 models on a task that performs fairly but still lacks the ranking of the generated suggestions. |
Evaluating D-MERIT of Partial-annotation on Information Retrieval (2024.emnlp-main)
Copied to clipboard
Royi Rassin, Yaron Fairstein, Oren Kalinsky, Guy Kushilevitz, Nachshon Cohen, Alexander Libov, Yoav Goldberg
| Challenge: | Using partially-annotated datasets for evaluation can lead to false conclusions . a dataset containing only a subset of relevant passages might result in misleading rankings . |
| Approach: | They propose to use a Wikipedia passage retrieval evaluation set to contain all relevant passages for each query. |
| Outcome: | The proposed dataset can be downloaded from https://d-merit.github.io. |
Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to assess data quality for training and testing large language models are lacking. |
| Approach: | They propose two approaches to assess the reliability of data for training large language models for external tool usage. |
| Outcome: | The proposed approaches outperform models trained on high-quality data on two popular benchmarks and an extrinsic evaluation that showcases the impact of data quality on model performance. |