Papers by Oren Kalinsky
Simple and Effective Multi-Token Completion from Masked Language Models (2023.findings-eacl)
Copied to clipboard
| Challenge: | Pre-trained neural masked language models are limited to predicting a single token . recent pre-tried LMs like T5 do allow predicting multi-token completions, but are more expensive to train and run. |
| Approach: | They propose two ways to adapt pre-trained masked language models to produce multi-token completions. |
| Outcome: | The proposed method surpasses current state-of-the-art models while being more parameter efficient. |
WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing summarization datasets are limited in their ability to evaluate output . a human evaluation is necessary to understand and improve summarizing systems . |
| Approach: | They propose a dataset based on how-to articles and coherent paragraph summaries written in plain language. |
| Outcome: | The proposed dataset makes human evaluation easier and more effective . the authors compare the proposed dataset to existing ones on PubMed and the literature. |
HotelQuEST: Balancing Quality and Efficiency in Agentic Search (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks for agentic search focus primarily on answer quality, overlooking efficiency factors that are critical for real-world deployment. |
| Approach: | They propose a benchmark for hotel search queries that includes 214 hotel query queries that range from simple factual requests to complex queries. |
| Outcome: | The proposed benchmarks show that LLM-based agents achieve higher accuracy than traditional retrievers, but at substantially higher costs due to redundant tool calls and suboptimal routing that fails to match query complexity to model capability. |
ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots (2025.naacl-short)
Copied to clipboard
Shani Goren, Oren Kalinsky, Tomer Stav, Yuri Rapoport, Yaron Fairstein, Ram Yazdi, Nachshon Cohen, Alexander Libov, Guy Kushilevitz
| Challenge: | Large Language Models (LLMs) have revolutionized many NLP applications. |
| Approach: | They propose an autocomplete evaluation framework for LLM-based chatbot interactions that includes a formal definition of the task and suitable metrics. |
| Outcome: | The proposed framework evaluates 11 models on a task that performs fairly but still lacks the ranking of the generated suggestions. |
Evaluating D-MERIT of Partial-annotation on Information Retrieval (2024.emnlp-main)
Copied to clipboard
Royi Rassin, Yaron Fairstein, Oren Kalinsky, Guy Kushilevitz, Nachshon Cohen, Alexander Libov, Yoav Goldberg
| Challenge: | Using partially-annotated datasets for evaluation can lead to false conclusions . a dataset containing only a subset of relevant passages might result in misleading rankings . |
| Approach: | They propose to use a Wikipedia passage retrieval evaluation set to contain all relevant passages for each query. |
| Outcome: | The proposed dataset can be downloaded from https://d-merit.github.io. |