| Challenge: | Existing benchmarks for semantic textual similarity (STS) use averaged human ratings as gold standard. |
| Approach: | They propose to use a Chinese sentence-to-sentence dataset to study collective human opinions in semantic textual similarity (STS) neither a scalar nor a single Gaussian fits a set of observed judgments adequately, they argue . |
| Outcome: | The proposed dataset does not capture disagreements on individual instances, but rather the confidence over the aggregate dataset. |
Similar Papers
What Can We Learn from Collective Human Opinions on Natural Language Inference Data? (2020.emnlp-main)
Copied to clipboard
| Challenge: | Despite the subjective nature of many NLU evaluations, little attention has been paid to the distribution of human opinions. |
| Approach: | They use a dataset with 464,500 annotations to study Collective HumAn OpinionS . they argue that models lack the ability to recover the distribution over human labels . |
| Outcome: | The proposed dataset examines the distribution of human opinions in NLU evaluation datasets. |
Beyond Averages: Learning with Annotator Disagreement in STS (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to capture and model disagreement in Semantic Textual Similarity (STS) ignore label dispersion and incentivize models to ignore uncertainty crucial for practical settings. |
| Approach: | They propose to capture and model disagreement in Semantic Textual Similarity (STS) a lightweight truncated Gaussian head and a cross-encoder are used to model disagreement . |
| Outcome: | The proposed approach improves accuracy and calibration of models to human judgments. |
Rethinking STS and NLI in Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. |
| Approach: | They propose to rethink semantic textual similarity (STS) and natural language inference (NLI) models with task-specific prompts and model overconfidence to capture disagreements between human judgements. |
| Outcome: | The proposed models are able to capture human opinions on individual examples without any parameter modifications. |
MUSTS: MUltilingual Semantic Textual Similarity Benchmark (2025.acl-short)
Copied to clipboard
| Challenge: | Existing benchmarks for semantic textual similarity (STS) are limited to high-resource languages and do not include datasets annotated focusing on relatedness instead of similarity. |
| Approach: | They propose to evaluate multilingual semantic textual similarity benchmarks which span 13 languages and annotated datasets to evaluate and compare them. |
| Outcome: | The proposed method is the most comprehensive benchmark of multilingual STS methods. |
From Semantics to Style: A Cross-Dataset Comparative Framework for Sentence Similarity Predictions (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing frameworks for analyzing text embedding models are limited. |
| Approach: | They propose a framework that uses lightweight poolers to analyze STS, PI, and Triplet datasets. |
| Outcome: | The proposed framework shows that the model captures semantic differences between sentences and is consistent across datasets. |
The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent studies comparing AI-generated and human-authored literary texts have produced conflicting results. |
| Approach: | They hypothesize that differences in reading quality can be explained by genuine differences in how readers interpret and value literature . |
| Outcome: | The authors show that the differences in reading quality are largely explained by differences in how readers interpret and value literature, rather than by an intrinsic quality of the texts evaluated. |
Linguistically Conditioned Semantic Textual Similarity (2024.acl-long)
Copied to clipboard
| Challenge: | Semantic textual similarity (STS) is a fundamental NLP task that measures the semantic similarity between two sentences. |
| Approach: | They propose to use a conditional STS dataset to measure sentences’ similarity conditioned on a certain aspect to reduce the inherent ambiguity posed by the sentences. |
| Outcome: | The proposed method improves the performance over baselines on the C-STS dataset with over 80% F1 score. |
Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Semantic similarity between two sentences depends on the aspects considered between those sentences. |
| Approach: | They propose a Conditional Semantic Textual Similarity task which measures the similarity between two sentences under a specified condition. |
| Outcome: | The proposed method improves Spearman correlation by 5.4% by training a supervised model on the re-annotated dataset. |
Compositional Evaluation on Japanese Textual Entailment and Similarity (2022.tacl-1)
Copied to clipboard
| Challenge: | Despite growing interest in linguistic universals, most NLI/STS studies focus on English. |
| Approach: | They propose a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. |
| Outcome: | The proposed datasets show that pre-trained language models are insensitive to word order and case particles. |
Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Text summarization is a key natural language generation task, but the high cost of inaccurate summaries raises concerns about the reliability of uncertainty estimation on text summarisation (UE-TS) evaluation methods. |
| Approach: | They propose a UE-TS benchmark that evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets. |
| Outcome: | The proposed benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. |