| Challenge: | Existing benchmarks for semantic textual similarity (STS) are limited to high-resource languages and do not include datasets annotated focusing on relatedness instead of similarity. |
| Approach: | They propose to evaluate multilingual semantic textual similarity benchmarks which span 13 languages and annotated datasets to evaluate and compare them. |
| Outcome: | The proposed method is the most comprehensive benchmark of multilingual STS methods. |
Similar Papers
Rethinking STS and NLI in Large Language Models (2024.findings-eacl)
Copied to clipboard
| Challenge: | Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. |
| Approach: | They propose to rethink semantic textual similarity (STS) and natural language inference (NLI) models with task-specific prompts and model overconfidence to capture disagreements between human judgements. |
| Outcome: | The proposed models are able to capture human opinions on individual examples without any parameter modifications. |
Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Semantic similarity between two sentences depends on the aspects considered between those sentences. |
| Approach: | They propose a Conditional Semantic Textual Similarity task which measures the similarity between two sentences under a specified condition. |
| Outcome: | The proposed method improves Spearman correlation by 5.4% by training a supervised model on the re-annotated dataset. |
C-STS: Conditional Semantic Textual Similarity (2023.emnlp-main)
Copied to clipboard
Ameet Deshpande, Carlos Jimenez, Howard Chen, Vishvak Murahari, Victoria Graf, Tanmay Rajpurohit, Ashwin Kalyan, Danqi Chen, Karthik Narasimhan
| Challenge: | Semantic textual similarity (STS) is a cornerstone task in natural language processing, but it is inherently ambiguous. |
| Approach: | They propose a task called conditional STS which measures similarity conditioned on an aspect elucidated in natural language. |
| Outcome: | The proposed task reduces subjectivity and ambiguity and enables fine-grained similarity evaluation using diverse conditions. |
Linguistically Conditioned Semantic Textual Similarity (2024.acl-long)
Copied to clipboard
| Challenge: | Semantic textual similarity (STS) is a fundamental NLP task that measures the semantic similarity between two sentences. |
| Approach: | They propose to use a conditional STS dataset to measure sentences’ similarity conditioned on a certain aspect to reduce the inherent ambiguity posed by the sentences. |
| Outcome: | The proposed method improves the performance over baselines on the C-STS dataset with over 80% F1 score. |
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multilingual benchmarks focus primarily on language understanding tasks. |
| Approach: | They develop a multi-way multilingual benchmark that measures critical capabilities of large language models across languages. |
| Outcome: | Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. |
Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss (2024.emnlp-main)
Copied to clipboard
| Challenge: | despite its efficiency, Sentence-BERT ignores the progressive nature of semantic relationships, despite a promising approach . contrastive learning methods have improved performance on renowned STS benchmarks, but they fail to leverage fine-grained information. |
| Approach: | They propose a regression framework that categorizes text pairs as either semantically similar or dissimilar . they propose two loss functions: Translated ReLU and Smooth K2 Loss to bridge this gap . |
| Outcome: | The proposed method achieves convincing performance across seven established STS benchmarks. |
Going Beyond Sentence Embeddings: A Token-Level Matching Algorithm for Calculating Semantic Textual Similarity (2023.acl-short)
Copied to clipboard
| Challenge: | Semantic Textual Similarity (STS) measures the degree to which the underlying semantics of paired sentences are equivalent. |
| Approach: | They propose a token-level matching inference algorithm which can be applied on top of any language model to improve its performance on STS task. |
| Outcome: | The proposed method improves the performance of almost all language models, with up to 12.7% gain in Spearman’s correlation. |
Mitigating Data Scarcity in Semantic Parsing across Languages with the Multilingual Semantic Layer and its Dataset (2024.findings-acl)
Copied to clipboard
Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Karim Ghonim, Lu Xu, Hee-Soo Choi, Alberte Fernández-Castro, Roberto Navigli
| Challenge: | Large Language Models (LLMs) have advanced significantly in understanding human text, but semantic representations remain crucial for various applications. |
| Approach: | They introduce a multilingual semantic layer which decouples from disambiguation and external inventories and simplifies the task. |
| Outcome: | The proposed model reduces performance gap between languages and annotators by enabling them to understand semantic relations between concepts in any language. |
Multilingual Corpus Creation for Multilingual Semantic Similarity Task (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing monolingual corpora are limited for semantic similarity tasks . a major factor affecting the success of deep neural networks is the availability of large and good quality corpors. |
| Approach: | They propose a semi-automated framework to create a multilingual corpus for a semantic similarity task. |
| Outcome: | The proposed framework can be applied to government, insurance, banking domains provided a bilingual website exists. |
From Semantics to Style: A Cross-Dataset Comparative Framework for Sentence Similarity Predictions (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing frameworks for analyzing text embedding models are limited. |
| Approach: | They propose a framework that uses lightweight poolers to analyze STS, PI, and Triplet datasets. |
| Outcome: | The proposed framework shows that the model captures semantic differences between sentences and is consistent across datasets. |