Paraphrase Generation Evaluation Powered by an LLM: A Semantic Metric, Not a Lexical One (2025.coling-main)
Copied to clipboard
| Challenge: | Existing measures for automatic paraphrase generation are based on lexical distances or semantic embedding alignments. |
| Approach: | They propose a measure based on a log likelihood ratio from an LLM to assess the quality of a potential paraphrase. |
| Outcome: | The proposed measure is better for sorting pairs of sentences by semantic proximity and provides an interpretable classification threshold between paraphrases and non-paraphrases. |
Similar Papers
PARAPHRASUS: A Comprehensive Benchmark for Evaluating Paraphrase Detection Models (2025.coling-main)
Copied to clipboard
| Challenge: | prevailing notion of paraphrase is simplistic, offering only limited view of vast spectrum of paraphrasing phenomena. |
| Approach: | They propose a benchmarking tool for paraphrase detection that provides a fine-grained evaluation lens. |
| Outcome: | The proposed benchmark enables rapid calibration of models to specific strictness levels. |
On the Evaluation Metrics for Paraphrase Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for paraphrase generation are not designed for the task, but adopted from other evaluation tasks. |
| Approach: | They propose a new evaluation metric for paraphrase generation that uses reference-based and reference-free metrics. |
| Outcome: | The proposed evaluation metric outperforms existing metrics and is more reliable than reference-based metrics. |
PAM: Paraphrase AMR-Centric Evaluation Metric (2025.findings-acl)
Copied to clipboard
| Challenge: | Current evaluation metrics for paraphrase generation are based on borrowed metrics from text-to-text tasks . this is not ideal for paraphrasing as we typically want variation in the lexicon while persisting semantics. |
| Approach: | They propose a Paraphrase AMR-Centric Evaluation Metric that uses AMR graphs extracted from the input text to evaluate paraphrases. |
| Outcome: | The proposed evaluation metric improves on different semantic textual similarity datasets on paraphrases with human semantic scores. |
Towards Better Characterization of Paraphrases (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models of natural language processing lack generalization and performance . existing models are often overreliant on learned spurious correlations resulting in poor generalization. |
| Approach: | They propose to use word position deviation and lexical deviation to characterize paraphrase pairs without expert human annotation. |
| Outcome: | The proposed metrics improve generalizability of models trained on the dataset and can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models. |
Controllable Paraphrase Generation for Semantic and Lexical Similarities (2024.lrec-main)
Copied to clipboard
| Challenge: | Lexically diverse paraphrases are crucial in data augmentation because they enhance the linguistic diversity of the corpus. |
| Approach: | They propose a controllable model for semantic and lexical similarities by attaching tags to the head of the input sentence. |
| Outcome: | The proposed model can paraphrase an input sentence according to the tags specified. |
A large-scale computational study of content preservation measures for text style transfer and paraphrase generation (2022.acl-srw)
Copied to clipboard
| Challenge: | Text style transfer and paraphrases generation are growing areas of NLP . many researchers still use BLEU-like measures to evaluate content preservation . |
| Approach: | They compare 57 different measures based on different principles on 19 annotated datasets . they find that measures relying on cross-encoder models outperform alternative approaches . |
| Outcome: | The proposed methods outperform traditional methods on 19 datasets. |
Neural-Driven Search-Based Paraphrase Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing non-supervised paraphrase generation models are biased toward specific problems like question answering or image captioning. |
| Approach: | They propose a search-based paraphrase generation scheme where candidate paraphrases are generated by iterated transformations from the original sentence and evaluated in terms of syntax quality, semantic distance, and lexical distance. |
| Outcome: | The proposed algorithms perform well against non-supervised baselines. |
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (P18-1)
Copied to clipboard
| Challenge: | Using neural machine translation, we generate more than 50 million sentential paraphrase pairs from a large parallel corpus. |
| Approach: | They use a dataset of more than 50 million English-English sentential paraphrase pairs to generate them automatically using neural machine translation. |
| Outcome: | The proposed dataset outperforms all supervised systems on every SemEval semantic textual similarity competition and shows how it can be used for paraphrase generation. |
Paraphrase Types for Generation and Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current approaches to paraphrase generation and detection ignore the intricate linguistic properties of language. |
| Approach: | They propose two tasks to consider specific linguistic perturbations at particular text positions. |
| Outcome: | The proposed tasks address the shortcoming of ignoring the linguistic properties of language. |
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges (2024.emnlp-main)
Copied to clipboard
| Challenge: | introducing Large Language Models (LLMs) has opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. |
| Approach: | They propose a taxonomy for organizing existing LLM-based evaluation metrics and a structured framework to understand and compare them. |
| Outcome: | The proposed taxonomy offers a framework to understand and compare LLM-based evaluation methods. |