| Challenge: | Current evaluation metrics for paraphrase generation are based on borrowed metrics from text-to-text tasks . this is not ideal for paraphrasing as we typically want variation in the lexicon while persisting semantics. |
| Approach: | They propose a Paraphrase AMR-Centric Evaluation Metric that uses AMR graphs extracted from the input text to evaluate paraphrases. |
| Outcome: | The proposed evaluation metric improves on different semantic textual similarity datasets on paraphrases with human semantic scores. |
Similar Papers
On the Evaluation Metrics for Paraphrase Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluation metrics for paraphrase generation are not designed for the task, but adopted from other evaluation tasks. |
| Approach: | They propose a new evaluation metric for paraphrase generation that uses reference-based and reference-free metrics. |
| Outcome: | The proposed evaluation metric outperforms existing metrics and is more reliable than reference-based metrics. |
Paraphrase Generation Evaluation Powered by an LLM: A Semantic Metric, Not a Lexical One (2025.coling-main)
Copied to clipboard
| Challenge: | Existing measures for automatic paraphrase generation are based on lexical distances or semantic embedding alignments. |
| Approach: | They propose a measure based on a log likelihood ratio from an LLM to assess the quality of a potential paraphrase. |
| Outcome: | The proposed measure is better for sorting pairs of sentences by semantic proximity and provides an interpretable classification threshold between paraphrases and non-paraphrases. |
Automated Paraphrase Lattice Creation for HyTER Machine Translation Evaluation (N18-2)
Copied to clipboard
| Challenge: | Existing machine translation evaluation metrics use synonyms and paraphrases to reward meaning-equivalent but lexically divergent translations. |
| Approach: | They propose a machine translation evaluation metric which exploits reference translations enriched with meaning equivalent expressions. |
| Outcome: | The proposed metric achieves medium performance on large and noisier datasets . it is compared with the existing HyTER evaluation metric . |
ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Paraphrase generation is a long-standing task in natural language processing (NLP). |
| Approach: | They propose to generate large-scale syntactically diverse paraphrase datasets by abstract meaning representation back-translation. |
| Outcome: | The proposed dataset is syntactically more diverse than existing datasets while maintaining good semantic similarity. |
Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer (D19-56)
Copied to clipboard
| Challenge: | Existing methods for textual transfer with no parallel corpora are insufficient to evaluate textual paraphrases with modified attributes or properties. |
| Approach: | They propose to add a metric for post-transfer classification accuracy and a method to combine them into a single overall score. |
| Outcome: | The proposed metrics correlate well with human judgments, at both the sentence-level and system-level. |
Neural-Driven Search-Based Paraphrase Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing non-supervised paraphrase generation models are biased toward specific problems like question answering or image captioning. |
| Approach: | They propose a search-based paraphrase generation scheme where candidate paraphrases are generated by iterated transformations from the original sentence and evaluated in terms of syntax quality, semantic distance, and lexical distance. |
| Outcome: | The proposed algorithms perform well against non-supervised baselines. |
Towards Better Characterization of Paraphrases (2022.acl-long)
Copied to clipboard
| Challenge: | Existing models of natural language processing lack generalization and performance . existing models are often overreliant on learned spurious correlations resulting in poor generalization. |
| Approach: | They propose to use word position deviation and lexical deviation to characterize paraphrase pairs without expert human annotation. |
| Outcome: | The proposed metrics improve generalizability of models trained on the dataset and can be used to generate specific forms of paraphrases for data augmentation or robustness testing of NLP models. |
Polly Want a Cracker: Analyzing Performance of Parroting on Paraphrase Generation Datasets (D19-1)
Copied to clipboard
| Challenge: | Paraphrase generation is an interesting and challenging task which has numerous practical applications. |
| Approach: | They analyze datasets commonly used for paraphrase generation research and show that simply parroting input sentences surpasses state-of-the-art models when evaluated on standard metrics. |
| Outcome: | The proposed model can generate paraphrases even without making any changes to the input sentence or even none at all, compared with other models. |
Evaluate AMR Graph Similarity via Self-supervised Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Current AMR metrics do not consider the entire structure of AMR graphs . |
| Approach: | They propose to learn automatic AMR graph similarity evaluation metric by encoding AMR to a pre-trained language model and using GNN adapters to capture structural information of AMR diagrams. |
| Outcome: | The proposed metric significantly improves the correlations with human semantic scores and remains robust under diverse challenges. |
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2021.eacl-main)
Copied to clipboard
| Challenge: | Abstract meaning representations are typically evaluated using surface matching metrics . however, there are problems with these metrics, since they allow multiple surface realizations . |
| Approach: | They propose a decomposable metric that measures the distance between the original and reconstructed AMRs. |
| Outcome: | The proposed metric reduces the complexity of the evaluation process by allowing for multiple surface realizations. |