Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2021.eacl-main)
Copied to clipboard
| Challenge: | Abstract meaning representations are typically evaluated using surface matching metrics . however, there are problems with these metrics, since they allow multiple surface realizations . |
| Approach: | They propose a decomposable metric that measures the distance between the original and reconstructed AMRs. |
| Outcome: | The proposed metric reduces the complexity of the evaluation process by allowing for multiple surface realizations. |
Similar Papers
Factorising AMR generation through syntax (N19-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic annotation framework which abstracts away from the surface form of text to capture the core 'who did what to whom' structure. |
| Approach: | They propose to decompose the generation process into two steps: first generate a syntactic structure, and then generate the surface form. |
| Outcome: | The proposed approach generates meaning-preserving syntactic paraphrases of the same graph, as judged by humans. |
A Human Evaluation of AMR-to-English Generation Systems (2020.coling-main)
Copied to clipboard
| Challenge: | a recent human evaluation of AMR generation systems is compared to automated metrics. |
| Approach: | They propose a human evaluation which collects fluency and adequacy scores and categorization of error types for AMR generation systems. |
| Outcome: | The results show that human evaluations are more nuanced than automated metrics. |
A Partially Rule-Based Approach to AMR Generation (N19-3)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a representation of a sentence as a labeled graph . because of these abstractions, it can be difficult to generate from AMR back to a fluent English sentence . |
| Approach: | They propose a new approach to generating English text from Abstract Meaning Representation (AMR) it is largely rule-based, supplemented by a language model and simple statistical linearization models . they also address difficulties of automatically evaluating AMR generation systems . |
| Outcome: | The proposed approach produces a fluent English sentence with a high quality . it is difficult to generate from an AMR back to a sentence which preserves original meaning . |
DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering (2023.acl-long)
Copied to clipboard
| Challenge: | Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. |
| Approach: | They propose a metric that evaluates natural language generation tasks as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models without training on evaluation datasets. |
| Outcome: | The proposed metric achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which exhibits strong dimension-level / task-level generalization ability and interpretability. |
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020.acl-main)
Copied to clipboard
Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, Salim Roukos
| Challenge: | Existing approaches to generating text from AMRs focus on training sequence-to-sequence or graph-tosequent models on annotated data. |
| Approach: | They propose a strong pre-trained language model with cycle consistency-based re-scoring to generate AMR text. |
| Outcome: | The proposed model outperforms existing methods on the English LDC2017T10 dataset. |
AMR Similarity Metrics from Principles (2020.tacl-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) graphs are rooted, acyclic, directed, and edge-labeled. |
| Approach: | They propose a canonical Smatch metric that aligns variables of two graphs and assesses triple matches. |
| Outcome: | The proposed metric is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria. |
Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on multilingual sentence embeddings focus on cross-lingual semantic textual similarity and transfer tasks. |
| Approach: | They propose a method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR) . they compare existing multi-lingual sentence embedded with AMR and improve their versions by reducing the surface variations across different languages and expressions. |
| Outcome: | The proposed method improves state-of-the-art multilingual sentence embeddings on transfer tasks and semantic textual similarity tests. |
SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features (2022.aacl-main)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (S3BERT) embeddings are composed of explainable sub-embeddings that emphasize various sentence meaning features. |
| Approach: | They propose to induce Semantically Structured Sentence BERT embeddings (S3BERT) that emphasize various sentence meaning features. |
| Outcome: | The proposed model shows high correlation to human similarity ratings, but lacks interpretability. |
DETECT: Determining Ease and Textual Clarity of German Text Simplifications (2026.eacl-long)
Copied to clipboard
| Challenge: | Current evaluation of German automatic text simplification relies on general-purpose metrics such as SARI, BLEU, and BERTScore. |
| Approach: | They propose a German-specific metric that holistically evaluates ATS quality across all three dimensions of simplicity, meaning preservation, and fluency. |
| Outcome: | The proposed metric achieves higher correlations with human judgments than widely used ATS metrics. |
Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing metrics for assessing the similarity of abstract meaning representations (AMRs) have complementary strengths and weaknesses, and are expensive to implement. |
| Approach: | They propose a Benchmark for AMR Metrics based on overt objectives that can be used to assess graph-based similarity metrics. |
| Outcome: | The proposed metrics match contextualized substructures and induce n:m alignments between their nodes. |