Papers with METEOR

49 papers
Context-Aware Neural Machine Translation Decoding (D19-65)

Copied to clipboard

Challenge: Existing approaches to enhance neural machine translation systems to take into account document-level information make the training process slower or require document- level annotated data.
Approach: They propose a decoding architecture that fuses the semantic space language model and a neural translation model.
Outcome: The proposed approach improves translation quality for English–Spanish using BLEU and METEOR.
“Knowledge is Power”: Constructing Knowledge Graph of Abdominal Organs and Using Them for Automatic Radiology Report Generation (2023.acl-industry)

Copied to clipboard

Challenge: conventional radiology workflows involve dictating diagnosis to transcriptionists, which is prone to delay and error.
Approach: They propose to generate a set of knowledge graphs from a large collection of free-text radiology reports and use them to generate automatic radiology report generation.
Outcome: The proposed model improves the reported BLEU-3, ROUGE-L, METEOR, and CIDEr scores by 2%, 4%, 2% and 2% respectively.
Generating Text through Adversarial Training Using Skip-Thought Vectors (N19-3)

Copied to clipboard

Challenge: Existing approaches to use word embeddings for text generation have been limited.
Approach: They propose to use GANs with word embeddings to reproduce writing style in text . they use a sentence embeddable vector to model people's way of expression .
Outcome: The proposed model outperforms baseline text generation networks across several metrics including BLEU-n, METEOR and ROUGE.
A Partially Rule-Based Approach to AMR Generation (N19-3)

Copied to clipboard

Challenge: Abstract Meaning Representation (AMR) is a representation of a sentence as a labeled graph . because of these abstractions, it can be difficult to generate from AMR back to a fluent English sentence .
Approach: They propose a new approach to generating English text from Abstract Meaning Representation (AMR) it is largely rule-based, supplemented by a language model and simple statistical linearization models . they also address difficulties of automatically evaluating AMR generation systems .
Outcome: The proposed approach produces a fluent English sentence with a high quality . it is difficult to generate from an AMR back to a sentence which preserves original meaning .
Multimodal Machine Translation with Embedding Prediction (N19-3)

Copied to clipboard

Challenge: Pretrained word embeddings improve multimodal machine translation of low-resource domains due to a shortage of training data.
Approach: They propose to combine pretrained word embeddings with search-based approaches to improve NMT of low-resource domains to better translate rare words.
Outcome: The proposed approach improves translation performance by 1.24 METEOR and 2.49 BLEU and achieves 7.67 F-score.
hinglishNorm - A Corpus of Hindi-English Code Mixed Sentences for Text Normalization (2020.coling-industry)

Copied to clipboard

Challenge: hinglishNorm is a human annotated corpus of Hindi-English code-mixed sentences for text normalization task.
Approach: They propose to annotate sentences in Hindi-English code-mixed sentences using a human annotated normalized form.
Outcome: The proposed corpus contains 13494 segments annotated for text normalization.
Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output (N18-5)

Copied to clipboard

Challenge: Many metrics have been proposed for Machine Translation (MT) that compare system translations against human references.
Approach: They propose to use BLEU and METEOR to evaluate machine translations against human translations.
Outcome: VisEval Metric Viewer (VEMV) provides visualisation of multiple evaluation scores so they can be easily interpreted by a user.
Converting the Point of View of Messages Spoken to Virtual Assistants (2020.findings-emnlp)

Copied to clipboard

Challenge: Using a voice message, virtual assistants extract the message and send it to the user’s contact, rather than properly converting it to “I love you.”
Approach: They propose to take a voice message from one user, convert it to “I love you” and deliver it to its target user.
Outcome: The proposed system can take a voice message from one user, convert the point of view of the message, and then deliver the result to its target user.
Evaluating Question Answering Evaluation (D19-58)

Copied to clipboard

Challenge: Existing n-gram based QA metrics have a number of drawbacks and are not suitable for all extractive tasks.
Approach: They propose to use BERTScore to evaluate translation for question answering (QA) they also explore whether existing n-gram based metrics are suitable for generative QA .
Outcome: The proposed BERTScore metric fails to provide stronger correlation with human judgements .
Uncertainty in Language Models: Assessment through Rank-Calibration (2024.emnlp-main)

Copied to clipboard

Challenge: Language Models (LMs) have shown promising performance in natural language generation . however, it is crucial to correctly quantify their level of uncertainty in responding to inputs.
Approach: They propose a framework to quantify uncertainty and confidence for Large Language Models . they use a Rank-calibration framework to measure uncertainty and confident responses .
Outcome: The proposed framework assesses uncertainty and confidence measures for LMs.
Let Me Know What to Ask: Interrogative-Word-Aware Question Generation (D19-58)

Copied to clipboard

Challenge: Existing models focus on generating questions based on text and the answer to the generated question.
Approach: They propose a pipelined system that predicts the type of interrogative word to be generated . they also propose qg models that can be used to generate questions based on text .
Outcome: The proposed system improves on the task of QG in SQuAD, improving from 46.58 to 47.69 in BLEU-1, 17.55 to 18.53 in blu-4, 21.24 to 22.33 in METEOR, and 44.53 to 46.94 in ROUGE-L.
An Error Analysis Framework for Shallow Surface Realization (2021.tacl-1)

Copied to clipboard

Challenge: BLEU and METEOR metrics fail to provide information on which linguistic factors impact performance of natural language generation models.
Approach: They propose a framework for error analysis which permits identifying which features of the input affect the models’ results.
Outcome: The proposed framework improves the performance of 174 system runs submitted to the Multilingual SR shared tasks.
Automating Human Evaluation of Dialogue Systems (2022.naacl-srw)

Copied to clipboard

Challenge: a recent study shows that human evaluations of dialogue systems weakly reflect human judgments.
Approach: They propose a BERT-based model that fine-tunes a model with three prediction heads to predict whether the system-generated output is natural, fluent, and informative.
Outcome: The proposed model achieves an average accuracy of 77% over the 3 labels . it also uses three different models to compute the labels compared to three separate models .
SmartMatch: Real-Time Semantic Retrieval for Translation Memory Systems (2026.eacl-demo)

Copied to clipboard

Challenge: Translation Memory (TM) systems are core components of computer-aided translation tools . however, they fail to retrieve semantically relevant content when surface similarity is low.
Approach: They propose an open-source demo and evaluation toolkit for TM retrieval that connects modern sentence encoders and strong lexical/fuzzy baselines with a vector database.
Outcome: The proposed toolkit exposes the end-to-end retrieval pipeline through a web-based UI for qualitative inspection and preference logging.
Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations (2022.aacl-main)

Copied to clipboard

Challenge: Existing approaches to generate text radiology reports are prone to errors and poor clinical accuracy.
Approach: They propose a two-step pipeline that subdivides the problem into factual triple extraction followed by free-text report generation.
Outcome: The proposed pipeline shows that the generated reports exhibit realistic style but lack clinical accuracy.
Training for Diversity in Image Paragraph Captioning (D18-1)

Copied to clipboard

Challenge: Existing image captioning models have a lack of diversity between sentences . current models have limited their effectiveness due to repetitive paragraphs .
Approach: They propose to apply sequence-level training to image paragraph captioning models . they find that standard self-critical training produces poor results .
Outcome: The proposed training improves on the Visual Genome dataset with no architectural changes.
Reading Between the Lines: Exploring Infilling in Visual Narratives (2020.emnlp-main)

Copied to clipboard

Challenge: Generating long form narratives from multiple modalities requires a model to learn surrounding contextual information by masking spans of input while decoding attempts in generating the entire text.
Approach: They propose to use infilling techniques to generate textual descriptions from images that are rich in contextual dependencies.
Outcome: The proposed model outperforms existing models in visual storytelling by generating text from a large scale dataset of 46,200 procedures and 340k pairwise images and textual descriptions.
Multilingual and Multimodal Learning for Brazilian Portuguese (2022.lrec-1)

Copied to clipboard

Challenge: Existing models that learn multimodal and multilingual representations perform better in many natural language tasks.
Approach: They use a multimodal and multilingual corpus to test its generalization ability for other languages . they achieve a BLEU score of 51.8 and a METEOR score of 78.0 on the test set .
Outcome: The proposed model outperforms the existing model on a Portuguese-English multimodal translation task.
On the Evaluation of Vision-and-Language Navigation Instructions (2021.eacl-main)

Copied to clipboard

Challenge: Existing instruction generators have not been evaluated using human wayfinders . BLEU, ROUGE, METEOR and CIDEr are ineffective for evaluating grounded navigation instructions.
Approach: They propose an instruction-trajectory compatibility model that operates without reference instructions to improve wayfinding performance.
Outcome: The proposed model shows the highest correlation with human wayfinding outcomes when scoring individual instructions.
Document Context Neural Machine Translation with Memory Networks (P18-1)

Copied to clipboard

Challenge: Experimental results show that our model exploits both source and target document context.
Approach: They propose a document-level neural machine translation model which takes both source and target document context into account using memory networks.
Outcome: The proposed model outperforms previous work in terms of BLEU and METEOR in English translations.
A Timestep aware Sentence Embedding and Acme Coverage for Brief but Informative Title Generation (2022.findings-naacl)

Copied to clipboard

Challenge: Existing methods for title generation are based on timestep aware sentence embeddings, but they are not effective for generating a title with appropriate information in the content.
Approach: They propose a Timestep aware Sentence Embedding mechanism which refreshes the sentences’ embeddings with corresponding key words in different decoding timesteps.
Outcome: The proposed framework outperforms existing methods on various title generation tasks and the evaluation scores are significantly higher than previous approaches.
GTR-LSTM: A Triple Encoder for Sentence Generation from RDF Data (P18-1)

Copied to clipboard

Challenge: Knowledge bases are becoming an enabling resource for many applications including Q&A systems, recommender systems, and summarization tools.
Approach: They propose a system to translate RDF triples into natural sentences using an encoder-decoder framework.
Outcome: The proposed model outperforms the baseline model by 17.6%, 6.0%, and 16.4% in terms of BLEU, METEOR, and TER scores.
HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning (2026.findings-eacl)

Copied to clipboard

Challenge: haptic captioning is the task of generating natural language descriptions from haptics, such as vibrations, for use in virtual reality and rehabilitation applications.
Approach: They propose a multimodal sensory language model that interprets vibration signals into descriptions in a given sensory, emotional, or associative category.
Outcome: The proposed model interprets vibration signals into descriptions in a given sensory, emotional, or associative category.
Accuracy is not enough: Evaluating Personalization in Summarizers (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing accuracy measures cannot evaluate the degree of personalization of summarization models.
Approach: They propose to use a PENS dataset to analyze the degree of personalization of ten different summarization models.
Outcome: The proposed measure can evaluate the degree of personalization of summarization models using the PENS dataset.
uFACT: Unfaithful Alien-Corpora Training for Semantically Consistent Data-to-Text Generation (2022.findings-acl)

Copied to clipboard

Challenge: uFACT (Un-Faithful Alien Corpora Training) is a training corpus construction method for data-to-text generation models.
Approach: They propose a training corpus construction method for data-to-text (d2t) generation models which augments a target corpus with alien corpora which have different semantic representations.
Outcome: The proposed method generates utterances which represent the semantic content of the data sources more accurately compared to models trained on the target corpus alone.
ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generation (2023.findings-emnlp)

Copied to clipboard

Challenge: Unlike previous methods, large language models produce entity-agnostic templates instead of copying the given example entities or validating/crafting the templates manually.
Approach: They propose an approach for structured data verbalisation into short template sentences in zero to few-shot settings that prompts Large Language Models to directly produce entity-agnostic templates.
Outcome: The proposed approach averages 66% parsing error rate reduction in generated verbalisations of RDF triples on the DART dataset.
Reinforced Multi-task Approach for Multi-hop Question Generation (2020.coling-main)

Copied to clipboard

Challenge: Empirical evaluation shows our model to outperform the single-hop question generation models on both automatic evaluation metrics such as BLEU, METEOR, and ROUGE and human evaluation metrics for quality and coverage of the generated questions.
Approach: They propose a question-aware reward function to maximize the utilization of supporting facts in the context.
Outcome: The proposed model outperforms single-hop neural question generation models on automatic evaluation metrics and human evaluation metrics for quality and coverage of the generated questions.
Argument Generation with Retrieval, Planning, and Realization (P19-1)

Copied to clipboard

Challenge: a novel argument generation framework is used to generate counter-arguments . CANDELA uses a text planning decoder to retrieve arguments of different perspectives .
Approach: They propose a powerful retrieval system and a novel two-step argument generation framework . they use a retrieval-based retrieval platform indexed with 12 million articles from Wikipedia .
Outcome: The proposed framework yields higher BLEU, ROUGE, and METEOR scores than state-of-the-art models.
Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation (2023.emnlp-main)

Copied to clipboard

Challenge: Existing methods for question generation over knowledge bases rely on annotated data for fine-tuning . emergence of Large Language Models (LLMs) has shown impressive generalization ability in few-shot tasks.
Approach: They propose to use a logical form to generate a question in a reasoning problem . they propose to extend the prompting method into a method that can generate questions in logical forms .
Outcome: The proposed method outperforms baselines on three public KBQG datasets.
CSEval: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLMs (2025.naacl-long)

Copied to clipboard

Challenge: Current evaluation methods do not capture complex attributes of counterspeech quality, such as contextual relevance, aggressiveness, or argumentative coherence.
Approach: They propose to use a dataset and framework to evaluate counterspeech quality across four dimensions: contextual relevance, aggressiveness, argument-coherence, and suitability.
Outcome: The proposed method outperforms ROUGE, METEOR, and BertScore in correlating with human judgement, indicating a significant improvement in automated counterspeech evaluation.
Geo-Aware Image Caption Generation (2020.coling-main)

Copied to clipboard

Challenge: Standard image caption generation systems do not take contextual information or world knowledge into account.
Approach: They propose to build an image-specific representation of the geographic context and adapt the caption generation network to produce appropriate geographic names in the image descriptions.
Outcome: The proposed system achieves significant improvements on a dataset that contains contextualized captions and geographic metadata and improves BLEU, ROUGE, METEOR and CIDEr scores.
Generative Pretraining for Paraphrase Evaluation (2022.acl-long)

Copied to clipboard

Challenge: ParaBLEU is a paraphrase representation learning model and evaluation metric for text generation.
Approach: They propose a paraphrase representation learning model and evaluation metric for text generation that uses generative conditioning as a pretraining objective.
Outcome: The proposed model outperforms existing models on the 2017 WMT Metrics Shared Task using only 50% of the available training data and surpasses BLEU, ROUGE and METEOR with only 40 examples.
Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering (2025.coling-main)

Copied to clipboard

Challenge: Hallucination remains a critical challenge in large language models (LLMs) in high-stake domains such as legal question answering.
Approach: They propose a method to mitigate hallucination in legal question answering by using behavior cloning and a novel Hard Sample-aware Direct Preference Optimization.
Outcome: The proposed method improves non-hallucinated Statute Rate, Statute Relevance Rate, Legal Claim Truthfulness, and traditional metrics.
Bridge Video and Text with Cascade Syntactic Structure (C18-1)

Copied to clipboard

Challenge: Using LSTM-CSS, we construct basic syntactic structure by completing syntastic structure.
Approach: They propose a video captioning approach that progressively completes syntactic structure by a conditional random field to construct basic syntaktic structure.
Outcome: The proposed method produces natural sentences with 42.3% and 28.5% accuracy compared to state-of-the-art methods.
An End-to-End Generative Architecture for Paraphrase Generation (D19-1)

Copied to clipboard

Challenge: Existing methods for generating paraphrases with linguistic knowledge are often domain specific and hard to scale, or yield inferior results.
Approach: They propose an end-to-end conditional generative architecture for generating paraphrases via adversarial training which does not depend on extra linguistic information.
Outcome: The proposed method outperforms existing models on automatic metrics and human evaluations on four public datasets.
QSTS: A Question-Sensitive Text Similarity Measure for Question Generation (2022.coling-1)

Copied to clipboard

Challenge: Existing measures for question generation have been inadequately evaluated . current research uses QA datasets containing pairs of (reference question, passage context) elements.
Approach: They propose a Question-Sensitive Text Similarity measure for comparing two questions . they also propose enabling question similarity research in QG contexts by using a dataset called SimQG.
Outcome: The proposed measure overcomes shortcomings of existing measures that depend on n-gram overlap scores and obtains superior results compared to existing measures on publicly-available QG datasets.
A Multi-Document Coverage Reward for RELAXed Multi-Document Summarization (2022.acl-long)

Copied to clipboard

Challenge: Multi-document summarization models are limited by limited references and with maximum-likelihood objectives.
Approach: They propose to fine-tune an MDS baseline with a reward that balances a reference-based metric such as ROUGE with coverage of the input documents.
Outcome: The proposed model improves on the Multi-News and WCEP datasets with a low-variance estimator . the proposed model also improves the coverage of the input documents .
Supervised Visual Attention for Multimodal Neural Machine Translation (2020.coling-main)

Copied to clipboard

Challenge: Existing studies show that a conventional visual attention mechanism trained in an unsupervised manner is not effective for multimodal neural machine translation.
Approach: They propose a supervised visual attention mechanism for multimodal neural machine translation that captures the relationship between a word and an image region more precisely than a conventional visual attention system.
Outcome: The proposed model improves on English-German and German-English translation tasks and English-Japanese and Japanese-English tasks using the Flickr30k Entities JP dataset.
Multimodal Differential Network for Visual Question Generation (D18-1)

Copied to clipboard

Challenge: Current dialog systems show improvement in visual question answering but this does not translate to improved human-AI dialog.
Approach: They propose to use a Multimodal Differential Network to generate natural questions from images using a multimodal differential network.
Outcome: The proposed approach significantly improves over state-of-the-art benchmarks on the quantitative metrics.
On the Importance of Diversity in Question Generation for QA (2020.acl-main)

Copied to clipboard

Challenge: In this paper, we show that textual diversity in automatic question generation is beneficial for downstream QA.
Approach: They propose to use textual diversity to promote automatic question generation as a quality measure for QA.
Outcome: The proposed measure of QG quality correlates well with evaluation on QA.
JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation (2022.coling-1)

Copied to clipboard

Challenge: Existing methods rarely consider cross-modal alignment between textual and visual features and ignore disease tags as auxiliary for report generation.
Approach: They propose a "Jointly learning framework for automated disease Prediction and radiology report Generation" the framework integrates cross-modal alignment between textual and visual features and disease tags to improve the quality of reports.
Outcome: The proposed framework improves the quality of radiology reports by combining the main task and auxiliary tasks.
Storyboarding of Recipes: Grounded Contextual Generation (P19-1)

Copied to clipboard

Challenge: Using a dataset for sequential procedural (how-to) text generation from images, we show that 61% of the users found our proposed model is better than the baseline model in terms of overall recipes.
Approach: They propose a dataset for sequential procedural (how-to) text generation from images in cooking domain.
Outcome: The proposed model achieves a METEOR score of 0.31, an improvement of 0.6 over the baseline model.
Interpretable Multi-headed Attention for Abstractive Summarization at Controllable Lengths (2020.coling-main)

Copied to clipboard

Challenge: Abstractive summarization at controllable lengths is a challenging task in natural language processing . high variance in screen-sizes often require extensive human supervision to perform these modifications.
Approach: They propose a supervised method to construct abstractive summaries of a text document at controllable lengths using an interpretable multi-headed attention mechanism.
Outcome: The proposed method outperforms baselines on two low-resource datasets in English by 14.70%.
Replace and Report: NLP Assisted Radiology Report Generation (2023.findings-acl)

Copied to clipboard

Challenge: Clinical practice frequently uses medical imaging for diagnosis and treatment.
Approach: They propose a template-based approach to generate radiology reports from radiographs . they use multilabel image classifiers to generate tags, pathological descriptions from tags .
Outcome: The proposed method improves on the most popular radiology report datasets.
Evaluating the Knowledge Dependency of Questions (2022.emnlp-main)

Copied to clipboard

Challenge: Existing evaluation metrics for MCQ generation focus on the n-gram based similarity of the generated MCq to the gold sample and disregard their educational value.
Approach: They propose to use a human survey to measure the MCQ’s answerability given knowledge of the target fact.
Outcome: The proposed methods measure the MCQ’s answerability given knowledge of the target fact.
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era (2025.findings-acl)

Copied to clipboard

Challenge: Image captioning has been a challenge for vision-language researchers for decades . current VLMs focus on tasks like visual question answering (YA) but image captioning is not as advanced as expected.
Approach: They evaluate VLMs' performance on image captioning using human annotations . they find that some metrics show high caption-level agreement with humans .
Outcome: The proposed model outperforms open-source models on image captioning . it achieves 93.4% correlation with human rankings at $4 per test .
Lost in Translation: Chemical Language Models and the Misunderstanding of Molecule Structures (2024.findings-emnlp)

Copied to clipboard

Challenge: chemistry and natural language processing (NLP) have advanced drug discovery.
Approach: They propose a framework for assessment of Chemistry LMs of different natures that relies on augmentations that preserve an underlying chemical.
Outcome: The proposed framework relies on augmentations that preserve an underlying chemical, such as kekulization and cycle replacements.
Extrapolating Multilingual Understanding Models as Multilingual Generators (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing multilingual understanding models are not capable of generating high-quality text compared with decoder-based causal language models.
Approach: They propose a method to adapt a multilingual encoder to a language generator with a small number of additional parameters.
Outcome: The proposed approach outperforms initialization-based methods with 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation.
Error Analysis of Multilingual Language Models in Machine Translation: A Case Study of English-Amharic Translation (2024.emnlp-main)

Copied to clipboard

Challenge: Multilingual large language models have significantly advanced machine translation, yet challenges remain for low-resource languages like Amharic.
Approach: They evaluated the performance of NLLB-200 and M2M in English-Amharic bidirectional translation using the Lesan AI dataset.
Outcome: The proposed models outperformed the existing models in English-Amharic bidirectional translation using the Lesan AI dataset.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations