Papers by Piyush Sharma
Neural Naturalist: Generating Fine-Grained Image Comparisons (D19-1)
Copied to clipboard
| Challenge: | a dataset of 41k sentences describes fine-grained differences between photographs of birds . human observers are adept at making fine-grain comparisons, but sometimes require aid in distinguishing visually similar classes. |
| Approach: | They propose a model that generates comparative language from a dataset of 41k sentences describing fine-grained differences between photographs of birds. |
| Outcome: | The proposed model can explain differences in visual embedding space using natural language . it evaluates the results with humans who must use the descriptions to distinguish real images . |
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (P18-1)
Copied to clipboard
| Challenge: | Practical applications of automatic image description systems include leveraging descriptions for image indexing or retrieval, and helping those with visual impairments by transforming visual signals into information that can be communicated via text-to-speech technology. |
| Approach: | They propose to extract and filter image caption annotations from billions of webpages and use them to train models. |
| Outcome: | The proposed model architectures perform better when trained on the Conceptual Captions dataset. |
COSMic: A Coherence-Aware Generation Metric for Image Descriptions (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to evaluate captions have limited learning of their output . previous methods focused on n-gram measures of similarity to reference output based on a ngram of similarities to the output metric. |
| Approach: | They propose a first discourse-aware learned generation metric for evaluating image descriptions. |
| Outcome: | The proposed metric predicts human ratings of captions on out-of-domain images. |
Informative Image Captioning with External Sources of Information (P19-1)
Copied to clipboard
| Challenge: | Current captioning models are trained to generate captions that only contain common object names, thus falling short on an important “informativeness” dimension. |
| Approach: | They propose a mechanism for integrating image information and fine-grained labels into a caption that describes the image in a fluent and informative manner. |
| Outcome: | The proposed model integrates image information with fine-grained labels to produce fluent captions . it can control the appearance of these labels in the output, resulting in fluent and informative captions. |
Denoising Large-Scale Image Captioning from Alt-text Data Using Content Selection Models (2022.coling-1)
Copied to clipboard
| Challenge: | Recent approaches to training large-scale image captioning (IC) models often fall short in terms of performance in leveraging noisy datasets in favor of clean annotations. |
| Approach: | They propose a technique that breaks down the task into two smaller, more controllable tasks - skeleton prediction and skelet-based caption generation. |
| Outcome: | The proposed method can generate better and denoised captions when using noisy datasets. |
Quality Estimation for Image Captions Based on Large-scale Human Evaluations (2021.naacl-main)
Copied to clipboard
| Challenge: | a problem with automatic image captioning is that it produces low quality captions when used in the wild. |
| Approach: | They propose to model caption quality from a human perspective and *without* access to ground-truth references. |
| Outcome: | The proposed model can detect and filter out low-quality captions on previously unseen images. |
Cross-modal Coherence Modeling for Caption Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for image captioning do not guarantee consistent image-text relations . current models do not provide enough data for training robust captioning models . |
| Approach: | They use an annotation protocol specifically devised for capturing image–caption coherence relations to study image captioning. |
| Outcome: | The proposed protocol improves image captioning models with coherence relations . the dataset is large enough to alleviate content hallucinations, the authors show . |
CrossVQA: Scalably Generating Benchmarks for Systematically Testing VQA Generalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual question answering models leverage spurious biases and take shortcuts to improve performance. |
| Approach: | They propose a semi-automatic framework for generating disentangled shifts by introducing a controllable visual question-answer generation module that generates highly-relevant question-announcer pairs with the desired dataset style. |
| Outcome: | The proposed framework generates highly-relevant and diverse question-answer pairs with the desired dataset style. |
Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering (D19-1)
Copied to clipboard
| Challenge: | Object detection is used in vision and language tasks but is expensive to learn . popular models rely on annotating ground-truths for bounding boxes and semantic labels . empirically, object detection leads to effective transfer learning and improved captioning and visual question answering models . |
| Approach: | They examine the effect of decoupling box proposal and featurization on down-stream tasks . they propose a family of "two-stage" object detectors that propose category-agnostic bounding boxes . |
| Outcome: | The proposed method improves image captioning and visual question answering models by leveraging large amounts of labeled annotations. |