Papers by R. Manmatha
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Encoder-decoder transformer models suffer from high inference latency due to auto-regressive decoding . Typically, the decoder takes up most of the latency because of the auto-decoding - a problem that is not solved by the current model. |
| Approach: | They propose an approach to perform Dynamic Early Exit on Decoder to reduce inference latency by 20%-74% by using a multi-exit encoder-decoder transformer model trained with deep supervision. |
| Outcome: | The proposed model reduces inference latency by 20%-74% with comparable or even higher accuracy compared to baseline models. |
DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models (2024.emnlp-main)
Copied to clipboard
Sungnyun Kim, Haofu Liao, Srikar Appalaraju, Peng Tang, Zhuowen Tu, Ravi Satzoda, R. Manmatha, Vijay Mahadevan, Stefano Soatto
| Challenge: | Existing methods for visual document understanding are limited by training on a small-scale, curated document dataset, compromising generalizability of VDU models to diverse documents. |
| Approach: | They propose a framework that integrates external document knowledge into the data generation process. |
| Outcome: | The proposed framework produces high-quality annotations and surpasses direct knowledge distillation approach. |
R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding (2025.findings-acl)
Copied to clipboard
Joonhyung Park, Peng Tang, Sagnik Das, Srikar Appalaraju, Kunwar Yashraj Singh, R. Manmatha, Shabnam Ghadar
| Challenge: | Existing vision-only GUI agents ground elements from large and cluttered screenshots, requiring them to process substantial irrelevant information that compromises their accuracy. |
| Approach: | They propose a visual agent model for GUI automation that leverages zoomed-in region proposals for precise element localization. |
| Outcome: | The proposed approach improves state-of-the-art grounding accuracy by 13% across diverse GUI platforms on the GUI grounding benchmarks ScreenSpot and AgentStudio. |