Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding (2025.naacl-long)
Copied to clipboard
| Challenge: | Flowcharts are typically presented as images, driving the trend of using vision-language models for end-to-end flowchart understanding. |
| Approach: | They propose a vision-language model (VLM) that generates textual representations from flowchart images and a textual Reasoner that performs question-answering based on the text representations. |
| Outcome: | Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow’s state-of-the-art performance as well as its robustness. |
Similar Papers
On the Perception Bottleneck of VLMs for Chart Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a perception bottleneck in large vision-language models is critical for chart understanding . instruction tuning improves the extraction capability of LVLMs, but the vision encoder remains a bottleneck . |
| Approach: | They propose to decompose the perception bottleneck into two components . the vision encoder bottleneck is where visual representation fails to encapsulate the correct information . |
| Outcome: | The proposed approach significantly mitigates the vision encoder bottleneck and improves the ability of LVLMs to comprehend charts. |
End-to-end Parsing of Procedural Text into Flow Graphs (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing flow graph parsers lack sufficient annotated data to train them . a lack of annotation can cause costly training, and poor flow graph training results in a large improvement. |
| Approach: | They propose a multi-task framework that performs tagging and graph generation simultaneously . they take advantage of the abundance of unlabelled recipes and generate noisy silver annotations . |
| Outcome: | The proposed model can unify the input representation and use compact encoders, resulting in small models with significantly fewer parameters than existing models. |
Code Generation From Flowcharts with Texts: A Benchmark Dataset and An Approach (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, researchers focus on generating codes from requirement documents. |
| Approach: | They propose to generate source code from flowcharts with texts instead of directly translating requirements into codes. |
| Outcome: | The proposed model improves on the baselines by transforming flowcharts into pseudo-code . the proposed model is based on 320 flowchartes with their corresponding source codes . |
StarFlow: Generating Structured Workflow Outputs From Sketch Images (2026.eacl-long)
Copied to clipboard
Patrice Bechard, Chao Wang, Amirhossein Abaskohi, Juan A. Rodriguez, Christopher Pal, David Vazquez, Spandana Gella, Sai Rajeswar, Perouz Taslakian
| Challenge: | Despite being widely used, building workflows can be complex, often requiring manual configuration through low-code platforms or visual programming tools. |
| Approach: | They propose a framework for generating structured workflow outputs from sketches using vision-language models to automate the process. |
| Outcome: | The proposed framework outperforms large vision-language models in the task of generating structured workflow outputs from sketches and diagrams. |
Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are highly sensitive to end-of-input artifacts in fine-tuning and inference data, e.g., whether input sequences end with punctuation or newline characters. |
| Approach: | They propose to convert generative LVLMs into vision-language encoders via contrastive learning objectives and use supervised contrastive objectives to train them. |
| Outcome: | The proposed approach improves visual and text representations and improves retrieval and (semantic) similarity tasks. |
Can VLMs Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision-language models integrate textual and visual information, enabling them to process visual inputs and generate predictions. |
| Approach: | They review work on modality collapse analysis to provide insights into the reason for this unintended behavior and review probing studies for fine-grained vision-language understanding. |
| Outcome: | The proposed models can achieve competitive performance in vision-language tasks despite relying heavily on textual information and ignoring visual information. |
Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning (2024.findings-emnlp)
Copied to clipboard
Mohammed Saidul Islam, Raian Rahman, Ahmed Masry, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, Enamul Hoque
| Challenge: | Recent studies have demonstrated that large vision language models (LVLMs) are not multi-modal and lack multi-tasking capabilities. |
| Approach: | They evaluate the performance of large vision language models (LVLMs) for chart understanding and reasoning tasks and compare them to open-source models. |
| Outcome: | The proposed models demonstrate impressive abilities in generating fluent texts covering high-level data insights, but they also encounter common problems like hallucinations, factual errors, and data bias. |
From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens (2025.emnlp-demos)
Copied to clipboard
Hala Sheta, Eric Haoran Huang, Shuyu Wu, Ilia Alenabi, Jiajun Hong, Ryker Lin, Ruoxi Ning, Daniel Wei, Jialin Yang, Jiawei Zhou, Ziqiao Ma, Freda Shi
| Challenge: | Existing vision-language models are based on exactmatch based accuracy and its derivations to evaluate performance. |
| Approach: | They propose a toolkit that supports systematic benchmarking, analysis, and interpretation of vision-language models by extracting intermediate outputs from any layer during the forward pass of open-source VLMs. |
| Outcome: | The proposed toolkit supports 16 state-of-the-art base VLMs and their over 30 variants and is extensible to accommodate new models without changing the core logic. |
Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing code generation benchmarks neglect flowchart-based code generation . existing benchmarks lack flowcharting-based evaluation, limiting the potential of large language models and minimizing human error. |
| Approach: | They propose to use flowcharts to evaluate existing LLMs' code generation capabilities. |
| Outcome: | The proposed benchmarks show that the supervised fine-tuning technique contributes greatly to the models’ performance. |
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Combining visual modality with pretrained language models has been effective for descriptive tasks such as image captioning. |
| Approach: | They ask: do multimodal models combine visual and visual adapted language models? they find that CLIP image representations and scaling of language models do not consistently improve self-rationalization in multimodal tasks. |
| Outcome: | The proposed model types do not consistently improve self-rationalization in multimodal tasks. |