Papers by Francesco Piccinno
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering (2023.acl-long)
Copied to clipboard
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Eisenschlos
| Challenge: | Visual language models that are pretraining on natural images or image-text pairs crawled from the web perform poorly on visual language tasks such as ChartQA and ChartQA. |
| Approach: | They propose to perform several pretraining tasks that cover plot deconstruction and numerical reasoning which are key capabilities in visual language modeling. |
| Outcome: | The proposed model outperforms state-of-the-art methods on benchmarks such as PlotQA and ChartQA by as much as 20%. |
Table-To-Text generation and pre-training with TabT5 (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) are limited when it comes to structured or semi-structured domains like tables. |
| Approach: | They propose an encoder-decoder model that generates natural language text based on tables and textual inputs. |
| Outcome: | TabT5 achieves 15% increase in sequence accuracy on spreadsheet formula prediction and data-to-text generation domains. |
STRUCTSUM Generation for Faster Text Comprehension (2024.acl-long)
Copied to clipboard
| Challenge: | Current large language models (LLMs) fail to adequately structure and organize information in a way that facilitates comprehension. |
| Approach: | They propose a taxonomy of problems around factuality, global and local structure common to both modalities and propose 'auto-QA' to improve the accuracy of generated structured representations. |
| Outcome: | The proposed models improve accuracy and speed without loss of accuracy. |
DePlot: One-shot visual language reasoning by plot-to-table translation (2023.findings-acl)
Copied to clipboard
Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun
| Challenge: | Existing models for visual language reasoning require tens of thousands of training examples and their reasoning capabilities are limited. |
| Approach: | They propose a one-shot solution to visual language reasoning by combining plot-to-text translation and reasoning over the translated text into a modality conversion module. |
| Outcome: | The proposed method improves on human-written queries on plots and charts compared with a fine-tuned SOTA model on human queries. |
Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs (2021.emnlp-main)
Copied to clipboard
| Challenge: | We present a new approach for weakly-supervised conversational Question Answering over Knowledge Graphs . |
| Approach: | They propose a Logical Form grammar that can model a wide range of queries on a Knowledge Graph while remaining sufficiently simple to generate supervision data efficiently. |
| Outcome: | The proposed grammar can model a wide range of queries while remaining simple to generate supervision data efficiently. |
mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models display surprising multilingual capabilities despite being pre-trained on English data. |
| Approach: | They propose a multilingual sequence-to-sequence model that disentangles language-specific information from language-agnostic information. |
| Outcome: | The proposed model outperforms existing models on representative natural language understanding and generation tasks in 40+ languages. |
Multimodal Chart Retrieval: A Comparison of Text, Table and Image Based Approaches (2024.naacl-long)
Copied to clipboard
| Challenge: | Multimodal chart retrieval is a task of retrieving information from a multimodal dataset, given a query. |
| Approach: | They compare four different approaches to multimodal chart retrieval . they use chart derendering, chart derending, table retrieval and Tab-GTR . |
| Outcome: | The proposed model outperforms the previous methods on the NQ-Tables benchmark and is significantly more efficient on in-distribution data. |
Answering Conversational Questions on Structured Data without Logical Forms (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to answering sequential questions based on structured objects do not use a logical form as an intermediate representation. |
| Approach: | They propose a novel approach to answering sequential questions based on structured objects without using a logical form as an intermediate representation. |
| Outcome: | The proposed approach is competitively tested on the Sequential Question Answering (SQA) task. |
Generating Logical Forms from Graph Representations of Text and Entities (P19-1)
Copied to clipboard
| Challenge: | Recent approaches to semantic parsing have cast it as a sequence-to-sequence task, with strong results. |
| Approach: | They propose a Graph Neural Network architecture to incorporate information about relevant entities and their relations during parsing. |
| Outcome: | The proposed approach outperforms the state-of-the-art in several tasks without pre-training and outperformed existing approaches when combined with BERT pre-trainment. |
TaPas: Weakly Supervised Table Parsing via Pre-training (2020.acl-main)
Copied to clipboard
| Challenge: | Answering natural language questions over tables is often seen as a semantic parsing task. |
| Approach: | They propose an approach to question answering over tables without generating logical forms by selecting table cells and optionally applying a corresponding aggregation operator. |
| Outcome: | The proposed approach outperforms or rivals existing models on three different datasets and performs on par with the state-of-the-art on WikiSQL and WikiTQ. |
Faithful Chart Summarization with ChaTS-Pi (2024.acl-long)
Copied to clipboard
| Challenge: | Chart-to-summary generation can help explore data, communicate insights, and help the visually impaired. |
| Approach: | They propose a reference-free chart summarization metric for scoring faithfulness . they use human raters to fix and rank candidate summaries from any chart-summarization model . |
| Outcome: | The proposed metric scores the summarization faithfulness according to human ratings better than reference-based metrics, either learned or n-gram based. |