| Challenge: | Existing models for data-to-text generation are wrongly generating estate in the output text. |
| Approach: | They propose a task that incorporates visual cues from tables and associated images to generate relevant text. |
| Outcome: | The proposed task incorporates visual cues from tables and associated images to generate relevant text. |
Similar Papers
ToTTo: A Controlled Table-To-Text Generation Dataset (2020.emnlp-main)
Copied to clipboard
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das
| Challenge: | Existing methods for data-to-text generation often hallucinate phrases not supported by the Wikipedia table. |
| Approach: | They propose a controlled task where annotators directly revise existing Wikipedia sentences to generate a one-sentence description. |
| Outcome: | The proposed task produces a one-sentence description from a Wikipedia table and highlighted cells. |
Beyond Embeddings: The Promise of Visual Table in Visual Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Visual representation learning has been a cornerstone in computer vision for decades. |
| Approach: | They propose a visual representation tailored for visual reasoning that provides instance-level world knowledge and detailed attributes that are essential for visual reason. |
| Outcome: | The proposed visual tables outperform existing models on 11 visual reasoning benchmarks. |
Table and Image Generation for Investigating Knowledge of Entities in Pre-trained Vision and Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing knowledge about entities acquired from natural language models is not retained in pre-trained vision & language models. |
| Approach: | They propose a task to verify how knowledge about entities acquired from natural language is retained in Vision & Language (V&L) models. |
| Outcome: | The proposed model forgets part of its entity knowledge by pre-training to improve image related tasks. |
PixT3: Pixel-based Table-To-Text Generation (2024.acl-long)
Copied to clipboard
| Challenge: | Table-to-text generation is a visual recognition task that uses textual descriptions from structured inputs. |
| Approach: | They propose to rethink data-to-text generation as a visual recognition task by removing the need for rendering the input in a string format. |
| Outcome: | The proposed model overcomes the challenges of linearization and input size limitations and is applicable to open-ended and controlled generation settings. |
Learning to Imagine: Visually-Augmented Natural Language Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for natural language generation are pre-trained on text-only corpora, resulting in visual commonsense. |
| Approach: | They propose a method that makes pre-trained language models learn to imagine for visually-augmented natural language generation. |
| Outcome: | The proposed method is compatible with Transformer-based architecture. |
Retrieving Multimodal Information for Augmented Generation: A Survey (2023.findings-emnlp)
Copied to clipboard
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, Shafiq Joty
| Challenge: | Large Language Models (LLMs) are increasingly using multimodality to augment their generation ability, but there is no unified perception of at which stage and how to incorporate different modalities. |
| Approach: | They propose to use multimodality to augment Large Language Models (LLMs) this will provide scholars with a deeper understanding of the methods' applications and encourage them to adapt existing techniques to the fast-growing field of LLMs. |
| Outcome: | The proposed methods improve factuality, reasoning, interpretability, and robustness of the generated content. |
OpenT2T: An Open-Source Toolkit for Table-to-Text Generation (2024.emnlp-demo)
Copied to clipboard
Haowei Zhang, Shengyun Si, Yilun Zhao, Lujing Xie, Zhijian Xu, Lyuhao Chen, Linyong Nan, Pengcheng Wang, Xiangru Tang, Arman Cohan
| Challenge: | Existing methods for table-to-text generation are limited and benchmarked on a limited number of datasets. |
| Approach: | They propose to use open-source tools to reproduce existing large language models for performance comparison and expedite the development of new models. |
| Outcome: | The proposed toolkit compares existing large language models on 9 table-to-text generation datasets and maintains a leaderboard to provide insights for future work. |
Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to condensing textual information into concise and structured tables are limited in their applicability in broader contexts. |
| Approach: | They propose a benchmark dataset for generating summary tables of competitions based on real-time commentary texts that incorporates large-scale textual information into concise and structured tables. |
| Outcome: | The proposed method exhibits strong generalization abilities, surpassing previous approaches on several other text-to-table datasets. |
Text-to-Table: A New Way of Information Extraction (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for information extraction are not well understood . text-to-table is a problem that aims to extract information from text data . |
| Approach: | They propose a new problem setting of information extraction, called text-to-table . they formalize text- to-table as a sequence-tosequence problem . |
| Outcome: | The proposed method outperforms existing methods on text-to-table tasks. |
TaKG: A New Dataset for Paragraph-level Table-to-Text Generation Enhanced with Knowledge Graphs (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing table-to-text generation benchmarks have some limitations, such as E2E and ToTTo focusing on singlesentence generation tasks. |
| Approach: | They propose a new table-to-text generation dataset called TaKG that uses a set of knowledge graphs to enhance table input. |
| Outcome: | The proposed model outperforms existing models for short-text generation tasks and shows reliable performance on long-text generated across a variety of metrics. |