ReportGPT: Human-in-the-loop Verifiable Table-to-Text Generation (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Recent advances in the quality and accessibility of large language models have precipitated a surge in user-facing tools for content generation. |
| Approach: | They propose a pipeline framework for verifiable human-in-the-loop table-to-text generation that is based on a domain specific language and a set of modules that use it as a representation for generating verifierable commentary. |
| Outcome: | The proposed framework learns from human feedback in real-time, needing only a few samples to improve performance. |
Similar Papers
OpenT2T: An Open-Source Toolkit for Table-to-Text Generation (2024.emnlp-demo)
Copied to clipboard
Haowei Zhang, Shengyun Si, Yilun Zhao, Lujing Xie, Zhijian Xu, Lyuhao Chen, Linyong Nan, Pengcheng Wang, Xiangru Tang, Arman Cohan
| Challenge: | Existing methods for table-to-text generation are limited and benchmarked on a limited number of datasets. |
| Approach: | They propose to use open-source tools to reproduce existing large language models for performance comparison and expedite the development of new models. |
| Outcome: | The proposed toolkit compares existing large language models on 9 table-to-text generation datasets and maintains a leaderboard to provide insights for future work. |
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation (2022.emnlp-main)
Copied to clipboard
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel Weld
| Challenge: | Effective human evaluation of text generation tasks remains an important, open area for research. |
| Approach: | They propose a system for running standardized human evaluations across different generation tasks. |
| Outcome: | The proposed system produces standardized human evaluations across tasks . it crowdsources predictions and ranks systems on leaderboards . the proposed system is not reproducible over time and different annotator populations . |
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text (2021.acl-long)
Copied to clipboard
| Challenge: | evaluators distinguish between human- and machine-authored text in three domains without training . evals' accuracy improved up to 55%, but it did not significantly improve across the three domain. |
| Approach: | They examine the role untrained human evaluations play in NLG evaluation and propose ways to improve their evaluations. |
| Outcome: | The evaluators distinguished between human- and machine-authored text at random chance level without training, but their accuracy did not improve across the three domains. |
Automatic and Human-AI Interactive Text Generation (with a focus on Text Simplification and Revision) (2024.acl-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we focus on text-to-text generation, a class of natural language generation tasks, that takes a piece of text as input and then generates a revision that is improved according to some specific criteria. |
| Approach: | This tutorial focuses on text-to-text generation, a class of natural language generation tasks that takes a piece of text as input and generates a revision that is improved according to some specific criteria. |
| Outcome: | This tutorial focuses on text-to-text generation, a class of natural language generation tasks, that takes a piece of text as input and generates a revision that is improved according to some specificcriteria. |
Credible without Credit: Domain Experts Assess Generative Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | ChatGPT has been criticized for its lack of accuracy and coherence . authors argue that language models could replace search engines and make college essays obsolete . |
| Approach: | a team of 10 domain experts conducts an initial assessment of language models using 100 expert-written questions. |
| Outcome: | The results show that language models are mixed in their accuracy. |
AnalystBench: Benchmarking professional long-form report generation with web-mined multimodal tasks (2026.findings-acl)
Copied to clipboard
Chau Minh Pham, Zichao Wang, Puneet Mathur, Alexa Siu, Akriti Jain, Aparna Garimella, Ananya B. Sai, Nedim Lipka, Mohit Iyyer, Varun Manjunatha
| Challenge: | Existing benchmarks decompose the end-to-end professional report generation into individual components. |
| Approach: | They propose a benchmarking tool that evaluates 20 real-world professional report generation tasks grounded in multimodal document collections. |
| Outcome: | The proposed model outperforms closed-source models on executive summarization tasks but drops significantly on long-horizon synthesis tasks. |
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020.acl-main)
Copied to clipboard
Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, Salim Roukos
| Challenge: | Existing approaches to generating text from AMRs focus on training sequence-to-sequence or graph-tosequent models on annotated data. |
| Approach: | They propose a strong pre-trained language model with cycle consistency-based re-scoring to generate AMR text. |
| Outcome: | The proposed model outperforms existing methods on the English LDC2017T10 dataset. |
MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation (2025.naacl-long)
Copied to clipboard
Satya Krishna Gorti, Ilan Gofman, Zhaoyan Liu, Jiapeng Wu, Noël Vouitsis, Guangwei Yu, Jesse C. Cresswell, Rasa Hosseinzadeh
| Challenge: | Recent advances in text-to-SQL generation rely on large closed-source models that present challenges in accessibility, privacy, and latency. |
| Approach: | They propose to use open-source text-to-SQL models to critique SQL queries . their method evaluates multiple outputs simultaneously and is competitive with larger models . |
| Outcome: | The proposed method achieves state-of-the-art performance compared to open-source models while remaining competitive with larger models at a much lower cost. |
FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Graph-to-text (G2T) generation is an important task in natural language generation as it renders graphs accessible to non-technical users in downstream applications such as question answering. |
| Approach: | They propose a metric that correctly identifies factual faithfulness and uses it to determine if a triple is present in a generated text. |
| Outcome: | The proposed metric achieves highest correlation with human annotations on data correctness, data coverage, and relevance. |
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. |
| Approach: | They propose a data augmentation technique that leverages large-scale language models to generate real text samples from a mixture of real samples. |
| Outcome: | The proposed method outperforms existing methods on diverse classification tasks. |