| Challenge: | Large language models (LLMs) have shown their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc. |
| Approach: | They propose to use GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. |
| Outcome: | The proposed framework compares GPT-4 with human data analysts to perform end-to-end data analysis with databases from a wide range of domains. |
Similar Papers
Is GPT-3 a Good Data Annotator? (2023.acl-long)
Copied to clipboard
| Challenge: | Data annotation is the process of labeling data that could be used to train machine learning models. |
| Approach: | They evaluate the performance of a large-scale language model developed by OpenAI . they compare it with traditional methods and analyze its output on a range of tasks . |
| Outcome: | The proposed model has shown impressive performance on a range of NLP tasks. |
Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent large language models such as ChatGPT and GPT-4 have shown exceptional capabilities of generalist models . however, their applicability and effectiveness in specific domains like finance needs a better understanding . |
| Approach: | They conduct empirical studies to compare the performance of ChatGPT and GPT-4 on financial text analytical problems using eight benchmark datasets from five categories of tasks. |
| Outcome: | The proposed models outperform the state-of-the-art models on a wide range of financial text analytical tasks. |
Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape . established automatic evaluation metrics are poor surrogates, correlating weakly with human judgement. |
| Approach: | They propose to use both automatic and human evaluation to evaluate generative LLMs on three NLP benchmarks: text summarisation, text simplification and grammatical error correction. |
| Outcome: | The proposed model outperforms many popular models according to human reviewers on the majority of metrics, while scoring much worse when using classic automatic evaluation metrics. |
GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing LLM leaderboards often reference scores reported in other papers without consistent settings and prompts, which may encourage cherry-picking favored settings and for better results. |
| Approach: | They propose an open-source and reproducible LLM evaluation suite built on top of OpenAI Evals that systematically evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
| Outcome: | The evaluation suite is built on top of OpenAI Evals and evaluates 10+ leading LLMs and OpenAI’s legacy models on 20+ curated benchmarks across 7 capability categories. |
Information Extraction from Legal Wills: How Well Does GPT-4 Do? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Using information extraction from legal wills is an important application of artificial intelligence (AI) |
| Approach: | They propose a manually annotated dataset for Information Extraction (IE) from legal wills . they also use it to evaluate the performance of large language models (LLMs) |
| Outcome: | The proposed dataset can be used to evaluate large language models on IE from legal wills . it shows that the model performs reasonably well, but inconsistent outputs and overgeneralization are observed . |
Does GPT-4 pass the Turing test? (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) such as GPT-4 seem well designed for Turing's game. |
| Approach: | They evaluated GPT-4 in a public online Turing test and found it passed in 49.7% of games. |
| Outcome: | The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%) participants’ decisions were based mainly on linguistic style (35%) and socioemotional traits (27%). |
Leveraging GPT-4 for Automatic Translation Post-Editing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Neural Machine Translation models still require translation post-editing to rectify errors and enhance quality under critical settings. |
| Approach: | They use GPT-4 to automatically post-edit NMT outputs across several language pairs . they show that GPT4 is adept at translation post- editing, producing meaningful edits . |
| Outcome: | The proposed translation post-editor improves on state-of-the-art language models on English-Chinese, English-German, Chinese-English and German-English language pairs. |
Can LLMs replace Neil deGrasse Tyson? Evaluating the Reliability of LLMs as Science Communicators (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) and AI assistants are experiencing exponential growth in usage among expert and amateur users. |
| Approach: | They propose to assess the reliability of current Large Language Models as science communicators . they use a dataset comprising 742 Yes/No queries embedded in complex scientific concepts . |
| Outcome: | The proposed model outperforms open-access models in scientific question-answering tasks . the model outpersforms GPT-4 Turbo models in many evaluation aspects . |
Evaluating Large Language Models on Wikipedia-Style Survey Generation (2024.findings-acl)
Copied to clipboard
Fan Gao, Hang Jiang, Rui Yang, Qingcheng Zeng, Jinghui Lu, Moritz Blum, Tianwei She, Yuang Jiang, Irene Li
| Challenge: | Recent studies have shown that large language models can perform well in general tasks, but their effectiveness and limitations in domainspecific tasks remain unclear. |
| Approach: | They examine the proficiency of Large Language Models (LLMs) in generating succinct survey articles specific to the niche field of NLP in computer science. |
| Outcome: | The LLMs perform better in generating succinct survey articles specific to the niche field of NLP in computer science, compared to human-authored surveys, but they exhibit bias in evaluation. |
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | ChatGPT and GPT-4 are popular as evaluation metric for complex generative tasks . however, they are not ready as human replacements due to significant limitations . |
| Approach: | They conduct extensive analysis to examine the stability and reliability of LLMs as automatic evaluators for abstractive summarization. |
| Outcome: | The proposed methods outperform the commonly used automatic metrics but are not ready for human evaluation due to significant limitations. |