Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are known to distill knowledge from vast datasets during the pretraining phase. |
| Approach: | They propose to use in-context concept definitions as prompts for zero-shot sentence labeling tasks. |
| Outcome: | The proposed model outperforms the leading open-source models and their proprietary APIs in most cases. |
Similar Papers
Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science (2024.lrec-main)
Copied to clipboard
Yida Mu, Ben P. Wu, William Thorne, Ambrose Robinson, Nikolaos Aletras, Carolina Scarton, Kalina Bontcheva, Xingyi Song
| Challenge: | Existing instruction-tuned Large Language Models (LLMs) have impressive language understanding and the capacity to generate responses that follow specific prompts. |
| Approach: | They evaluate the zero-shot performance of two publicly accessible LLMs, ChatGPT and OpenAssistant, in the context of six Computational Social Science classification tasks. |
| Outcome: | The proposed LLMs perform better than state-of-the-art models on social science tasks. |
Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptions (2025.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning has improved performance of large language models, but descriptive instructions are still under-explored. |
| Approach: | They propose an ensemble prompt framework to describe selection criteria of multiple in-context examples. preliminary experiments on machine translation confirm that this framework boosts ICL performance. |
| Outcome: | The proposed framework improves on commonsense, math, logical reasoning and hallucination tasks with three LLMs. |
Concept Space Alignment in Multilingual LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual large language models generalize somewhat across languages, but it is unclear whether this is a result of improved, implicit alignment, or of something else, e.g., linguistic overlap or semi-parallel subsets of training data. |
| Approach: | They hypothesize that implicit alignment is the reason for generalization in multilingual large language models. |
| Outcome: | The proposed model generalizes well across languages, but lacks linearity. |
Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent studies have focused on specific domains or types of persuasion, but a general study has focused on how LLMs produce persuasive text. |
| Approach: | They construct a dataset to measure and benchmark the ability of Large Language Models (LLMs) to produce persuasive text. |
| Outcome: | The proposed model can be used to generate persuasive text across domains and domains. |
Language Models for Text Classification: Is In-Context Learning Enough? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research on text classification models with prompts is limited in scale and lacks understanding of how these methods compare to more established methods. |
| Approach: | They compare the performance of large and smaller language models with prompts to achieve state-of-the-art performance in many NLP tasks. |
| Outcome: | The proposed models outperform the more standard approaches in binary, multiclass, and multilabel tasks in a large scale evaluation of 16 text classification datasets. |
Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines (2025.findings-acl)
Copied to clipboard
Do Xuan Long, Duong Ngoc Yen, Do Xuan Trong, Anh Tuan Luu, Kenji Kawaguchi, Shafiq Joty, Min-Yen Kan, Nancy F. Chen
| Challenge: | In-context learning is an important but not fully understood ability of pre-trained large language models. |
| Approach: | They propose a tool that generates two streams of guidelines capturing task language and format distributions and prompts them to define them by prompting. |
| Outcome: | The proposed model improves both strong open- and closed-source LLMs by over 5% in both zero- and few-shot settings. |
LLMs for Low Resource Languages in Multilingual, Multimodal and Dialectal Settings (2024.eacl-tutorials)
Copied to clipboard
| Challenge: | Recent advances in AI can be attributed to the remarkable performance of Large Language Models (LLMs) success of LLMs depends on specific training techniques, such as instruction tuning and prompting . |
| Approach: | They explore the capabilities of Large Language Models (LLMs) in various tasks and languages . they also examine their performance, fine-tuning, instructions tuning, and close vs. open models . |
| Outcome: | The proposed model can be used for speech and multimodal tasks across modalities, languages, and dialects. |
How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on enhancing the factualness of large language models using context knowledge. |
| Approach: | They propose to use ChatGPT to construct probing datasets that provide diverse and coherent evidence corresponding to various facts. |
| Outcome: | The proposed model can encode knowledge across different layers, and it is compared with existing models. |
Prompting Large Language Models for Counterfactual Generation: An Empirical Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have made remarkable progress in a wide range of natural language understanding and generation tasks, but their ability to generate counterfactuals has not been examined systematically. |
| Approach: | They propose a framework to evaluate LLMs' ability to generate counterfactuals based on key factors including intrinsic properties and prompt design. |
| Outcome: | The proposed framework examines the strengths and weaknesses of large language models (LLMs) and identifies factors that influence their ability to generate counterfactuals. |
Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Study (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation approaches to evaluate Large Language Models are affected by potential biases within LLMs. |
| Approach: | They propose two many-shot In-Context Learning (ICL) prompt templates to help LLM evaluators mitigate potential biases. |
| Outcome: | The proposed templates reduce biases by using in-context examples with model-generated rationales as references. |