Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Evaluating generalisation capabilities of multimodal models based solely on performance on out-of-distribution data fails to capture their true robustness . proposed framework examines the role of instructions and inputs in generalisation abilities of such models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. |
| Approach: | They propose a framework that examines the role of instructions and inputs in the generalisation abilities of multimodal models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. |
| Outcome: | The proposed framework examines the role of instructions and inputs in the generalisation abilities of multimodal models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. |
Similar Papers
Robustness of Learning from Task Instructions (2023.findings-acl)
Copied to clipboard
| Challenge: | traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples. |
| Approach: | a new study investigates the system robustness when instructions are manipulated and paraphrased . task instructions give the model the definition of the task and allow it to output the appropriate answer . |
| Outcome: | a new study shows that supervised learning is robust when instructions are manipulated, paraphrased or iii from different levels of conciseness. |
Exploring Compositional Generalization of Large Language Models (2024.naacl-srw)
Copied to clipboard
| Challenge: | a recent study has found that large language models can generalize compositional instructions from simple instructions to complex ones. |
| Approach: | They study the generalization ability of large language models with respect to compositional instructions . they first construct a dataset with the help of ChatGPT guided by the self-instruct technique . |
| Outcome: | The proposed model can generalize from simple instructions to more intricate ones, the authors show . their results show that training LLMs on higher-order compositional instructions improves performance on lower-order ones, but not on higher order ones. |
Methods for Estimating and Improving Robustness of Language Models (2022.naacl-srw)
Copied to clipboard
| Challenge: | Large language models suffer from weak generalisation ability due to shallow textual relations over full semantic complexity of the problem. |
| Approach: | They propose to incorporate some of these measures into training objectives to enhance distributional robustness of LLMs. |
| Outcome: | The proposed models outperform human models on complex tasks and outperformed other models on deep networks. |
Robustness of Fusion-based Multimodal Classifiers to Cross-Modal Content Dilutions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on understanding the robustness of vision-and-language models to imperceptible variations on benchmark tasks. |
| Approach: | They develop a model that generates additional dilution text that maintains relevance and topical coherence with the image and existing text, and when added to the original text, leads to misclassification of the multimodal input. |
| Outcome: | The proposed model outperforms fusion-based classifiers on Crisis Humanitarianism and Sentiment Detection tasks by 23.3% and 22.5% in presence of dilutions generated by the model. |
Data Factors for Better Compositional Generalization (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent diagnostic datasets on compositional generalization expose severe problems . state-of-the-art models trained on larger and more general datasets show better generalization ability . |
| Approach: | They conduct an empirical analysis by training Transformer models on a variety of training sets with different data factors including dataset scale, pattern complexity, example difficulty, etc. |
| Outcome: | The proposed model training on larger datasets improves on compositional generalization tasks. |
Multimodal Instruction Tuning with Conditional Mixture of LoRA (2024.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have demonstrated proficiency in diverse tasks across different domains. |
| Approach: | They propose a method that integrates multimodal instruction tuning with Conditional Mixture-of-LoRA. |
| Outcome: | Experimental results show that MixLoRA outperforms LoRA with the same or higher ranks . MLLMs have demonstrated remarkable proficiency in diverse tasks across domains . |
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022.emnlp-main)
Copied to clipboard
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen
| Challenge: | a benchmark of 1,616 diverse NLP tasks and their expert-written instructions is used to test generalization of models to unseen tasks . a recent study shows that instruction-following models outperform instruction-based models by over 9% . |
| Approach: | They build a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. |
| Outcome: | The proposed model outperforms existing instruction-following models by over 9% on the benchmark despite being smaller. |
Sequential Compositional Generalization in Multimodal Models (2024.naacl-long)
Copied to clipboard
| Challenge: | a growing number of multimodal models have a limited capacity for generalization . however, prior studies into compositionality have focused on visual grounding and downstream tasks like image captioning. |
| Approach: | They examine compositional generalization using egocentric kitchen activity videos . they find bi-modal and tri-modal models exhibit a clear edge over their text-only counterparts . |
| Outcome: | The proposed model outperforms text-only models in a multimodal setting. |
JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing text-to-text methods struggle with issues such as generalization, robustness, and controllability due to their lack of explicit task structures. |
| Approach: | They propose a structure-to-structure approach that uses JSON structures to represent tasks. |
| Outcome: | The proposed method outperforms TextTuning in terms of performance, robustness, and controllability across different scenarios. |
MultiSkill: Evaluating Large Multimodal Models for Fine-grained Alignment Skills (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing evaluation settings for large multimodal models focus on coarse-grained evaluation without considering skill composition required by specific instructions. |
| Approach: | They propose an evaluation protocol that assesses large multimodal models across multiple fine-grained skills for alignment with human values. |
| Outcome: | The proposed evaluation protocol decomposes coarse-level scoring to fine-grained skill set-level score tailored to each instruction. |