Papers by Hamish Ivison
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors (2023.findings-acl)
Copied to clipboard
| Challenge: | Prior work shows training models on multitask data augmented with task descriptions transfers knowledge to new tasks. |
| Approach: | They propose to use unlabeled target-task data to train models on task descriptions . they use only 2% of the data from the P3 pool without labeled target task data . |
| Outcome: | The proposed model outperforms baseline models on 12 out of 14 datasets . it also provides better initialization than single model on target-task data . |
TESS: Text-to-Text Self-Conditioned Simplex Diffusion (2024.eacl-long)
Copied to clipboard
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, Arman Cohan
| Challenge: | Existing models for diffusion generation are expensive and discrete, resulting in a large number of diffusion steps to generate text. |
| Approach: | They propose a text diffusion model that is fully non-autoregressive and employs a new form of self-conditioning and applies the diffusion process on the logit simplex space rather than the learned embedding space. |
| Outcome: | The proposed model outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models. |
HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation (2023.acl-long)
Copied to clipboard
| Challenge: | Recent NLP models have shown the remarkable ability to generalise ‘zero-shot’ to new tasks using only natural language instructions as guidance. |
| Approach: | They introduce Hypernetworks for INstruction Tuning (HINT) which converts task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder. |
| Outcome: | The proposed models outperform strong state-of-the-art models by over 10% when controlling for compute. |
TESS 2: A Large-Scale Generalist Diffusion Language Model (2025.acl-long)
Copied to clipboard
| Challenge: | Existing instruction-following diffusion models are predominantly trained using an autoregressive paradigm. |
| Approach: | They propose a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models and matches and sometimes exceeds strong autoregressive (AR) models. |
| Outcome: | The proposed model outperforms and sometimes exceeds existing autoregressive (AR) models on a number of tasks. |
Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |