Papers by Ivan Kobyzev
Resonance RoPE: Improving Context Length Generalization of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks. |
| Approach: | They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions. |
| Outcome: | The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios. |
OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for detecting hallucinations and omissions in Machine Translation systems focus on analyzing the model’s internal states or relying on external tools. |
| Approach: | They propose an Optimal Transport-based word aligner specifically designed to enhance the detection of hallucinations and omissions in Machine Translation systems. |
| Outcome: | The proposed method is competitive with state-of-the-art methods across 18 language pairs on the HalOmi benchmark and shows promising features. |
Efficient Classification of Long Documents via State-Space Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing transformer-based models can only process long documents with limited computational resources due to their quadratic computation time and space. |
| Approach: | They propose to use state-space models for long document classification tasks instead of using sparse or hierarchical structures to solve this problem. |
| Outcome: | The proposed model performs comparable to self-attention models while being 36% more efficient. |
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that the flatness of the local minimum correlates well with better generalization. |
| Approach: | They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs. |
| Outcome: | The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost. |
Polarized-VAE: Proximity Based Disentangled Representation Learning for Text Generation (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for learning disentangled representations of real-world data focus on attribute labels or unsupervised methods that manipulate factorization in the latent space of models such as the variational autoencoder (VAE). |
| Approach: | They propose an approach that disentangles select attributes in the latent space based on proximity measures reflecting the similarity between data points with respect to these attributes. |
| Outcome: | The proposed method outperforms the VAE baseline and is competitive with state-of-the-art approaches while being more a general framework applicable to other attribute disentanglement tasks. |
ReGLA: Refining Gated Linear Attention (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) are known for their computational and storage requirements due to the quadratic computation complexity of softmax attention. |
| Approach: | They propose to reduce the quadratic computation complexity of softmax attention by using feature maps, normalization and the gating mechanism to improve performance. |
| Outcome: | The proposed model outperforms existing gated linear attention models in extensive tasks including training from scratch and post-linearization with continual pre-training. |
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) do not mitigate the noise in the teacher’s output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. |
| Approach: | They propose a method that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds. |
| Outcome: | The proposed method achieves state-of-the-art performance on NLU and computer vision tasks. |
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for regularizing deep neural networks rely on weight decay, dropout, batch/layer normalization to converge faster and generalize. |
| Approach: | They propose a framework for training with label regularization which includes conventional LS but can also model instance-specific variants. |
| Outcome: | The proposed approach consistently yields better results than conventional regularization on seven machine translation and three image classification tasks while maintaining training efficiency. |
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Existing hybridization schemes use sliding-window attention (SWA) to reduce KV cache usage and improve latency. |
| Approach: | They propose a training-free method that decomposes a large neighborhood search problem into three subproblems and a method that uses black-box binary optimization for short-context head selection. |
| Outcome: | Extensive experiments on 4 LLMs show that BOSCH outperforms layer-level heuristics and 6 strong static head-level methods with larger gains at higher SWA ratios. |
Do we need Label Regularization to Fine-tune Pre-trained Language Models? (2023.eacl-main)
Copied to clipboard
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh, Tianda Li, Alan Do-Omri, Peng Lu, Pascal Poupart, Ali Ghodsi
| Challenge: | Knowledge Distillation (KD) is a label regularization technique that can be replaced with lighter teacher-free variants such as the label-smoothing technique. |
| Approach: | They propose to use knowledge distillation to train student models by deploying the teacher network during training. |
| Outcome: | The proposed method can be replaced with lighter teacher-free variants on PLMs with more than 600 distinct trials and ran each configuration five times. |
Integral Transformer: Denoising Attention, Not Too Much Not Too Little (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to reduce attention noise by integrating signals from logit distributions are prone to attention noise. |
| Approach: | They propose a self-attention mechanism that integrates signals from the logit distribution to denoise attention. |
| Outcome: | The proposed model outperforms vanilla, Cog, and Differential attention variants on knowledge and reasoning benchmarks. |
Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Speculative decoding uses a small draft model to generate a single input token, instead of sequentially generating tokens until completion. |
| Approach: | They propose a method that generates varying draft models adapted to the input context using simple rules. |
| Outcome: | The proposed method is competitive with the current SOTA for self-speculative decoding while being a truly plug-and-play method. |
Attribute Controlled Dialogue Prompting (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-tuning is an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks. |
| Approach: | They propose an instance-specific prompt-tuning algorithm for dialog generation that generates prompts based on instance-level control code rather than the conversation history. |
| Outcome: | The proposed prompt-tuning module is a fraction of the size of the pretrained language model and saves memory and expensive storage space. |
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry. |
| Approach: | They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank. |
| Outcome: | The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA . |