Papers by Koushik Sen
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations? (2024.findings-acl)
Copied to clipboard
| Challenge: | Language models are more proficient at code generation, but they still generate incorrect programs. |
| Approach: | They define a group of models that have a high log-probability and weak correctness checks. |
| Outcome: | The proposed model samples fail to understand counterfeits through three clear failure modes . counterfeits are confusing to the model as they are to other models, the authors say . |
Benchmarking Language Models for Code Syntax Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models capture the syntactic rules of natural languages without fine-tuning on syntax understanding tasks. |
| Approach: | They propose a benchmarking test to compare pre-trained language models with a large-scale dataset of programs annotated with syntactic relationships in their corresponding abstract syntax trees. |
| Outcome: | The proposed model fails to match baselines based on positional offsets and keywords. |
PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are a powerful transfer learning approach for knowledge graph (KG) completion. |
| Approach: | They propose a parameter-lite transfer learning approach for pretrained language models for knowledge graph (KG) completion. |
| Outcome: | The proposed model outperforms the state-of-the-art models on a knowledge graph completion benchmark by tuning 1% of the parameters. |
SlimFit: Memory-Efficient Fine-Tuning of Transformer-based Models Using Training Dynamics (2024.naacl-long)
Copied to clipboard
Arash Ardakani, Altan Haan, Shangyin Tan, Doru Thom Popovici, Alvin Cheung, Costin Iancu, Koushik Sen
| Challenge: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
| Approach: | They propose a tool that analyzes transformer-based models and freezes less-contributory layers during fine-tuning to reduce the overall on-device memory usage. |
| Outcome: | SlimFit reduces the memory requirements of transformer-based models by analyzing their training dynamics and freezing less-contributory layers during fine-tuning. |
LangProBe: a Language Program Benchmark (2025.findings-emnlp)
Copied to clipboard
Shangyin Tan, Lakshya A Agrawal, Arnav Singhvi, Liheng Lai, Michael J Ryan, Dan Klein, Omar Khattab, Koushik Sen, Matei Zaharia
| Challenge: | Composing language models into multi-step language programs is a mainstream paradigm for building AI systems, but tradeoffs in this space have only scarcely been studied before. |
| Approach: | They propose a benchmarking tool to evaluate the architectures and optimization strategies for language programs . they find that optimized language programs offer strong cost-quality Pareto improvement . |
| Outcome: | The proposed framework evaluates the impact of program architectures and optimizers on quality and cost. |