Papers by Ganesh Jawahar
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Neural architecture search (NAS) uses weight-sharing supernets to generate diverse subnetworks without retraining. |
| Approach: | They propose a weight-sharing supernet that leverages mixture-of-experts to enhance supernet model expressiveness with minimal training overhead. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) performance in NAS for fast machine translation models, surpassing NAS-BERT and AutoDistil across various model sizes. |
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation (2023.findings-acl)
Copied to clipboard
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks Lakshmanan, V.S., Ahmed Hassan Awadallah, Sebastien Bubeck, Jianfeng Gao
| Challenge: | Existing MoE designs do not consider computational constraints (e.g., FLOPs, latency) Existing works in MoE consider homogeneous design where the same number of experts of the same size are placed uniformly throughout the network. |
| Approach: | They propose a framework for designing heterogeneous MoEs under computational constraints. |
| Outcome: | The proposed framework achieves 4x inference speedup and FLOPs reduction over manual models and within 1 BLEU point of MoE SwitchTransformer over benchmark datasets for NMT. |
What Does BERT Learn about the Structure of Language? (P19-1)
Copied to clipboard
| Challenge: | BERT is a language representation model that has performed well in diverse language understanding benchmarks. |
| Approach: | They perform experiments to unpack the elements of English language structure learned by BERT. |
| Outcome: | The proposed model outperforms state-of-the-art models in the GLUE benchmark by a significant margin. |
Contrastive Learning of Sociopragmatic Meaning in Social Media (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent progress in representation and contrastive learning in NLP has not considered the class of sociopragmatic meaning (i.e., meaning in interaction within different language communities). |
| Approach: | They propose a framework for learning task-agnostic representations transferable to a wide range of sociopragmatic tasks. |
| Outcome: | The proposed framework outperforms other contrastive learning frameworks for both in-domain and out-of-domain data, across both the general and few-shot settings. |
Automatic Detection of Entity-Manipulated Text using Factual Knowledge (2022.acl-short)
Copied to clipboard
| Challenge: | Current fake news detectors that exploit stylometric signals from the text are insufficient for distinguishing manipulated text from human written text. |
| Approach: | They propose a neural network detector that detects manipulated news articles by reasoning about the facts mentioned in the article. |
| Outcome: | The proposed detector outperforms the state-of-the-art detector in accuracy. |
Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora (2020.acl-main)
Copied to clipboard
| Challenge: | comparing two corpus texts and searching for words that differ in their usage between them is a common problem in digital humanities and computational social science. |
| Approach: | They propose an alternative approach that does not use vector space alignment, and instead considers the neighbors of each word. |
| Outcome: | The proposed method is interpretable and stable in 9 different setups and is highly reliable. |
LLM Performance Predictors are good initializers for Architecture Search (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have diverse applications, encompassing both open-ended tasks (e.g., brainstorming and chat) and closed-ended ones (eg. question answering). |
| Approach: | They construct PP prompts for Large Language Models (LLMs) that estimate the performance of specific deep neural network architectures on downstream tasks. |
| Outcome: | The proposed model achieves a SoTA mean absolute error and a slight degradation in rank correlation coefficient compared to baseline predictors in machine translation tasks. |
Automatic Detection of Machine Generated Text: A Critical Survey (2020.coling-main)
Copied to clipboard
| Challenge: | Current text generative models excel in producing text that matches the style of human language reasonably well. |
| Approach: | They conduct an in-depth error analysis of the state-of-the-art detector and discuss research directions to guide future work in this exciting area. |
| Outcome: | The proposed detectors can distinguish between human and text generated by the model and can be used to generate fake news and fake product reviews. |