Papers by Mike Zhang
Measuring and Narrowing the Compositionality Gap in Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a language model can correctly answer all sub-problems but not generate the overall solution. |
| Approach: | They propose a method that asks itself and then answers follow-up questions to narrow the compositionality gap by reasoning explicitly instead of implicitly. |
| Outcome: | The proposed method improves on chain of thought by asking itself and answering follow-up questions. |
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data (2026.acl-long)
Copied to clipboard
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera, Sara Hincapié Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Akinyi Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Jayne Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob Van Der Goot, Lanwenn ar C’horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin L Rice, Azril Hafizi Amirudin, Jesujoba Oluwadara Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, null Akshata, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Rufaro Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray, Sarah K. K. Luger
| Challenge: | Language identification (LID) is a fundamental step in curating multilingual corpora. |
| Approach: | They introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. |
| Outcome: | The proposed benchmark covers 109 languages and shows that existing evaluations overestimate accuracy for many languages in the web domain. |
ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain (2023.acl-long)
Copied to clipboard
| Challenge: | Increasing number of NLP benchmarks highlight need for multilingual models for job-related tasks. |
| Approach: | They introduce a language model called ESCOXLM-R that uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations taxonomy. |
| Outcome: | The proposed model outperforms XLM-R-large on short spans and entity-level and surface-level span-F1 tasks on entity- and surface level. |
SkillSpan: Hard and Soft Skill Extraction from English Job Postings (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing studies on Skill Extraction (SE) use crowd-sourced labels or annotations from a predefined skill inventory. |
| Approach: | They propose a dataset that contains 14.5K sentences and over 12.5K annotated spans. |
| Outcome: | The proposed model outperforms non-adapted models and single-task outperformed multi-task learning. |
MDCR: A Dataset for Multi-Document Conditional Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | ConditionalQA is limited to questions on single documents, neglecting harder cases that may require *cross-document reasoning* and *optimization*. |
| Approach: | They propose to use a dataset to evaluate models' ability to answer eligibility questions on single documents. |
| Outcome: | The proposed dataset can reflect real-world challenges and serve as a test bed for complex conditional reasoning that requires optimization. |
Kompetencer: Fine-grained Skill Classification in Danish Job Postings via Distant Supervision and Transfer Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Several studies focus on Skill Identification, but there is little work in further categorizing the identified skills. |
| Approach: | They propose a Danish job posting dataset annotated for nested spans of competences . they use the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy API to obtain fine-grained labels via distant supervision. |
| Outcome: | The proposed dataset outperforms existing models in the Danish job postings. |
BLADE: Benchmarking Language Model Agents for Data-Driven Science (2024.findings-emnlp)
Copied to clipboard
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai Zhang, Lanyi Zhu, Mike Merrill, Jeffrey Heer, Tim Althoff
| Challenge: | Language model-based agents can be used to conduct and support data-driven science, but evaluating them on open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. |
| Approach: | They propose a benchmark to automatically evaluate agents’ multifaceted approaches to open-ended research questions. |
| Outcome: | BLADE evaluates agents’ multifaceted approaches to open-ended research questions using data from 12 datasets and research questions drawn from existing scientific literature. |
Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls (2026.acl-industry)
Copied to clipboard
Rasmus T. Aavang, Rasmus Tjalk-Bøggild, Alexandre Iolov, Giovanni Rizzi, Mike Zhang, Johannes Bjerva
| Challenge: | Earnings calls are a key source of financial information about public companies. extracting information from earnings calls is difficult. |
| Approach: | They propose to use LLMs to perform open-ended extraction from unstructured call transcripts to provide a baseline for this valuable domain through the consistent tracking of emergent KPIs. |
| Outcome: | The proposed method provides a baseline for this valuable domain through the consistent tracking of emergent KPIs. |
Cartography Active Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to label data are limited in their notion of informativeness, due to post-training model uncertainty and batch diversity. |
| Approach: | They propose a new Active Learning algorithm that exploits the behavior of the model on individual instances during training as a proxy to find the most informative instances for labeling. |
| Outcome: | The proposed method is competitive to other common AL methods, showing that training dynamics derived from small seed data can be successfully used for AL. |
Experimental Standards for Deep Learning in Natural Language Processing Research (2022.findings-emnlp)
Copied to clipboard
Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, Barbara Plank
| Challenge: | a lack of common experimental standards remains an open challenge to the field at large . |
| Approach: | They propose to distill discussions on experimental standards into a single, widely-applicable methodology. |
| Outcome: | Using best practices, we can strengthen experimental evidence, improve reproducibility and enable scientific progress. |
Can Humans Identify Domains? (2024.lrec-main)
Copied to clipboard
Maria Barrett, Max Müller-Eberstein, Elisa Bassignana, Amalie Brogaard Pauli, Mike Zhang, Rob van der Goot
| Challenge: | Textual domain is a crucial property within the Natural Language Processing community due to its effects on downstream model performance. |
| Approach: | They examine the level of human disagreement and the relative difficulty of each annotation task by training classifiers to perform the same task. |
| Outcome: | The authors show that human proficiency in identifying related intrinsic textual properties is low and that disagreements are high. |
BTS: Harmonizing Specialized Experts into a Generalist LLM (2025.emnlp-main)
Copied to clipboard
Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X. Cai, Jakob Nicolaus Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, Suchin Gururangan, Mike Lewis
| Challenge: | Branch-Train-Stitch (BTS) is an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. |
| Approach: | They propose an efficient and flexible training algorithm for combining large language model (LLM) experts into a single, capable generalist model using lightweight stitch layers. |
| Outcome: | The proposed model can generalize to new domains despite being frozen . it yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts. |
Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning (2024.acl-long)
Copied to clipboard
Shivalika Singh, Freddie Vargus, Daniel D’souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O’Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, Sara Hooker
| Challenge: | Existing datasets in the English language are mostly in the realm of instruction fine-tuning . aya dataset, the Aya Collection, and the AYa Evaluation Suite are key resources . |
| Approach: | They aim to build a human-curated instruction-following dataset spanning 65 languages . they work with fluent speakers of languages from around the world to collect natural instances of instructions and completions . |
| Outcome: | The goal is to build a human-curated instruction-following dataset spanning 65 languages. |
Entity Linking in the Job Market Domain (2024.findings-eacl)
Copied to clipboard
| Challenge: | In Natural Language Processing, entity linking (EL) has centered around Wikipedia, but yet remains underexplored for the job market domain. |
| Approach: | They propose to use a bi-encoder and an autoregressive model to link fine-grained span-level skill mentions to a specific taxonomy entry to quantify labor market demands. |
| Outcome: | The proposed model outperforms GENRE in strict evaluation, but performs better in loose evaluation. |
Effective Long-Context Scaling of Foundation Models (2024.naacl-long)
Copied to clipboard
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, Hao Ma
| Challenge: | Large language models (LLMs) are rapidly deployed and continue to evolve through scaling. |
| Approach: | They propose a method to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. |
| Outcome: | The proposed model can surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks with a cost-effective instruction tuning procedure that is free of expensive annotations. |
Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Method (2025.acl-long)
Copied to clipboard
| Challenge: | Existing RAG solutions address the alignment problem in a limited manner . ARM explores relationships among data objects, enabling a retrieve-all-at-once solution for complex queries . |
| Approach: | Experimental results show that ARM improves alignment of open-domain questions with available data . ARM explores relationships among data objects, enabling a retrieve-all-at-once solution for complex queries. |
| Outcome: | Experimental results show that ARM outperforms existing RAG methods on complex open-domain questions. |
Follow the Path: Reasoning over Knowledge Graph Paths to Improve Large Language Model Factuality (2026.findings-acl)
Copied to clipboard
| Challenge: | fs1 improves factuality of reasoning traces by sourcing them from large reasoning models and conditioning them on knowledge graph paths. |
| Approach: | They propose a method that improves the factuality of reasoning traces by sourcing them from large reasoning models and grounding them by conditioning on knowledge graph (KG) paths. |
| Outcome: | The proposed method outperforms instruction-tuned models on open-domain questions . it significantly improves model performance over more complex questions and numerical answer types compared to baselines. |
NNOSE: Nearest Neighbor Occupational Skill Extraction (2024.eacl-long)
Copied to clipboard
| Challenge: | a new method for extracting occupational skills from text is needed to overcome the scarcity of skills across datasets. |
| Approach: | They propose a method that leverages multiple datasets to extract occupational skills from text . they propose NNOSE to extract neighboring skills from other datasets . |
| Outcome: | The proposed method improves infrequent skill extraction without additional fine-tuning. |
SHADES: Towards a Multilingual Assessment of Stereotypes in Large Language Models (2025.naacl-long)
Copied to clipboard
Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aimée Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, Nikita Nangia, Anaelia Ovalle, Giada Pistilli, Dragomir Radev, Beatrice Savoldi, Vipul Raheja, Jeremy Qin, Esther Ploeger, Arjun Subramonian, Kaustubh Dhole, Kaiser Sun, Amirbek Djanibekov, Jonibek Mansurov, Kayo Yin, Emilio Villa Cueva, Sagnik Mukherjee, Jerry Huang, Xudong Shen, Jay Gala, Hamdan Al-Ali, null Tair Djanibekov, Nurdaulet Mukhituly, Shangrui Nie, Shanya Sharma, Karolina Stanczak, Eliza Szczechla, Tiago Timponi Torrent, Deepak Tunuguntla, Marcelo Viridiano, Oskar Van Der Wal, Adina Yakefu, Aurélie Névéol, Mike Zhang, Sydney Zink, Zeerak Talat
| Challenge: | Large Language Models reproduce and exacerbate social biases present in training data, and resources to quantify this issue are limited. |
| Approach: | They propose a multilingual parallel dataset to examine culturally-specific stereotypes that may be learned by LLMs. |
| Outcome: | The proposed dataset includes stereotypes from 20 regions around the world and 16 languages, spanning multiple identity categories subject to discrimination worldwide. |
Evidence > Intuition: Transferability Estimation for Encoder Selection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LM transferability have focused on a priori tuning of encoders . prior work has examined the different yet related tasks of performance prediction . |
| Approach: | They propose to generate quantitative evidence to predict which LM will perform best on a target task without fine-tuning all candidates. |
| Outcome: | The proposed model outperforms the standard of human practitioner ranking in 94% of the setups. |