Papers by Eric Davis
KoBEST: Korean Balanced Evaluation of Significant Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | a well-formulated benchmark allows objective and precise evaluation of diverse models. |
| Approach: | They propose a benchmark for Korean balanced evaluation of significant tasks that requires advanced Korean linguistic knowledge. |
| Outcome: | The proposed benchmarks are based on five Korean-language downstream tasks . the data is annotated by humans and thoroughly reviewed to guarantee high data quality. |
What, When, and How to Ground: Designing User Persona-Aware Conversational Agents for Engaging Dialogue (2023.acl-industry)
Copied to clipboard
| Challenge: | a personalized dialogue system can generate user-customized responses based on long-term memory about the user's persona. |
| Approach: | They propose a method for building a personalized open-domain dialogue system . they combine weighted dataset blending and negative persona information augmentation methods . |
| Outcome: | The proposed method balances dialogue fluency and tendency to ground while introducing a response-type label to improve controllability and explainability of the grounded responses. |
TelBench: A Benchmark for Evaluating Telco-Specific Large Language Models (2024.emnlp-industry)
Copied to clipboard
Sunwoo Lee, Dhammiko Arya, Seung-Mo Cho, Gyoung-eun Han, Seokyoung Hong, Wonbeom Jang, Seojin Lee, Sohee Park, Sereimony Sek, Injee Song, Sungbin Yoon, Eric Davis
| Challenge: | a growing demand for Large Language Models (LLMs) is requiring specialized models to augment customer service agents' skills. |
| Approach: | They propose a methodology for developing a specialized Telecommunications LLM . they use a dataset to evaluate customer service expertise in the telecommunications domain . |
| Outcome: | The proposed model improves the efficiency of customer service agents and reduces response times. |
TelAgentBench: A Multi-faceted Benchmark for Evaluating LLM-based Agents in Telecommunications (2025.emnlp-industry)
Copied to clipboard
Sunwoo Lee, Daseong Jang, Dhammiko Arya, Gyoung-eun Han, Injee Song, SaeRom Kim, Sangjin Kim, Seojin Lee, Seokyoung Hong, Sereimony Sek, Seung-Mo Cho, Sohee Park, Sungbin Yoon, Wonbeom Jang, Eric Davis
| Challenge: | Large Language Models (LLMs) are becoming powerful agentic systems . generic benchmarks fail to assess realistic, non-English performance . |
| Approach: | They propose to evaluate five core agentic capabilities: Reasoning, Planning, Action (tool-use), Retrieval-Augmented Generation, and Instruction Following. |
| Outcome: | The evaluations reveal significant performance disparities between models that employ explicit reasoning and those that do not. |
The Indigenous Languages Technology project at NRC Canada: An empowerment-oriented approach to developing language software (2020.coling-main)
Copied to clipboard
Roland Kuhn, Fineen Davis, Alain Désilets, Eric Joanis, Anna Kazantseva, Rebecca Knowles, Patrick Littell, Delaney Lothian, Aidan Pine, Caroline Running Wolf, Eddie Santos, Darlene Stewart, Gilles Boulianne, Vishwa Gupta, Brian Maracle Owennatékha, Akwiratékha’ Martin, Christopher Cox, Marie-Odile Junker, Olivia Sammons, Delasie Torkornoo, Nathan Thanyehténhas Brinklow, Sara Child, Benoît Farley, David Huggins-Daines, Daisy Rosenblum, Heather Souter
| Challenge: | This paper describes the first, three-year phase of a project at the National Research Council of Canada that is developing software to assist Indigenous communities in preserving their languages and extending their use. |
| Approach: | They describe the first phase of a project at the National Research Council of Canada that is developing software to assist Indigenous communities in preserving their languages. |
| Outcome: | The proposed software will help Indigenous communities preserve and revitalize their languages and extend their use. |
Adaptively profiling models with task elicitation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language model evaluations fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. |
| Approach: | They propose a method that automatically builds new evaluations to profile model behavior. |
| Outcome: | The proposed method finds that language models fail in hundreds of tasks . it also finds that o3-mini is prone to hallucination when fabrications are repeated . |