Papers by Christopher Lucas
Non-Compositionality in Sentiment: New Data and Analyses (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Many studies on sentiment analysis focus on the fact that sentiment computations are compositional . linguistic utterances often do not adhere to strict patterns and can be surprising when looking at the individual words involved. |
| Approach: | They propose a method for obtaining non-compositionality ratings for phrases with respect to their sentiment . they also propose evaluating computational models for sentiment analysis using the rating resource . |
| Outcome: | The proposed method enables non-compositional ratings for phrases with respect to their sentiment . the results are compared with a new resource of ratings for 259 phrases . |
Discovering Language Model Behaviors with Model-Written Evaluations (2023.findings-acl)
Copied to clipboard
Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
| Challenge: | Prior work creates evaluations with crowdwork or existing data sources, which are not always available. |
| Approach: | They generate evaluations automatically with language models (LMs) using crowdwork or existing data sources to find out how they behave . |
| Outcome: | The results show that large LMs repeat back a dialog user’s preferred answer and express greater desire to pursue concerning goals like resource acquisition and goal preservation. |
Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation (2022.acl-long)
Copied to clipboard
| Challenge: | Unlike literal expressions, idioms’ meanings do not follow from their parts, posing a challenge for neural machine translation (NMT). |
| Approach: | They examine the mechanics of the dominant NMT model, Transformer, and their effect on their understanding of idioms. |
| Outcome: | The proposed model over-generates compositional, literal translations and is unable to translate idioms accurately. |