Papers by Boris Katz
Assessing Language Proficiency from Eye Movements in Reading (N18-1)
Copied to clipboard
| Challenge: | a novel approach to determine second language proficiency uses behavioral traces of eye movements during reading . over 1.5 billion people are learning English as a second language worldwide . traditional approaches to language proficiency testing have several drawbacks, including the fact that they are typically prepared manually and require extensive resources for test development . |
| Approach: | They propose a method which uses behavioral traces of eye movements during reading to determine learners’ second language proficiency. |
| Outcome: | The proposed approach correlates with standardized English proficiency tests and is validated by eyetracking with eye movements from other readers. |
Measuring Social Biases in Grounded Vision and Language Embeddings (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to measure social biases in word embeddings are limited to visually grounded word embeds . a new study generalizes word embedment associations to visually ground word embeddas . |
| Approach: | They generalize word embeddings' biases to visually grounded word embeds . they propose two generalizations that answer questions about how biase, language, and vision interact . |
| Outcome: | The proposed measures are applied to a new dataset that includes 10,228 images from COCO, Conceptual Captions, and Google Images. |
The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank (2022.emnlp-main)
Copied to clipboard
Adam Yaari, Jan DeWitt, Henry Hu, Bennett Stankovits, Sue Felshin, Yevgeni Berzak, Helena Aparicio, Boris Katz, Ignacio Cases, Andrei Barbu
| Challenge: | Existing treebanks derived from text include only text and are based on single-modality texts. |
| Approach: | They propose to use audio-visual transcripts and part of speech tags to create an English language treebank based on dialog in Hollywood movies. |
| Outcome: | The proposed treebank is the 3rd largest UD English treebank and the only multimodal treebank in UD. |
Grounding language acquisition by training semantic parsers using captioned videos (D18-1)
Copied to clipboard
| Challenge: | a new method for parsing sentences using captioned videos is being developed . we use video clips to ground the semantics of language, but without annotations . |
| Approach: | They develop a semantic parser that is trained in a grounded setting using captioned videos . they use a corpus of sentences paired with videos without other annotations to train it . |
| Outcome: | The proposed parser recovers the meaning of English sentences despite no annotations . learning a grounded semantic parsers can expand the range of data that parseurs can be trained on . |
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study. |
| Approach: | They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains. |
| Outcome: | a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not. |