Research

I'm generally interested in grounding knowledge and learning. This work lies at the intersection of vision, language and motor control; and deals with integrating knowledge from multiple modalities to facilitate robust learning, manipulation, and visual perception. In many ways this is attempting to do what humans are really good at, and computers have been traditionally terrible at, getting the big picture. A mix of detectors for visual perception are organized with the help of a stochastic grammar that applies the physical constraints of the structures that the robot is dealing with. Language naturally integrates into this system as a constraint on the possible structures, and can also be generated as an output of the system by looking for the most likely sentences given some mix of linguistic and visual input. There are two very different tasks driving this work, learning board games and manipulating structures from assembly toys.