Social interactions are a blind spot for AI/ML

Machines have great difficulty understanding them!

Machines, robots, vision systems are not social. A lot of the frustrations we have with our machines, a lot of the dangers of automation, and many of the limitations of robotics all stem from a lack of social understanding.

From a machine learning point of view social interactions pose a difficult challenge. Exactly the same actions may have completely different social meaning depending on subtle changes in how the actions are taken, their physical context, prior interactions, the reaction of another agent, etc. These kinds of non-local problems that are extended in time with vague labels where the same idea can be expressed in radically different ways which you might need to recognize zero-shot are exactly where modern ML is weakest.

We have three key ideas:

1. You cannot make progress without social simulators. Just like we have simulators in robotics and physics, we need them to explore social interactions. ( Citation: Netanyahu, Shu & al., 2020 Netanyahu, A., Shu, T., Katz, B., Barbu, A. & Tenenbaum, J. (2020). PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception. The AAAI Conference on Artificial Intelligence.
   
   @inproceedings{netanyahu2021phase,
       title={{PHASE}: PHysically-grounded Abstract Social Events for Machine Social Perception},
       author={Netanyahu, Aviv and Shu, Tianmin and Katz, Boris and Barbu, Andrei and Tenenbaum, Joshua B},
       booktitle={The AAAI Conference on Artificial Intelligence},
       year={2020}
      } )
2. Formal models for social interactions are possible, and we propose the first generic model, Social MDPs. Robotics benefited tremendously from MDPs because we could all understand and agree on the underlying problem. ( Citation: Tejwani, Kuo & al., 2022 Tejwani, R., Kuo, Y., Shu, T., Katz, B. & Barbu, A. (2022). Social interactions as recursive MDPs. Conference on Robot Learning. 949–958.
   
   @inproceedings{tejwani2022social,
       title={Social interactions as recursive {MDP}s},
       author={Tejwani, Ravi and Kuo, Yen-Ling and Shu, Tianmin and Katz, Boris and Barbu, Andrei},
       booktitle={Conference on Robot Learning},
       pages={949--958},
       year={2022},
       organization={PMLR}
      } )
3. You can create models of social interactions that mix and many match social and physical goals. That’s what gives rise to the rich interactions we see in humans. ( Citation: Tejwani, Kuo & al., 2023 Tejwani, R., Kuo, Y., Shu, T., Stankovits, B., Gutfreund, D., Tenenbaum, J., Katz, B. & Barbu, A. (2023). Zero-shot linear combinations of grounded social interactions with Linear Social MDPs. AAAI Conference on Artificial Intelligence.
   
   @inproceedings{tejwani2023zero,
       title={Zero-shot linear combinations of grounded social interactions with Linear Social MDPs},
       author={Tejwani, Ravi and Kuo, Yen-Ling and Shu, Tianmin and Stankovits, Bennett and Gutfreund, Dan and Tenenbaum, Joshua B and Katz, Boris and Barbu, Andrei},
       booktitle={AAAI Conference on Artificial Intelligence},
       year={2023}
      } )

A social simulator #

Since social interactions are so vague and poorly defined, we need playgrounds to explore empirically what they are. This usually means building a simulation, but we have almost no social simulators today. No way to generate and model physical social interactions. We set out to develop one of the first grounded social simulators, an environment where we can generate interactions and determine if they are physical, social, or a combination of the two. ( Citation: Netanyahu, Shu & al., 2020 Netanyahu, A., Shu, T., Katz, B., Barbu, A. & Tenenbaum, J. (2020). PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception. The AAAI Conference on Artificial Intelligence.

@inproceedings{netanyahu2021phase,
    title={{PHASE}: PHysically-grounded Abstract Social Events for Machine Social Perception},
    author={Netanyahu, Aviv and Shu, Tianmin and Katz, Boris and Barbu, Andrei and Tenenbaum, Joshua B},
    booktitle={The AAAI Conference on Artificial Intelligence},
    year={2020}
   } )

Social MDPs #

Our core idea is that social interactions occur when agents recursively model other agents, and that all social interactions can be modeled this way. Practically, we implement this with a novel extension of MDPs (Markov decision processes) ( Citation: Tejwani, Kuo & al., 2022 Tejwani, R., Kuo, Y., Shu, T., Stankovits, B., Gutfreund, D., Tenenbaum, J., Katz, B. & Barbu, A. (2022). Incorporating rich social interactions into MDPs. International Conference on Robotics and Automation (ICRA). 7395–7401.

@inproceedings{tejwani2022incorporating,
    title={Incorporating rich social interactions into {MDPs}},
    author={Tejwani, Ravi and Kuo, Yen-Ling and Shu, Tianmin and Stankovits, Bennett and Gutfreund, Dan and Tenenbaum, Joshua B and Katz, Boris and Barbu, Andrei},
    booktitle={International Conference on Robotics and Automation (ICRA)},
    pages={7395--7401},
    year={2022},
    organization={IEEE}
   } ) , Social MDPs. We put forward Social MDPs as the definition of what a social interaction is. Even if we’re wrong, simply being able to discuss this topic formally and mathematically with a falsifiable model is a big step forward. Hopefully, this ushers in an era of formal models for social interactions that allows us to discover what they really are. A Social MDP is nested, it reasons recursively about the actions, physical goals, and social goals of another agent ( Citation: Tejwani, Kuo & al., 2022 Tejwani, R., Kuo, Y., Shu, T., Katz, B. & Barbu, A. (2022). Social interactions as recursive MDPs. Conference on Robot Learning. 949–958.

@inproceedings{tejwani2022social,
    title={Social interactions as recursive {MDP}s},
    author={Tejwani, Ravi and Kuo, Yen-Ling and Shu, Tianmin and Katz, Boris and Barbu, Andrei},
    booktitle={Conference on Robot Learning},
    pages={949--958},
    year={2022},
    organization={PMLR}
   } ) . Remarkably, after we formulated Social MDPs we discovered economists already use a similar model, but this idea had not yet been applied to robotics.

Combining social interactions #

Many cognitive scientists and psychologists had long conjectured that social interactions consist of a core set of social skills which are then combined together into the rich social behavior we observe in humans and animals. We produced the first computational implementation of this idea. ( Citation: Tejwani, Kuo & al., 2023 Tejwani, R., Kuo, Y., Shu, T., Stankovits, B., Gutfreund, D., Tenenbaum, J., Katz, B. & Barbu, A. (2023). Zero-shot linear combinations of grounded social interactions with Linear Social MDPs. AAAI Conference on Artificial Intelligence.

@inproceedings{tejwani2023zero,
    title={Zero-shot linear combinations of grounded social interactions with Linear Social MDPs},
    author={Tejwani, Ravi and Kuo, Yen-Ling and Shu, Tianmin and Stankovits, Bennett and Gutfreund, Dan and Tenenbaum, Joshua B and Katz, Boris and Barbu, Andrei},
    booktitle={AAAI Conference on Artificial Intelligence},
    year={2023}
   } ) For the first time we understand mathematically what it means to combine social interactions: you take linear combinations of Social MDPs each of which is conditioned on a mixture of social and physical goals. This way one agent can help you assemble a chair, while preventing you from hurting yourself, and making sure you beat someone else who is trying to do the same. The fact that linear combinations of Social MDPs lead to models which have rational behaviors is a good hint that this direction is reasonable.

Cooperation Conflict Competition Coercion Exchange

The future of social interactions #

We still don’t have a full theory of social interactions, but we have some hints that such a theory exists and a start on some social interactions. What we didn’t talk about here is that these models agree with human judgments about social interactions. They are not models that engage socially, they are models of how humans engage socially. This means that we could use them to understand human social interactions and our social limitations.

Many disorders are defined largely in terms of dysfunctions in social interactions. We talk about someone being on the Autism spectrum. But, this is not a satisfying scientific description. There is no unit of measure for the Autism spectrum. The tests that determine roughly were you might be on the spectrum aren’t based on a mechanistic understanding of social interactions. And worse, we have no idea if there is even just one spectrum. You might imagine that social skills have many components that could go wrong. For example, if you look at Social MDPs maybe someone has more difficulty with one of the core social interactions, or with perceiving the difference between a social or physical interaction, or a limitation with attention.

This is what our lab is doing now. Trying to build fine-grained tests of social and physical reasoning based on Social MDPs. We create scenarios from Social MDPs and judge them with the Social MDP. These are then calibrated to have a certain difficulty range. We’re showing these to neurotypical subjects, subjects with visual impairments and subjects with ASD. Hopefully soon ASD will have units, models, and mechanisms so that we can take the next leap forward.

Of course, there are many other exciting topics in social robotics, not least of which is building Transformer approximations of Social MDPs and combining them with other models to recognize social interactions in videos or text and to carry them out efficiently. This isn’t quite the dawn of social computing, but there’s a glimmer on the horizon that the relationship between us and machines can be totally different than it’s been in the past. One day it will be hard to imagine a machine that doesn’t work with you and adapt to you, we just need to make social computing much more mathematical and formal than we have until now.