Methods for agents that understand, ask, and adapt

The Tutorial

Recent breakthroughs in Multimodal Large Language Models (MLLMs), Vision-Language-Action (VLA) models, and emerging world models have dramatically advanced Embodied AI. Yet, while agents can now perform complex autonomous tasks in controlled settings, deploying them in unstructured, dynamic, and human-centric environments requires a fundamental leap in how they perceive and interact with the world and human.

To operate alongside humans, in applications like elderly care, co-working in warehouses and industrial automation, agents must master multimodal interaction beyond perception. The Human-centric Embodied Multimodal Interaction (HEMI) focuses on three core pillars regarding embodied multimodal interaction: i) Multimodal Perception & Reasoning: how agents fuse multimodal sensory inputs (visual, spatial, linguistic) to comprehend the physical world, identifying critical uncertainty and confusion, ensuring their actions are executable, effective, and safe; ii) Proactive Multimodal Interaction: To resolve ambiguous human intents and prevent physical risks, agents must proactively interact with users, leveraging both verbal and non-verbal cues to clarify goals; iii) Personalized Multimodal Interaction: interpreting situated contexts to adapt agent behaviors, actions, and communication styles according to individual user preferences.

By bridging vision-language representation, advanced multimodal human-agent interaction, and the social sciences, the HEMI tutorial establishes an interdisciplinary foundation for researchers with scientific and technical basis required to build intelligent, collaborative, and trustworthy embodied agents in the shared physical world.