Physical AI Working Group

The working groups invite all members to join four monthly meetings. The goal is to map Notre Dame's existing strengths in data science, foundation models, and robotics, and to identify key opportunities in Physical AI. Tentative plan: (1) discussions in January, (2) student oral or poster presentations in February, (3) report outlines in March, and (4) findings summary in April.

In addition, the group will host three seminars featuring distinguished external speakers.

Please let us know your thoughts about the Working Group. Stay tuned for time and location of the events!

When and Where: 3:30-4:30pm, April 29 (Wednesday); 120 DeBartolo Lecture Hall

Abstract: The autonomous driving stack divides into two halves: perception and planning. While planning has well-defined outputs — trajectories and control signals — perception must decide how to describe the world. This talk explores perception as a translator from raw sensor data to semantic information, covering panoptic segmentation, LiDAR vs camera paradigms, and the challenges of defining what the model should output. We also discuss prediction, world models, and why even modern "end-to-end" systems still need intermediate labels. Throughout, we highlight practical insights from production autonomous driving, where the hardest problems are often in problem definition, not model architecture.

Bio: Mingcheng Chen received his Ph.D. in Computer Science from the University of Illinois at Urbana- Champaign, where his research focused on Computer Graphics. After graduation, he joined the self- driving project (Chauffeur) at Google [x], which soon became Waymo. After five and a half years on the Perception team at Waymo, Mingcheng joined AutoX, which later became Tensor Auto, where he has led Perception at Tensor for over four years. In this talk, Mingcheng will briefly introduce autonomous driving systems and touch on several interesting topics, with a clear emphasis on the perception system.

When and Where: 2:00-3:00pm, April 30 (Thursday); Virtual (Zoom)

Abstract: Robot foundation models have rapidly emerged as a central focus across industry and academia, driving major investment and research. At TRI, we've been building Large Behavior Models (LBMs) through large-scale imitation learning on diverse manipulation skills. In this talk, I'll present recently published results from this effort, focusing on the challenges of scaling data collection, developing rigorous evaluation frameworks for real-world policy performance, and extending these systems to humanoid platforms including efforts on the Boston Dynamics Atlas. While LBMs perform well across many tasks and often recover from failures, contact-rich manipulation remains a core challenge. Poor regulation of behavior near and during contact leads to robots frequently missing contact entirely or exhibiting unstable physical interactions. High-fidelity contact sensing through multimodal tactile sensors offers a path forward. I'll introduce TRI's award-winning visuo-tactile sensor work: SoftBubble and Punyo, highly deformable tactile grippers, and PolyTouch, a robust, low-cost sensor combining gel-based high-resolution touch sensing, peripheral vision, and contact audio. Together, these platforms enable HiLBMs: haptic-informed behavior models that integrate touch with vision and proprioception, helping robots navigate complex physical interactions through both feedback and prediction. The final piece is VLA Foundry, TRI's recently released open-source framework that unifies LLM, VLM, and VLA training in a single codebase. Most existing toolkits address only the action fine-tuning stage, leaving researchers to stitch together incompatible pretraining pipelines. VLA Foundry is a push for open source, open weights, open data, open design, and open evaluation for robot foundation models. If we want training truly performant dexterous manipulation to be less of an art and more of a science, a robust shared tool base is essential. While publications share the ideas, a solid codebase is what allows us to actually collaborate and build together.

Bio: Dr. Naveen Kuppuswamy is a Senior Research Scientist in the Large Behavior Models division at the Toyota Research Institute. His research explores how tactile and haptic sensing can enhance foundation models for contact-rich manipulation in unstructured environments. Over the past decade, he has worked across whole-body tactile sensing and control, soft robotics, and Large Behavior Models (LBMs) for robot manipulation. He previously served as Robot Data Lead for the LBM project and as Tactile Perception and Control Lead at TRI. His current research focuses on haptic-informed foundation models and multimodal LBMs for humanoid manipulation, integrating touch with large-scale learning. His key contributions include the Soft-bubble (Punyo) gripper and PolyTouch sensor systems, which advance how robots perceive and reason through touch. His work spans 50+ publications (900+ citations), multiple patents, and several awards, including Best Paper (Field and Service Robotics) at ICRA 2025, Best Paper from IEEE RA-L 2019, and Best Student Paper Finalist at RSS. Naveen holds a PhD in Artificial Intelligence from the University of Zurich, an MS from KAIST, and a BE from Anna University. He is driven by the long-term goal of building robotic systems that enhance human capability and independence.

When and Where: 2:00-3:00pm, May 8 (Friday); 140 DeBartolo Lecture Hall

Abstract: Reinforcement learning (RL) has shown promising performance across a variety of complex domains; however, its high sample complexity limits broader application---particularly in real-world, sparse reward settings. In this talk, I will describe a body of work that addresses this challenge by drawing on demonstrations and, increasingly, raw action-free video, an abundant and underexplored source of supervision for robot learning. I will begin by presenting a class of imitation learning algorithms capable of directly learning from demonstrations, even when they are suboptimal. Key to these algorithms is their ability to adaptively determine when and how to rely on different demonstrators, and to transition from imitation- to reinforcement-based learning, enabling the learner to outperform the demonstrators. I will then describe work that learns a task-agnostic, progress-based reward function from action-free video demonstrations---without access to action labels or manually specified rewards. Pretrained on large-scale egocentric human videos, this reward model generalizes across tasks and even across embodiments, supporting goal-conditioned policy learning from the large-scale, uncurated video data available on the internet. Finally, I will present very recent work that takes advantage of action-free visual demonstrations to learn a world model jointly with continuous latent action representations. By leveraging adversarial regularization and diffusion-based video generation, this approach learns structured, semantically meaningful action representations that support both imitation learning from observation and goal-directed planning.

Bio: Matthew R. Walter is an associate professor at the Toyota Technological Institute at Chicago (TTIC). His interests revolve around the realization of intelligent, perceptually aware robots that are able to act robustly and effectively in unstructured environments, particularly with and alongside people. His research focuses on machine learning-based solutions that allow robots to learn to understand and interact with the people, places, and objects in their surroundings. Matthew has investigated these areas in the context of various robotic platforms, including autonomous underwater vehicles, self-driving cars, voice-commandable wheelchairs, mobile manipulators, and autonomous cars for (rubber) ducks. Matthew obtained his Ph.D. from the Massachusetts Institute of Technology and the Woods Hole Oceanographic Institution, where his thesis focused on improving the efficiency of inference for simultaneous localization and mapping.

When and Where: 11:00-12:15am, May 13 (Wednesday); 117 DeBartolo Lecture Hall

Abstract: Robot learning fundamentally depends on access to abundant, high-quality, and low-cost data. Humanoid robots present unique challenges and opportunities—combining locomotion over rigid terrains (mostly) with manipulation of diverse, often deformable, objects. While synthetic data has driven remarkable progress in locomotion through deep reinforcement learning, manipulation remains limited by data scarcity and simulation fidelity. In this talk, I will discuss our recent advances in simulation technology inspired by our breakthroughs in computer graphics, aimed at enabling more effective humanoid learning for complex loco-manipulation tasks. Our new simulation engine delivers over 100× improvements in both speed and accuracy for deformable object dynamics, unlocking a wide range of contact-rich tasks previously deemed infeasible. I will conclude by outlining how these advances may shape the next frontier of humanoid intelligence, where realistic synthetic data bridges the gap between simulation and the real world.

Bio: Fan Shi is an Assistant Professor in the Department of Electrical and Computer Engineering at NUS, where he holds the prestigious NUS Presidential Young Professorship. His research focuses on AI for robotics, with particular interests in physical simulation and robot learning. He has received several international recognitions, including awards and support from leading organizations such as the NVIDIA Academic Grant Program, Google Research Funding, and Swiss AI Initiative. Before joining NUS, he was a Postdoctoral Researcher at ETH Zurich. He earned his Ph.D. and M.S. degrees at the University of Tokyo, and his B.S. degree at Peking University.