While recent advances in generative policies have enabled robots to learn complex tasks from demonstrations, a critical gap exists in allowing humans to guide these policies at deployment-time to incorporate use-case and task-specific constraints without full policy retraining. This project will advance our recently-developed Inference-Time Policy Steering (ITPS) framework that enables real-time human guidance of pre-trained policies without retraining. Our work will build directly upon and extend the types of diffusion policies that will be developed in “T2.4 DexNex Diffusion Policies”; the real-time refinement mechanisms we propose don’t require policy retraining, making them particularly suitable for deployment alongside DexNex policies. We will work in collaboration with Friesen, Klatzky and Orta as they pursue theT3 project “Essential haptic cues for dexterous tasks,” and will generalize our ITPS approach to incorporate haptic feedback and guidance. Finally, the project is synergistic with the “T3: Robot Atlas” project’s focus on adoption barriers; our project will develop the technical capabilities needed to make robots more adaptable according to end-user needs.

Inference-Time Policy Steering (ITPS). We present a novel framework to unify various forms of human interactions to steer a frozen generative policy. User interactions “prompt” pre- trained policies to synthesize aligned behaviors at inference time.