Doob-Based Generative Policy Optimization

Diffusion and flow models provide expressive policy classes for robotic and sequential decision-making systems.

However, their action likelihoods are generally intractable, limiting the direct use of widely adopted policy optimization methods such as PPO and GRPO.

We propose Doob Policy Optimization (DoobPO), a framework that leverages Doob's $h$-transform for likelihood-free reinforcement learning fine-tuning of generative policies.

DoobPO separates policy improvement from generative policy learning.

It first learns a lightweight ratio model that represents the desired policy update, and then distills the resulting ratio-tilted target policy into the generative policy via conditional drift matching.

This formulation applies uniformly to both diffusion and flow policies and supports a broad family of policy improvement rules, including PPO and GRPO.

We prove that the practical conditional drift matching objective is gradient-equivalent to the ideal Doob drift matching objective.

We further show that the induced diffusion process solves an endpoint-constrained stochastic optimal control problem.

This establishes that it achieves the minimum-energy modification required to realize the desired policy tilt.

Across MuJoCo continuous-control benchmarks, DoobPO achieves strong performance against competitive diffusion- and flow-policy baselines, showing a principled and practical route to RL fine-tuning for generative policies.