Mapping Offline Policies with OptiFlow
OptiFlow improves offline reinforcement learning by using optimal transport to align policies with data distributions rather than relying on direct critic maximization.
Offline reinforcement learning typically forces a policy to maximize a critic's value estimates directly. When a dataset contains multiple valid ways to solve a task, this greedy approach causes the agent to collapse onto a single action mode or chase high-value noise in regions where the training data is sparse. OptiFlow replaces this with a structured allocation approach using entropic optimal transport to map actions from a reference policy to a target policy. It functions as a one-step flow policy learning method that couples action samples via state-wise entropic optimal transport.
To achieve this, OptiFlow learns a value-aware reference policy and a target policy simultaneously. Instead of just chasing high values, the model estimates a transformation. It uses a neural network to represent the target policy as a flow-based transformation of the reference policy. The network takes the state as input and generates the transformation parameters that map source action samples to target action samples. This ensures the output remains coherent relative to the learned value landscape. The training process relies on computing the optimal transport plan directly rather than minimizing an arbitrary two-part loss function.
| Feature | Standard Value Guidance | OptiFlow (Optimal Transport) |
|---|---|---|
| Action Pairing | Direct Critic Maximization | State-wise Entropic Transport |
| Stability | Prone to Mode Collapse | Anchored to Data Modes |
| Bias Risk | High (Out-of-Distribution) | Low (In-Distribution Focus) |
| Target Strategy | Greedy Value Selection | Value-aware Sample Allocation |
Solving for the Transport Plan
The mapping is solved as an entropic optimal transport problem. The Sinkhorn algorithm calculates the transport plan by taking a cost matrix and applying iterative row and column normalization. This iterative process acts as a projection: each row normalization enforces that the sum of the transport plan for a source action equals its original probability, while each column normalization ensures the target distribution matches the desired value-weighted allocation. Through repeated scaling, the matrix converges to a doubly stochastic form that satisfies the marginal constraints of the reference and target policies.
Because the algorithm includes an entropy term, it produces a soft transport plan. Rather than forcing a singular, discrete mapping, the entropy term spreads the probability across a range of valid, high-reward actions. This prevents the model from collapsing onto a point estimate, as the objective function effectively requires the transport plan to maintain a broader set of actions consistent with the reference policy. The cost matrix entry for a given state-action pair is constructed as the negative of the critic's value estimate, balanced by a penalty term proportional to the L2 distance between the reference and target action. By scaling the critic value by a temperature parameter, the model converts these units into a unified cost value that makes high-reward actions more attractive relative to the distance penalty.
Geometric Constraint in Practice
Consider a robotic arm reaching for an object where the dataset contains 100 successful demonstrations. A standard critic might identify a specific joint configuration—a region not well-represented in the logs—that it predicts will lead to higher rewards. In OptiFlow, the cost matrix measures the L2 distance between this potential action and the nearest viable action within the reference distribution. If the distance to the outlier is high, the cost term in the Sinkhorn optimization grows and outweighs the value gain provided by the critic. Consequently, the transport plan assigns minimal probability mass to the out-of-distribution configuration. The agent is forced to allocate its probability mass to actions that exist within the reference distribution, essentially choosing the best action that remains statistically grounded in the training data.
What remains unknown is how OptiFlow scales to significantly more complex, high-dimensional action spaces where the cost of computing the Sinkhorn iteration may become a bottleneck during real-time inference. While the framework provides a robust way to constrain policy improvement to the support of the offline dataset, its effectiveness is intrinsically tied to the quality of the learned critic. If the critic assigns high values to noisy or non-representative trajectories that happen to be geometrically close to the reference policy, the transport plan will still move mass into those regions. Relying on this approach requires a critic that is calibrated well enough to distinguish between true improvements and the artifacts of a limited dataset.