Verifying Policy Robustness Without Extra Rollouts

Adversarial Importance Sampling, or Advis, provides a way to estimate and optimize verifiable worst-case returns for deep reinforcement learning agents using existing training data. Traditionally, verifying a policy against input perturbations meant training separate neural networks to act as adversaries or running new, costly simulations for every potential attack vector. This required significant computational overhead, as each new threat model necessitated fresh environment rollouts. Advis instead operates on offline logged data, bypassing the need for environment interaction during the verification phase.

The Mechanism of Importance Sampling

Advis functions by reweighting outcomes from a baseline policy to estimate performance under a specific adversarial condition. If you have a trajectory generated by a baseline policy, you can approximate how that trajectory would look under an adversarial policy by multiplying the total cumulative reward of the trajectory by an importance weight. This weight is the product of likelihood ratios for every action taken throughout the sequence. For each step, you calculate the probability of the action under the adversarial policy divided by the probability of the same action under the baseline policy. The adversarial policy in this context is typically defined as a perturbation function that maps states to action distributions, often biased toward lower rewards or higher failure probabilities.

Because these ratios are multiplied together for each step of the trajectory, the resulting scalar accounts for the divergence between the policies across the entire episode. By calculating these weights offline using logged trajectory data, you can estimate performance without the adversary ever interacting with the environment. Beyond simple evaluation, the differentiability of this importance-sampling estimator allows you to optimize policies to be more robust by directly minimizing the estimated worst-case returns during training.

Stress-Testing the Adversarial Landscape

By using 6-14x more configurations than prior work, the authors show that the adversarial landscape is often filled with small, localized pockets of failure. Because the importance weights are calculated offline based on existing trajectories, you can stress-test an agent against thousands of these specific, varied attack parameters without the cost of running thousands of new simulations.

Consider an autonomous navigation agent trained to follow a path. A naive robustness test might perturb the sensor inputs by a single, fixed noise value. With Advis, you can evaluate how that agent behaves against thousands of potential perturbations derived from the original trajectory data. To translate these weights into a safety verdict, you compare the weighted return against a predefined threshold: if the reweighted return of a trajectory falls below the minimum acceptable reward, that specific trajectory is flagged as a failure point. The sum of these weighted failures across your dataset provides an estimate of the policy's failure rate under the adversarial condition.

This precision depends on the alignment between the baseline and adversarial policies. If the policies diverge too significantly, the product of many likelihood ratios becomes unstable. This leads to vanishing or exploding weights, which creates high variance in the estimator and renders the return estimate unreliable. Identifying how to maintain stability when the target adversarial policy significantly deviates from the baseline remains an open challenge in verifying complex, non-linear policies. Future research will likely need to address whether adaptive clipping or variance-reduction techniques can extend the effective range of these estimates for more volatile environments.