Offline Reinforcement Learning for Wind Farm Control
MTD3-BC optimizes wind farm yaw control by learning policies from static datasets instead of iterative simulator cycles.
Wind farm operators manage wake steering by adjusting the yaw misalignment of upstream turbines to redirect wake flow, aiming to increase the total power output of downstream units. Traditional systems require iterative loops against high-fidelity wake models to calculate how specific yaw angles impact downstream wind speed. This approach is computationally taxing and requires site-specific calibration, making it sensitive to latency and difficult to scale across varying terrain. MTD3-BC, or Modified Twin Delayed Deep Deterministic Policy Gradient with Behavior Cloning, attempts to solve this by reframing the control task as an offline reinforcement learning problem. It learns an optimal control policy directly from pre-collected operational logs, removing the need for a live, real-time simulation environment during training.
Constrained Learning from Logs
A central challenge in offline reinforcement learning is ensuring the agent does not propose actions that fall outside the distribution of the training data. Because the model cannot interact with the environment to explore, it has no way to gather new information. MTD3-BC uses behavior cloning to anchor the policy. The training objective functions as a weighted trade-off. The actor optimizes a combined objective, seeking to maximize the Q-value provided by the critic while minimizing the divergence from the logged expert action. The actor uses a loss function that penalizes deviation from the ground truth recorded in the logs. If the actor's proposed yaw setpoint deviates from the log, the penalty increases, effectively forcing the agent to prioritize the behavior of previous human or automated operators when value estimates are uncertain.
To manage mechanical constraints, the loss function includes an action consistency term. This is a mean-squared error penalty applied to the difference between consecutive yaw setpoints. By minimizing this term, the agent prioritizes smooth adjustments over rapid shifts that could accelerate wear on turbine components. Once training is complete, the actor can be deployed directly into the turbine control loop. Instead of running a complex wake simulation, the actor takes real-time sensor inputs—such as local wind speed, incoming wind direction, and the current yaw angles of neighboring turbines—and outputs the optimal yaw setpoint for the next operational window.
| Feature | Online RL | Model-Based Control | MTD3-BC (Offline RL) |
|---|---|---|---|
| Data Source | Real-time Simulator | Physical Model | Pre-collected Dataset |
| Training Cost | Very High | Low | Reduced |
| Wake Model Req. | None | Required | None |
| Performance Gain | Baseline | Variable | 10% over greedy |
Extrapolation and Performance
The utility of MTD3-BC is limited by the information density of the training logs. If a state falls outside the training distribution, the critic's value estimations degrade, leading the actor to follow inaccurate gradients. In wind tunnel experiments, the agent yielded a 10% power output increase compared to a greedy baseline, where each turbine operates in isolation. This matches the efficacy of calibrated, model-based control schemes.
Engineers evaluating this approach must secure datasets that explicitly contain high-variance operational scenarios. If logs consist only of steady-state conditions, the model will fail to adapt to shifts in ambient turbulence. The transition from a wind tunnel to a full-scale asset requires ensuring that the training logs capture the necessary breadth of environmental conditions. Whether this method can maintain its 10% gains across diverse, sparse log environments remains the primary barrier to wider industrial adoption.