Decoupling LLM Training and Inference with LoRA
AsyncGRPOTrainer decouples training and inference by syncing LoRA adapters via shared storage instead of direct network fabric coupling.
Traditional reinforcement learning for large language models requires tight coupling between training and inference workers. Systems often use high-speed fabrics to broadcast full model weights, creating a hard dependency on physical cluster topology. Any training update mandates a global synchronization event across the entire inference fleet. Switching to LoRA reduces the synchronization payload from gigabytes of full weights to a few megabytes of adapter files. This architecture uses a shared filesystem for adapter data and an orchestrator to keep training and inference as independent jobs.
| Feature | Traditional Async RL | Async GRPO with LoRA |
|---|---|---|
| Sync Target | Full Model Weights | LoRA Adapter Only |
| Data Size | ~3 GB (1.5B model) | ~Few MB |
| Network Dependency | NCCL / Cluster Fabric | Shared Storage Bucket + Proxy |
| Deployment | Single Node / Tight Coupling | Independent HF Jobs |
When the trainer finishes an iteration, it writes a new LoRA adapter file to a target directory, naming the folder with a monotonically increasing version index. An orchestration proxy monitors this directory and updates a global state indicator pointing to the most recent version. The inference nodes, running vLLM, periodically poll the proxy to check for new version markers. When a node detects a newer version, it pulls the corresponding adapter files from the shared storage.
To manage the transition, vLLM loads the new adapter weights into memory without interrupting active requests. If a request arrives during an adapter swap, the engine continues processing it with the existing weights to ensure stability. Once the swap is complete, the engine uses the new adapter for subsequent requests. This architecture relies on the fact that the KV cache, which stores the intermediate states of a sequence, remains valid across adapter swaps for the same base model. This allows for continuous serving without needing to flush the cache or restart the process.
In this architecture, vLLM keeps the base weights frozen in GPU memory while dynamically swapping the small adapter layers. By decoupling the trainer and inference fleet, you remove the requirement for specialized cluster interconnects or synchronized state across thousands of nodes. The bandwidth savings are substantial when dealing with large models. You also gain operational freedom to run the training and inference jobs as distinct, asynchronous workloads.
While this approach solves the synchronization bottleneck, the impact on long-term training convergence remains a practical consideration. We know that frequent swapping and the reliance on shared storage latency can introduce small overheads that aggregate over millions of steps. What is less clear is how this asynchronous lag affects the stability of algorithms that rely on strict policy synchronization, such as those sensitive to stale gradient updates in a reinforcement learning context.