Hiding PCIe Latency in Mixture-of-Experts Models

Large Mixture-of-Experts (MoE) models often exceed the available VRAM on a single GPU, forcing systems to offload inactive experts to system memory. Traditional offloading treats memory as a cache, fetching an expert only when the forward pass demands it. Because PCIe latency typically exceeds the compute time for a single expert layer, this approach forces the GPU to stall while waiting for data. SeqMoE moves this process from reactive loading to predictive scheduling. It achieves 80.22% of the performance of a model held entirely in device memory while keeping only 45% of the expert weights in VRAM.

Forecasting Expert Usage

To avoid stalls, SeqMoE treats expert access as a predictable sequence. The system uses a sequence-to-sequence model to forecast future expert activation patterns. This predictor consumes hidden states from the input tokens—specifically the outputs of preceding transformer layers—to map these internal representations directly to specific expert indices. Because routing decisions across layers share strong correlations within the same input sequence, the predictor generates a probability distribution for upcoming expert usage. This provides the scheduler with a list of target experts before the primary router processes the relevant tokens. The predictor introduces minimal computational overhead, allowing the scheduler to prepare transfers while the GPU is busy with current layer computations.

Scheduling and Execution

Once the system predicts a future need, it converts the load request into a Job Sequencing with Deadlines formulation. The scheduler manages the timing of PCIe transfers by treating expert loading as a background task. It modifies the execution graph by inserting non-blocking asynchronous copy kernels that prioritize data movement before the model layer requires the weights. These kernels do not compete with compute kernels for the same GPU streaming multiprocessors; instead, they occupy the PCIe bus during cycles where compute kernels are not saturating the interconnect. By mandating a 'ready' status for the weights before the GPU kernel execution window begins, the system ensures that compute and data movement are effectively decoupled.

Eviction Logic

For cache management, SeqMoE implements a modified Belady policy. Traditional Belady assumes perfect knowledge of future access, which the predictor provides here as an estimated likelihood. The system calculates the distance to the next expected access by evaluating the probability output of the predictor. The eviction engine ranks experts currently in VRAM based on this distance. If the predictor assigns a high probability to an expert for a near-term layer, the system retains it regardless of how long it has been since its last actual use. When two experts have similar predicted arrival times, the system prioritizes the one with the higher probability score. If scores are identical, it applies a simple least-recently-used tie-breaker to clear the space.

Performance is bound by the accuracy of the predictor. When the system fails to anticipate a sudden shift in routing, it defaults to a standard reactive stall. The system observed a 96.97% hit rate in standard evaluation, meaning the remaining percentage results in unavoidable latency penalties. This approach is currently validated against models where the active expert subset fits within the available local VRAM cache at a 45% ratio. Whether this scheduling logic remains efficient as the gap between total expert size and local memory widens remains a matter for future stress testing, as the overhead of the prediction itself could theoretically scale if the model architecture becomes significantly more sparse.