PACE Reduces RAG Latency With Predictive Filler Control
The PACE framework cuts perceived latency in RAG systems by generating temporary filler text based on a cached query-embedding map while the primary retriever works.
PACE uses a filler controller to generate temporary text while the retriever fetches data, minimizing perceived latency in RAG systems. It was designed for a humanoid-robot sales service where waiting for a retrieval-augmented generation pipeline to process a request and generate a response can lead to idle gaps in interaction. Instead of showing a spinner or a blank screen, the framework uses a joint control mechanism to manage both the primary response retrieval and a temporary, non-blocking filler stream. This shifts the waiting period into a UI opportunity.
| Metric | Standard RAG | PACE Framework | Improvement |
|---|---|---|---|
| P95 PTFR (s) | 0.53 | 0.29 | ~45% Reduction |
| Cache Stale Rate | 86% | 0% | Near-total elimination |
| Filler Conflicts | High | Zero | 94% call reduction |
Projection Through Semantic Caching
The controller projects potential outcomes by referencing a pre-indexed map of query embeddings. Before the retriever is triggered, the system calculates the semantic vector of the user's query and maps it against a set of historical query-response pairs cached in a local vector database. This provides a 'projected range'—a shortlist of top-k results that are statistically likely to be retrieved. The controller then generates filler text based on these likely candidates rather than the raw retrieval output.
This architecture prevents hallucinations by decoupling the filler from the final LLM chain. The filler is treated as a temporary interface component, restricted to a predefined semantic buffer. If the semantic distance between the filler's topic and the actual document fetched by the retriever exceeds a threshold, the system triggers a hard cut to the real data, preventing the filler's placeholder logic from bleeding into the final generation context. The 'schema' here is a localized index of topic clusters that the controller uses to stay within valid conversational bounds. This map is updated asynchronously, ensuring that the filler controller only predicts within topics where the system has sufficient data to generate coherent, low-variance bridging text.
Balancing System Constraints
The framework uses a cascading router to enforce performance bounds. The controller never holds a request for longer than a single RAG turn; if the retrieval process finishes, the filler ends immediately. This hard-coded limit ensures that the system never adds latency to the actual generation process, keeping the tradeoff strictly focused on filling idle time rather than delaying the output.
We know it works well for humanoid-robot customer service, but its utility in unpredictable environments remains unproven. In highly dynamic document sets—such as live news or shifting technical documentation—the controller may default to silence if the variance of the retrieval pool is too broad. For anyone building with this, the bottleneck is not the compute required for the filler, but the maintenance of that query-embedding map. If the schema is outdated, the controller effectively guesses blindly, which forces the system to either produce irrelevant filler or drop the feature entirely. The next step for this approach will be determining if such a system can handle a wider range of high-entropy topics without falling back to a silent state.