Routing Agent Tools via Latent State Projections

Gavel extracts tool-routing decisions directly from the hidden states of a frozen LLM. Instead of relying on a separate retriever model or an external reranker to identify the correct tool, Gavel trains two lightweight linear maps that project the model's high-dimensional internal representation into a lower-dimensional routing space. By treating the model as a latent router that has already processed the task, this approach requires negligible compute compared to existing methods that utilize external retrieval components totaling between 1.2 billion and 16 billion parameters. Gavel isolates internal activation patterns that correlate with specific functional requirements.

Mapping Internal Representations to Tools

Existing routing pipelines often rely on external retrieval systems to process task metadata. Gavel replaces this infrastructure by training linear projection matrices that map the hidden state vector to a tool-selection space. Mathematically, the map applies a matrix multiplication to the hidden state vector from a middle transformer layer, rotating and scaling the vector to align with pre-calculated centroids in the target space. By the time a transformer reaches its middle layers, it has performed sufficient self-attention operations to aggregate input tokens into a compact latent vector. These layers synthesize the constraints of a request because the attention heads have already identified the relevant entities in the prompt, acting as a summary of the model's trajectory before the final prediction head generates a specific token.

To build the skill banks for these projections, the system processes a static dataset of instruction-tool pairs. During this calibration phase, the system records the internal hidden states of the frozen model for each known query-tool pair. These states are averaged per tool to form a stable centroid within the routing space. This creates a latent signature—a mathematical anchor point representing the semantic intent to call a specific API—without modifying the model's weights or requiring additional fine-tuning. The map acts as a bridge, pointing the model's latent understanding toward the specific coordinate where a tool resides.

The Glance and Verdict Mechanism

Gavel executes selection in two distinct steps. The first operation, known as the 'glance', projects the hidden state vector through the trained linear map to compute a cosine similarity score against the pre-computed tool centroids. The second phase, the 'verdict', evaluates the model's confidence for the tool-call token. In this implementation, the system monitors the logit distribution during the standard forward pass of the base model. The model is trained to output a specific tool-call trigger token whenever its internal state identifies the need for external data.

If the model assigns high probability to that token, it serves as a confirmation of the 'glance' signal. The final decision is calculated by taking the product of the glance similarity score and the model's probabilistic confidence for that tool-call token. This dual-verification method allows the agent to filter out noise, as a random activation spike might score high in the glance phase but fail to align with the model's probabilistic intent. By using the model's own output distribution as a gate, Gavel ensures that only signals the model is statistically 'confident' about are treated as valid routing events.

Limits and Unknowns

This method relies on the assumption that the target model produces distinct, consistent activations for different tool intents during its forward pass. If the internal states do not diverge significantly between two similar tools, the linear map will struggle to differentiate them. Furthermore, the performance of Gavel remains tethered to the underlying base model, such as Qwen3-32B. It cannot recover signal that the base model has failed to encode during its pre-training phase. While research has utilized Qwen3-32B, it remains an open question how this method scales across different model architectures or significantly different pre-training objectives. The geometry of latent space is highly dependent on the model's original training data and optimization landscape, and whether these linear projections remain stable across disparate model families is not yet known. The transition from one architecture to another may require re-calibration of the centroids or entirely new linear maps, as the internal 'language' of the transformer depends on its specific configuration and depth.