VIP-Router Dynamically Selects Vision Pruning Strategies
VIP-Router replaces static vision token pruning with an adaptive routing module that assigns compression strategies per input.
VIP-Router introduces a sample-adaptive router that selects the optimal pruning strategy for each specific input. Multimodal Large Language Models typically process images by feeding every visual patch into an encoder, a process that is computationally expensive for high-resolution inputs. By training a lightweight router to choose between different compression methods on the fly, the system improves average accuracy by a 26.9% relative margin compared to traditional static pruning strategies.
The Problem of Fixed Pruning
Most existing MLLM efficiency efforts rely on fixed pruning strategies. A model might be configured to drop 50% of tokens based on static criteria like attention scores or spatial density. This approach treats all data distributions identically, failing to account for how visual feature importance shifts between different domains, such as high-resolution medical imagery and standard web-scraped photographs. Even when a specific pruning configuration wins on a benchmark's average score, other methods consistently outperform it on specific subsets of the test data.
Comparison of Pruning Paradigms
| Feature | Fixed Pruning Strategy | VIP-Router Adaptive Strategy |
|---|---|---|
| Decision Point | Pre-inference | Per-input inference |
| Strategy Selection | Single strategy for all | Input-specific routing |
| Overhead | Negligible | 0.017% of backbone parameters |
| Best Case Performance | Average across all images | Optimal for each specific image |
How Routing Works
VIP-Router operates as a plug-and-play module that sits in front of the model's visual encoder. The router acts as a classifier trained to map specific input features to an optimal pruning policy. It is conditioned on low-cost visual and textual features to decide which strategy in a pre-defined library should be applied to the current input. The model is trained end-to-end to minimize accuracy loss for specific visual features, effectively learning which signals are critical for downstream task performance and ensuring the selected pruning mask preserves those tokens.
Because the module introduces parameters representing only 0.017% of the underlying model's backbone, it adds almost zero latency to the overall inference pipeline. This approach addresses the accuracy loss associated with aggressive token reduction by ensuring the model does not apply a high-compression policy to an image where every visual signal is necessary for interpretation. By learning to associate input features with performance outcomes, the model maintains high accuracy without requiring a full-token pass on every input.
For engineers building MLLM-based applications, this means you no longer have to select one compression setting that balances accuracy and speed for your entire dataset. You can maintain a library of pruning strategies and let the router act as the dispatcher. The core trade-off here is the initial investment in training the router to recognize the specific image distributions relevant to your use case, which is a departure from using off-the-shelf static pruning methods. It remains unclear how the router scales as the library of available pruning strategies expands, or if there is a diminishing return where the added complexity of managing a larger library outweighs the marginal accuracy gains.