Sparse Models Built During Active Learning

The Improve & Prune (I&P) method extracts sparse subnetworks from deep models during active learning without requiring separate search phases. This approach embeds iterative magnitude pruning into the standard retraining cycles of active learning. The authors show that on image classification tasks, these sparse models reach 95% sparsity while maintaining the accuracy of their dense counterparts, eliminating the compute tax usually associated with searching for efficient architectures.

Deep active learning involves training a model, evaluating an unlabeled pool to select informative samples, and repeating the process as new data arrives. Every cycle requires a training run, which is computationally expensive. Traditionally, researchers use iterative magnitude pruning to find sparse models by training, pruning weights, and retraining until convergence. These two workflows share a structure, but they have historically been treated as distinct operations with independent resource costs. I&P merges these cycles into a single unified pipeline.

In the I&P workflow, the pruning operation happens as the model reaches convergence during the retraining cycle. When a model completes a training run on the current dataset, the system calculates the magnitude of the weights. The smallest weights are pruned, and the mask is applied to the model weights before proceeding to the next acquisition step. The model is trained on the expanded dataset using this sparse weight mask. The sparse structure persists and is further refined across acquisition rounds, allowing the model to adapt to new data points while maintaining the sparsity discovered in earlier phases.

This process functions by progressively refining the weight mask as the dataset grows. During the initial training rounds, the model identifies connections that provide the most utility for the current training distribution. As the active learning loop incorporates new data, these weights undergo further fine-tuning. Because the pruning criteria are based on magnitude, the weights that contribute the most to the gradient descent objective are preserved across cycles. The model effectively accumulates a subnetwork that remains performant even as the training distribution expands. The sparsity is not a static result but a dynamic constraint that evolves in parallel with the data acquisition process.

For anyone building machine learning pipelines, the primary gain here is efficiency during the iterative process. Training dense models at every active learning step is often a bottleneck when dealing with large architectures. By applying pruning within the I&P framework, you gain a model that is faster to score and cheaper to train in subsequent iterations. This allows for more frequent acquisition rounds or the use of larger, more capable base architectures that would otherwise be impractical to cycle repeatedly. If you are building a system for image classification, this workflow enables the continuous refinement of a sparse, deployable model without the overhead of re-starting the search for a winning architecture after every new data acquisition.

While this method reduces training costs by maintaining a sparse structure across cycles, its current implementation is focused on specific architectures and tasks. I&P effectiveness remains unverified on transformer-based architectures and domain-specific datasets. Future work will need to demonstrate whether these sparsity patterns translate to more complex, multi-modal tasks or if the pruning criteria must be adjusted for different loss landscapes.