Compressing 3D Video Models with Label-Guided Distillation

Label-Guided Knowledge Distillation (LGKD) introduces a mechanism for compressing 3D convolutional neural networks (CNNs) used in action recognition. Traditional distillation methods in this field import techniques from 2D image analysis, which treat video as a static stack and ignore the temporal dynamics unique to motion data. By incorporating ground truth labels into the training loop, LGKD allows the student model to focus on teacher-provided features that correlate with action classes, effectively filtering out background noise that usually clutters a teacher model's internal feature maps. This approach uses labels as a heuristic for where the student should focus its limited parameter budget. It avoids forcing the student to replicate the teacher's activations across every frame indiscriminately.

The Mechanism of LGKD

Existing video distillation typically forces a lightweight student to match the feature activations of a massive teacher model. This is inefficient because teacher activations often contain noise or irrelevant temporal data that do not contribute to accurate classification. LGKD addresses this through two specific pipelines operating in parallel.

First, the sample-wise distillation pipeline applies a temporal mask to the student's training. This mask identifies temporal segments based on ground truth labels. For each segment, the student's loss is weighted according to the presence of the correct class label. These weights ensure the gradient updates prioritize segments where the teacher captures the discriminative action. By doing so, the student learns to emphasize temporal segments containing key motions rather than trying to replicate every activation of the teacher model.

Second, the class-wise distillation pipeline structures the feature space by aligning student representations with class prototypes. These prototypes represent the center of gravity for each class within the teacher's learned representation. They are maintained as dynamic averages, updated throughout the training process to reflect the teacher's evolving understanding of each action category. Because the student model is smaller, the alignment ensures its internal representation is pulled toward the teacher's established class clusters. This focuses the student's capacity on the features that best distinguish between action classes.

Comparison of Approaches

A Worked Example

Consider an action recognition model identifying a pole vault clip. A standard distillation method forces the student to replicate the teacher's representation for every frame, including the static grass in the background. The teacher model has learned to filter this noise, but the student lacks the capacity to reach that same level of abstraction. The mask in LGKD looks at the ground truth label across the time dimension to identify the segments where the athlete is actually vaulting. It scales those segments up in the student's training objective and scales down the low-information segments where only static grass is visible. The student model then spends its limited parameter budget on the frames where the teacher shows high confidence, leading to a sharper focus on the actual motion.

Impact on Performance

LGKD requires fully supervised settings with a reliable, dense set of labels for training clips. Because the method relies on ground truth labels to generate the temporal masks and define the prototype clusters, it is currently unsuitable for unsupervised or weakly-supervised video datasets. This dependency means that any noise or inaccuracy in the original dataset labels propagates directly into the mask generation, potentially leading the student to focus on incorrectly labeled frames. When compared to a standard student baseline on the Kinetics-400 dataset, models trained with LGKD consistently demonstrate a higher mean average precision (mAP) for the same parameter count, proving that guiding a student via labels is more effective than blind mimicry.