Boosting Sign Language Translation with Motion Vectors

Explicitly augmenting skeletal pose data with frame-to-frame motion differences improves sign language translation. Standard spatial embeddings capture the static position of a signer, but these coordinate snapshots fail to represent the temporal dynamics that define grammatical meaning. By calculating the delta between consecutive coordinate frames and concatenating those values to the input, the model receives a direct signal for movement without requiring the transformer to reconstruct velocity from separate spatial observations. This technique yielded a sharper improvement in BLEU scores in our testing compared to baseline models that relied on spatial data alone.

The Mechanism of Motion Augmentation

Sign language translation maps a sequence of 3D skeletal points to a natural language string. A standard transformer encoder processes these points using self-attention. This mechanism calculates pairwise dot products between token embeddings to measure similarity between positions across different time steps. While effective for identifying long-range correlations, self-attention lacks an explicit inductive bias for calculating derivatives like velocity.

Self-attention is mathematically designed to compare the static relevance of one position to another. When a transformer processes absolute spatial coordinates, it must learn to reconstruct relative trajectories through its weights across multiple layers. This requires significant parameter capacity to approximate what is effectively a simple subtraction. The attention mechanism sees a coordinate pair and must map the relationship into a temporal feature, which is an inefficient way to extract information already inherent in the input sequence. By providing the delta as a distinct feature, the model treats movement as a primary signal rather than a derived property.

Moving Beyond Linear Deltas

Raw arithmetic subtraction of coordinates assumes a linear relationship between frames, which fails to capture non-linear acceleration or deceleration in natural signing. A learned motion embedding improves on this by passing the delta through a small multi-layer perceptron or a learned projection layer before concatenation. This allows the model to apply non-linear activation functions, enabling it to map raw velocity into a higher-dimensional space where acceleration patterns are more discriminative. Instead of a fixed subtraction, the model learns a feature representation of the motion that is tuned to the specific velocity profiles of the signs in the training set.

Consider the sign for 'ask' versus 'request'. Both share similar hand configurations, but the trajectory of the movement differentiates the intensity and direction of the action. Without explicit motion vectors, a model must infer the path from two distinct coordinate snapshots, often leading to confusion between near-identical spatial poses. By injecting the learned velocity feature, the model immediately registers the difference in speed and trajectory, resulting in higher precision for verbs defined by motion rather than shape.

Building these models requires high-fidelity pose estimation. If the input data is noisy, the delta feature compounds this error across the sequence, introducing instability. The next step for those building these systems is to determine if joint-specific temporal smoothing prior to calculating the delta provides a more robust signal for the transformer to process.