AdamX Replaces Momentum with Cosine Similarity

AdamX acts as a drop-in optimizer replacement that modifies how weight updates are scaled. While standard Adam relies on the moving average of gradients to smooth training, AdamX uses the cosine similarity between the current gradient and the moving average to regulate the learning rate. By focusing on directional consistency rather than raw momentum, the optimizer attempts to filter out noisy updates that do not align with the recent training trajectory.

The Mechanism of Alignment

Standard optimizers like Adam use the first and second moments to normalize updates, which can be sensitive to gradient oscillations. AdamX stabilizes this process by treating the cosine similarity as a scalar multiplier for the update step. The similarity score is calculated as the inner product of the current gradient vector and the moving average of previous gradients, normalized by the product of their magnitudes. If the score is 1, the vectors are perfectly aligned; as the score approaches 0, the optimizer suppresses the update magnitude. The multiplier is computed as a direct product of the learning rate and the cosine similarity score, ensuring that updates occurring in a direction contradictory to the recent history are dampened.

Consider an engineer debugging a model that experiences sudden gradient spikes during training. With standard Adam, these spikes are incorporated into the momentum buffer, potentially skewing the next few update steps. In AdamX, the optimizer calculates the directional alignment between the spiked gradient and the stable moving average. Because the spike typically deviates from the historical trend, the cosine similarity score drops, which automatically shrinks the learning rate multiplier for that specific step. This prevents the noisy signal from being fully reflected in the weight update, effectively smoothing the loss curve without requiring manual intervention or separate learning rate schedules.

Implementation Trade-offs

AdamX introduces a dependency on historical gradient alignment that is not present in standard optimizers. While the scalar multiplier logic is straightforward, the optimizer does not provide an explicit definition for its 'localized variance' metric, which serves as a secondary control for signal dampening. For an engineer, this creates a black box: when training becomes unstable, it is difficult to determine whether the instability is caused by the loss landscape or by the optimizer's internal dampening thresholds. Without a transparent way to monitor how this variance affects the learning rate, tuning the optimizer involves trial and error rather than analytical adjustment. The impact of this alignment-based approach on extremely sparse architectures also remains unverified, as the frequency of updates in those models may cause the denominator in the similarity calculation to fluctuate significantly, potentially leading to erratic learning rate behaviors.