Replacing Output Layers with Geodesic Decoding

Riemannian Language Models (RiLM) eliminate the standard output projection layer in small-scale language models by replacing weight-based decoding with geodesic distance calculations on a manifold. This approach removes the output projection matrix, which typically consumes one-third of the parameter budget in sub-million parameter models. Instead, it derives token likelihoods by calculating the distance between a model's current state and fixed vocabulary embeddings, then applying a softmax function over the negative squared geodesic distances to normalize these values into a probability distribution.

In conventional architectures, an embedding width of 128 with a vocabulary of 2,000 creates a massive bottleneck where the output matrix dominates the parameter count. By utilizing a shared embedding space for both input and output, RiLM treats the decoding process as a geometric operation rather than a learned linear transformation. To perform this, the model maps its hidden state vector into the hyperbolic manifold using an exponential map—a local isometry that preserves the geometry of the tangent space—placing the vector at a coordinate on the Poincare ball. The next-token prediction then corresponds to the negative squared geodesic distance between this coordinate and the points representing the vocabulary.

Using the Poincare ball as the underlying manifold, referred to as HypRiLM, outperformed flat Euclidean space significantly in these tests. This works because hyperbolic space accommodates the exponential growth of nodes as you move from the center to the boundary, allowing the model to embed hierarchical linguistic relationships more efficiently than Euclidean space. When deploying this, you must account for the instability of naive hyperbolic recurrence. Because hyperbolic space has infinite volume near the boundary, training updates can easily push vectors into regions where gradients vanish. To counter this, the authors utilize Mobius stabilization, a re-normalization process that pulls vectors back toward the center of the manifold to keep training trajectories from collapsing.

While current findings are limited to small models, a stress test on a 10,000-vocabulary set confirms that geodesic decoding transfers across larger vocabulary sizes. The primary reason this remains viable at scale is that both standard matrix multiplication and geodesic distance calculations share O(V) complexity, where V is the vocabulary size. The geometry does not introduce a higher theoretical complexity class, though it does add constant-time overhead for the distance calculation itself. Whether these geometric advantages persist at the massive scale of transformer layers remains the primary open question, as the compute cost of these non-linear geometric operations at high dimensionality has not been fully benchmarked against highly optimized hardware kernels used for matrix multiplications.