Why Rank Deficiency Stalls Tree Tensor Networks

Tree tensor networks represent functions by decomposing high-dimensional tensors into a hierarchical tree of smaller core tensors. While these models can represent complex functions, they struggle to optimize when the internal rank—the bond dimension—is insufficient. This does not replace the challenges of the loss landscape, but rather manifests as a specific geometric hurdle: the rank deficiency creates degenerate saddle points that halt gradient-based learning.

The optimization objective is to minimize a loss function by updating tensor weights via backpropagation. Each step involves a sequence of tensor contractions, which are effectively chains of matrix multiplications across the tree edges. If the bond dimension between two tensors is smaller than the required rank of the function being learned, the resulting product of those matrices is mathematically constrained to that lower rank. No matter how you adjust the individual entries within the tensors, the contraction result cannot capture the higher-order dependencies of the target function. The expressivity is hard-capped by the geometry of the architecture itself.

Degenerate Saddles and Parity

This structural bottleneck alters the loss surface by introducing degenerate saddle points. In standard neural networks, saddle points are often traversed by gradients that indicate a descent path. In these networks, the gradient with respect to specific weights becomes identically zero because the model lacks the representational capacity to reduce the error further. The loss surface effectively flattens in these directions, trapping the optimizer because no infinitesimal update can overcome the missing dimensions of the tensor bond.

A parity function provides a clear illustration of this phenomenon. Computing the parity of N bits requires access to the correlations between all input bits. If you decompose this into a tree tensor network, the contraction process aggregates information as you move up the tree branches. If the bond dimension is restricted, the network is forced to truncate the correlation information at each bottleneck. Once the chain of contractions hits a dimension lower than the requirements of the parity function, the network effectively loses the dependency information for the remaining bits. It is not that the model is "trying" and failing; it is that the math of the contraction prohibits the information from ever being represented in the final output.

What Stays Unknown

We know that these networks possess conditional benignity: if the model finds a minimum-norm local minimum, it is guaranteed to be a global minimum. However, the exact conditions that force the optimizer to favor these minimum-norm solutions over others in practice remain an open question. We have a solid grasp on why the rank constraint creates an impassable wall, but we have yet to fully map the transition between the basin of attraction that leads to a global solution and the plateaus that cause these networks to stall.