Fixing Structural Generalization in Transformers
Recent research demonstrates that compositional generalization in Transformers depends less on lexical variety and more on the diversity of structural constructors present during training.
Transformers often struggle with compositional generalization because their training data lacks structural variance. When a dataset is built using a single, consistent method for nesting expressions, the model interprets that specific geometry as a predictive signal. It fails to decouple the core logic of the task from the specific syntax used to represent it because it has never encountered the same rule expressed in a different shape. This creates a reliance on shortcuts that break the moment the input structure deviates from the training distribution.
Structural Constructors and Positional Bias
A constructor defines how individual inputs are bound together into a larger expression. Consider the operation of addition in a nested sequence. You can represent (5 + 3) + 2 as a left-associative tree where the first pair is grouped, or as a right-associative structure like 5 + (3 + 2). If a model is trained exclusively on left-associative trees, it learns to associate the addition operator with fixed positional offsets. In standard attention, the softmax function normalizes scores to focus on relevant tokens. When the input geometry remains constant, the model learns to place high probability mass on the exact indices that historically held the components of the operation. The model effectively hard-codes its attention to look at those specific spots, ignoring the recursive nature of the logic because it never needs to look elsewhere to minimize training loss.
How Diversity Forces Generalization
When training data incorporates high type diversity by mixing multiple constructors, the model can no longer rely on fixed attention paths to minimize loss. If the training set contains both left-associative and right-associative trees, a weight configuration that locks onto a fixed index for one structure will produce high error rates when it encounters the other. When a model trained on mixed tree structures encounters an instance that does not fit its current template, the gradients are penalized because the fixed-index strategy cannot capture the relational dependency across both structures. To minimize the loss across this varied set, the model is forced to shift its attention heads away from fixed absolute indices toward relative-position or dependency-based pointers. These pointers identify the logic by tracking the relationship between the operator and its operands, regardless of whether they appear at index 2 or index 5. This shift in internal representation means the model is no longer relying on a geometric template, but on a rule that is invariant to the tree structure.
| Variable | Historical View | New Finding |
|---|---|---|
| Primary Difficulty | Inherent structural limitation | Low type diversity |
| Key Metric | Compound divergence | Contradicted as an explanation |
| Generalization Source | Lexical variety | Constructor type diversity |
Designing Diverse Pipelines
For those building pipelines to fine-tune models, this shifts the focus from lexical variation to structural variation. Simply swapping variable names is insufficient if the underlying syntax trees remain uniform. If you want to improve performance on recursive tasks, your training data must include a broad distribution of constructor types. If your dataset for a logic task is composed of one thousand examples, ensure that at least one-quarter of those instances use alternative nesting or grouping strategies. This forces the attention mechanism to move beyond memorizing the layout and requires the model to develop a generalized understanding of the underlying operations.