Automating Symmetry Discovery with Groupoids
A groupoid-based reinforcement learning framework enables agents to discover local, context-dependent symmetries by mapping distinct raw states into shared canonical representations.
This framework uses groupoids to discover state-dependent symmetries, mapping regional state-space equivalencies to a single canonical form. Traditional reinforcement learning forces an agent to learn policies for every orientation of a task separately because it relies on global symmetry assumptions. Groupoids are mathematical structures that generalize groups by allowing composition only when targets and sources align. They allow the agent to automate what previously required manual symmetry definitions. By partitioning the state space into orbits, the agent identifies which regions are equivalent and maps them to a single canonical representative.
Shifting from Global to Local Symmetry
Standard algorithms assume a transformation remains valid across the entire state space. When a symmetry exists only locally, such as in modular environments with repeating sub-tasks, these agents fail to generalize. The groupoid framework partitions the state space into orbits where an orbit representative acts as a canonical state. Unlike fixed group actions, groupoids allow the agent to treat different sub-tasks as the same problem if their underlying transition dynamics match.
| Feature | Standard Q-Learning | Groupoid-Based RL |
|---|---|---|
| Symmetry Scope | Global / Uniform | Local / State-Dependent |
| Abstraction Basis | Fixed Group Actions | Dynamic Equivalence Discovery |
| State Mapping | None (Raw States) | Canonical Forms via Transporters |
| Sample Efficiency | Low | Improved |
Automating Symmetry Discovery
The framework learns symmetries by identifying states that exhibit identical transition dynamics. It employs a siamese-style architecture where the network learns an embedding space. To determine that two states are equivalent before the loss function is applied, the agent compares the transition outcomes observed during environment interaction. If applying an action at state A leads to an outcome consistent with applying an action at state B, the network treats them as part of the same orbit. The network minimizes the distance between these states in the embedding space, essentially forcing them to converge to the same canonical representation.
To bridge the gap between the canonical space and the actual environment, the agent uses a learned transporter function. This function maps a state back to its specific real-world coordinates. When the policy network makes a decision in the normalized, canonical frame, the transporter acts as a decoder or projection layer that shifts the action intent into the specific action required for the current state's geometry. The policy never sees the variations in the raw state; it operates on the canonical form, while the transporter handles the geometric translation required to execute the action in the local coordinate frame.
Scaling the Framework
The complexity of managing these equivalence classes grows with the diversity of the environment. In cases where the environment contains highly heterogeneous sub-tasks, the number of distinct orbits increases. This places a higher demand on the model to maintain consistent mappings for each unique type of transition logic present in the environment.
We currently lack a clear understanding of the limits of functional approximation for these transporters. While neural networks can learn to approximate these mappings, it remains unclear if they can maintain perfect precision when the environment features an extremely high number of distinct symmetries. The primary challenge for future development is determining if these transporters can be unified into a single, continuous function that covers the entire state space, or if they will remain tied to the density of the learned transition dynamics.