Merging Multimodal Models with TAC-Merge

The researchers behind TAC-Merge have shifted the strategy for combining multimodal models from independent layer aggregation to a cross-layer influence approach. Traditional methods for model merging, such as averaging weights or simple parameter interpolation, treat each layer as an isolated component. When merging models with distinct visual and language capabilities, this approach fails because updates are coupled across layers, a factor that standard techniques ignore. By acknowledging this dependency, the model keeps the early layer's visual feature-map geometry intact for the language layer to process correctly.

The Problem with Layer-Isolation

Most current merging techniques combine updates within individual layers. In a multimodal system, this is problematic because an update to an early vision-processing block changes the latent vector passed to the subsequent cross-attention or language-processing blocks. If you merge experts independently, you ignore the fact that these updates are coupled. Because the output of the vision encoder acts as the base language for the subsequent layers, if the base changes, the downstream logic loses its context.

Consider the practical challenge of merging a vision-language model trained on specific image-text pairs with one tuned for generic reasoning. If you simply average the weights of a vision encoder that maps images to a specific coordinate space, the downstream language model receives a shifted input that it no longer recognizes. Traditional averaging treats each layer as a stationary object, but in deep neural networks, layers act as a pipeline. Each layer’s output is the input to the next, and any change at an early stage affects all subsequent layers.

Comparing Merging Paradigms

How TAC-Merge Works

TAC-Merge manages this coupling through two technical modules: Multimodal Influence Mapping (MIM) and Coupled Merge Control (CMC). MIM constructs a graph representing how an update at any depth influences the final output of the model. To characterize the relationship between disparate layer updates, it utilizes Ricci curvature. In this context, curvature measures how trajectories of activation patterns—which represent how data flows through the layers—converge or diverge across the latent manifold. A high curvature score signifies that a parameter change will cause significant, non-linear divergence in downstream representations. MIM uses these scores to define a shared fusion objective that identifies which layers have the strongest effect on the final output.

Once the influence is mapped, CMC takes over. It treats the merge as a unified optimization problem. The objective function aggregates information from multiple layers to solve for weight coefficients. Rather than simply averaging, it uses the curvature metrics to weight the contribution of each expert. If a layer has high curvature, the objective function places stricter constraints on how that layer's weights are modified to ensure that the merged output does not deviate from the target task's performance. By jointly optimizing these regional weights, the process seeks to minimize the global loss across all tasks.

Engineering Implications

For an engineer, this approach shifts the cost from simple averaging to a constraint-satisfaction problem. The process of calculating curvature requires that you understand which layers are most sensitive to change. You must decide whether you prioritize the preservation of the original experts' performance or the alignment of the new merged model. Because the merge is treated as a global optimization problem, you have a mechanism to define how much weight to give each expert based on their influence on the output. This provides more control than simple interpolation, as you are not assuming that all layers contribute equally to the final task success.