Benchmarking Multimodal Federated Learning in Healthcare

OmniMed-FL provides a controlled sandbox to test how vision and language models can be fused across distributed hospital networks without centralizing sensitive patient data. It tackles a primary obstacle in clinical AI: standard algorithms lack the design to simultaneously process both medical imaging and patient records, while privacy regulations like HIPAA and GDPR strictly forbid moving raw patient records to a central server. The framework uses a proxy corpus of 3,000 chest radiographs paired with synthetic clinical notes to measure performance under non-IID, or non-independently and identically distributed, data skew. These results serve as descriptive proxy comparisons, not estimates of diagnostic performance or clinical deployment readiness.

To bridge the gap between image and text, OmniMed-FL employs a late-fusion approach. The system extracts separate feature vectors from the vision and language encoders, concatenates them into a single joint representation, and feeds this into a final classification head. This architecture allows the hospital to keep local models specialized to their specific imaging equipment and clinical documentation styles, only communicating the joint weights to the global server.

Performance and Stability Across Strategies

The choice of aggregation strategy dictates whether a global model converges or collapses when individual hospitals have vastly different patient populations. In a clinical setting, label skew creates a scenario where one hospital may have an abundance of pneumonia cases while another specializes in fractures, forcing the global model to resolve conflicting gradients. The table below compares the performance of different aggregation methods at a client count of K=5 with a Dirichlet parameter of alpha=0.1.

FedProx achieves a higher F1 score than standard FedAvg. FedProx works by adding a proximal term, typically a squared L2 distance, to the local loss function. This term penalizes the difference between the local weights and the global weights, effectively restricting local model updates from drifting too far from the global state. By constraining this divergence, the global model avoids becoming overly biased toward the idiosyncratic patterns of a single, large-volume clinic.

Scaling and Computational Trade-offs

Engineers designing these systems face a strict limit on diagnostic accuracy. They must also manage communication overhead and the negative impact of data drift. Increasing the number of participating hospitals from 3 to 20 causes a linear increase in bidirectional data volume, which peaks at 183.5 GiB in the largest tested configuration.

Label skew remains the most dangerous variable for model stability. The study demonstrates that shifting the label distribution degrades F1 scores by 0.27, whereas increasing the number of clients sevenfold only impacts the score by 0.10. Data heterogeneity represents the true bottleneck for convergence. When the underlying labels are inconsistent, local models in different environments learn to map the same visual features to mismatched vector representations. This prevents the global classifier from identifying common patterns in the joint feature space, leading to degradation that simple ensemble methods cannot compensate for.

Multimodal fusion consistently outperforms unimodal baselines, reaching a macro-F1 of 0.956 compared to 0.934 for text-only models on this proxy corpus. This accuracy comes with a measurable cost. The multimodal models require 2.3 times the memory of their unimodal counterparts to store the parameters of both the image and text encoders, which necessitates local edge hardware capable of supporting high-VRAM throughput.