DualMLC: Fusing Encoders and Decoders for Classification

DualMLC processes documents through two distinct model architectures to capture complementary semantic evidence for multi-label text classification. By merging the output of a bidirectional encoder with an autoregressive decoder, the approach addresses the limitations of pipelines that rely on a single feature representation. The framework passes the same input document into two separate architectures, with each branch producing a vector of relevance scores, known as logits, across the entire label set. If you are building high-accuracy classifiers, this approach replaces the standard practice of relying on one model architecture for all patterns.

The Mechanism of Late Logit Fusion

The bidirectional branch uses an encoder-only model where each token attends to every other token simultaneously. This full-context visibility allows the model to identify specific markers or keywords anywhere in the text. Simultaneously, the autoregressive branch uses causal masking to restrict each token's attention to only those that came before it. This allows the model to capture dependency chains, such as the relationship between a condition and a required action described later in a paragraph.

DualMLC uses a weighted sum of these logits to determine the final assignment. The fused logit is calculated as L = α * Logit_encoder + (1 - α) * Logit_decoder. In this equation, α is a vector of learnable weights where each individual label has its own corresponding parameter. During backpropagation, the gradient of the loss with respect to α measures how much each branch contributed to the correct classification. If the encoder provides a more accurate logit for a specific label, the gradient update adjusts α to increase the encoder's weight for that class.

Performance Trade-offs and Scaling

The primary constraint of this architecture is the computational cost. Because you must execute both the encoder and decoder passes, the latency is additive. These branches are generally executed in parallel on separate compute streams to maximize throughput, but the total inference time for a request is constrained by the slower of the two branches, typically the autoregressive decoder. The fusion process itself adds negligible latency, as it is a simple element-wise multiplication and summation operation performed after the final logits are generated.

Rare labels present a specific difficulty for the learnable weight vector. When a label appears infrequently, the training signal is insufficient for the optimizer to settle on an optimal α. This often results in the weight oscillating during training, which creates high variance and poor classification performance for those categories. Practitioners usually handle this by initializing α at 0.5 to keep both branches equally active for rare labels, effectively hedging against the lack of specific signal until the model observes enough instances to distinguish the branches. It remains an open question whether structural priors—such as forcing a bias toward the encoder for short, keyword-dense labels—could stabilize these weights better than purely data-driven training.

The effectiveness of this method depends on whether the bidirectional and autoregressive representations provide genuinely different information for your specific labels. If your dataset relies entirely on keyword matching, the decoder may provide little value beyond the encoder, potentially adding noise. The benefit is most pronounced when the label definition requires both local context and sequential logic.