Kraken: Decoupling Audio Reconstruction from Translation

Most speech models generate audio by predicting waveforms directly. This is computationally expensive because waveforms require high-frequency sampling, forcing the transformer to learn the nuances of individual samples rather than higher-level semantic features. Kraken uses single-layer vector quantization to map audio into a compact token space derived from self-supervised learning features. These tokens act as a lossy filter, retaining high-level phonetic information like phoneme boundaries and broad spectral trends, while discarding transient noise and phase details that carry little semantic weight. The decoder maps these discrete linguistic tokens to continuous spectral patterns using a codebook-based lookup mechanism, which feeds the result into a generative neural vocoder to reconstruct the final waveform.

Decoupling Speaker Identity

The Kraken architecture handles speaker identity through the Autowave-X module. This component fuses translated linguistic tokens from the Qwen3-8B model with acoustic features extracted from the input source. The fusion occurs via cross-attention layers, where the acoustic features act as keys and values that provide the decoder with specific pitch contours and formant shifts. By injecting these spectral markers directly into the latent representation, the model anchors the generated phonemes to the original speaker's vocal characteristics. The decoder performs synthesis by using the cross-attention output to condition the spectral generation process. Because the LLM is responsible only for text-based translation, the decoder manages waveform synthesis independently. When translating English to French, the LLM determines the target words, while the cross-attention layers pull the spectral envelope from the English input to ensure the French output retains the original speaker's vocal characteristics. You can tune the Autowave-X decoder independently to improve voice fidelity without retraining the underlying language model. This is particularly visible when a speaker uses nuanced prosody; where other models might flatten a pitch curve, Kraken preserves the specific upward inflection at the end of French interrogatives by mapping the source acoustic keys directly to the target spectral tokens.

Preserving Vocal Fidelity

Kraken achieves improved prosody transfer capabilities compared to Qwen3-8B. Because the identity signal does not compete with the linguistic translation task for the LLM's attention, the resulting audio maintains stable prosody across varied speaker demographics. The cross-attention mechanism preserves source-specific texture by bypassing the training bias found in monolithic models, which often regress to a generic average voice when faced with rare dialects or distinct vocal textures. While we have confirmed this performance gain, the long-term stability of these transfers across highly divergent language pairs remains an open question for future architecture audits.