Continuous-Time Acoustic Modelling with Neural CDEs

Continuous-time acoustic modelling using Neural Controlled Differential Equations (CDEs) replaces discrete frame-level duration expansion in text-to-speech systems. Traditional models rely on a length-regulation step where predicted durations repeat encoder states to match frame counts, forcing the system to output static vectors for the entire duration of a phone. By reformulating phone representations as a continuous control path, CDE-based models evolve hidden states dynamically as the system progresses through the acoustic interval. This trajectory is sampled during decoding, resulting in more fluid transitions between states than frame-based repetition allows.

Comparing Discrete and Continuous Approaches

In standard TTS pipelines, duration prediction forces a sequence of phone embeddings to match target audio length. If an encoder produces 10 phonemes for a 100-frame audio clip, the system repeats those 10 vectors to fill the space. Neural CDEs treat the latent space as a continuous vector field, where the acoustic state changes continuously over time.

Technically, the model maps a sequence of phoneme embeddings into a continuous path through cubic spline interpolation. This spline defines the path X(t) that drives the evolution of the hidden state H(t). The network f(H(t)) acts as a steering mechanism; it takes the current acoustic state and outputs a weight matrix that determines how the incoming phoneme information, dX, should nudge the current state H(t) forward. The integrated hidden state H(t) is then passed through a projection layer, often a linear transformation, to map the high-dimensional latent vector to the specific parameters of a target mel-spectrogram frame. Because H(t) changes continuously, the resulting output features evolve smoothly, avoiding the discrete jumps typical of frame-duplication.

Implications for Model Architecture

For those building speech synthesis systems, this transition changes how you handle temporal resolution. Instead of tuning the duration predictor to force audio into a rigid target shape, you tune the vector field that governs the derivative of the acoustic features. You modify the control path magnitude by scaling the values of the interpolated phone embeddings before they enter the differential equation solver. When you increase the slope of this path, the hidden state moves through the latent space at a higher velocity. This velocity translates directly to the rate of change in the output acoustic features, allowing for more expressive acoustic gradients that would otherwise be smoothed out or forced into static blocks by frame repetition.

Consider the synthesis of an emotional prosody curve. In a discrete model, shifting from neutral to emphatic tone is constrained by the frame rate; you can only change the output when moving to the next discrete vector, often resulting in stair-step artifacts. With a CDE approach, the model continuously solves for the hidden state across the entire interval, ensuring that the acoustic features evolve according to the learned vector field throughout the phoneme.

This approach introduces a computational cost: the need for numerical integration. Every step of the decoding process requires calling a solver to compute the next state based on the differential equation. This is more expensive than the simple index-lookups used in standard duration-expanded systems. The performance burden scales with the number of evaluation steps required by the solver to maintain stability. For production systems, the primary engineering challenge is balancing the integration step-size against the audio quality requirements, ensuring that the continuous trajectory remains coherent without exceeding latency budgets. It remains unclear how these models will scale to ultra-long-form speech compared to autoregressive baselines, as the cumulative error in integration over long sequences is not yet fully characterised in the literature.