Serving LLMs on Tenstorrent Hardware

The vLLM Tenstorrent plugin integrates models into the vLLM ecosystem by leveraging the platform’s out-of-tree execution mechanism. Instead of replacing the existing vLLM stack, it maps model architectures to Tenstorrent hardware using a prefix-based convention. This allows operators to run familiar inference workloads while delegating compute to the Tenstorrent tt-metal runtime. The integration relies on the underlying ttnn environment for device discovery, which automatically detects the available silicon topology upon initialization, allowing the plugin to bind the model to the detected mesh fabric without manual runtime configuration.

The Shift to Compiled Mesh Execution

Tenstorrent hardware organizes compute across a 2D fabric of cores. Programs are traced and compiled specifically for a fixed hardware topology. This forces a departure from the typical GPU inference pattern, where parallelism is often adjusted via runtime flags or host-side collective operations. On Tenstorrent hardware, the parallelism is baked into the mesh program during compilation.

Plugin Scheduling and Constraints

Because the mesh operates as a single, unified execution context, standard vLLM parameters like tensor-parallel-size are ignored. Instead, the MESH_DEVICE variable dictates the physical grid resources, effectively locking the model to a specific hardware footprint. This creates a friction point with vLLM's standard request scheduler. vLLM is built to manage fluid token budgets, but Tenstorrent mesh programs are rigid. The execution is a fixed, traced step, which eliminates the overhead of host-orchestrated communication but demands strict adherence to the pre-compiled tensor shapes.

Device-Side Sampling Mechanism

Performance on this stack hinges on keeping data movement within the mesh fabric. In a standard GPU implementation, the model generates a massive logit tensor that must be transferred to the CPU host. The host then performs the argmax or top-k operation to select the next token. This transfer creates a bottleneck that grows with every token. The Tenstorrent plugin instead performs sampling directly on the device. Logits are generated within the specific hardware cores where the final layers reside, and the sampling logic is executed locally on those cores. Only the final token ID is transmitted back to the host. By confining the massive probability distribution to the internal mesh memory, the system keeps the host-device interconnect clear for weight streaming.

Handling Batch Mismatches

Compiled mesh programs require fixed tensor shapes. If a request arrives that deviates from the pre-compiled sequence length or batch size, the runtime cannot simply pad the tensors on the fly. When a request does not fit the active trace, the system must either pad the input to reach the pre-defined shape or, in cases of severe mismatch, trigger a re-compilation. Padding is the primary management strategy, filling the remaining capacity with empty tokens to satisfy the fixed-shape requirement. If the discrepancy is too large, the system is forced to re-compile the mesh program. This is a disruptive event, causing a multi-second latency spike that effectively stalls the pipeline. Builders must decide if their workload stability justifies this rigid throughput. What remains unknown is the effectiveness of various 'bucketed' compilation strategies—where operators pre-compile for a set of common sequence lengths—in masking these re-compilation costs for real-world, unpredictable request distributions.