Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP
A developer built a native TPU inference rig for Gemma 4 E2B using Rust on a Cloud TPU v5e-1 VM, with all 35 layers, ring cache, and decode loop written in Rust and reaching TorchTPU through a 154-line C++ shim over ATen instead of the tch crate. The setup boots in 4 minutes 29 seconds from Python 3.14 to serving an OpenAI-compatible endpoint, decodes at a median 12.20 tokens per second, and matches Hugging Face Transformers on 64 of 64 greedy tokens on CPU. A Claude Code MCP server provisions the TPU, and the VM builds the Rust crate as a PyO3 extension on boot, with Python hosting the HTTP server and process.