Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway

9.1 relevance

Multi-tenant GPU serving architecture with Rust, highly technical and directly relevant.

AI/ML dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform engineers, this is a concrete reference architecture for multi-tenant LLM serving that solves GPU utilization, tenant isolation, and usage-based billing with a minimal Rust gateway and vLLM's scheduling.

Key Takeaways

Evaluate vLLM's Multi-LoRA and priority scheduling combined with a lightweight Rust gateway to build cost-effective, multi-tenant AI inference services on shared GPU hardware.

Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway

Summary

A Rust gateway (axum, 245 lines) fronts a shared GCP T4 running Gemma 4 E2B via vLLM Multi-LoRA, enforcing per-tenant API keys, adapters, and prefix caches. vLLM's priority scheduling mitigates contention, dropping TTFT from 29.9s to 6.52s for other tenants when one sends long prompts. The system bills by GPU time via a ledger matching vLLM's counters, turning a single VM into a multi-tenant business.

Author

xbill