Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

8.1 relevance

Benchmarking six LLM serving engines on limited GPU is highly relevant for AI deployment.

AI/ML dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform engineers deploying small models on consumer GPUs, this head-to-head comparison reveals that engine choice—not model quantization—determines throughput under load, with vLLM and SGLang dominating batching scenarios while llama.cpp remains competitive for single-user inference.

Key Takeaways

Choose vLLM or SGLang for multi-request serving on 8 GB GPUs; reserve llama.cpp for single-user or low-concurrency workloads to avoid wasting GPU memory.

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

Summary

A developer benchmarked six inference serving engines—llama.cpp, vLLM, SGLang, mistral.rs, plain PyTorch, and LiteRT-LM—all running Gemma 4 E2B with Google's QAT 4-bit weights on a single 8 GB RTX 3050. Answer quality was nearly identical across engines (GSM8K 0.8992–0.9030, BFCL 0.9125–0.925), but throughput varied dramatically under load: vLLM and SGLang reached 1,651–1,711 tokens/second at 64 concurrent requests, while llama.cpp managed 633 and mistral.rs 308. The E2B speculative decoder boosted vLLM 1.90x but slowed mistral.rs to 0.72x, and llama.cpp showed no benefit from using more GPU memory beyond 2.6 GB.

Author

xbill