Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp
9.1 relevance
Score Breakdown
technical depth 10
novelty 9
actionability 9
community 8
strategic 7
personal 10
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Cutting-edge optimization for running large models on constrained hardware, highly actionable.
Summary
On a 4 GB GTX 1650 Ti, Gemma 4 E2B does not fit in 16-bit, and Transformers' own 4-bit path runs out of memory on its first forward pass. Kept in Google's QAT 4-bit end to end, with a few Triton kernels and a CUDA graph, plain PyTorch decodes at 88 tok/s, 1.07x llama.cpp on the same card, and holds the full 131,072-token context.