Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

8.5 relevance

Running large LLM on limited hardware, highly novel and actionable for AI/ML engineers.

Open Source github.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform and AI engineers running agentic or RAG workloads on constrained hardware, this eliminates GPU dependency for massive MoE models, enabling local inference of 284B-parameter checkpoints on commodity desktops or servers with 64 GB RAM and AVX-512 support.

Key Takeaways

Evaluate llama-modchip v2.4.0 for CPU-only MoE inference if you need to run 284B DeepSeek or 30B Nemotron models on sub-64 GB RAM systems, leveraging Fractal-64 quantization and multi-column tiling to avoid GPU costs.

🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code...

Summary

Llama-modchip v2.4.0 enables CPU-first inference of 284B DeepSeek-V4-Flash and 30B MoE models on as little as 64GB RAM by applying custom ggml/llama.cpp kernel patches, quantization transcoders, and a 4-level register/L1/L2/L3 multi-column tiling schedule that boosts prefill by 77% and decode by 53%. The 284B checkpoint is compressed from 76.87 GiB to 56.56 GiB (1.71 bpw) via Fractal-64 quantization, fitting entirely in system RAM with zero NVMe swap, while a 1.64 MB L2/L3 Zobrist n-gram drafter and /dev/shm KV slot snapshots enable 13.1 ms agent context restore for tool-use workflows.