Janas: Run LLMs larger than RAM by streaming experts from SSD

8.9 relevance

Open source tool to run LLMs larger than RAM by streaming experts from SSD, highly actionable.

AI/ML github.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building AI-powered applications, Janas demonstrates that large MoE models can run on commodity hardware, reducing cloud dependency and enabling local inference for latency-sensitive or privacy-critical agent workflows.

Key Takeaways

Evaluate Janas for local inference of large MoE models to reduce cloud costs and improve latency in agent-based systems.

37 tok/s from Qwen3-Next-80B (48 GB) on a 32 GB laptop, 24 without drafts vs 2.8 for llama.cpp; reading 62 vs 16 tok/s. Faster than llama.cpp on all 13 models compared in Q4_K_M. LLMs bigger than y...

Summary

Janas-LLM runs 80B-parameter MoE models on a 32GB laptop by streaming expert weights from an NVMe SSD and caching frequent ones in memory, achieving 37 tok/s vs llama.cpp's 2.8 tok/s for Qwen3-Next-80B. Written in C with no dependencies, it outperforms llama.cpp across all tested models with 2-10x speedups for both prompt processing and text generation. The experimental open-source tool by Maurizio Cammalleri is available on GitHub and supports Linux and Windows.