[GitHub Trending] JustVugg/colibri

7.4 relevance

Novel hierarchical inference engine for MoE models, highly technical and relevant to AI/ML infrastructure.

AI/ML github.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform engineers and architects, Colibri demonstrates a practical approach to running state-of-the-art MoE models on commodity hardware without GPU dependency, which could reshape how teams deploy LLMs in edge, CI/CD, or cost-sensitive environments.

Key Takeaways

Evaluate Colibri for running 100B+ parameter MoE models on CPU-only machines by leveraging disk-based expert swapping, starting with the GLM-5.2 744B model requiring ~10GB RAM.

A hierarchical inference engine for frontier MoE models: hot experts on the GPU (CUDA, Vulkan), warm ones in RAM, cold ones streamed from NVMe. Written in C, runs on the hardware you already own. -...

Summary

Colibri is a free, open-source inference engine written in pure C that runs massive mixture-of-experts models (up to 744B parameters) on consumer hardware by keeping only the active expert in RAM and streaming the rest from disk. It supports 13 model families including GLM-5.2, DeepSeek V4 Flash, Qwen3.6, and OLMoE, requires no GPU (though Vulkan/CUDA are optional), and needs just 8GB RAM with 22GB disk for the smallest model. The setup auto-detects hardware, recommends a fitting model, builds or downloads the engine, and provides an OpenAI-compatible API endpoint at localhost:8000/v1.