Rbitnet

A warren for bits

Rbitnet memory-maps the GGUF, runs the transformer, and serves tokens over an OpenAI-compatible API on port 8080. The whole runtime is Rust.

What ships today

bitnet-core Llama / Qwen3 / BitNet b1.58 GGUF, CPU quantized kernels, paged & quantized KV hooks.
rbitnet-server OpenAI /v1 HTTP, SSE streaming, health / ready / Prometheus metrics, hot reload.
rbitnet CLI Curated catalog, Hub download, quickstart, TUI chat, BitNet recipes, optional train helpers.

What's new in 0.2.0

Released 8 October 2026. The everyday path is unchanged: install, download a GGUF, rbitnet serve. The new pieces are opt-in and aimed at an NVIDIA GPU.

CUDA Llama serving Several chat requests can share GPU projections. Turn on continuous batching and fused multi-seq. Live per-token streaming is a separate opt-in.
Prompt restore Llama and Qwen prefixes can sit in RAM, then on disk. A measured Llama 3.2 1B prompt returned in tens of milliseconds instead of a full recompute.
Block prefill CUDA block prefill for Llama, Qwen, GPT-OSS, and GLM, plus paged KV so a 16 GiB GPU can hold more than one Llama request.

Changelog · How to enable it

Status

The HTTP server and the main GGUF paths are in daily use. CUDA Llama multi-request serving shipped in 0.2.0 as an opt-in. Other architectures still run one forward at a time.

Area	State
OpenAI-shaped HTTP + limits + auth + metrics	Done
CI / release binaries on tag	Done
GGUF Llama forward + tokenizer + BitNet native path	Done
CUDA Llama fused prefill and decode	Opt-in, 0.2.0
Llama / Qwen RAM and SSD prefix restore	Opt-in, 0.2.0
Qwen, GPT-OSS, GLM shared multi-request forward	Not in 0.2.0
Full production-ready claim	Not claimed

Full status & roadmap →

Roadmap focus

CPU remains the default when no NVIDIA GPU is present. 0.2.0 closed the Llama CUDA serving slice. The next work is the same idea on Qwen and MoE models.

Near term Live SSE and adaptive admission are opt-in on Llama CUDA. Publish the same 1/4/8 numbers on your own GPU before you rely on them.
Next Shared multi-request decode for Qwen and for GPT-OSS / GLM experts. CUDA graphs stay experimental.
Not in this release Grammar-constrained JSON, image chat, and Vulkan / Metal inference.

Advanced guide →

Install

One checkout, one binary. Python only if you convert checkpoints to GGUF upstream.

Linux and macOS:

curl -fsSL https://raw.githubusercontent.com/azerothl/Rbitnet/main/scripts/install.sh | sh
rbitnet models install tinyllama:q4 --dir ./models
rbitnet up tinyllama:q4 --dir ./models
rbitnet serve

Windows (PowerShell):

irm https://raw.githubusercontent.com/azerothl/Rbitnet/main/scripts/install.ps1 | iex
rbitnet models install tinyllama:q4 --dir ./models
rbitnet up tinyllama:q4 --dir ./models
rbitnet serve

User guide · FR quickstart

Documentation

Two guides cover the product. The user guide is enough to install and chat. The advanced guide has GPU flags, concurrent requests, and deployment.