Rust LLM Serving Engine

A focused inference engine with clear subsystem boundaries

Hetero-Paged-Infer concentrates on the core serving path: paged KV cache management, continuous batching, and OpenAI-compatible HTTP APIs.

Read ArchitectureQuick Start

Rust

Language

Memory-safe systems code with explicit interfaces.

Paged KV Cache

Memory

Block-based allocation and accounting.

Continuous Batching

Scheduling

Prefill/decode flow with decode-priority behavior.

OpenAI-Compatible

Serving

Completions and chat endpoints with operational probes.

Documentation map

Architecture

Engine structure, scheduling model, and memory management design.

Setup

Installation, configuration, and local usage.

API

Core types and HTTP API references.

Deployment

Docker and production-oriented deployment notes.

Development

Contributing and validation workflow.

Current scope

• The repository already provides a usable local engine, scheduler, KV cache manager, and HTTP serving layer.
• The docs site now keeps only current implementation guidance and omits historical, whitepaper, and showcase content.
• Real CUDA kernel execution remains future work and is not documented as available functionality.