Nano-vLLM

Nano-vLLM is a minimal single-GPU inference engine for Qwen3. It includes continuous batching, paged KV cache management, prefix caching, chunked prefill, and temperature sampling in a compact PyTorch implementation.

Requirements

• Python 3.10-3.12
• PyTorch 2.4 or newer
• One CUDA-capable NVIDIA GPU
• Local Qwen3 model weights in safetensors format

Install the package:

pip install -e .

Download Qwen3-0.6B:

huggingface-cli download Qwen/Qwen3-0.6B \
  --local-dir models/Qwen3-0.6B

Example

example.py runs batched text generation:

python example.py

To use a model stored elsewhere:

MODEL_PATH=/path/to/Qwen3-0.6B python example.py

Benchmark

python bench.py

The benchmark also reads MODEL_PATH and defaults to models/Qwen3-0.6B.

API

from nanovllm import LLM, SamplingParams

llm = LLM("models/Qwen3-0.6B", max_num_batched_tokens=256)
outputs = llm.generate(
    ["Hello, Nano-vLLM."],
    SamplingParams(temperature=0, max_tokens=32),
)
print(outputs[0]["text"])

License

This fork is based on GeeeekExplorer/nano-vllm and is distributed under the MIT License. See LICENSE for the original and fork copyright notices.