custom-llm-serving-engine

Production-oriented LLM inference server built from scratch in Python.

Continuous batching | SSE streaming | KV cache tracking | Prometheus metrics

---

Highlights

• OpenAI-compatible API with /v1/completions and /v1/chat/completions
• Streaming via Server-Sent Events, token-by-token delivery
• Continuous batching foundation sharing GPU across concurrent requests
• KV cache management with per-request memory tracking
• Prometheus metrics at /metrics for monitoring and alerting
• Health check at /health for orchestration readiness probes
• Graceful shutdown that completes in-flight requests before cleanup
• Structured JSON logging for observability pipelines

For full architecture details, trade-offs and limitations see DESIGN.md.

For the complete verification report covering all 6 end-to-end scenarios see summary.txt.

---

Architecture

HTTP Request
     |
     v
FastAPI (uvicorn)
     |
     +-- /health, /metrics               -> immediate response
     +-- /v1/completions (stream=false)  -> InferenceEngine (thread pool)
     +-- /v1/completions (stream=true)   -> RequestScheduler
     |                                      -> BatchedInferenceEngine
     |                                      -> asyncio.Queue -> SSE
     +-- /v1/chat/completions            -> same paths as above

Two engine modes:

Mode	Flag	Description
Simple	LLM_USE_BATCHING=false	HuggingFace generate() in thread pool. Supports temperature, top_p.
Batched	LLM_USE_BATCHING=true	Background async loop, token-by-token with KV cache reuse. Required for streaming.

---

Quick Start

git clone https://github.com/JohnScheuer/custom-llm-serving-engine
cd custom-llm-serving-engine
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

Server without model (route testing)

python -m uvicorn src.server.main:app
curl http://localhost:8000/health

Server with model (CPU)

LLM_LOAD_MODEL_ON_STARTUP=true \
LLM_DEVICE=cpu \
LLM_DTYPE=float32 \
python -m uvicorn src.server.main:app

Server with model and batching (GPU)

LLM_LOAD_MODEL_ON_STARTUP=true \
LLM_USE_BATCHING=true \
python -m uvicorn src.server.main:app

---

Usage

Completion

curl -X POST http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2-0.5B-Instruct",
    "prompt": "The key insight of transformers is",
    "max_tokens": 30,
    "temperature": 0.0
  }'

Chat

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2-0.5B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is attention in neural networks?"}
    ],
    "max_tokens": 60
  }'

Streaming (requires LLM_USE_BATCHING=true)

python examples/streaming_client.py "Explain quantum computing"

Python client

python examples/python_client.py

---

Configuration

All settings use environment variables with the LLM_ prefix or a .env file.

Variable	Default	Description
LLM_MODEL_NAME	Qwen/Qwen2-0.5B-Instruct	HuggingFace model ID
LLM_DEVICE	cuda	cuda or cpu
LLM_DTYPE	float16	float16, bfloat16, float32
LLM_MAX_BATCH_SIZE	16	Max concurrent requests in batch
LLM_KV_CACHE_SIZE_MB	4096	KV cache budget in MB
LLM_LOAD_MODEL_ON_STARTUP	false	Load model during startup
LLM_USE_BATCHING	false	Enable continuous batching engine

---

Running Tests

python -m pytest -q

# Verbose
python -m pytest -v

Current status: 33/33 passing.

Test coverage includes:

• FastAPI endpoints (status codes, JSON schemas, error guards)
• RequestScheduler (enqueue, promote, complete, fail, submit)
• BatchedInferenceEngine (token step, EOS detection, streaming callbacks)
• SimpleKVCacheManager (allocate, release, clear, size estimation)
• InferenceEngine (greedy vs sampling, finish reasons, async execution)

---

Load Test

pip install aiohttp
python benchmarks/load_test.py --concurrent 10 --total 50

Output includes: throughput (req/s), median/p95/p99 latency, success rate.

---

Metrics

Prometheus metrics available at http://localhost:8000/metrics:

Metric	Type	Description
llm_requests_total	Counter	Requests by endpoint and status
llm_request_duration_seconds	Histogram	Latency distribution
llm_active_requests	Gauge	In-flight requests
llm_throughput_tokens_total	Counter	Total generated tokens
llm_kv_cache_usage_bytes	Gauge	KV cache memory usage
llm_batch_size	Gauge	Current batch size
llm_pending_requests	Gauge	Scheduler queue depth

---

Docker

docker build -t custom-llm-serving-engine -f docker/Dockerfile .
docker run -p 8000:8000 \
  -e LLM_LOAD_MODEL_ON_STARTUP=true \
  -e LLM_DEVICE=cpu \
  -e LLM_DTYPE=float32 \
  custom-llm-serving-engine

---

Project Structure

custom-llm-serving-engine/
+-- src/
|   +-- server/
|   |   +-- main.py              # FastAPI app with lifespan
|   |   +-- config.py            # pydantic-settings configuration
|   |   +-- schemas.py           # Pydantic request/response models
|   |   +-- routes/
|   |       +-- completions.py   # /v1/completions (batch + stream)
|   |       +-- chat.py          # /v1/chat/completions
|   |       +-- health.py        # /health
|   |       +-- metrics.py       # /metrics
|   +-- engine/
|   |   +-- model_loader.py      # HuggingFace model loading
|   |   +-- inference.py         # Simple generation engine
|   |   +-- batching.py          # Continuous batching engine
|   |   +-- scheduler.py         # Request queue and scheduling
|   |   +-- kv_cache.py          # KV cache manager
|   +-- utils/
|       +-- logging.py           # Structured JSON logger
|       +-- metrics.py           # Prometheus metric definitions
|       +-- streaming.py         # SSE chunk helpers
+-- tests/
|   +-- test_endpoints.py
|   +-- test_engine.py
|   +-- test_batching.py
|   +-- test_kv_cache.py
+-- benchmarks/
|   +-- load_test.py
+-- examples/
|   +-- curl_completion.sh
|   +-- python_client.py
|   +-- streaming_client.py
+-- docker/
|   +-- Dockerfile
+-- DESIGN.md                    # Full architecture and trade-offs
+-- summary.txt                  # End-to-end verification report
+-- requirements.txt
+-- pyproject.toml
+-- LICENSE
+-- README.md

---

Documentation

Document	Purpose
DESIGN.md	Architecture deep-dive, component descriptions, configuration reference, known limitations, future work, and related projects
summary.txt	Complete project verification report with all 6 end-to-end test scenarios, bug fixes applied, and final status

---

Limitations

This is an MVP serving engine. Key limitations documented honestly:

1. No true GPU batching - requests processed sequentially per step
2. No paged KV cache - full cache per request, potential OOM on large batches
3. Greedy decoding only in batch loop - sampling only in simple engine
4. No request cancellation - disconnected clients still consume GPU
5. Single model per process - no multi-model routing
6. No multi-GPU - single device only

See DESIGN.md for full details and future work.

---

Related Projects

Kernel-level (potential integration as optimized backends):

• mini-llm-inference-engine - C++/CUDA runtime, 87% Tensor Core ceiling
• flash-attention-sm75-fused - register-fused attention, 5 TFLOPS peak
• sm75-tensorcore-microkernel - PTX-level GEMM primitives
• fused-backward-kernels-sm75 - training-adjacent backward kernels

Compiler/Optimization:

• llm-fusion-compiler - kernel fusion and code generation

Analysis (informed design decisions):

• llm-serving-sim - end-to-end serving simulator
• admission-pressure-predictor-bench - admission control policies
• graceful-degradation-bench - degradation strategies under overload

Together these form a coherent full-stack LLM systems portfolio spanning GPU kernels, compiler infrastructure, serving architecture, and system analysis.

---

License

MIT - Joao Felipe De Souza, 2026