A complete notebook-based AI engineering system that snapshots official vLLM documentation, builds a semantic retrieval index, serves a quantized local language model, routes questions between documentation and web-search tools, evaluates concurrent workloads, and monitors system resources in real time.
Data acquisition → preprocessing → indexing → inference serving → agent orchestration → evaluation → observability
- Overview
- Key Features
- Architecture
- System Workflow
- Core Components
- Technology Stack
- Prerequisites
- Quick Start
- Server Configuration
- Agent Routing Policy
- Example Questions
- Generated Outputs
- Troubleshooting
- Current Limitations
- Future Improvements
- Acknowledgments
This repository contains a Jupyter Notebook named vllm-cleare.ipynb that implements an end-to-end developer-assistant pipeline around the vLLM ecosystem.
The system can:
- Download the official vLLM documentation from a selected GitHub release tag.
- Convert the documentation into a reproducible local dataset.
- Build a semantic vector index with LlamaIndex and Hugging Face embeddings.
- Serve an AWQ-quantized Qwen2.5 model through a local OpenAI-compatible API.
- Use an agent to select either local documentation retrieval or web search.
- simulate multiple concurrent users and measure request latency.
- Display live CPU, RAM, GPU, VRAM, process, and server-health metrics.
Unlike a basic inference or RAG demo, this project connects the major stages of a production-oriented AI pipeline in a single, reproducible notebook.
| Capability | Description |
|---|---|
| Reproducible documentation dataset | Downloads documentation from a version-pinned vLLM GitHub release. |
| Semantic RAG retrieval | Searches documentation by meaning rather than exact keyword matching. |
| Local model serving | Runs an AWQ-quantized Qwen2.5 model through vLLM. |
| OpenAI-compatible API | Exposes local inference through standard /v1 endpoints. |
| Tool-based agent routing | Selects local documentation retrieval or web search based on the query. |
| Concurrent request testing | Simulates multiple users and measures latency and parallel speedup. |
| Live observability | Tracks server state, CPU, RAM, GPU, VRAM, temperature, power, and process memory. |
| Private local inference | Keeps model execution and documentation retrieval on local infrastructure. |
flowchart TD
A[Official vLLM GitHub Repository<br/>Versioned Release Tag] --> B[Documentation Snapshot Pipeline]
B --> C[Markdown Extraction and Cleaning]
C --> D[Structured Dataset<br/>raw files + corpus.jsonl + manifest.json]
D --> E[Embedding Pipeline<br/>BAAI/bge-m3]
E --> F[LlamaIndex VectorStoreIndex]
F --> G[Local Documentation Retriever]
H[User Question] --> I[Tool-Calling Agent]
G --> I
J[Web Search Tool<br/>DuckDuckGo] --> I
I --> K[vLLM OpenAI-Compatible Server]
K --> L[Qwen2.5-7B-Instruct-AWQ]
K --> M[Grounded Final Answer]
K --> N[Live Monitoring Dashboard]
N --> O[Server Health]
N --> P[CPU and RAM]
N --> Q[GPU, VRAM, Temperature, and Power]
N --> R[vLLM Process Memory]
1. Select a vLLM release tag
↓
2. Download and snapshot the official documentation
↓
3. Build raw Markdown, JSONL corpus, and manifest files
↓
4. Generate embeddings and construct the vector index
↓
5. Start the local vLLM OpenAI-compatible server
↓
6. Initialize the documentation and web-search tools
↓
7. Route each user question through the agent
↓
8. Generate a response using the selected information source
↓
9. Evaluate concurrent requests and monitor system resources
A reliable RAG system requires a clean, traceable, and reproducible knowledge source. Instead of manually copying documentation, the notebook downloads the official vLLM repository at a selected release tag, such as v0.13.0.
The snapshot stage:
- downloads the repository archive for a selected tag;
- extracts the archive locally;
- collects relevant Markdown documentation files;
- preserves source paths and version information;
- creates a JSONL corpus for retrieval;
- generates a manifest for reproducibility and auditing.
| Artifact | Purpose |
|---|---|
raw/ |
Stores the original documentation files. |
corpus.jsonl |
Stores structured retrieval records in JSON Lines format. |
manifest.json |
Stores snapshot metadata, version details, file counts, and output paths. |
A typical corpus.jsonl record contains:
{
"doc_id": "unique-document-id",
"path": "docs/path/to/file.md",
"version": "v0.13.0",
"url": "official-source-url",
"text": "documentation content"
}This stage converts documentation into a versioned dataset that can be indexed, reproduced, inspected, and compared across releases.
The notebook builds a vector-based retrieval system over the documentation corpus.
BAAI/bge-m3
The embedding model converts documentation chunks and user questions into semantic vector representations. This allows the retriever to find conceptually related content even when the wording differs.
LlamaIndex VectorStoreIndex
The retrieval pipeline:
- loads the structured documentation corpus;
- splits documents into retrieval-friendly chunks;
- generates an embedding for each chunk;
- stores the resulting vectors in a semantic index;
- retrieves the top matching chunks for each technical question;
- passes the retrieved context to the language model.
- How can vLLM be launched as an OpenAI-compatible server?
- How should GPU memory utilization be configured?
- What does tensor parallelism do in vLLM?
- How can the maximum context length be changed?
- Which settings affect concurrent request throughput?
Using retrieved documentation reduces unsupported answers and makes technical responses easier to validate.
The project serves a local instruction-tuned model through vLLM rather than relying on an external inference API.
Qwen/Qwen2.5-7B-Instruct-AWQ
AWQ quantization reduces model memory requirements while retaining strong instruction-following capability, making the model more practical for local GPU inference.
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--dtype half \
--max-model-len 4096 \
--max-num-seqs 8 \
--gpu-memory-utilization 0.55 \
--enforce-eager \
--tensor-parallel-size 2 \
--attention-backend TRITON_ATTN \
--port 8000Important
--tensor-parallel-size 2 assumes that two compatible GPUs are available. Set it to 1 for a single-GPU system, or adjust it to match the intended GPU configuration.
curl http://localhost:8000/v1/modelsA successful JSON response indicates that the server is online and exposing the model through the OpenAI-compatible API.
http://localhost:8000/v1
The notebook creates a ReAct-style agent that dynamically chooses the most appropriate information source.
| Tool | Intended use |
|---|---|
vLLM_Documentation |
vLLM installation, configuration, internals, APIs, deployment, and behavior. |
Web_Search |
Time-sensitive, external, general, or non-vLLM information. |
The documentation tool:
- receives the user's technical question;
- retrieves the most relevant indexed documentation chunks;
- injects the retrieved context into the model prompt;
- generates a documentation-grounded response.
The web-search tool:
- receives an external or time-sensitive question;
- searches the web through DuckDuckGo;
- collects relevant result information;
- provides that information to the agent for answer generation.
This design combines the precision of local retrieval with the flexibility of external search.
The notebook includes a parallel request test that simulates multiple users interacting with the system at the same time.
- latency for each request;
- total wall-clock runtime;
- sequential-versus-parallel runtime comparison;
- estimated parallel speedup;
- successful and failed request counts;
- model-server behavior under concurrent load.
Concurrency testing is especially relevant for vLLM because its scheduling and batching capabilities are designed to improve throughput under multi-request workloads.
The monitoring dashboard provides production-style visibility into both the inference server and the host system.
- online or offline status;
- model name reported by
/v1/models; - API request failures;
- vLLM process detection;
- vLLM process RAM consumption.
- CPU utilization;
- total and used system memory;
- system memory percentage.
- GPU utilization;
- used and total VRAM;
- GPU temperature;
- power consumption;
- per-GPU status when multiple devices are available.
The dashboard helps identify out-of-memory conditions, overloaded resources, stalled processes, unavailable servers, and inefficient configuration choices.
| Layer | Technology |
|---|---|
| Notebook environment | Jupyter Notebook |
| Documentation source | Official vLLM GitHub repository |
| Dataset format | Markdown, JSONL, JSON |
| RAG framework | LlamaIndex |
| Embedding model | BAAI/bge-m3 |
| Inference server | vLLM |
| Language model | Qwen/Qwen2.5-7B-Instruct-AWQ |
| API style | OpenAI-compatible REST API |
| Agent pattern | ReAct-style tool-calling agent |
| External search | DuckDuckGo |
| Monitoring | CPU, RAM, NVIDIA GPU, VRAM, process, and API health metrics |
Before running the notebook, ensure that the environment provides:
- a Linux environment suitable for vLLM;
- Python and Jupyter Notebook or JupyterLab;
- a compatible NVIDIA GPU;
- a working NVIDIA driver and CUDA-compatible runtime;
- enough storage for the model, documentation snapshot, and vector index;
- internet access during documentation download, model acquisition, and web-search steps;
- access to
nvidia-smiwhen GPU monitoring is enabled.
Note
Hardware requirements depend on the selected model, context length, concurrency level, quantization format, and tensor-parallel configuration.
vllm-cleare.ipynb
Execute the notebook's setup cells and confirm that the required libraries are available.
Select the intended vLLM release tag and run the snapshot pipeline. The pipeline should generate the raw documentation directory, corpus.jsonl, and manifest.json.
Run the embedding and LlamaIndex cells to construct the vector index over the documentation corpus.
Run the server command from the notebook or a terminal.
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
--dtype half \
--max-model-len 4096 \
--max-num-seqs 8 \
--gpu-memory-utilization 0.55 \
--enforce-eager \
--tensor-parallel-size 2 \
--attention-backend TRITON_ATTN \
--port 8000curl http://localhost:8000/v1/modelsRun the cells that create:
- the local documentation retrieval tool;
- the web-search tool;
- the tool-calling agent;
- the local vLLM client connection.
Use vLLM-focused questions to test local RAG retrieval and general or time-sensitive questions to test web-search routing.
Execute the multi-user simulation and inspect request latency, total runtime, and parallel speedup.
Run the dashboard cells to observe server health and system resource utilization while requests are processed.
| Option | Purpose | Tuning guidance |
|---|---|---|
--dtype half |
Uses half-precision model execution. | Appropriate when the selected GPU and model support it. |
--max-model-len 4096 |
Sets the maximum context length. | Reduce it when memory usage is too high. |
--max-num-seqs 8 |
Limits concurrently active sequences. | Lower it to reduce memory pressure; increase carefully for throughput testing. |
--gpu-memory-utilization 0.55 |
Defines the fraction of GPU memory vLLM may use. | Increase gradually when sufficient VRAM is available. |
--enforce-eager |
Uses eager execution rather than CUDA graph capture. | Useful for compatibility and debugging, but may affect performance. |
--tensor-parallel-size 2 |
Distributes the model across two GPUs. | Set it to the number of GPUs intended for tensor parallelism. |
--attention-backend TRITON_ATTN |
Selects the Triton attention backend. | Change or remove it when unsupported by the environment. |
--port 8000 |
Sets the HTTP server port. | Change it when port 8000 is already occupied. |
| Question category | Preferred tool | Example |
|---|---|---|
| vLLM configuration | Local documentation | “How does --gpu-memory-utilization work?” |
| vLLM API behavior | Local documentation | “How do I list models from the local server?” |
| vLLM deployment | Local documentation | “How can I configure tensor parallelism?” |
| General knowledge | Web search | “What is the latest NVIDIA data-center GPU?” |
| Current prices or releases | Web search | “What is the current price of a specific GPU?” |
| External framework comparison | Web search | “How does vLLM compare with another serving framework today?” |
For mixed questions, the agent may need both local documentation and external information before producing the final answer.
How do I serve a model with vLLM using an OpenAI-compatible endpoint?
What is the purpose of tensor parallelism in vLLM?
How can I reduce GPU memory usage when serving a model?
Which configuration options affect concurrent sequence processing?
What are the latest developments in open-source LLM serving?
Which current GPUs are suitable for local inference workloads?
What recent changes have been announced for a related AI framework?
After the snapshot stage, the output structure is expected to resemble:
vllm_docs_snapshot/
└── v0.13.0/
├── raw/
│ ├── docs/
│ ├── README.md
│ ├── RELEASE.md
│ └── ...
├── corpus.jsonl
└── manifest.json
Depending on the notebook implementation, additional generated artifacts may include:
- persisted vector-index files;
- cached model or embedding files;
- concurrency-test result tables;
- latency summaries;
- dashboard outputs;
- request logs;
- retrieved source snippets.
Check the following:
nvidia-smiThen verify that:
- the NVIDIA driver is working;
- the selected model is accessible;
- the configured port is available;
- the vLLM process has not exited with an error;
- the API URL matches the configured port;
- the selected attention backend is supported.
Try reducing resource requirements:
--gpu-memory-utilization
--max-model-len
--max-num-seqs
--tensor-parallel-size
Possible fixes:
- reduce
--max-num-seqs; - reduce
--max-model-len; - lower
--gpu-memory-utilization; - use a smaller or more aggressively quantized model;
- close other GPU-intensive processes;
- verify that tensor parallelism matches the available GPU count.
Start the server on another port:
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --port 8001Update the notebook's API base URL accordingly:
http://localhost:8001/v1
Try the following:
- increase
similarity_top_k; - improve chunk sizes and chunk overlap;
- split Markdown by headings and sections;
- remove navigation noise and duplicated content;
- preserve source metadata during chunking;
- add a reranking stage;
- evaluate retrieval separately from answer generation.
Review the tool descriptions and routing prompt. Tool descriptions should clearly separate:
- vLLM-specific documentation questions;
- external or general questions;
- time-sensitive questions;
- mixed questions that may need multiple sources.
Confirm that:
- the host has an NVIDIA GPU;
nvidia-smiis available;- the notebook process has permission to query GPU metrics;
- the required monitoring dependencies are installed;
- the notebook is not running inside an environment without GPU passthrough.
Verify the endpoint:
curl http://localhost:8000/v1/modelsAlso inspect the vLLM terminal output for model-loading, CUDA, quantization, attention-backend, or memory-allocation errors.
- The complete system is notebook-oriented rather than packaged as a standalone application.
- Retrieval quality depends on documentation cleaning and chunking choices.
- Web-search quality depends on the search results available at request time.
- Hardware requirements may be significant even with quantization.
- The example server configuration may require adjustment for different GPU counts and memory capacities.
- A notebook dashboard is useful for experimentation but does not replace a production monitoring stack.
- Agent routing can fail when a question ambiguously mixes documentation and external information.
- persist the vector index for faster startup;
- add hybrid dense and keyword retrieval;
- add a cross-encoder or LLM-based reranker;
- include source citations in every documentation-grounded answer;
- add automated retrieval-quality evaluation;
- add answer-faithfulness and hallucination checks;
- export concurrency results to CSV or JSON;
- report throughput, time to first token, and tokens per second;
- add streaming responses;
- add authentication and request-rate limits;
- package the notebook logic as reusable Python modules;
- provide a command-line interface or web UI;
- containerize the complete pipeline;
- integrate Prometheus and Grafana for production monitoring;
- support automatic comparison between multiple vLLM documentation releases;
- add fallback behavior when the local server or web-search tool is unavailable.
This project is built around the following open-source ecosystems:
- vLLM for high-throughput local language-model serving;
- Qwen for the instruction-tuned language model;
- LlamaIndex for retrieval and agent orchestration;
- BAAI/bge-m3 for semantic embeddings;
- Hugging Face for model distribution and tooling;
- DuckDuckGo for external search integration;
- Jupyter for interactive experimentation and system demonstration.
A reproducible local AI assistant pipeline for documentation-grounded question answering, high-performance inference, concurrent evaluation, and live observability.