FakeGPU

Validate CUDA-facing applications, estimate GPU memory, and simulate distributed GPU workflows without a production GPU cluster.

English · 简体中文 · 繁體中文

Report a bug · Request a feature

Important

FakeGPU is a development, compatibility-testing, and capacity-planning tool. It does not provide numerical or performance parity for arbitrary CUDA kernels. Passthrough, hybrid, and calibration workflows still require a real CUDA stack.

About the project

FakeGPU simulates CUDA-facing environments for development, CI, compatibility checks, and capacity planning. It exposes configurable NVIDIA-like devices to applications while maintained operations run on CPU, records simulated memory and communication, and provides static estimators for workloads that should not be loaded at all.

Physical GPUs are optional for the simulation and analysis paths. A compatible physical CUDA stack is required only for passthrough, hybrid, and calibration runs.

What FakeGPU answers

Question	Recommended path	Physical GPU
Does PyTorch code follow the expected CUDA-facing control flow?	Python FakeCUDA runtime	No
Can an unmodified process load and call CUDA-family shared libraries?	Native interception	No
Will a selected GPU profile fit a workload?	Preflight or static memory estimator	No
How much checkpoint, KV-cache, adapter, or MoE memory should an LLM use?	LLM estimator	No
Which homogeneous or mixed-length online LLM requests fit under a memory budget?	Serving planner	No
Does a target plus draft model for speculative decoding fit in memory?	Serving planner	No
How do resolution, batch size, CFG, VAE tiling, and offload affect diffusion generation memory?	Diffusion estimator	No
Where are the GPU-only entry points and dependencies in a repository?	Repository analyzer	No
What does a distributed training configuration imply for rank-local memory?	Training planner	No
Where do compute, communication, wait, and memory overlap in a trace?	Trace replay	No
How does an estimate compare with an actual CUDA run?	Passthrough or hybrid calibration	Yes

Typical use cases

When this is useful	What FakeGPU provides	Start with
Choosing a GPU before renting capacity or starting a long job	Profile-aware checkpoint, KV-cache, activation, optimizer, and workspace estimates	`estimate-llm`, `preflight`
Sizing chat, RAG, completion, summarization, or draft-assisted traffic before deployment	Per-request prompt/generation lengths, continuous-batch admission, chunked-prefill transients, shared-prefix KV groups, speculative target/draft memory, and explicit memory headroom	`plan-serving`, `validate`
Developing CUDA-oriented PyTorch code on a laptop or CPU-only CI runner	CUDA-visible control-flow checks while maintained tensor operations execute on CPU	`fakegpu.init(...)`, `demo`, `validate`
Comparing full fine-tuning, LoRA, QLoRA, checkpointing, offload, or sharding plans	Phase-aware and rank-local memory estimates before allocating a cluster	`plan-training`, Python memory estimator
Comparing UNet and diffusion-transformer generation shapes and memory optimizations	Architecture-specific text encoder, denoising, and VAE-decode estimates from fixed profiles or a local pipeline	`estimate-diffusion`, `validate`
Reviewing an unfamiliar GPU repository or native extension	GPU entry-point, dependency, kernel, and unsupported-API inventory	`analyze-repo`, `analyze-kernel`, `capabilities`
Designing or debugging a distributed workflow	Collective routing, link contention, rank waits, memory timelines, and TCP payload validation	`simulate-topology`, `replay-trace`, `bandwidth`
Observing simulated devices and processes in CI or a local lab	Bounded Prometheus metrics, exporter health, and short in-memory history	`nvidia-smi`, `metrics`
Turning a small real-GPU trial into evidence for repeated runs	Prediction-versus-observation reports and signature-scoped calibration data	`calibrate`, `preflight --memory-calibration`

Real-GPU memory-estimation evidence

Note

In the recorded validation envelopes below, the stack-calibrated static estimator stayed within 0.08% on 26 controlled GPU observations and within 1.921% across ten Qwen full/LoRA SFT cases.

Absolute percentage error is |predicted - observed| / observed × 100%. “Agreement” below is its complement, 100% - error, shown only as a more intuitive reading of the same measurement.

Validated envelope	Real-GPU reference	Evidence	Absolute percentage error	Agreement
Controlled ATen MLP and Transformer grid with backend-resident calibration	RTX 3090 Ti and RTX PRO 5000; PyTorch/CUDA 2.12/13.0 and 2.9/12.8	13 workloads, 26 observations	0.08% maximum	≥99.92%
Qwen3-8B BF16 SDPA inference	RTX PRO 5000; PyTorch 2.9.1/CUDA 12.8	Model load and inference peak	0.0129% load; 0.0672% peak	99.9871%; 99.9328%
Qwen 0.8B/2B full and LoRA SFT	RTX PRO 5000; PyTorch 2.8/CUDA 12.8	10 training cases	0.102%–1.921%	98.079%–99.898%
Qwen 0.8B/2B native NF4 QLoRA	RTX PRO 5000; PyTorch 2.8/CUDA 12.8	10 quantized training cases	0.628%–1.732%	98.268%–99.372%

How to read these numbers:

The Qwen rows use torch.cuda.max_memory_allocated() as the reference. CUDA context memory and reserved-but-unused allocator memory are excluded.
The controlled ATen row adds one backend-resident measurement from the exact GPU and software stack; that value must not be reused on another stack.
Ranges show the minimum and maximum case error, not an average. Maximum underestimation is the important failure mode when evaluating OOM risk.
A 99.x% agreement value is not spare capacity. Capacity decisions should still apply a workload-specific safety margin or factor.

These are fixed-workload measurements, not a universal accuracy claim. Different models, shapes, attention backends, quantization kernels, allocators, PyTorch/CUDA versions, or GPUs require a matching calibration. The links above point to the immutable validation snapshot containing the full configurations and measured byte counts. A machine-readable evidence summary is checked against the README in CI.

On a CUDA host, regenerate the maintained controlled comparison with:

python3 scripts/validation/static_memory_validation.py \
  --output build/static-memory-validation.json \
  --markdown build/static-memory-validation.md \
  --max-underestimate-percent 5

On a CPU-only host, add --static-only; this checks the estimation path but does not produce a real-GPU accuracy measurement. For a plan-serving report, normalize repeated real-GPU phase peaks before comparison:

python3 -m fakegpu calibrate observe-serving \
  build/speculative-serving-plan.json \
  --prefill-peak-bytes 24800000000,24900000000,25000000000 \
  --prefill-peak-bytes 24950000000,25050000000 \
  --decode-peak-bytes 25200000000,25300000000,25250000000 \
  --decode-peak-bytes 25350000000,25400000000 \
  --source torch.cuda.max_memory_reserved \
  --strict \
  --json build/serving-observation.json

The command retains every sample and uses the maximum for each phase. Five samples per phase are required by default; --strict returns status 1 when that minimum is not met. ready_for_comparison records sample sufficiency, not a GPU-validated claim. The plan must contain --target-profile so the observation keeps the same GPU identity.

Automate the repeated executions with a framework-specific runner:

python3 -m fakegpu calibrate collect-serving \
  build/speculative-serving-plan.json \
  --repetitions 5 \
  --timeout-seconds 900 \
  --strict \
  --json build/serving-observation.json \
  -- python3 benchmark_serving.py

For a homogeneous, non-speculative Transformers workload, FakeGPU includes a runner, so no separate benchmark file is needed. The plan must describe the same execution semantics and use a dynamic KV cache:

python3 -m fakegpu plan-serving \
  --model-dir /mnt/z/models/Qwen/Qwen3-0.6B \
  --active-sequences 4 \
  --max-batch-size 8 \
  --prompt-tokens 2048 \
  --generated-tokens 128 \
  --dtype bfloat16 \
  --attention-implementation sdpa \
  --kv-cache-strategy dynamic \
  --target-profile a100 \
  --json build/transformers-serving-plan.json

python3 -m fakegpu calibrate collect-serving \
  build/transformers-serving-plan.json \
  --repetitions 5 \
  --timeout-seconds 900 \
  --strict \
  --json build/transformers-serving-observation.json \
  -- python3 -m fakegpu calibrate sample-transformers

The model path keeps the Hugging Face owner/repository hierarchy. On hosts without /mnt/z, change only the models root and retain Qwen/Qwen3-0.6B.

sample-transformers rechecks the local model configuration and Safetensors headers against the plan, then measures full prefill and token-at-a-time decode with the returned KV cache. Its default metric is torch.cuda.max_memory_reserved; use --metric allocated for tensor-only allocator peaks. It deliberately rejects mixed request sets, speculative decoding, paged or quantized caches, shared-prefix or chunked-prefill plans, quantized checkpoints, and adapters until those execution paths have matching measurement implementations. PyTorch and Transformers remain optional and must already be installed in the CUDA benchmark environment.

vLLM or custom instrumentation that already has trustworthy phase peaks can use the protocol builder without recreating CUDA metadata:

python3 -m fakegpu calibrate emit-serving-sample \
  build/vllm-serving-plan.json \
  --prefill-peak-bytes 24900000000 \
  --decode-peak-bytes 25300000000 \
  --metric nvml.process_family_peak_bytes \
  --framework vllm \
  --framework-version 0.10.2

The same adapter is available as fakegpu.build_cuda_serving_sample(...) for use inside a Python benchmark. vLLM reserves engine and KV-cache memory from its configured GPU budget, while its cache-usage metric is a fraction of blocks rather than a process-memory byte peak. FakeGPU therefore requires the vLLM benchmark to provide real phase bytes instead of labeling cache usage as prefill or decode memory.

The runner is invoked in a new process for every repetition and must emit one fakegpu.serving_peak_sample.v1 JSON object. It may print the object as its last line or prefix a one-line object with FAKEGPU_SERVING_SAMPLE= when the framework also writes logs to stdout:

{
  "schema_version": "fakegpu.serving_peak_sample.v1",
  "workload_signature": "<FAKEGPU_SERVING_WORKLOAD_SIGNATURE>",
  "run_index": 1,
  "metric": "torch.cuda.max_memory_reserved",
  "phases": {
    "prefill": {"peak_bytes": 24900000000},
    "decode": {"peak_bytes": 25300000000}
  },
  "environment": {
    "backend": "cuda",
    "simulated": false,
    "gpu_name": "NVIDIA A100-SXM4-80GB",
    "gpu_uuid": "GPU-...",
    "compute_capability": [8, 0],
    "total_memory_bytes": 85899345920,
    "software": {
      "framework": "transformers",
      "framework_version": "...",
      "cuda_version": "...",
      "torch_version": "..."
    }
  }
}

FakeGPU supplies the plan path, workload signature, target profile, compute capability, run index, and run count through FAKEGPU_SERVING_* environment variables. The same protocol can wrap Transformers, vLLM, or a custom server without adding them as FakeGPU dependencies. The built-in Transformers runner and vLLM/custom protocol builder populate this contract automatically. Collection fails if the runner reports simulated CUDA, a target capacity outside a 2% profile tolerance, a different compute capability, inconsistent software metadata, a timeout, or a nonzero exit. The report retains the actual/profile capacity difference. Raw command arguments are not stored; the report retains only the executable name and a command fingerprint. simulated: false is still runner-supplied metadata rather than independent hardware attestation, so retained benchmark logs and calibration gates remain necessary for a GPU-validated claim.

To compare compatible prediction and observation reports for any workload:

python3 -m fakegpu calibrate compare \
  build/prediction.json \
  build/observation.json \
  --json build/calibration-comparison.json

python3 -m fakegpu calibrate verify \
  build/calibration-comparison.json \
  --max-underestimate-percent 5 \
  --max-absolute-percentage-error-percent 5 \
  --min-interval-coverage-percent 90 \
  --capacity-bytes 25769803776 \
  --json build/calibration-verification.json

The comparison reports per-phase signed and absolute error, interval coverage, and a recommended memory safety margin and factor. calibrate verify exits with status 1 when a configured gate fails. It checks maximum underestimation, median/p95/maximum absolute percentage error, prediction-interval coverage, false-safe fit decisions at the supplied capacity, and workload-dimension consistency. Apply results only to the same workload signature, shapes, dtype, software stack, and GPU profile.

Research workload reliability report

FakeGPU reports reliability per workload and environment signature. A GPU-validated result applies only to the recorded model revision, shapes, dtype, attention backend, allocator, software stack, and GPU. CPU-validated means that maintained execution or analysis behavior passed without a physical GPU. Modeled identifies analytical coverage without matching real-GPU evidence, while Planned is not yet a supported accuracy claim. Serving plans expose a path-independent workload_signature so calibration evidence cannot silently cross serving configurations. Use calibrate observe-serving to retain repeated phase samples before comparison and reliability gates. Use calibrate collect-serving with the built-in sample-transformers runner, or with vLLM/custom measurements wrapped by build_cuda_serving_sample, when repeated real-CUDA evidence is needed.

Current repository verification

This repository state was verified on 2026-08-05 with every suite from scripts/test.sh and both declarative validation manifests on macOS 26.5 arm64, Python 3.11.9, and PyTorch 2.9.1 CPU. The Transformers sampling adapter was also exercised on an RTX PRO 5000 Blackwell with PyTorch 2.11.0+cu128 and Transformers 5.14.1:

Validation layer	Maintained check	Result
Python runtime, estimators, CLIs, schemas, and README contracts	Complete `pytest` suite	223 passed
Declarative validation matrices	7 smoke executions plus 39 research architecture, cache, generic/vLLM serving, training, calibration, and diffusion executions	46 passed
Native interception	Build, library boundaries, exports, preload, memory types, coordinator, and unsupported-API policy	Passed
FakeGPU-SMI diagnostics	Bounded state, topology/NVLink/MIG views, NVML peer/MIG queries, health fields, and event reporting	Passed
Monitoring exporter	Prometheus/JSON snapshots, bounded history/cardinality, malformed-state degradation, and HTTP endpoints	Passed
Native capability inventory	5 groups, 26 explicit APIs, 24 policy-enforced APIs	Passed
GPU profile catalog	82 profiles across 15 compute capabilities	Passed
CPU numerical simulation	GEMM, cuBLASLt, batched GEMM, BLAS1/2, and FP16: 8 maintained test groups	Passed
CUDA-enabled PyTorch native matmul	Requires a CUDA build of PyTorch	Not run on this CPU-only host
Transformers real-CUDA serving sampler	Qwen/Qwen3-0.6B, BF16 SDPA, 2 × 128-token prefill + 8 generated tokens, 3 isolated samples per phase	Passed; `ready_for_comparison`

GitHub CI separately runs the Python suite on Python 3.10–3.12 and the native smoke and CPU simulation suites on Linux and macOS. Published accuracy results above come from their linked immutable validation snapshots. The new Transformers sampler result validates repeated execution, CUDA identity, and report collection; without a comparison gate, it is not an accuracy claim.

Maintained research workload matrix

Workload class	Covered variations	Evidence	Status
Offline decoder inference	Qwen3-8B, BF16, SDPA, model load, prefill, and decode peak	RTX PRO 5000 prediction-versus-observation data	`GPU-validated`
Full and adapter SFT	Qwen 0.8B/2B full fine-tuning and LoRA	Ten RTX PRO 5000 training cases	`GPU-validated`
Quantized adapter SFT	Qwen 0.8B/2B native NF4 QLoRA	Ten RTX PRO 5000 training cases	`GPU-validated`
General decoder analysis	Dense and MoE metadata; MHA, GQA, MQA, and compressed-latent MLA; adapters, quantized checkpoints, eager/SDPA attention, KV cache, and expert-parallel traffic	Formula, fixture, CLI regression tests, and the four-case architecture matrix	`CPU-validated` + `Modeled`
Distributed training plans	DeepSpeed, Accelerate, FSDP/FSDP2, sharding, checkpointing, and CPU/NVMe offload	Configuration, byte-accounting, topology, and trace tests	`CPU-validated` + `Modeled`
KV-cache allocation	Dynamic growth, static reservation, 2/4/8-bit quantized storage, paged block rounding, and sliding-window limits	Formula, API, `--kv-cache-strategy` CLI, and `tests/data/research_validation.yaml` matrix tests	`CPU-validated` + `Modeled`
Online serving capacity	Homogeneous and mixed-length request sets under continuous batching, ordered admission, chunked prefill, grouped prefix caching, paged KV blocks, generic or vLLM runtime budgets, profile or explicit KV capacity, and admission headroom	Checkpoint headers, architecture and block-allocation formulas, `plan-serving --runtime vllm`, `--vllm-kv-cache-memory-bytes`, `--vllm-non-kv-cache-memory-bytes`, unit tests, and generic/vLLM chat/RAG cases in `tests/data/research_validation.yaml`	`CPU-validated` + `Modeled`
Diffusion image generation	Stable Diffusion v1.5, SDXL Base, and PixArt-Sigma; local UNets, cross-attention DiTs, and SD3/Flux-style joint-attention transformers; CFG, offload, attention/VAE slicing, and VAE tiling	Fixed-revision and local component headers, architecture configs, phase formulas, CLI/unit tests, and a three-case local-architecture matrix in `tests/data/research_validation.yaml`	`CPU-validated` + `Modeled`
speculative decoding	Independent autoregressive draft-model weights and KV cache, target lookahead KV, batched verification transients, optional fixed acceptance assumptions, and homogeneous/mixed request admission	Checkpoint headers, architecture formulas, `plan-serving --draft-model-dir`, `--speculative-tokens`, `--speculative-acceptance-rate`, and unit tests; no maintained real-GPU observation yet	`CPU-validated` + `Modeled`
Multi-GPU LLM execution	TP, PP, CP, EP, MoE imbalance, and combined FSDP/ZeRO execution	Analytical topology and coordinator coverage only	`Modeled`
Diffusion training	Optimizer, gradient, activation, EMA, and parameter-efficient tuning	No dedicated estimator or real-GPU evidence yet	`Planned`

Architecture-aware estimate comparisons

The previous “reproducible modeled effects” label only meant that fixed inputs produce the same formula result; it did not mean the estimate matched a real GPU observation. This section is now named “architecture-aware estimate comparisons” to make that distinction explicit. The examples are generated by checked-in formulas and verified by README contract tests. GiB values are binary units, and the results do not replace same-configuration GPU calibration.

Research scenario	Controlled comparison	Modeled effect	Validation
LLM inference KV cache	32-layer GQA, 8 KV heads, head dim 128, batch 1, BF16; context 4K → 32K	0.50 → 4.00 GiB (8× cache)	Exact byte formula
Attention cache architecture	32 layers, 8 active sequences, 4K BF16 paged cache; standard head dim 128; MHA 32 KV heads / GQA 8 / MQA 1 / MLA latent 512 + RoPE 64	16.00 / 4.00 / 0.50 / 1.125 GiB	Architecture-specific cache-layout formulas and four-case manifest
Online serving prefix cache	Same decoder, 8 active sequences, 4K prompt + 256 generated tokens, paged BF16 KV; independent caches → one shared 1K prefix	4.25 → 3.38 GiB (−20.6%)	Shared/private paged-block formula
Mixed chat/completion serving	Same decoder; prompts 4K/8K/2K and generation 256/512/1024; independent KV → two chat requests sharing a 1K system prefix	1.97 → 1.84 GiB decode KV (−6.3%)	Heterogeneous request-set formula and chat/RAG matrix
Speculative target calls	5 draft tokens per verification step and an independent 70% per-token acceptance assumption	1.00 → 2.94 expected output tokens per target step; not a throughput claim	Geometric acceptance formula plus dual-model memory tests
LLM training	8B BF16 parameters, AdamW, 4 GPUs, checkpointed activations; replicated → full shard	104.31 → 26.54 GiB per rank (−74.6%)	Training-plan phase model
Stable Diffusion v1.5 generation	512², batch 1, FP16, CFG; all weights resident → model offload	2.30 → 1.65 GiB (−28.1%)	Component + phase model
SDXL batch generation	1024², batch 4, FP16, CFG; regular VAE decode → VAE slicing	11.49 → 7.74 GiB (−32.7%)	Sequential VAE batch-shape model
SDXL high-resolution generation	2048², batch 1, FP16, CFG; full-frame VAE → 512² VAE tiles	11.49 → 7.27 GiB (−36.7%)	Sequential VAE tile-shape model
PixArt-Sigma DiT generation	1024², batch 1, FP16, CFG, model offload; eager → SDPA denoise phase	2.32 → 1.28 GiB (−44.7%)	Patch-token, cross-attention, and phase model

The diffusion profiles use component parameter counts from fixed revisions of Stable Diffusion v1.5 and SDXL Base 1.0, and PixArt-Sigma XL 2. Runtime context, allocator fragmentation, backend-private workspaces, and transfer overlap remain outside these numbers. Architecture-aware reports remain Modeled with accuracy.status set to uncalibrated until a matching observation exists. Local architecture inspection is exercised through estimate-diffusion --model-dir.

The cache formulas follow the workload shapes exposed by Transformers cache strategies. The serving planner models homogeneous shapes and ordered heterogeneous active request sets. Its speculative mode follows the independent draft/target flow described by the original speculative-decoding paper and Transformers assisted decoding: a smaller model proposes several tokens and the target verifies them in one forward pass. Request arrival distributions, cache eviction, preemption, adaptive draft length, KV rollback workspaces, and end-to-end throughput or latency remain outside the supported estimate. The workload terminology follows vLLM serving. Binary CUDA extensions and arbitrary kernels remain outside CPU FakeCUDA execution; those workloads require analysis plus a passthrough or hybrid real-GPU observation.

New or refreshed public validation rows should record:

at least five isolated observations and the maximum observed peak;
predicted and observed bytes for every reported phase;
maximum underprediction as the primary OOM-risk metric, with a publication target of at most 5% for a GPU-validated row;
median, 95th-percentile, and maximum absolute percentage error;
prediction-interval coverage and the count of false-safe decisions, where FakeGPU predicted a fit but the real workload reached OOM; and
the model revision, command, shapes, dtype, backend, allocator settings, GPU, driver, CUDA, PyTorch, and framework versions.

Use calibrate verify to apply these limits to machine-readable comparison reports before publishing a result.

Rows that miss the target remain Modeled or are marked experimental instead of being presented as validated. Agreement percentages remain secondary to maximum underprediction and false-safe OOM decisions.

How it works

Path	What the application sees	What actually runs
Python FakeCUDA	CUDA devices, CUDA-looking tensors, memory APIs, and common training flows	Maintained PyTorch operations execute on CPU through `FakeCudaTensor`
Native interception	`libcuda`, `libcudart`, `libcublas`, `libnvidia-ml`, and `libnccl` entry points	Selected operations use host memory or CPU math; unsupported behavior is classified and reported
Analysis and reporting	Memory, FLOP, roofline, topology, and communication reports	ATen graphs, safetensors metadata, runtime traces, calibration data, and coordinator events are analyzed

Built with

Python 3.10+ for the runtime, estimators, CLI, and reports
C++17 and CMake for native interception libraries and the coordinator
PyTorch for CPU-backed FakeCUDA execution and ATen graph capture
YAML and JSON schemas for GPU profiles, validation manifests, and reports

(back to top)

Getting started

Prerequisites

Linux or macOS
Python 3.10 or newer
CMake 3.14 or newer
A C++17 compiler
PyTorch for the Python FakeCUDA runtime

On Debian or Ubuntu, install build-essential. On macOS, install the Xcode Command Line Tools.

Installation

Clone the repository:

git clone https://github.com/FanBB2333/FakeGPU.git
cd FakeGPU

Build the native libraries and install the package:

scripts/build.sh
FAKEGPU_BUILD_DIR="$PWD/build" python3 -m pip install .

For development directly from a checkout:

python3 -m pip install pytest PyYAML jsonschema ruff
export PYTHONPATH="$PWD"

Verify the installation

python3 -m fakegpu doctor --list-profiles
python3 -m fakegpu demo --profile l4

doctor checks the profile catalog, native libraries, and PyTorch environment. demo performs a small forward, backward, and optimizer step on CPU while the program sees a CUDA device.

(back to top)

Usage

Run PyTorch code on FakeCUDA

Initialize FakeGPU before importing PyTorch:

import fakegpu

fakegpu.init(runtime="fakecuda", profile="a100", device_count=2)

import torch

device = torch.device("cuda:0")
model = torch.nn.Linear(8, 4).to(device)
x = torch.randn(2, 8, device=device)
loss = model(x).square().mean()
loss.backward()

print(torch.cuda.device_count())      # 2
print(torch.cuda.get_device_name(0))  # NVIDIA A100
print(loss.item())

Maintained operations execute on CPU while device placement, memory limits, training control flow, and error handling use the simulated CUDA surface.

Inspect FakeGPU devices and processes

State publishing is opt-in. Set a state directory before the workload calls fakegpu.init(...) or starts through the native launcher; build/ is ignored by Git:

export FAKEGPU_SMI_STATE_DIR=build/smi

# Python FakeCUDA workload.
python3 your_script.py

# Unmodified native CUDA/NVML workload.
python3 -m fakegpu --build-dir build ./your_native_workload

From another terminal, inspect the running workload:

# Compact device and process table; add "-l 1" to refresh every second.
python3 -m fakegpu nvidia-smi --state-dir build/smi

# Device inventory and detailed runtime/profile/allocator report.
python3 -m fakegpu nvidia-smi --state-dir build/smi -L
python3 -m fakegpu nvidia-smi --state-dir build/smi -q

# NVIDIA-style modeled topology and NVLink views.
python3 -m fakegpu nvidia-smi topo -m --state-dir build/smi
python3 -m fakegpu nvidia-smi nvlink -s --state-dir build/smi

# Modeled faults, compatibility warnings, publisher failures, and stale state.
python3 -m fakegpu nvidia-smi events --state-dir build/smi

# Modeled MIG GPU and compute instances.
python3 -m fakegpu nvidia-smi mig -lgi --state-dir build/smi
python3 -m fakegpu nvidia-smi mig -lci --state-dir build/smi

# Script-friendly GPU and process queries.
python3 -m fakegpu nvidia-smi --state-dir build/smi \
  --query-gpu=index,name,uuid,pci.bus_id,profile.id,compute_cap,memory.total,memory.used,memory.free,allocator.model,native.kernel_launches,native.gemm_calls,native.io_bytes \
  --format=csv
python3 -m fakegpu nvidia-smi --state-dir build/smi \
  --query-compute-apps=pid,process_name,gpu_uuid,used_gpu_memory,peak_gpu_memory,stage,status \
  --format=csv,noheader,nounits
python3 -m fakegpu nvidia-smi --state-dir build/smi \
  --query-runtime=pid,fakegpu.version,runtime.backend,runtime.mode,policy.oom,tracking.dispatch,catalog.profiles,catalog.native_apis,dispatch.calls,publisher.failed_writes,status \
  --format=json

The detailed report includes the FakeGPU version, runtime backend and policies, Python/PyTorch/CUDA versions, state freshness, profile catalog and native API coverage, synthetic device identity, compute properties, memory categories, allocator activity, dispatch tracking, and per-process peaks. Use -i with an index, UUID, PCI bus ID, or profile ID to select devices; use --json for the complete normalized inventory. State schema v2 is emitted, while v1 files remain readable. --query-runtime exposes the same runtime, policy, software, tracking, catalog, dispatch, and publisher-health details as script-friendly CSV or JSON. Use --help-query-runtime to list every supported field.

Native interception additionally publishes allocation lifetime, transfer volume, kernel launches, GEMM calls/FLOP, compatibility events, and unsupported API counts. State is refreshed while the process is running and marked exited when the process shuts down. FAKEGPU_SMI_DETAIL_LIMIT bounds retained detail entries and FAKEGPU_SMI_MAX_STATE_BYTES limits each state file; -q reports publisher write counts, failures, latency, and serialized size.

NVLink modeling is disabled by default. Set FAKEGPU_NVLINK_GROUPS="0,1;2,3" before starting the workload to create full-mesh peer relationships inside each semicolon-separated group. The optional FAKEGPU_NVLINK_BANDWIDTH_GBPS value defaults to 900. The same model drives state files, topo -m, nvlink -s, the topology.*/nvlink.* query fields, and native NVML link-state, capability, remote-device-type, and remote-PCI queries. Invalid groups leave every link inactive and publish a configuration error.

MIG modeling is disabled by default. Configure instances before starting the workload with FAKEGPU_MIG_LAYOUT="DEVICE:PROFILE:MEMORY_MIB[:COUNT];...", for example FAKEGPU_MIG_LAYOUT="0:1g.10gb:10240:2;1:2g.20gb:20480". Each modeled GPU instance owns one compute instance. The layout drives state files, -L, -q, mig -lgi, mig -lci, the mig.* query fields, and native NVML MIG mode, handle, UUID, parent, instance-ID, and memory-capacity queries. Layouts that exceed parent memory, eight slices, or eight instances per device are rejected without creating partial instances. Per-instance runtime allocation attribution is not implemented, so state files mark used/free instance memory unobserved.

Fault injection is also disabled by default. Set entries using FAKEGPU_FAULT_EVENTS="DEVICE:CODE:SEVERITY[:COUNT];...", for example FAKEGPU_FAULT_EVENTS="0:XID_79:critical;1:NVLINK_CRC:error:3". Codes are labels; supported severities are info, warning, error, and critical. The events view combines configured fault events with unsupported native API calls, publisher failures, stale state, and model-configuration errors. health.* query fields expose each device's modeled status, maximum severity, event counts, and the fact that hardware health is unobserved. Invalid input activates no faults and is reported as a configuration error.

UUIDs and PCI bus IDs are stable simulated identifiers. Temperature, fan speed, live power draw, and hardware GPU utilization remain N/A because the CPU-backed runtime cannot observe them; profile power and clock values are shown separately as static specifications. Topology labels and configured bandwidth, fault codes, and health status are modeled inputs, not hardware measurements or observed ECC/Xid data.

Export monitoring metrics

The normalized FakeGPU-SMI state can be exported without adding a monitoring dependency. A one-shot command emits Prometheus text by default or a normalized JSON snapshot:

python3 -m fakegpu metrics --state-dir build/smi
python3 -m fakegpu metrics --state-dir build/smi --json

For local collection, start the bounded in-memory exporter:

python3 -m fakegpu metrics --state-dir build/smi --serve \
  --host 127.0.0.1 --port 9400 --interval 1 \
  --history-size 300 --max-process-series 128

curl http://127.0.0.1:9400/metrics
curl http://127.0.0.1:9400/healthz
curl http://127.0.0.1:9400/api/v1/history

/metrics exposes the latest device, process, runtime, MIG, topology, health, and publisher values; /healthz reports source and scrape status; /api/v1/history returns normalized recent samples. Process series retain the highest-memory processes first and are limited to 128 by default and 256 at most; --max-process-series 0 disables them. History keeps 300 samples by default and at most 1,440. Both limits are enforced in memory, and no monitoring history is written to disk or tracked by Git. Use Prometheus or another scraper for durable retention.

The server listens only on 127.0.0.1 by default and has no authentication. Place it behind an authenticated proxy or network policy before binding a non-loopback address. Exported values preserve the same modeled-versus-observed semantics as FakeGPU-SMI; they do not turn unavailable hardware telemetry into measurements.

Intercept native CUDA libraries

Build the native libraries, then let the module launcher prepare LD_PRELOAD or DYLD_INSERT_LIBRARIES for an unmodified command:

python3 -m fakegpu --build-dir build --profile a100 \
  python3 your_script.py

python3 -m fakegpu --build-dir build --devices "a100:2,h100:2" \
  python3 your_script.py

python3 -m fakegpu --build-dir build \
  --mode simulate \
  --unsupported-api error \
  python3 your_script.py

Unsupported native calls can be recorded, warned about, or returned as cudaErrorNotSupported or CUDA_ERROR_NOT_SUPPORTED.

Check memory before a run

Run a command to a target stage and write reports under an ignored build directory:

python3 -m fakegpu preflight \
  --runtime fakecuda \
  --profile a100 \
  --stage forward \
  --report-dir build/preflight \
  --strict \
  -- python3 train.py

Preflight tracks the memory visible on the executed path and classifies whether the selected profile fits the workload.

Plan online LLM serving capacity

Estimate a homogeneous active-request pool without loading checkpoint tensors:

python3 -m fakegpu plan-serving \
  --model-dir /models/example \
  --active-sequences 16 \
  --max-batch-size 64 \
  --prompt-tokens 4096 \
  --generated-tokens 256 \
  --prefill-chunk-tokens 512 \
  --shared-prefix-tokens 1024 \
  --kv-cache-strategy paged \
  --kv-cache-block-tokens 16 \
  --target-profile a100 \
  --memory-utilization 0.9 \
  --json build/serving-plan.json

The report separates model weights, prefill and decode transients, shared and private KV segments, runtime/scheduler overhead, and usable device headroom. --device-memory-gib can replace --target-profile for a custom capacity. Prefix sharing is supported for dynamic and paged caches; paged shared and private segments are rounded independently. Capacity search never exceeds --max-batch-size.

Match vLLM runtime reservation

vLLM preallocates a paged KV pool instead of growing only with the active requests. Select its memory policy when the deployment target is vLLM:

python3 -m fakegpu plan-serving \
  --model-dir /mnt/z/models/Qwen/Qwen3-0.6B \
  --active-sequences 16 \
  --max-batch-size 64 \
  --prompt-tokens 4096 \
  --generated-tokens 256 \
  --runtime vllm \
  --kv-cache-strategy paged \
  --kv-cache-block-tokens 16 \
  --target-profile rtx-pro-5000-blackwell \
  --memory-utilization 0.92 \
  --json build/vllm-serving-plan.json

Automatic sizing follows the vLLM V1 policy: requested model-executor memory minus profiled non-KV memory, rounded down to whole cache blocks. The report keeps logical request KV demand separate from the reserved pool and exposes num_gpu_blocks, cache token capacity, block-rounding tail, initialization fit, and concurrency at max_model_len. If a matching vLLM startup profile is available, pass its weights + peak activations + non-torch + CUDA-graph total with --vllm-non-kv-cache-memory-bytes; this replaces FakeGPU's modeled non-KV value. --vllm-kv-cache-memory-bytes provides an exact per-GPU cache override and, like vLLM, ignores --memory-utilization for cache sizing.

When --memory-utilization is omitted, vLLM mode uses 0.92, matching the current vLLM CacheConfig reference. Specify the value explicitly for a version-locked report. The budget formula tracks vLLM's documented GPU-worker profiling path, but initial free memory, other processes, tensor/pipeline parallelism, hybrid cache groups, allocator fragmentation, and version-specific kernels still need same-stack observation. The research manifest checks both automatic and explicit-cache modes. Speculative decoding currently remains available only with the generic runtime model; vLLM-specific speculative schedulers and cache managers are not represented yet.

Add an independent autoregressive draft checkpoint to estimate speculative decoding memory:

python3 -m fakegpu plan-serving \
  --model-dir /models/target \
  --draft-model-dir /models/draft \
  --active-sequences 8 \
  --max-batch-size 32 \
  --prompt-tokens 4096 \
  --generated-tokens 256 \
  --speculative-tokens 5 \
  --speculative-acceptance-rate 0.7 \
  --kv-cache-strategy paged \
  --target-profile a100 \
  --json build/speculative-serving-plan.json

The report keeps target and draft weights plus both KV caches resident, adds conservative lookahead slots, and compares target verification with draft proposal transients. The acceptance assumption reports expected output tokens per target call; it does not change the memory peak or claim an end-to-end speedup. Equal vocabulary sizes are required, while tokenizer identity remains unverified because checkpoint headers cannot prove token-ID equivalence.

Every homogeneous or mixed-request serving report also emits a path-independent workload_signature. It covers target/draft architecture and weight storage, request shapes, KV/prefill settings, and speculative configuration while excluding absolute model paths and device capacity. Put the same field in a measurement report before using calibrate verify; the calibration gate compares the signature and checks target.profile as a separate dimension, so evidence from a different proposal length or GPU profile is rejected.

For chat, RAG, completion, or summarization traffic with different lengths, provide an ordered request manifest:

{
  "schema_version": "fakegpu.serving_requests.v1",
  "requests": [
    {
      "id": "chat-a",
      "prompt_tokens": 4096,
      "generated_tokens": 256,
      "prefix_group": "system-prompt",
      "shared_prefix_tokens": 1024
    },
    {
      "id": "chat-b",
      "prompt_tokens": 8192,
      "generated_tokens": 512,
      "prefix_group": "system-prompt",
      "shared_prefix_tokens": 1024
    },
    {
      "id": "completion",
      "prompt_tokens": 2048,
      "generated_tokens": 1024
    }
  ]
}

python3 -m fakegpu plan-serving \
  --model-dir /models/example \
  --requests serving-requests.json \
  --max-batch-size 64 \
  --prefill-chunk-tokens 512 \
  --prefill-concurrency 2 \
  --kv-cache-strategy paged \
  --target-profile a100 \
  --memory-utilization 0.9 \
  --json build/mixed-serving-plan.json

Manifest mode calculates every request independently, stores each named prefix group once, and admits the longest fitting prefix of the file without reordering it. The report includes admitted and rejected request IDs, per-request transients, component-wise concurrent-prefill bounds, grouped KV segments, and the requested capacity headroom. Group members must use the same positive shared_prefix_tokens value; shared prefixes require dynamic or paged KV storage. Named groups are treated as resident cache hits; cache population, lookup probability, and eviction behavior are not predicted.

Arrival timing, cache eviction, preemption/reordering, tensor parallelism, adaptive speculative scheduling, KV rollback workspaces, throughput, and latency are listed as unmodeled. Until a matching online-serving observation is supplied, validation_status remains Modeled and accuracy.status remains uncalibrated.

Analyze a repository or model

# Find GPU entry points, dependencies, native sources, and compatibility risks.
python3 -m fakegpu analyze-repo .

# Estimate checkpoint, KV-cache, transient, adapter, and MoE memory.
python3 -m fakegpu estimate-llm \
  --model-dir /models/example \
  --batch-size 1 \
  --prompt-tokens 128 \
  --generated-tokens 32 \
  --dtype bfloat16 \
  --kv-cache-strategy paged \
  --kv-cache-block-tokens 16 \
  --target-profile a100 \
  --json build/llm-estimate.json

# Compare diffusion generation phases and memory optimizations.
python3 -m fakegpu estimate-diffusion --list-profiles
python3 -m fakegpu estimate-diffusion \
  --model-profile stable-diffusion-xl-base-1.0 \
  --height 1024 --width 1024 --batch-size 4 \
  --attention-backend sdpa --vae-slicing \
  --offload model --target-profile a100 \
  --json build/diffusion-estimate.json

# Inspect a local Diffusers pipeline without loading tensor payloads.
python3 -m fakegpu estimate-diffusion \
  --model-dir /models/pixart-or-flux \
  --height 1024 --width 1024 --text-tokens 300 \
  --dtype bfloat16 --attention-backend sdpa \
  --offload model --target-profile a100 \
  --json build/local-diffusion-estimate.json

# Audit source and built native exports against the capability manifest.
python3 -m fakegpu capabilities \
  --source-root . \
  --build-dir build \
  --strict

The LLM estimator reads safetensors headers without materializing checkpoint weights. Choose dynamic, static, quantized, or paged with --kv-cache-strategy; the JSON report separates logical storage, quantization savings, static reservation, paged-block overhead, and optional sliding-window limits. Quantized cache accounting retains 128 recent tokens at the compute dtype by default; change it with --kv-cache-residual-tokens. MHA, GQA, and MQA use separate key/value storage. When the model config contains kv_lora_rank, qk_nope_head_dim, qk_rope_head_dim, and v_head_dim, the estimator identifies MLA and uses its compressed KV latent plus decoupled RoPE component for cache bytes. MLA backend workspaces and projection absorption remain runtime-specific and require calibration.

The diffusion estimator separates text encoding, repeated denoising, and VAE decode phases. With --model-dir, it reads only model_index.json, component config.json files, and selected safetensors headers. It does not import remote custom code or read tensor payloads. Checkpoint storage and runtime weight bytes at the requested dtype are reported separately. The built-in pixart-sigma-xl-2-1024-ms profile provides a fixed-revision transformer example alongside the two Stable Diffusion UNet profiles.

UNets, cross-attention patch transformers (DiT/PixArt), and SD3/Flux-style joint-attention transformers use distinct activation formulas. Image-token counts include the VAE scale, patch size, and Flux latent packing; CFG doubles the denoiser batch only for architectures that use positive and negative branches. Use --weight-variant fp16|bf16|fp32 when a local directory contains multiple checkpoint families.

--offload model, --attention-slicing, --vae-slicing, and --vae-tiling expose common Diffusers memory trade-offs. Until a matching real-GPU comparison is supplied, the report remains Modeled, accuracy.status is uncalibrated, and no unsupported error percentage or prediction interval is emitted.

(back to top)

Command reference

Command	Purpose
`fakegpu doctor`	Check the installation, native libraries, PyTorch, and profiles
`fakegpu demo`	Run a small CPU-backed, CUDA-visible training step
`fakegpu preflight`	Execute a workload to a target stage and classify fit or OOM
`fakegpu analyze-repo`	Inventory repository entry points and GPU-only risks
`fakegpu analyze-kernel`	Inspect CUDA, PTX, and SASS resources and operations
`fakegpu estimate-llm`	Estimate decoder memory, communication, and FLOPs
`fakegpu plan-serving`	Plan homogeneous or mixed-request admission, chunked prefill, and grouped shared-prefix KV storage
`fakegpu estimate-diffusion`	Estimate diffusion text-encode, denoise, and VAE-decode memory phases
`fakegpu estimate-roofline`	Produce a profile-aware analytical latency interval
`fakegpu plan-training`	Normalize distributed training configs and estimate rank memory
`fakegpu simulate-topology`	Model collective routes and link contention
`fakegpu replay-trace`	Summarize compute, communication, wait, and memory timelines
`fakegpu calibrate`	Build serving observations, compare memory reports, and enforce reliability gates
`fakegpu capabilities`	List or strictly audit native API classifications
`fakegpu nvidia-smi`	Inspect devices, processes, modeled topology/MIG, health status, and reliability events
`fakegpu metrics`	Export bounded Prometheus/JSON metrics and serve short in-memory history
`fakegpu workspace-profiles`	Validate and inspect workspace estimation profiles
`fakegpu validate`	Run a declarative JSON, TOML, or YAML validation matrix
`fakegpu coordinator`	Manage the distributed simulation coordinator
`fakegpu bandwidth`	Validate simulated TCP payloads and report throughput

Use python3 -m fakegpu --help for the complete list and python3 -m fakegpu <command> --help for command-specific options.

(back to top)

GPU profiles

The catalog contains 82 YAML profiles covering consumer, workstation, data-center, and embedded NVIDIA GPUs from Maxwell through Blackwell. Profiles are shared by the Python and native runtimes.

python3 -m fakegpu doctor --list-profiles
python3 -m fakegpu demo --profile rtx4090
python3 -m fakegpu --build-dir build --devices "t4,a100:2,h100" \
  python3 your_script.py
python3 scripts/update_nvidia_gpu_catalog.py --check

Set FAKEGPU_PROFILE or pass --profile to select one profile. Use --devices for a heterogeneous list.

(back to top)

Development

Build

All reusable native build behavior is exposed through one script:

scripts/build.sh
scripts/build.sh --release
scripts/build.sh --debug
scripts/build.sh --build-dir build-custom -- -DSOME_CMAKE_OPTION=value

Build directories and compiled artifacts are ignored by Git.

Test

The maintained regression surface is grouped into four commands:

scripts/test.sh python
scripts/test.sh smoke
scripts/test.sh cpu
scripts/test.sh all

Suite	Coverage
`python`	Maintained Python regression tests
`smoke`	Native loading, reports, capabilities, SMI topology/health, and coordinator
`cpu`	CPU-backed cuBLAS simulation
`all`	All maintained suites

Run a declarative validation manifest directly when needed:

python3 -m fakegpu validate \
  --manifest tests/data/validation_smoke.yaml \
  --report-dir build/validation-smoke \
  --strict

python3 -m fakegpu validate \
  --manifest tests/data/research_validation.yaml \
  --report-dir build/validation-research \
  --strict

Reusable scripts

Path	Purpose
`scripts/build.sh`	Configure and compile native targets
`scripts/test.sh`	Run maintained test suites
`scripts/update_nvidia_gpu_catalog.py`	Check or update profile metadata
`scripts/validation/`	Shared report and artifact validators
`scripts/linux/`	Linux GPU-management helpers
`scripts/macos/`	macOS-to-Linux-VM helpers

(back to top)

Project structure

FakeGPU/
├── fakegpu/    Python package, CLI, runtimes, and estimators
├── profiles/   YAML GPU profile catalog
├── schemas/    JSON report and validation schemas
├── scripts/    Reusable build, test, platform, and validation tools
├── src/        Native C++ interception and coordinator implementation
└── tests/      Maintained regression tests and minimal native fixtures

Generated build directories, compiled libraries, test reports, caches, local environments, binary assets, and design drafts are excluded through .gitignore.

(back to top)

Limitations

Native simulation does not execute arbitrary CUDA kernels.
FakeCUDA covers maintained Python and PyTorch behavior, not binary CUDA extensions.
Static analysis cannot resolve every dynamic import, generated kernel, runtime shape, or data-dependent branch.
Memory estimates can miss backend-private allocations, custom operators, allocator policies, and unmatched workspaces.
Diffusion profiles model fixed reference pipelines. Custom ControlNet, IP-Adapter, LoRA, safety-checker, refiner, video, and DiT components require additional component metadata and matching real-GPU validation.
Online-serving plans accept explicit mixed lengths and independent autoregressive draft models. They do not model request arrival distributions, cache eviction, preemption or scheduler reordering, self-speculative/Medusa/ EAGLE variants, tokenizer remapping, tensor-parallel execution, throughput, or latency.
Roofline output is an analytical interval, not measured kernel latency.
Distributed timing includes coordinator work, memory copies, sockets, and process scheduling; it is not an NCCL, NVLink, or RDMA benchmark.
Modeled fault events are report-only control-plane inputs. They do not alter CUDA execution or represent NVML ECC counters, Xid observations, or hardware failures.
Hybrid and passthrough modes require a compatible physical CUDA stack.
macOS System Integrity Protection can remove DYLD_* variables from system binaries. Prefer a Homebrew, conda, or pyenv Python for native interception.

(back to top)

Roadmap

See the open issues for proposed features and known limitations.

(back to top)

Contributing

Bug reports, focused test cases, profile corrections, documentation improvements, and implementation patches are welcome.

Fork the repository.
Create a branch: git checkout -b feat/your-change.
Add or update tests for the changed behavior.
Run scripts/test.sh all.
Commit with a clear Conventional Commit message.
Push the branch and open a pull request.

For estimation or compatibility issues, include the exact command, selected profile, software versions, and generated report.

Name		Name	Last commit message	Last commit date
Latest commit History 386 Commits
.github/workflows		.github/workflows
fakegpu		fakegpu
profiles		profiles
schemas		schemas
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
README.zh-CN.md		README.zh-CN.md
README.zh-TW.md		README.zh-TW.md
pyproject.toml		pyproject.toml
setup.cfg		setup.cfg
setup.py		setup.py

Folders and files

Latest commit

History

Repository files navigation

FakeGPU

Table of contents

About the project

What FakeGPU answers

Typical use cases

Real-GPU memory-estimation evidence

Research workload reliability report

Current repository verification

Maintained research workload matrix

Architecture-aware estimate comparisons

How it works

Built with

Getting started

Prerequisites

Installation

Verify the installation

Usage

Run PyTorch code on FakeCUDA

Inspect FakeGPU devices and processes

Export monitoring metrics

Intercept native CUDA libraries

Check memory before a run

Plan online LLM serving capacity

Match vLLM runtime reservation

Analyze a repository or model

Command reference

GPU profiles

Development

Build

Test

Reusable scripts

Project structure

Limitations

Roadmap

Contributing

License

Acknowledgments

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages