A package that provides an aggregated view of the NVIDIA GPU information on several hosts.
You can install nvidb using pip. First, clone the repository:
git clone https://github.com/FanBB2333/nvidb.git
cd nvidb
pip install .Or install directly from PyPI:
pip install nvidb
# If the specified version is unavailable in your custom repository, use pypi.org as the source:
pip install nvidb -i https://pypi.org/simpleUse the interactive command to add servers:
nvidb addThis will guide you through adding a new server with prompts for host, port, username, authentication method, etc.
To manually configure remote servers, create or edit the configuration file at ~/.nvidb/config.yml:
mkdir -p ~/.nvidb/
cp config.example.yml ~/.nvidb/config.yml
# Edit the file with your server detailsConfiguration file template:
servers:
- hostname: "example1.com" # Server hostname or IP address
port: 22 # SSH port number
username: "user1" # SSH username for authentication
nickname: "Production GPU" # Human-readable nickname for display
auth: "auto" # Authentication method: auto | key | password
identityfile: "~/.ssh/id_ed25519" # Optional, used only when auth is auto/key
proxyjump: "login.example.com" # Optional OpenSSH ProxyJump host or alias
gpus: [0, 1] # Optional queue placement allowlistConfiguration Options:
hostname: Server hostname or IP address (required)port: SSH port, default is 22 (required)username: SSH username (required)nickname: Human-readable server nickname (optional)auth: Authentication method -auto,key, orpassword(optional, default:auto)identityfile: SSH private key path (optional, only effective whenauthisautoorkey)password: SSH password (optional, will prompt if needed)proxyjump: OpenSSH jump host, alias, or comma-separated chain (optional; for exampleloginorlogin-a,alice@login-b:2222)gpus: GPU indices available to new queue placements on this server (optional). Monitoring still shows every card, and running jobs outside the allowlist continue normally. An empty list disables GPU placement on the node while still permitting CPU-only queue jobs.
The server-level gpus list is a placement policy. The job option --gpus N
still means "request N cards"; it does not name individual indices.
Warning: Storing passwords in plaintext in the configuration file is NOT RECOMMENDED for security reasons. Consider using SSH key-based authentication (
auth: key) instead.
proxyjump uses the local OpenSSH client, so aliases and credentials configured
in ~/.ssh/config are reused. Connect to the jump host once with ssh login to
accept its host key and verify key/agent authentication before running the
non-interactive job queue. nvidb import copies ProxyJump from OpenSSH config:
Host login
HostName login.example.com
User jump-user
Host training-a100
HostName 10.0.0.42
User gpu-user
ProxyJump loginnvidb import
nvidb --remoteThe same file also holds a view section that nvidb maintains itself. Persistent
layout keys (v, d, s, f, g, u, t, and p in single-line mode)
write the new state back so the next run opens with the same view. Pane focus and
the temporary Detailed-mode process collapse are not persisted. See
2.6 Interactive TUI Navigation for what each
setting does.
You can customize the working directory by setting NVIDB_HOME:
export NVIDB_HOME=/path/to/custom/nvidbDefault working directory is ~/.nvidb/.
nvidb # Monitor local GPU only (interactive TUI)
nvidb --remote # Monitor local and remote servers
nvidb --once # Print GPU stats once and exit
nvidb --once --remote # Print all servers once and exit
nvidb --version # Show versionTip: Set
remote: trueunder thebasicsection of~/.nvidb/config.ymlto make plainnvidbinclude remote servers by default (same fornvidb log). Pass--no-remotefor a one-off local-only run.
GPU status is collected directly from NVML. Local collection uses nvidia-ml-py;
remote collection keeps a standard-library Python agent open over SSH and calls
libnvidia-ml.so.1 directly, so no Python package needs to be installed on the
remote host. nvidia-smi -q -x is retained only as a compatibility fallback when
NVML cannot be initialized.
nvidb add # Interactively add a new server
nvidb import [path] # Import servers from SSH config (default: ~/.ssh/config)
nvidb info # Show configuration info and server listContinuously log GPU statistics to an SQLite database:
nvidb log # Log local GPU with default settings
nvidb log --remote # Log local and remote GPUs
nvidb log --interval 10 # Set logging interval to 10 seconds
nvidb log --db-path /path/to/db # Specify custom database pathPress Ctrl+C to stop logging and save data.
Open a Dash-based interactive web dashboard to view live GPU info and browse log sessions:
pip install dash
nvidb web # Web dashboard (Live + Logs)
nvidb web --db-path /path/to/db
nvidb web --port 8502After the server starts (http://localhost:8501 by default):
- Live: per-server GPU tables plus rolling utilization / VRAM charts; toggle
include remote, pick the refresh interval, or pause auto-refresh. (basic.remote: trueornvidb --remote webenables remote by default.) - Logs: pick a session in the left table, then filter by node / metric / time range. Charts support zoom, pan and legend isolation; click any chart point to inspect that snapshot. The raw table supports filtering, sorting and CSV export.
nvidb log web is deprecated; use nvidb web instead.
Remove server configurations or delete log data:
nvidb clean # Interactive cleanup menu
nvidb clean all # Delete all data (requires double confirmation)When viewing GPU stats, use these keyboard shortcuts:
| Key | Action |
|---|---|
v |
Switch unified/per-node view |
d |
Toggle unified row detail |
s |
Cycle unified GPU sorting |
f |
Cycle unified GPU filters |
g |
Toggle unified per-node grouping |
u |
Show/hide nodes without GPU support |
t |
Toggle GPU and selected-process history |
Enter / Space / l / → |
Show and enter the selected GPU's process pane |
h / ← |
Return focus to the GPU/node pane |
Tab |
Switch between visible GPU and process panes |
p |
Show/hide the selected GPU's process pane |
j / ↓ |
Move the active-pane selection down |
k / ↑ |
Move the active-pane selection up |
PgUp / PgDn |
Move the active-pane selection by a page |
[ / ] |
Page through a long wrapped command |
/ |
Edit a live process filter |
o / F6 |
Cycle process sorting |
O |
Reverse the current process sort |
+ / - |
Show more/fewer process rows |
i / T / K |
Arm SIGINT / SIGTERM / SIGKILL for the selected process |
Esc |
Cancel a signal or clear the process filter |
? |
Open context-sensitive help |
| Mouse | Select rows, sort headers, click actions, or scroll either pane |
Enter (signal armed) |
Confirm the pending process signal |
Enter / Space (per-node view) |
Toggle the selected server details |
a |
Expand all servers |
c |
Collapse all servers |
q |
Quit |
The default per-node view keeps each server's summary and expandable detail
table. The unified view places GPUs from every node in one table. By default,
single-line rows are grouped into per-node blocks: a band names the node in
bold cyan followed by its hostname/IP, GPU count, free GPUs, average
utilization, and VRAM, with a dim rule filling the rest of the line. The rows
below it drop the redundant Node / Hostname/IP columns so more width goes
to the GPU metrics. Press g to turn grouping off (or sort by anything other
than node order) and the flat table with Node and Hostname/IP columns comes
back.
Detailed cards include node identity themselves and omit the redundant band.
Columns adapt to the terminal width, with core identity, utilization, model, and
VRAM fields kept ahead of secondary metrics.
In the unified view, press d to switch between the single-line table and
Detailed cards. Each card is divided into four labelled rows: GPU identity,
LOAD, MEM/TEMP, and I/O. Utilization and VRAM gain block bars on terminals
at least 100 columns wide. The palette stays deliberately quiet: grey for
structure and secondary values, cyan for the active focus and normal metrics,
and yellow/red for warnings. Green is reserved for a running/healthy state.
The node name comes before the GPU index so the machine is the first thing you
read.
PCIe throughput is scored against the link the card is actually running on. The
PCIe column names that link (4.0x16, 4.0x4), and the RX/TX cells fill
up with their share of it — green under 40%, yellow past 40%, red past 70%, at
which point the interconnect is shaping transfers. Detailed cards spell the same
thing out as Link 4.0x4 7.34GB/s of 4.0x16 plus a percentage per direction,
where the trailing of ... is the widest link the card itself supports. A card
wired through a narrower slot or riser reports the narrower link, and a card
sitting idle reports the reduced generation it drops to for power saving; both
are the real ceiling for the traffic measured alongside them. SXM boards are
scored on their host PCIe link, which is what these counters measure — NVLink
traffic between GPUs is not included.
The capacity line summarizes available and busy GPUs, average utilization, and
used/total/free VRAM. Press s to cycle between node order, available GPUs
first, and highest utilization first. A GPU is considered available when its
utilization is below 5% and its VRAM usage is below 10%.
Unified pages are sized from the current terminal height. The title shows the
current focus and visible GPU range, and › marks the active selection.
Press f to cycle through all, available, busy, and error-only views. Busy
means at least 50% GPU utilization. Error-only mode hides GPU rows and lists
nodes whose latest refresh failed. Because the filter is restored from the
config on the next run, a warning line above the table spells out how many GPUs
it is hiding.
Detailed mode places the selected GPU's process pane directly below the cards;
there is no separate drill-down screen. Enter, Space, →, or l shows the
pane when necessary and moves focus into its task list. Repeating one of these
keys keeps process focus instead of collapsing the pane. Press p to hide or
show it explicitly; ← or h returns to GPU/node selection without hiding it.
The same navigation applies in single-line mode. The process table shows PID,
user, process VRAM, percentage of total GPU VRAM, CPU%, host MEM%, RSS, elapsed
time, state, and command. Narrow terminals discard secondary columns first. The
selected process gets an htop-style low-contrast grey background, and its block
below the table spells out both percentage of the whole GPU and percentage of
currently used GPU memory, threads, state, elapsed time, and the complete
wrapped command line.
Press / to filter the process list as you type. The query matches PID, user,
type, state, process name, and the full command. Enter keeps the current
filter and leaves edit mode; Esc clears it. Press o or F6 to cycle through
VRAM, CPU, host memory, RSS, elapsed time, PID, and command sorting. Uppercase
O reverses the current order. The visible table headers are also clickable:
click a field once to sort by it and again to reverse it.
Use + and -, or the row controls in the action bar, to change how many
processes stay visible. The request is capped at 12 rows and reduced
automatically when the selected-process details need the space.
When a wrapped command would push the action bar off-screen, it automatically
uses height-aware pages (at most five command lines on terminals shorter than
28 rows). Use [/], the clickable command buttons, or the wheel over the
command to see every line.
Use Enter, →, or l to enter process focus, ← or h to return to
GPU/node focus, and Tab to switch between visible panes. Then use j/k or
the arrow keys to move the highlighted row. The active pane uses solid borders;
the inactive pane uses dashed borders. Only the active pane displays a
selected-row background, so process rows are not highlighted while GPU/node
selection has focus. Mouse reporting is on by default: click any GPU card or
process row to select it, use the wheel over either pane to scroll that pane,
and click the action buttons directly. Most terminals need Shift (or
Option) held down for their own drag-to-select while mouse reporting is
active; set mouse: false under view to disable TUI mouse handling.
The process action bar exposes SIGINT, SIGTERM, and SIGKILL as i, T,
and K. Every signal requires a second identical click/key press or Enter
within five seconds; Esc cancels it. Signals run on the node that owns the
selected GPU and report permission or command failures in the pane. If
GPU/node selection has focus, the first signal key only transfers focus to the
process pane; press it again to arm the action.
Press ? for a help panel tailored to the current per-node, GPU, or process
focus. Click anywhere inside the panel, or press ?, Esc, or q, to close
it. Closing help with q does not quit the monitor.
Press t (or click History) to show utilization, VRAM, and temperature for the
selected GPU plus CPU, GPU VRAM share, host memory, and RSS for the selected
process. Both histories retain the latest 60 successful refresh samples in
memory and do not add remote requests. On terminals shorter than 36 lines, the
history rows temporarily replace the selected process's command block so the
action buttons remain visible; toggle History off to restore the command.
Machines without NVIDIA GPUs (a macOS laptop, a CPU-only host) are not expanded
by default in the per-node view and are collapsed into a single "hidden" line in
the unified node status. Press u to show them.
Layout toggles are written back to the view section of ~/.nvidb/config.yml,
so the next nvidb run starts with the same layout. Process filters, sorting,
row counts, and help visibility are session-only.
Use the @nvidb.monitor decorator to track GPU usage during function execution:
import nvidb
@nvidb.monitor
def train_model():
# Your training code here
pass
# With custom options
@nvidb.monitor(sample_interval=0.05, gpu_indices=[0, 1])
def multi_gpu_training(epochs: int = 100):
pass
# Async function support
@nvidb.monitor
async def async_training():
passAfter function execution, it outputs:
======================================================================
[nvidb.monitor] Function completed: train_model
Signature: train_model()
Location: /path/to/file.py:14
----------------------------------------------------------------------
Duration: 125.3s
----------------------------------------------------------------------
GPU 0: NVIDIA GeForce RTX 3090 Ti
Memory:
Peak: 8192.00 MiB / 24.00 GiB
Delta: +6144.00 MiB
Utilization:
Avg: 85.0%
Temperature:
Peak: 72C
Power:
Peak: 320.5W
======================================================================
Decorator Options:
sample_interval: Sampling interval in seconds (default: 0.1)gpu_indices: List of GPU indices to monitor (default: all GPUs)enabled: Enable/disable monitoring (default: True)
nvidb queue adds a small slurm-like scheduler on top of the servers already in
config.yml. Jobs are submitted from any machine that can SSH to the nodes, wait
until a GPU has room for them, run detached on the node, and report back.
The queue's only shared state is one SQLite file (~/.nvidb/queue.db). Several
independent clients — several Claude Code sessions, a script, an open TUI —
coordinate purely by reading and writing that file, so none of them has to stay
running for the queue to work.
There is no daemon. Any command that touches the queue performs a tick:
probe every node, settle finished jobs, then place whatever now fits. Ticks are
rate-limited and guarded by a lease, so a burst of concurrent clients results in
one round of SSH traffic rather than one per client. The open TUI ticks on a
timer, and nvidb queue tick --watch 10 gives you a background ticker if you
want one.
GPUs are allocated by VRAM budget, not whole cards:
free = total − memory used by non-queue processes − reservations of queue jobs − headroom
Counting foreign processes is what lets the queue share machines with work you
started by hand: a card that someone else has filled simply reports no capacity,
and jobs go elsewhere until it frees up. A job is charged the larger of what it
reserved and what it actually uses, so understating --vram cannot oversubscribe
a card. CPU-only jobs (--gpus 0) reserve nothing, so they are bounded by a
count instead: max_cpu_jobs_per_node, four by default.
A job cancelled or timed out while its node was unreachable keeps its record
final but its process alive. The pid is remembered separately from the job — it
has to outlive both job requeue and job purge — and the first probe that
reaches that node again kills it, provided the pid is still running that job's
run.sh. nvidb queue status --json reports how many such cleanups are
outstanding as pending_reaps.
On nodes where the driver accounts for none of the memory in use — WSL, where
the GPU is driven from Windows, and which either names no processes at all or
names them without memory figures — the split between foreign work and the
queue's own jobs cannot be measured. Those GPUs are marked blind and their own
jobs are credited up to their reservation, which keeps the queue from charging a
job twice.
The queue's own bookkeeping is only half the picture: these are workstations,
and most of what runs on them was started by hand. nvidb queue nodes reports
the whole card.
nvidb queue nodes # capacity, plus the unmanaged work behind it
nvidb queue nodes --procs # every process on every GPU, with its owner
nvidb queue nodes --json # the same as structured data workstation [up] 10.0.0.2 seen 00:00:00 ago
GPU MODEL UTIL MEM UNMANAGED QUEUE RESERVED JOBS FREE
GPU0 NVIDIA GeForce RTX 3090 Ti 10% 23.2G/24.0G 97% 15.2G (blind) 0M 8.0G 1 316M
unmanaged ~15.2G of 23.2G in use; this driver reports no per-process memory
pid 1608997 /python3.11 alice job 12 sweep
UTIL and MEM describe the card itself, whoever is using it. UNMANAGED is
what the queue did not start — (2p) counts the processes behind it, (blind)
means the figure is inferred rather than measured. QUEUE is what the queue's
own jobs hold right now, against the RESERVED they were promised.
In JSON each GPU carries mem_used_mb, mem_total_mb, mem_used_percent,
util_percent, external_mem_mb, queue_mem_mb, reserved_mb, free_mb, and
a processes list naming every process with managed and job_id.
# Put the command last, after `--`, or pass it as one quoted string
nvidb job submit --name train --vram 20G -- python train.py --epochs 10
nvidb job submit --node gpu-node --vram 8G --workdir ~/proj -- python eval.py
nvidb job submit --gpus 0 -- python prepare_data.py # CPU-only
nvidb job submit --after 12 --vram 4G -- python report.py # runs after job 12
nvidb job submit --script run.sh --vram 12G # a local file's contents
nvidb job ls # everything, newest state
nvidb job ls --active # only pending and running
nvidb job show 12 --logs 40 # detail plus the tail of stdout
nvidb job logs 12 -f # follow the output
nvidb job wait 12 13 # block until both finish (0 ok, 1 failed, 2 timed out)
nvidb job cancel 12 # kill the remote process group
nvidb job requeue 12 # run a finished job again
nvidb job priority 12 5 # set the priority (bare number sets, +1/-2 adjust)
nvidb job priority 12 --up 2 # move a pending job two slots earlier instead
nvidb job purge # forget finished job records
nvidb queue status # nodes, capacity and jobs in one view
nvidb queue status --procs # the same, with every GPU process listed
nvidb queue events --since 42 # replay what happened while you were away
nvidb queue drain gpu-node # stop scheduling onto a node (`resume` undoes it);
# jobs already on it keep running and still report back
nvidb queue ignore offline-node # stop probing/scheduling and hide a disconnected node
nvidb queue unignore offline-node # restore it; `nodes --include-ignored` lists hidden nodes
nvidb queue tick # force one scheduler passUseful submit options: --priority N (higher goes first), --timeout SECONDS
(kill an overrunning job), --retries N (restart if the process vanishes),
--env KEY=VALUE, --tag, --note, and --wait to block until the job
finishes.
Read commands refresh the queue themselves before printing, so nvidb job show
never reports stale state. Pass --no-tick for a pure database read, or
--tick to force a refresh.
Every job runs with CUDA_VISIBLE_DEVICES set to its allocation, plus
NVIDB_JOB_ID, NVIDB_JOB_NAME, NVIDB_NODE, NVIDB_JOB_DIR, and
NVIDB_STATUS_FILE.
A job carries two independent pieces of free text, kept apart because they have different authors and neither should be able to overwrite the other:
note — what you or a client says about the job. Set it at submit time and
edit it whenever, including long after the job has finished:
nvidb job submit --note "baseline A, lr=1e-4" -- python train.py
nvidb job note 12 # read it
nvidb job note 12 --append "loss plateaued at epoch 30"
nvidb job note 12 "superseded by job 19" # replace
nvidb job note 12 --clearprogress — what the job says about itself. The job writes one line to
$NVIDB_STATUS_FILE and the scheduler collects it on every probe, so you can
watch a long run without tailing its log:
# inside a training script
import os
with open(os.environ["NVIDB_STATUS_FILE"], "w") as handle:
handle.write(f"epoch {epoch}/{total} loss {loss:.3f}")echo "epoch $i/$n loss $loss" > "$NVIDB_STATUS_FILE" # or from shellOnly the last line of the file is read, so overwriting is the normal pattern.
Both fields appear in job ls, queue status, job show, the TUI and every
--json payload. The last thing a job reported is kept on its finished record,
which is often the quickest explanation of how far a lost or failed job got. A
retry clears the stale status line but keeps your note.
When something goes wrong on a node, the queue records an alert locally, classified by what actually happened:
| Kind | Raised when | Severity |
|---|---|---|
job_failed |
the command exited non-zero | error |
job_lost |
the process vanished with no exit status | error |
job_timeout |
the job passed --timeout and was killed |
error |
dependency_failed |
a job can never run because a dependency failed | error |
job_unschedulable |
no GPU in the cluster is big enough to ever hold it | error |
job_retried |
the process vanished but a retry remains | warning |
launch_failed |
the job could not be started (disk full, permissions) | warning |
node_down |
a node stopped answering | error |
For a failed job the queue pulls the tail of its stderr (falling back to stdout) onto the alert, so the reason is readable locally without another round trip:
nvidb queue alerts # what needs attention
nvidb queue alerts --detail # ... with the captured output
nvidb queue ack 3 # acknowledge one
nvidb queue ack --all # acknowledge everythingnvidb queue alerts exits non-zero while anything is unacknowledged, so a shell
or an agent can branch on it. Alerts stay until acknowledged; they are never
re-raised for the same failure, and nvidb queue status and the TUI both lead
with them.
Recording an alert and delivering it are separate. Recording happens on whatever scheduler pass notices the failure, so nothing is lost when nobody is watching. Delivery is the daemon's job:
nvidb queue daemon # tick every 15s and push failures
nvidb queue daemon --interval 5
nvidb queue daemon --once --json # a single pass, for cronThe daemon is optional — the queue works exactly as before without it, since every command runs a scheduler pass. Run it when you want failures pushed to you rather than waiting to be asked, and for prompt timeout enforcement: without it, a job that overruns is only killed the next time some command happens to tick.
Each alert is delivered once, whether or not the daemon restarts. Channels are
configured under queue.notify: a desktop notification, a JSON-lines file at
$NVIDB_HOME/alerts.log, and a command hook that receives the alert as JSON
on stdin — which is how you route failures to anything else.
Every command accepts --json and prints one JSON document on stdout, which is
the intended way for tools — Claude Code sessions in particular — to use the
queue:
nvidb queue status --json # nodes, per-GPU budgets, job table, counts
nvidb job submit --json --vram 20G -- python train.py
nvidb job wait 12 --json --logs 40
nvidb queue events --json --since 42Three things make the database usable as a coordination channel between processes that never talk directly:
- Dependencies.
--after 12,13records the ordering in the queue, so a client can lay out a pipeline and exit; the jobs still run in order. - Results. A job that writes
$NVIDB_JOB_DIR/result.jsonhas that payload collected on completion and served bynvidb job result <id>. Clients can also write one directly withnvidb job result <id> --set '{"note":"..."}', which passes structured data between them without a shared filesystem. - Events.
nvidb queue events --since <id>replays every state change, so a client that was not running can catch up on exactly what it missed.
skills/nvidb-queue/ is an Agent Skill that
teaches Claude Code, Codex and other skill-aware tools to route GPU work through
the queue instead of starting it over raw SSH. Install it by symlinking, so both
agents track the repository:
ln -s "$PWD/skills/nvidb-queue" ~/.claude/skills/nvidb-queue
ln -s "$PWD/skills/nvidb-queue" ~/.codex/skills/nvidb-queuenvidb queue # or: nvidb queue tuiThe screen stacks node capacity, the job table, and a detail or log pane for the selected job. All SSH work happens on a worker thread, so an unreachable node slows the numbers down but never freezes the interface.
| Key | Action |
|---|---|
j / k / arrows |
Move the selection in the focused pane |
PgUp / PgDn |
Move a page at a time |
Tab |
Switch focus between the node and job panes |
Enter |
Show or hide the selected job's detail pane |
[ / ] |
Page through wrapped detail or log text |
L |
Toggle a live tail of the selected job's log |
c |
Cancel the selected job (press twice) |
r |
Re-queue the selected finished job |
+ / - |
Raise or lower the selected job's priority |
K / J |
Move a pending job up/down the dispatch order |
s / S |
Cycle the sort column / flip its direction |
t |
Force a scheduler tick now |
a |
Toggle automatic ticking |
f |
Cycle the job filter |
p |
GPU processes: unmanaged only / all / none |
d |
Drain or resume the selected node |
A |
Acknowledge every open alert |
? |
Help |
q |
Quit |
| Mouse click | Select rows, activate actions, sort by column |
| Mouse wheel | Move selections or page detail/log text |
The job table opens in queue order — running jobs first, then pending jobs
exactly as the scheduler would dispatch them — so K/J visibly reorder the
line a job is waiting in. Reordering rewrites the priorities of the pending
jobs the moved one passes; the PRI column always shows the real values.
Clicking a column header sorts by that column (a second click flips it), and
each GPU line draws one memory bar whose segments distinguish foreign memory
(amber) from this queue's reservations (teal) and free space (dim). Colours
are deliberately muted: healthy values render grey or plain, and saturation
is reserved for states that need attention.
Mouse reporting is enabled by default. Click anywhere in a node card to select
that node, or click a job row to select it; clicking the selected job again
shows or hides its detail pane. The wheel acts on the pane under the pointer.
Status counts select their matching job filter, and clicking a job alert opens
that job's log. The bottom action bar exposes the actions that apply to the
current selection, including the second confirmation required for cancellation.
Most terminals reserve normal text selection for Shift-drag or Option-drag
while mouse reporting is active. Set mouse: false under view in
~/.nvidb/config.yml to disable it in both nvidb TUIs.
Nothing is installed. A generated run.sh is delivered over SSH and started
with setsid, so the job outlives the client that launched it. Each job keeps a
directory on its node (~/.nvidb/jobs/<id>/ by default) holding the script,
stdout.log, stderr.log, its pid, and the exit status with the time it
finished. Every later interaction — checking liveness, reading output, killing a
job — is a single shell round trip.
Tuning lives under queue: in config.yml; see
config.example.yml for the full set of keys.
Everything above assumes some client eventually runs a scheduler pass. That
stops being enough when the point is to submit work and walk away: a job queued
just before a laptop is closed stays pending until something looks again.
Jobs already running are unaffected — they were detached from the SSH
session that started them — so the only thing missing is somebody to dispatch.
Move the queue onto a machine that stays on, and the laptop becomes a client of
it. The queue's own configuration lives in its own file, so one ~/.nvidb can
hold both a monitor watching many machines and a queue scheduling onto a few:
# ~/.nvidb/queue.yml on the queue host — see queue.example.yml
servers:
- hostname: "gpu-node.example.com"
port: 2222
username: "user"
nickname: "gpu-node"
gpus: [0, 1] # optional: only new queue jobs are restricted
queue:
include_local: true
local_node_name: "queue-host"The keeper keeps that machine scheduling. Its default shell supervisor is
written into ~/.nvidb, restarts nvidb queue daemon whenever it stops, and
detaches from the session that installed it — no root, cron entry, or service
manager:
nvidb queue keeper install --start # write ~/.nvidb/queue-keeper.sh and run it
nvidb queue keeper status # non-zero while nothing is keeping the queue moving
nvidb queue keeper logs -n 50
nvidb queue keeper stopOn a Linux host with a working user systemd session, install a user service
instead. --start enables and starts nvidb-queue.service; systemd then
restarts the daemon after failures and starts it with the user's next session:
nvidb queue keeper install --systemd --start
systemctl --user status nvidb-queue.serviceStarting the user service during boot, before that user logs in, additionally requires lingering where the host permits it:
loginctl enable-linger "$USER"The shell supervisor does not survive a reboot. Any later client command starts
it again. A client configured with remote: opens no database of its own and
forwards the whole command line to the queue host, running the keeper's ensure
in the same round trip. A missing or failed keeper is reported on stderr rather
than silently leaving the queue idle:
# ~/.nvidb/queue.yml on the laptop
remote:
host: "queue-host.example.com"
nvidb: "/home/user/.local/bin/nvidb" # absolute: a non-interactive shell has a thin PATHnvidb job submit, job wait, logs --follow, queue status and the TUI then
behave exactly as they do locally — it is the same CLI, running over there — and
the exit codes come back unchanged. --local runs one command against this
machine's own queue instead, and nvidb queue status reports keeper up or
keeper DOWN wherever a keeper is installed.
Back up the queue with SQLite's online backup API, which includes committed WAL state without pausing the daemon:
nvidb queue backup # ~/.nvidb/backups/queue-<time>.db
nvidb queue backup /secure/queue-copy.db # explicit destination, never overwrittenThe daemon can create and rotate backups automatically:
queue:
backup:
enabled: true
interval_hours: 24
keep: 7
directory: null # defaults to $NVIDB_HOME/backupsEach snapshot is checked with PRAGMA quick_check before its complete temporary
database is atomically published without replacing an existing path.
nvidb queue status --json includes last_backup_at.
- NVIDIA driver with NVML (
libnvidia-ml.so.1) - Python 3.8+
- Python 3.8+ and SSH access on remote servers
- Local OpenSSH client when
proxyjumpis configured nvidia-smiis optional and used only as an NVML failure fallback
- The live header shows
Source: nvmlduring normal collection andSource: nvidia-smiif the compatibility fallback was needed - Database files are stored in
~/.nvidb/gpu_log.dbby default - Configuration and logs are stored in
~/.nvidb/directory
- Monitor local info with
nvidb:
- Monitor remote info with
nvidb --remote:
- Monitor on web panel with
nvidb web:
Local info:
Remote info:
- Thanks to NVIDIA for providing NVML and nvidia-ml-py, used for direct GPU telemetry collection.
- Thanks to nvitop for demonstrating efficient direct NVML polling and metric caching patterns.
- Thanks to NVIDIA for providing
nvidia-smi, retained as a compatibility fallback. - Thanks to Paramiko for powering SSH connections for remote monitoring.
- Thanks to PyYAML for YAML-based configuration loading and saving.
- Thanks to pandas for parsing and processing GPU stats and log data.
- Thanks to blessed for building the interactive terminal UI.
- Thanks to termcolor for colored terminal output.
- Thanks to Dash and Plotly for powering the web dashboard.