Monitoring Template

A reusable, batteries-included monitoring stack for any backend project running on Docker — Swarm or plain Compose. Copy this folder next to your project, fill in one .env file, run ./deploy.sh — and you get metrics, logs, dashboards, and Telegram alerts.

O'zbekcha qo'llanma: README_UZ.md

What you get

Two independent pipelines feeding one Grafana:

metrics (PULL):  exporters ──◄── Prometheus ──► Grafana (dashboards + alerts)
logs   (PUSH):  containers ──► Promtail ──► Loki ──► Grafana (Explore + alerts)

Component	Role	Included
Prometheus	Scrapes and stores metrics (15s interval, 30d retention)	core
Grafana	Dashboards, Explore, alert evaluation → Telegram	core
Loki	Log storage (7d retention)	core
Promtail	Ships every container's logs to Loki, glues tracebacks	core
node-exporter	Host CPU / memory / disk / network	core
cAdvisor	Per-container CPU / memory	core
redis-exporter	Redis memory, clients, ops/sec, hit rate	optional
postgres-exporter	Connections, TPS, cache hit, DB size (one per instance)	optional
celery-exporter	Task successes/failures, queue length, runtimes	optional
nginx-exporter	Connections and request rate via stub_status	optional
blackbox-exporter	External uptime probes + SSL expiry	optional

One stack per server. node-exporter, cAdvisor, and Promtail observe the whole host, so a single monitoring stack covers every project on the machine. To monitor a second project on the same server, add scrape jobs to prometheus/prometheus.yml — do not deploy a second copy of this stack.

Deploy modes

Set DEPLOY_MODE in .env; deploy.sh handles the rest.

	swarm (default)	compose
Compose file	docker-compose.monitoring.yml	docker-compose.standalone.yml
Deploy command	docker stack deploy	docker compose up -d
Network	attachable overlay	bridge (apps must declare it as external)
Promtail config	promtail.yml (swarm labels)	promtail.standalone.yml (compose labels)
Multi-node	yes (mode: global agents)	no — one server
APP_LOG_SELECTOR matches	<stack>_<service> names	short compose service names

Log labels are normalized: in both modes Grafana queries use {stack="...", service="..."}.

Requirements

• swarm mode: Swarm initialized (docker swarm init); an attachable overlay network shared with your app stacks (deploy.sh creates it if missing).

• compose mode: just Docker + Compose v2; a shared bridge network (deploy.sh creates it) that your app's compose file declares as external:

  # in your app project's docker-compose.yml
  networks:
    default:
      name: shared_net
      external: true

Quick start

# 1. Copy the template to the server (or git clone your copy of it)
scp -r monitoring-template user@server:/projects/monitoring && ssh user@server
cd /projects/monitoring

# 2. Configure
cp .env.example .env
nano .env                      # set DEPLOY_MODE first, then see "Configuration"

# 3. One-time PostgreSQL setup (per DB instance; skip if not using Postgres)
docker exec -it $(docker ps -qf name=<your_db_container> | head -1) \
  psql -U postgres -c "CREATE USER monitor WITH PASSWORD 'STRONG_PASSWORD'; GRANT pg_monitor TO monitor;"

# 4. Trim to your project
#    - delete unused exporters from docker-compose.monitoring.yml
#    - delete their jobs from prometheus/prometheus.yml
#    - fill the EDIT ME blocks in prometheus/prometheus.yml (app metrics, blackbox URLs)

# 5. Deploy (always through the script — it loads .env, Swarm does not)
./deploy.sh

# 6. Open Grafana through an SSH tunnel
ssh -N -L 3001:localhost:3001 user@server   # then http://localhost:3001

Verify it works

# swarm mode
docker stack services <STACK_NAME>          # every service 1/1 (or global N/N)
docker service logs --since 2m <STACK_NAME>_postgres-exporter   # no errors

# compose mode
docker compose -p <STACK_NAME> ps           # everything Up
docker compose -p <STACK_NAME> logs --since 2m postgres-exporter

In Grafana → Explore → Prometheus, run up — every target should be 1. In Explore → Loki, run {stack=~".+"} — logs should stream in.

Configuration (.env)

Variable	What it is
DEPLOY_MODE	swarm or compose
STACK_NAME	Stack / compose-project name (services become <name>_grafana, ...)
MONITOR_NETWORK	Shared network (overlay or bridge depending on mode)
GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD	Grafana login (password required)
GRAFANA_PORT / GRAFANA_ROOT_URL	Published port / external URL
PROMETHEUS_RETENTION	Metrics retention (default 30d)
ALERT_BOT_TOKEN / ALERT_GROUP_ID / ALERT_TOPIC_ID	Telegram delivery (see Alerts)
APP_LOG_SELECTOR	Regex of service names needing traceback reassembly, e.g. myproj_(app|worker)_?.*
REDIS_ADDR / REDIS_PASSWORD	redis-exporter target
POSTGRES_DSN	postgresql://monitor:PASS@<db_dns>:5432/<db_name>?sslmode=disable
CELERY_BROKER_URL / CELERY_QUEUES	celery-exporter broker + comma-separated queues
NGINX_STATUS_URL	nginx stub_status URL

Gotchas that bite in practice:

• POSTGRES_DSN database name must be the real one — check DB_NAME / POSTGRES_DB in the project's own .env. A wrong name fails every scrape with pq: database "..." does not exist.

• DNS names differ by mode: in swarm a service db in stack myproj is reachable as myproj_db; in compose it is just the service name (db), as long as both projects share MONITOR_NETWORK.

• Celery workers must run with --events, otherwise celery-exporter sees nothing.

• nginx stub_status needs a tiny server block in your nginx:

  server {
      listen 8080;
      location /stub_status { stub_status; }
  }

Application metrics (RED: rate / errors / duration)

Expose /metrics from your backend and uncomment the app-backend job in prometheus/prometheus.yml:

• Django: django-prometheus
• FastAPI: prometheus-fastapi-instrumentator
• Express: prom-client

Scrape via tasks.<stack>_<service> DNS in swarm mode, or the plain compose service name in compose mode — both examples are in the EDIT ME block, and both resolve to every replica rather than a single load-balanced address.

Dashboards

Import via Grafana → Dashboards → New → Import → ID:

ID	Dashboard	For
1860	Node Exporter Full	host
193 / 14282	Docker monitoring / cAdvisor	containers
763	Redis Dashboard (redis_exporter 1.x)	redis
9628	PostgreSQL Database	postgres
17508 / 17509	Celery Tasks Overview / by Task	celery
12708	NGINX exporter	nginx
7587	Blackbox Exporter	uptime/SSL

To version dashboards in git, export their JSON into grafana/dashboards/ — they auto-load within 30s (provisioned into the "Infra" folder).

Alerts → Telegram

1. Create a bot with @BotFather, copy the token into ALERT_BOT_TOKEN.
2. Add the bot to your alerts group. Get the chat ID from https://api.telegram.org/bot<TOKEN>/getUpdates (send one message in the group first). Groups have negative IDs, e.g. -100123....
3. Redeploy. Test: Grafana → Alerting → Contact points → Telegram → Test.

Included starter rules (Infra folder): Scrape Target Down, Service Down (blackbox), Host CPU High, Host Memory Low, Disk Space Low. Add project-specific rules (error-log alerts from Loki, latency SLOs, queue backlogs) in grafana/provisioning/alerting/rules.yml following the same schema. Once blackbox targets are real, flip Service Down to noDataState: Alerting so a dead endpoint that stops reporting still alerts.

Securing Grafana

Both files publish Grafana as 127.0.0.1:3001:3000. In compose mode this bind works as written — Grafana is reachable only from the server itself. In swarm mode, ingress ignores the IP bind and listens on 0.0.0.0 — and Docker's published ports bypass ufw. So in swarm mode pick one:

• SSH tunnel (default, safest): block the port in the cloud firewall / security group, access via ssh -N -L 3001:localhost:3001 user@server.
• Reverse proxy on a subdomain: remove the ports: block entirely, point your nginx at <STACK_NAME>_grafana:3000 over the shared network, add TLS (Let's Encrypt needs the subdomain in public DNS), set GRAFANA_ROOT_URL. Grafana Live needs WebSocket headers (Upgrade/Connection) proxied.
• VPN (WireGuard/Tailscale) if you already run one.

Troubleshooting

Symptom	Cause / fix
postgres-exporter: pq: database "..." does not exist every 15s	DSN database name ≠ real DB name — fix POSTGRES_DSN
postgres-exporter: password authentication failed / role ... does not exist	Monitoring user not created or wrong password — rerun the one-time SQL
postgres-exporter: collector failed name=stat_replication	Known PG 17 incompatibility — already disabled via --no-collector.stat_replication
Promtail: burst of 400 ... entry too far behind after a redeploy	Harmless: rejected entries are duplicates already stored in Loki. The positions volume prevents recurrence; the very first deploy after adding the volume produces one final burst
Deploy "works" but env values are empty inside services	You ran docker stack deploy directly — it does not read .env. Use ./deploy.sh
Prometheus can't reach app containers (compose mode)	The app project doesn't join MONITOR_NETWORK — declare it as an external network there
up == 0 for one exporter	That exporter can't reach its target — docker service logs <STACK_NAME>_<exporter>
No app logs in Loki	Promtail is mode: global but needs the docker socket — check it runs on each node; check the service label with {stack="<your app stack>"}
Tracebacks split across log entries	APP_LOG_SELECTOR regex doesn't match your service names, or your log lines don't start with a timestamp

Updating

Config change → ./deploy.sh (only changed services restart). Force-restart one service: swarm — docker service update --force <STACK_NAME>_grafana; compose — docker compose -p <STACK_NAME> restart grafana.