feat(local): local dev stack via docker compose - #357
Open
you06 wants to merge 5 commits into
Open
Conversation
Per @tmgg06 in #mem9-discussion:efde4126 (2026-06-09): preparing for the code memory v0 spike, need a one-command local mem9 setup so dogfood doesn't depend on TiDB Cloud or a hand-rolled TiDB on the laptop. Approach (option A from my msg=35951712, ack msg=59173ccb): docker-compose with 3 services instead of a single multi-process image. Cleaner separation, lets us upgrade TiDB and mem9 independently, matches Docker-one-process-per-container convention. Services: - `tidb`: `pingcap/tidb:v8.4.0` in `unistore` single-process mode. No PD/TiKV containers — unistore is the embedded backend TiDB uses for testing and supports the VECTOR(N) column types in server/schema.sql. Vector INDEX still needs TiFlash and stays commented out in the schema. - `schema-init`: one-shot mysql:8.0 job. Waits for tidb healthcheck (mysqladmin ping), creates the `mnemo` database, pipes server/schema.sql in. Exits 0 on success. - `mem9`: `local/mnemo-server:dev` (built via `make docker`). Waits for schema-init via `service_completed_successfully` so it never starts against an empty or half-loaded schema. MNEMO_DSN points to the in-compose tidb container. Healthcheck on tidb uses `mysqladmin ping` (canonical TiDB readiness probe — only succeeds after SQL accept). README "Local Quickstart with Docker Compose" section added under "Self-Hosting > Docker" with the 3-command bring-up recipe and notes on overriding the mem9 image via MEM9_IMAGE. This is a spike; smoke-verify on the operator's machine before merging — no docker on this dev box so the compose file was authored by inspection. Iterate on TiDB version / storage backend if unistore doesn't accept the VECTOR types. fix(local): address @rossi review — tenant seed, polling, env safety Per @rossi msg=4a48a342 in #mem9-discussion:efde4126, the v1 compose spike (e8290c7) had 4 issues that would surface only when an operator actually `docker compose up`'d it: 1. `tidb` healthcheck used `mysqladmin` but the official `pingcap/tidb` image ships without mysql client tools — the healthcheck would fail closed forever. Dropped tidb healthcheck; moved the readiness poll into `schema-init` which already has mysql:8.0 with mysqladmin baked in. Loop polls every 1s for up to 60s. 2. `schema-init` entrypoint was `/bin/sh -c 'set -euo pipefail; ...'`. `pipefail` is a bash-ism; dash/busybox sh barf on it. Switched to `/bin/bash` (mysql:8.0 is Debian-based and has /bin/bash) and tightened the set flags to `set -eu`. 3. The big one: v1alpha2 routes resolve the `X-API-Key` header to a `tenants` row via `ResolveApiKey` middleware (server/internal/middleware/auth.go:327). The first commit loaded schema but seeded zero tenants, so every API call would 401. schema-init now inserts a `local-dev` tenant pointing at the same in-compose tidb (single-DB local setup). With the default MNEMO_ENCRYPT_TYPE=plain, the empty `db_password` survives the decrypt path unchanged. ON DUPLICATE KEY UPDATE keeps the seed idempotent across recreates. 4. The `mem9` service had no env hardening. Three defaults that silently break local-only operation are now overridden: - `MNEMO_TIDB_ZERO_ENABLED=false` — the Zero provisioner is on by default (config.go:182) and would try to call https://zero.tidbapi.com on startup. Wrong for a self-contained stack where we already seeded the tenant. - `MNEMO_INGEST_MODE=raw` — default `smart` invokes an LLM during ingest; local stack has no LLM key by default. Smart mode is still one env flip away. - (left embedding/FTS at their `disabled` defaults; operators who want vector recall flip them per env.) README smoke now exercises X-API-Key end-to-end (pinned store + query) so a passing run actually proves "agent can use this", not just "server didn't crash on startup". Still operator-side todo: a real `docker compose up` on a host that has docker, to confirm pingcap/tidb v8.4 in unistore mode accepts the VECTOR(1536) column from schema.sql. No docker on this dev box so the compose is still authored by inspection. feat(local): self-contained build — one-command `docker compose up --build` @tmgg06 hit `pull access denied for local/mnemo-server` because the v1 compose only referenced `local/mnemo-server:dev` without telling Docker how to produce it — that image had to be pre-built via `make build-linux && make docker REGISTRY=local COMMIT=dev`, which also required a Go toolchain on the host. Rossi proposed the fix (msg=44b6b380) and I'm taking it: multi-stage Dockerfile + compose build directive so a clean checkout can boot the whole stack with one command and no host dependencies besides Docker. Changes: - New `server/Dockerfile.local`: builds the binary inside the image (golang:1.24 builder → alpine:3.19 runtime). Kept as a separate file from `server/Dockerfile` so the existing release pipeline (`make docker`, which assumes a pre-built `server/bin/mnemo-server`) stays untouched. - `docker-compose.yml` mem9 service: adds `build: { context: ., dockerfile: server/Dockerfile.local }` and defaults `image` to `mem9:local` instead of `local/mnemo-server:dev`. Closer to the product name as @tmgg06 noted; binary name inside the image stays `/mnemo-server` for code-path compatibility. `MEM9_IMAGE` override still works for prebuilt images. - README quickstart: drops the `make build-linux` and `make docker` prerequisites; new flow is just `docker compose up --build`. Updated the "override mem9 image" example to use a more realistic prebuilt-image use case (`ghcr.io/...`). Still operator-side: nobody has actually `docker compose up`-ped this yet because I don't have docker on this host. Tagging @tmgg06 to retry; expectation now is just `docker compose up --build` from a clean tree with no prior `make` invocations. fix(local): rename in-compose database mnemo → mem9 for user-visible naming @tmgg06 (msg=6eea6e2d) saw `mnemo` in `show databases;` and asked why the user-visible name doesn't match the product. He's right — the historical Go module / binary / env-var prefix is `mnemo`/`MNEMO_*` (can't change those without touching the codebase), but the things an operator sees in their docker stack should say `mem9`. Renamed in docker-compose.yml: - `CREATE DATABASE IF NOT EXISTS mem9;` - `mysql ... -D mem9` for schema load + tenant seed - seeded tenant `db_name='mem9'` - `MNEMO_DSN=...@tcp(tidb:4000)/mem9?parseTime=true` (replace_all also caught the DSN — the env-var prefix `MNEMO_` is the read-side internal name and stays unchanged) Also cleaned up the top-of-file usage comment that still pointed at the pre-multi-stage workflow (`make build-linux && make docker REGISTRY=local COMMIT=dev`) from before 5badb54. Replaced with the actual one-liner: `docker compose up --build`. Verified by grep: nothing in server/ hardcodes `mnemo` as a DB name (server reads the DB from MNEMO_DSN), so this rename is purely cosmetic for compose surface area. The binary inside the image is still `/mnemo-server`, matching server/Dockerfile / server/Dockerfile.local and the existing release pipeline. feat(local): add .env.example + thread compose vars through it @tmgg06 (msg=c937526b) asked for a .env.example. @rossi (msg=a57590a3) shaped the design — surface all the knobs an operator actually wants through compose's auto-loaded `.env`, with `${VAR:-default}` substitution inside docker-compose.yml so defaults still work for operators who skip the `cp .env.example .env` step. New `.env.example` covers: - Image: MEM9_IMAGE (use prebuilt instead of in-place build) - Host ports: MEM9_PORT, TIDB_PORT (re-map if 8080/4000 are taken) - Local DB + key: MEM9_DB_NAME, MEM9_LOCAL_API_KEY (keep schema-init seed and DSN in sync via a single source of truth) - Server toggles (verified to exist in server/internal/config/config.go): - MNEMO_DB_BACKEND=tidb - MNEMO_FTS_ENABLED=false - MNEMO_DISABLE_SESSION_SAVE=false - MNEMO_INGEST_MODE=raw - LLM/embed knobs (passed through empty by default so server stays in no-key code paths): MNEMO_LLM_*, MNEMO_EMBED_*, MNEMO_EMBED_AUTO_* Verified each env name against `config.go`: - MNEMO_LLM_TIMEOUT was on @tmgg06's wish list (msg=7d072cf8) but is NOT read anywhere in server/internal — llm package hardcodes 120s. Per his follow-up (msg=3a60aaba) and Rossi's confirmation, omitted to avoid the false-promise problem. Adding it would also require a Config.LLMTimeout field + llm.Client plumbing. docker-compose.yml now: - `cp .env.example .env` workflow documented in the file header - All operator-facing strings (ports, DB name, seed key, ingest mode, LLM/embed keys, FTS toggle, session-save toggle, db backend) are `${VAR:-default}` substituted - schema-init reads `DB_NAME` and `API_KEY` from compose env vars derived from `.env`, so the tenant row + the mem9 service's DSN always reference the same database name and API key (no risk of schema-init seeding tenant_id='local-dev' while the DSN points at `mnemo` instead of `mem9`) README updates the quickstart to the `cp .env.example .env` + `docker compose up --build` pair and briefly lists the most-used knobs. fix(local): surface MNEMO_TIDB_ZERO_ENABLED in .env.example + thread it through compose @tmgg06 (msg=d0fb707a) noticed `MNEMO_TIDB_ZERO_ENABLED` wasn't in .env.example even though docker-compose.yml relies on it being false. Rossi (msg=caa9a485) proposed surfacing it with a clear "do not flip" warning, which is what I'm taking. Why this var matters: server defaults it to true (config.go:182) which registers the TiDB Zero provisioner, and any POST /v1alpha1/mem9s call then tries to hit https://zero.tidbapi.com. The compose stack is designed to be self-contained (single seeded tenant), so the var must stay false locally — letting an operator silently override it to true would break self-containment without an obvious failure mode. Changes: - `.env.example`: add MNEMO_TIDB_ZERO_ENABLED=false in the "Core DB / search toggles" block with the strongest warning comment in the file (explicit "MUST stay false" + describes the failure mode). - `docker-compose.yml`: change `MNEMO_TIDB_ZERO_ENABLED: "false"` from hardcoded to `${MNEMO_TIDB_ZERO_ENABLED:-false}`. Default stays safe; an operator who explicitly wants to test the provisioner path (e.g. to validate a TiDB Cloud Pool setup) can flip it via .env rather than editing compose. No other variables changed. Single-tenant local compose contract from 6f22dd6 unchanged. fix(local): bind published ports to loopback by default @magallan's review of the compose branch (msg=04e718ab) flagged that `${TIDB_PORT:-4000}:4000` / `${MEM9_PORT:-8080}:8080` publish on all interfaces. Combined with the stack's dev-only security posture — root TiDB user with empty password, a seeded API key that is public knowledge because it ships in this repo — running the stack on a machine with a public or LAN-facing NIC would expose both the DB and the API to the network. Both port mappings now default to 127.0.0.1: - "${TIDB_BIND:-127.0.0.1}:${TIDB_PORT:-4000}:4000" - "${MEM9_BIND:-127.0.0.1}:${MEM9_PORT:-8080}:8080" MEM9_BIND / TIDB_BIND are surfaced in .env.example with a warning about when overriding is reasonable. Zero impact on the local dev loop (everything in the README uses localhost), pure risk reduction for the misuse case. This was Magallan's preferred mitigation over removing the default `local-dev` API key (his argument: loopback binding addresses the realistic failure mode — dev stack on a network-exposed machine — while keeping the one-command UX; explicit-key forcing punishes every operator for a risk loopback already closes). Default-key question left open for @tmgg06 if he still wants Option B on top. fix(local): README smoke search uses ?q= not ?query= @Kaltsit's review (P1, msg=b5ac7384): the smoke GET used ?query=smoke but the handler reads q.Get("q") (server/internal/handler/memory.go:688), so the command silently fell through to list mode — it would look green after the POST while never exercising the search path at all. A smoke test that can't fail on broken search is worse than none. One-character family fix: ?query=smoke → ?q=smoke, matching the existing docs and tests elsewhere in the repo. chore(local): bump TiDB image v8.4.0 → v8.5.6 Per @tmgg06's request in #mem9-pr-review:af8cf80a, with @Kaltsit / @magallan agreement: v8.5.6 is the newer LTS patch release, a better base for validating VECTOR(1536)-on-unistore behavior than v8.4.0, and reduces exposure to fixed-in-8.5 bugs. Pure version bump — both docker-compose.yml and the README quickstart reference updated. Smoke gating unchanged: docker compose up --build + schema-init clean of VECTOR errors + README's 3 curl checks.
you06
force-pushed
the
feat/local-docker-compose
branch
from
June 10, 2026 06:25
3c87c77 to
7caf49e
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7caf49ecd0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Code comments should describe the code, not where it was discussed. Removed the tracking-thread / attribution lines from server/Dockerfile.local and docker-compose.yml; the design background lives in the PR discussion. No functional change.
The compose build uses the repo root as Docker build context. Without a .dockerignore the context tarball includes .env — harmless with a local daemon, but with a remote Docker context or BuildKit cloud builder any secrets the operator put in .env would be transmitted off-machine. The final image was never affected (Dockerfile.local only copies server/), this closes the context-transfer channel. Scope intentionally minimal per review discussion: just .env and .git for now; build-context size trimming (site/, dashboard/, node_modules/) left for later if anyone cares.
.gitignore tracks three secret files (.env, .env.local, .publish.env); a bare .env entry left the other two in the build context. .env* covers all three. .env.example is also matched, which is fine — the build doesn't copy it.
Per review decision: enumerate .env / .env.local / .publish.env (the three secret files .gitignore tracks) rather than the .env* glob. Same protection, more readable, and .env.example stays in the build context where it's harmless either way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a self-contained local development stack so anyone can run a real
mem9 server with one command — no Go toolchain, no external TiDB Cloud:
Services:
tidb— pingcap/tidb:v8.5.6, unistore single-process modeschema-init— one-shot job: waits for TiDB, loads server/schema.sql,seeds a
local-devtenant so the v1alpha2 X-API-Key path worksimmediately
mem9— built in-place from the new server/Dockerfile.local(multi-stage Go build; release pipeline's server/Dockerfile untouched)
Security defaults: published ports bind to 127.0.0.1 only (root
no-password TiDB + a public seed key must not face a network), TiDB
Zero provisioning disabled, ingest mode
rawso no LLM key is needed.All operator knobs (image, ports, bind address, DB name, API key,
ingest/LLM/embedding config) are documented in .env.example.