diff --git a/docs/SELF_HOSTING_COMPOSE_PLAN.md b/docs/SELF_HOSTING_COMPOSE_PLAN.md new file mode 100644 index 0000000000..2f38a3a096 --- /dev/null +++ b/docs/SELF_HOSTING_COMPOSE_PLAN.md @@ -0,0 +1,442 @@ +# Docker Compose self-hosting plan + +## Goal and scope + +The self-hosted edition should have one boring entry point: + +```sh +docker compose up -d +``` + +That command should produce a usable, persistent Macro installation with a web +UI, local sign-in, documents, channels/messages, uploads, and live updates. An +operator should be able to add capabilities with Compose profiles without +replacing the base installation or moving its data. + +This is a proposal, not a claim that the command works today. In particular, +this plan does not attempt to reproduce Macro's production AWS topology. It +selects local substitutes and explicitly labels features that still require an +external provider. + +## What the repository supports today + +There are two useful but different foundations: + +1. `docker/docker-compose.yml` describes most application containers. It + includes Postgres, Redis, Kafka, OpenSearch, and FusionAuth definitions and + builds the Rust, sync, lexical, and worker images. +2. `just stack up --no-doppler` is the closest thing to a working self-contained + product. The xtask layer generates secrets and Compose overrides, builds and + stages the frontend, mounts host-built Rust binaries, provisions LocalStack, + writes the FusionAuth kickstart and Caddy configuration, runs migrations, + initializes search, and waits for readiness. `docs/RUNNING_LOCALLY.md` + documents this path. + +This demonstrates that the main product can run without Doppler and without +real AWS. It also shows which orchestration must be moved into images and +one-shot Compose services before plain Compose is sufficient. + +### What plain Compose currently provides + +| Area | Present in the checked-in Compose graph | Usable after only `docker compose up`? | +| --- | --- | --- | +| Postgres, Redis, Kafka, OpenSearch | Yes, through `docker/docker-compose-databases.yml` | No. Networks and volumes are declared `external`, so a fresh host must create them first; application schemas and search indices are not initialized by Compose. | +| FusionAuth | Yes, through an included Compose file | No. Its volumes and `auth` network are external, and the generated kickstart/client configuration is normally prepared by xtask/Pulumi-oriented setup. | +| Local AWS substitutes | No base `localstack` service | No. The just recipes start an out-of-band container and use the host AWS CLI to create buckets, queues, tables, CORS, policies, and S3 notifications. | +| Rust APIs and workers | Mostly | Not reliably. They need a large resolved environment and initialized dependencies. The bundle image also omits `search_processing_service`, although Compose tries to execute it. | +| Document editing | `sync_service` and `lexical_service` are present | Only after secrets, database initialization, auth, and routing are supplied. The lexical image additionally needs `GITHUB_PACKAGES_TOKEN` at build time. | +| Browser application | No | No. The base file exposes service ports but has neither a built web application nor the generated Caddy single-origin proxy. | +| Passwordless email | FusionAuth is present | No convenient flow. The working local stack adds Mailpit; base Compose does not. | +| Uploads/static files | Relevant services are present | No. They depend on absent LocalStack resources and initialization. | +| Search | OpenSearch and the embedded DSS search API are present; an indexing worker is declared | No. Index creation and the complete event/indexing path are orchestrator work today. | +| Gmail/Google/GitHub/Stripe | Code paths are present | No with stub credentials, by design. These require real provider credentials and callback configuration. | +| AI editing/chat | Cognition and AI editing services are present | No useful model responses without a supported model provider and credentials. | +| Calls, push, production email delivery, cloud CDN | Some domain/API support exists | Not as local equivalents. These depend on provider infrastructure or services not represented by the base graph. | + +### Why the current command is not a self-hosting interface + +A clean machine fails before product behavior can be evaluated: + +- the included files expect pre-created `databases` and `auth` networks and + named external volumes; +- LocalStack is created outside Compose; +- provisioning is a collection of `just`, AWS CLI, migration, FusionAuth, and + search-init steps; +- application configuration normally comes from a generated env file; +- the browser UI and reverse proxy are generated/staged by xtask, not declared + in the base file; +- `depends_on` mostly expresses start order, not readiness or successful + initialization; and +- all unprofiled services start, including expensive or credential-dependent + services, so the default is neither minimal nor predictably healthy. + +The correct strategy is therefore to productize the already-working local +orchestrator decisions, not to expose every production microservice by default. + +## Capability tiers + +Tiers are cumulative. A higher tier adds containers and configuration while +retaining the same database and object-store volumes. + +### Tier 0 — evaluation shell + +**Command:** `docker compose --profile demo up -d` + +**Adds:** `web`, `gateway`, and an optional immutable demo dataset or seed job. + +**Works:** UI exploration with sample content. This may be useful for a very +small prebuilt image distribution, but it is not a real multi-user deployment. + +**Does not work:** durable edits, signup, uploads, integrations, or email. + +This tier is optional. Do not make it the default because a surprising +read-only product is worse than a small functional product. + +### Tier 1 — core self-hosted product (default) + +**Command:** `docker compose up -d` + +**Containers:** + +- `gateway` (Caddy or nginx) serving the web build and reverse-proxying one + origin; +- `postgres`, `redis`, `localstack`, `fusionauth-db`, `fusionauth`, and + `mailpit`; +- `init-secrets`, `init-localstack`, `migrate`, and `init-auth` one-shot jobs; +- `authentication-service`, `connection_gateway`, `document_storage_service`, + `document_upload_finalizer`, `static_file_service`, `static_file_cdn`, + `sync_service`, and `lexical_service`; and +- optionally `contacts_service`, `unfurl_service`, and `image_proxy_service` if + the UI cannot cleanly feature-detect them. They are low-cost compared with + search and AI. + +**Works:** local signup/login (codes in Mailpit), user/team basics, channels and +messages backed by MacroDB, document creation/editing/sync, permissions, +uploads/downloads, static assets, and websocket live updates. + +**Intentionally unavailable:** full-text search/indexing, linked external +inboxes, real outbound mail, AI actions, billing, push notifications, and +production observability. + +**Resource target:** document an initial target of 4 CPU, 8 GiB RAM, and 20 GiB +disk, then measure it in CI. The target is a budget to validate, not a current +guarantee. + +### Tier 2 — local search and discovery + +**Command:** `docker compose --profile search up -d` + +**Adds:** `opensearch`, `kafka`, `search_processing_service`, search bootstrap, +and any queue consumers required to turn document/email events into index +updates. + +**Works:** indexed title/content search, search filters, and reindexing. PDF +content search only works after the relevant extractor/converter worker is +packaged; the local no-default-features build currently drops PDF processing and +must be labeled accordingly. + +**Operational requirement:** include a repeatable `search-reindex` job and pin +OpenSearch rather than using a floating image. Budget another 2–4 GiB RAM. + +### Tier 3 — communications and notifications + +**Command:** `docker compose --profile communications up -d` + +**Adds:** `email_service`, `email_pubsub_workers`, `notification_service`, and +the necessary LocalStack queues. If scheduled reminders are included, add a +small scheduler container rather than requiring the operator to run the current +manual minute loop. + +**Works without external credentials:** locally injected/test email, email +thread UI over seeded or imported data, in-app notifications, and Mailpit +delivery. + +**Works after adding provider secrets:** Gmail account linking/sync and real +outbound SMTP/transactional delivery. Provider-backed functions must expose a +clear disabled state when credentials are absent rather than failing service +startup. + +### Tier 4 — AI and document intelligence + +**Command:** `docker compose --profile ai up -d` + +**Adds:** `document_cognition_service` and `ai_editing_worker`; optionally local +embedding/inference containers in a separate `local-models` profile. + +**Works after adding credentials:** chat/completions, AI document editing, +summarization, and other tools whose dependencies are included in lower tiers. + +**Does not automatically become local:** a cloud model remains an external +dependency unless a supported local model adapter is implemented. Compose +should validate one of `OPENAI_API_KEY`, another supported provider, or a local +model endpoint before starting this profile. + +### Tier 5 — external integrations + +**Commands:** profiles should be provider-specific, for example +`--profile gmail`, `--profile smtp`, `--profile github`, or `--profile stripe`. + +**Adds:** only provider-specific relays/workers. Most value comes from enabling +configuration on existing services, so these profiles may primarily select +validation jobs and documentation rather than duplicate services. + +**Works:** the selected integration after its keys, public callback URL, TLS, +and provider console configuration are supplied. + +**Boundary:** calls/media infrastructure, mobile push, enterprise SSO, cloud +CDN, and production billing should remain explicitly unsupported until each has +a documented local adapter and an end-to-end test. Do not imply that “all +profiles” equals production parity. + +### Operations add-ons (orthogonal) + +These are not a product tier: + +- `--profile observability`: Jaeger and optional metrics/log collection; +- `--profile backup`: scheduled `pg_dump` plus object-store backup; +- `--profile edge`: TLS ingress for a configured hostname; and +- `--profile devtools`: Redis UI, Mailpit UI, and direct database/search ports. + +Only the gateway should publish a host port by default. Database, Redis, +LocalStack, FusionAuth, and internal APIs should be private Compose-network +services. This is both safer and easier to deploy remotely. + +## Proposed Compose layouts + +There are three reasonable representations. Use the first as the public +interface and keep the other two for development or advanced operators. + +### Configuration A — one `compose.yml` with profiles (recommended) + +Place a root `compose.yml` at the conventional path. Keep infrastructure, +initializers, and Tier 1 unprofiled. Put optional services behind one or more +profiles: + +```yaml +name: macro + +services: + gateway: { profiles: [] } + web: { profiles: [] } + postgres: { profiles: [] } + redis: { profiles: [] } + localstack: { profiles: [] } + migrate: { profiles: [] } + init-localstack: { profiles: [] } + fusionauth: { profiles: [] } + init-auth: { profiles: [] } + authentication-service: { profiles: [] } + connection_gateway: { profiles: [] } + document_storage_service: { profiles: [] } + sync_service: { profiles: [] } + lexical_service: { profiles: [] } + static_file_service: { profiles: [] } + + opensearch: { profiles: [search] } + kafka: { profiles: [search] } + search_processing_service: { profiles: [search] } + + email_service: { profiles: [communications] } + email_pubsub_workers: { profiles: [communications] } + notification_service: { profiles: [communications] } + + document_cognition_service: { profiles: [ai] } + ai_editing_worker: { profiles: [ai] } + + jaeger: { profiles: [observability] } +``` + +Compose automatically starts dependencies of explicitly selected services, but +profile dependencies can be subtle. Every supported invocation must be checked +with `docker compose config` and an end-to-end smoke test. + +Advantages: the advertised command is literal, discovery is easy, and upgrades +have one versioned graph. Disadvantage: the file becomes long. Manage that with +YAML anchors and generated documentation, not nested includes that make the +default entry point dependent on repository layout. + +### Configuration B — base plus capability overlays + +Keep `compose.yml` as Tier 1 and publish additive files: + +```sh +docker compose up -d +docker compose -f compose.yml -f compose.search.yml up -d +docker compose -f compose.yml -f compose.search.yml \ + -f compose.communications.yml up -d +``` + +This provides the cleanest diffs and lets an operator audit exactly what a tier +adds. It is also suitable if Compose profile merging proves awkward. Its main +cost is command length and a larger compatibility test matrix. Never require an +overlay to *remove* a broken default service; overlays should only add or +override deliberate configuration. + +### Configuration C — production-oriented external dependencies + +Provide `compose.external.yml` for operators who already run PostgreSQL, Redis, +S3-compatible storage, OpenSearch, OIDC, SMTP, and an ingress controller: + +```sh +docker compose -f compose.yml -f compose.external.yml up -d +``` + +The override disables bundled stateful services and reads explicit URLs from +Docker secrets/environment. This is an advanced configuration and should be +tested separately. It is not a reason to make the default depend on external +services. + +## Required implementation work + +### 1. Make images distributable + +- Publish immutable multi-architecture images for the web app, gateway config, + every Tier 1 service, and each optional worker. Pin third-party images by + version (ideally digest). +- Remove build-time dependence on private package credentials from the default + path. Vendor/build the lexical artifact in CI or publish its image. +- Make the Rust bundle inventory and Compose inventory share one checked source + of truth so Compose cannot request a missing executable. +- Generate an SBOM and document image provenance and upgrade compatibility. + +Building 80+ crates on a small self-hosting machine is not an acceptable first +start. `build:` may remain as a contributor override, but released Compose must +default to `image: ghcr.io/...:${MACRO_VERSION}`. + +### 2. Replace host orchestration with idempotent init jobs + +Add one-shot services with health/exit conditions: + +1. `init-secrets` creates persistent internal signing keys when absent and + refuses insecure known defaults outside an explicit demo mode. +2. `migrate` runs all required MacroDB migrations under a Postgres advisory + lock. +3. `init-localstack` creates buckets, queues, DynamoDB tables, policies, CORS, + and notifications idempotently. +4. `init-auth` supplies a versioned FusionAuth kickstart and emits/records the + client configuration consumed by auth and web. +5. `init-search` creates mappings/templates; `search-reindex` is separately + operator-invoked. + +Application services should wait for healthy dependencies and successful init +jobs using `condition: service_healthy` and +`condition: service_completed_successfully`. Every init job must be safe after +restart and upgrade. + +### 3. Ship a single origin + +Turn the generated Caddy routing table into a checked-in or image-generated +production template. The gateway must: + +- serve the static web bundle at `/app/` (and redirect `/`); +- proxy the existing `/auth`, `/dss`, `/sync`, `/lexical`, `/static-file`, and + other enabled routes; +- support websocket upgrades; +- set forwarded headers and request-size/time-out limits suitable for uploads; +- hide internal service ports; and +- optionally terminate TLS for `MACRO_PUBLIC_URL`. + +The frontend should render a capability response from the backend. Disabled +profiles should hide or clearly disable their UI, not generate repeated network +errors. + +### 4. Define a small, stable configuration contract + +Commit `.env.example` with only operator decisions, for example: + +```dotenv +MACRO_VERSION=1.0.0 +MACRO_PUBLIC_URL=http://localhost:3000 +POSTGRES_PASSWORD=change-me +MACRO_ADMIN_EMAIL=admin@example.com +# Optional: SMTP_URL, OPENAI_API_KEY, GOOGLE_CLIENT_ID, ... +``` + +Derive internal URLs inside Compose. Store passwords and generated signing keys +as Docker secrets/files, not in image layers. Fail with a specific message for +missing required settings; tolerate absent optional integrations. Doppler must +not be part of the self-hosted runtime contract. + +### 5. Persistence, upgrades, and recovery + +- Use Compose-managed named volumes by default, not `external: true` volumes. +- Document the authoritative data sets: PostgreSQL and S3/object storage are + backed up; Redis, search indices, Kafka, and queues should be reconstructible + where possible. +- Provide `backup`, `restore`, `migrate`, and `reindex` runbooks and containers. +- Test rollback limitations and publish a supported upgrade sequence. Never + silently run a destructive migration on startup. +- Add storage health and disk-space guidance. + +### 6. Security baseline + +- Bind only the gateway by default and remove development UIs/public ports. +- Generate unique secrets, disable development modes, configure secure cookies, + validate the public origin, and support trusted reverse-proxy CIDRs. +- Run containers as non-root with read-only root filesystems where feasible; + use dropped capabilities and resource limits. +- Do not ship dummy AWS/provider credentials as though they were secure + credentials. LocalStack's dummy credentials are acceptable only on its + isolated network. +- Document that public internet exposure requires TLS, backups, mail abuse + controls, patching, and monitoring. + +## Delivery plan and acceptance gates + +### Phase 1 — prove the minimal graph + +1. Capture the services actually used by a clean local E2E run. +2. Add capability flags to the frontend and disable search, email, AI, billing, + and provider integrations. +3. Establish Tier 1 E2E scenarios: first admin signup, second user invite/login, + document edit/reload, upload/download, channel message, and websocket update. +4. Record CPU, memory, disk, image download, cold-start, and restart behavior. + +**Exit gate:** those scenarios pass with no host tools except Docker Compose and +`curl`, on both amd64 and arm64. + +### Phase 2 — create the released Compose artifact + +1. Publish images and root `compose.yml` plus `.env.example`. +2. Move generated proxy, LocalStack, auth, migration, and search setup into + images/init jobs. +3. Remove external volumes/networks and unnecessary host port bindings. +4. Add healthchecks, dependency conditions, restart policies, and a + `docker compose ps` troubleshooting table. + +**Exit gate:** on an empty Docker host, the exact README command reaches a +healthy app; repeating it is a no-op; `docker compose down` preserves data; and +restarting the host preserves login, edits, and uploads. + +### Phase 3 — add and verify profiles one at a time + +Implement `search`, then `communications`, then `ai`. For each profile: + +- add a capability contract and UI state; +- test upgrade from the previous tier without data reset; +- test startup without optional provider secrets; +- document resource deltas and unsupported edges; and +- add a profile-specific E2E workflow. + +**Exit gate:** every documented command is tested from both an empty volume set +and an upgraded lower-tier installation. + +### Phase 4 — operating lifecycle + +Add backup/restore drills, versioned migrations, security hardening, release +notes, and an external-dependencies overlay. Run a quarterly restore test and a +Compose compatibility matrix against current Docker Engine/Compose releases. + +## Recommended public documentation wording + +Until Phase 2 passes, describe the repository as **locally runnable through the +Nix/xtask development orchestrator**, not as `docker compose up` self-hostable. +After Phase 2, advertise: + +> `docker compose up -d` runs Macro Core. Search, communications, AI, and +> provider integrations are optional profiles with separate resource and +> credential requirements. Self-hosted does not currently include every managed +> Macro feature. + +That sets a useful, testable promise while leaving room to expand the product +incrementally.