Skip to content

JITSU-108: build console as a production dist in the dev chart - #1515

Open
sahiltyagi-jitsu wants to merge 2 commits into
newjitsufrom
feat/jitsu-108-console-prod-build
Open

sahiltyagi-jitsu wants to merge 2 commits into
newjitsufrom
feat/jitsu-108-console-prod-build

Conversation

@sahiltyagi-jitsu

Copy link
Copy Markdown
Contributor

Closes JITSU-108.

The dev chart ran the console as next dev behind a background loop that rsynced webapps/, libs/ and types/ from the host mount every 2 seconds. That made it the largest memory consumer in the dev stack, and the hot-deploy was unreliable — races, missed changes, stale .next — on top of an apt-get install rsync in every console pod.

This builds the console from the mounted source with next build in an init container and serves it with next start — the same model bulker and rotor already use: compile in an init container, run the output in the main container. Rebuild happens on redeploy.

Measured

Minikube, before and after, same cluster and session.

Before (next dev) After (next start)
cgroup current 2,165 MiB 415 MiB −81%
cgroup peak 2,499 MiB 416 MiB −83%
next-server RSS 1,418 MiB 333 MiB −76%

The trade

The cost moves from runtime to deploy rather than simply growing:

Before After
Pod created → Ready 1m 01s 2m 20s
First hit on / 3.48s 0.004s
First hit on /signin 0.40s 0.021s
Subsequent hits ~0.007s ~0.002s

next dev compiles routes lazily, so "Ready" only meant the healthcheck route had compiled — every other page paid a compile on first visit, after every restart.

Build time falls as .next warms on the node-cache PVC: 2m 39s cold, then 1m 58s, then 1m 15s.

You only pay it when console is redeployed, and not at all in host-UI mode (scaling.console.replicas: 0) where the Deployment isn't rendered.

What this gives up

In-cluster hot reload. For console UI iteration use pnpm console:dev on the host — faster and lighter than any in-cluster dev server. Documented in a new "Developing the console" section in helm/README.md.

One behaviour change worth knowing: a build failure now fails the init container, so the pod won't start rather than showing the error in the browser. ./dev-deploy.sh build-logs console works for console now — the init container is named build, matching bulker/rotor.

Kept deliberately

The seed retry and the events-log-init loop. Nothing else in the dev chart creates the ClickHouse events-log tables that power Live Events — in prod the admin/events-log-init route does it.

Also drops the now-unused /project hostPath mount from the console pod.

Verified

  • next build succeeds in the init container with the console env
  • seed and events-log-init both still run (events-log ClickHouse tables ready)
  • /api/healthcheck → 200, Ready in 75ms
  • ./dev-deploy.sh restart console rebuilds and comes back healthy
  • .next persists across restarts, so rebuilds are incremental

Notes

Standalone output (NEXTJS_STANDALONE_BUILD=1) was left unexplored — at an 81% reduction it isn't worth the extra moving part.

The ticket also asks for the onboarding doc to point console iteration at pnpm console:dev. That lives outside this repo and isn't covered here.

Distinct from JITSU-48 (prod mode = published images for self-hosting); that one runs console from an image rather than building it. Both need next build, so this lands the build plumbing first.

🤖 Generated with Claude Code

The dev chart ran the console as `next dev` behind a background rsync loop
that re-synced webapps/, libs/ and types/ from the host mount every 2
seconds. That made it the largest memory consumer in the dev stack, and the
rsync hot-deploy was unreliable — races, missed changes, stale .next — on top
of an apt-get install in every console pod.

Build the console from the mounted source with `next build` in an init
container and serve it with `next start`, the same model bulker and rotor
already use: compile in an init container, run the output in the main
container. Rebuild happens on redeploy. The init container is named `build`,
so `./dev-deploy.sh build-logs console` now works for console too.

Measured on minikube, before and after, same cluster and session:

  cgroup current   2,165 MiB -> 415 MiB   (-81%)
  cgroup peak      2,499 MiB -> 416 MiB   (-83%)
  next-server RSS  1,418 MiB -> 333 MiB   (-76%)

The cost moves from runtime to deploy rather than simply growing:

  pod created -> Ready   1m 01s -> 2m 20s
  first hit on /         3.48s  -> 0.004s
  first hit on /signin   0.40s  -> 0.021s

`next dev` compiles routes lazily, so Ready only meant the healthcheck route
had compiled; every other page paid a compile on first visit, after every
restart. Build time falls as .next warms on the node-cache PVC: 2m 39s cold,
then 1m 58s, then 1m 15s.

In-cluster hot reload is gone. For console UI iteration use `pnpm console:dev`
on the host with `scaling.console.replicas: 0`, which is faster and lighter
than any in-cluster dev server — documented in helm/README.md. In that mode
the console Deployment is not rendered at all, so no build runs either.

The seed retry and the events-log-init loop are both kept: nothing else in the
dev chart creates the ClickHouse events-log tables that power Live Events.
Also drops the now-unused /project hostPath mount from the console pod.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jitsu-code-review[bot]
jitsu-code-review Bot previously approved these changes Sep 14, 2026

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the console Helm deployment change and its source/cache lifecycle.\n\nFinding:\n- Console restarts rebuild stale cached sources, so the documented post-edit workflow does not apply changes.

Comment thread helm/templates/console.yaml
The install job is a pre-install/pre-upgrade helm hook, so it only runs on
deploy or upgrade. `dev-deploy.sh restart console` is a plain rollout restart
and never triggers it, which left /cache/workspace holding whatever source was
last copied. The rsync loop this PR removes had been keeping it fresh, so
dropping it meant a restart silently rebuilt stale code — and the README told
people to use exactly that command after editing.

Sync webapps, libs and types from /project before building, the same reason
bulker's init container copies from /project before it compiles. Dotfiles are
excluded, so the cached .next survives and a stale host build is never copied
over it.

Verified on minikube: wrote a marker file into webapps/console, ran
`./dev-deploy.sh restart console`, and read it back from
/cache/workspace/webapps/console in the new pod.

Caught by jitsu-code-review on #1515.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sahiltyagi-jitsu

Copy link
Copy Markdown
Contributor Author

Good catch from the review — that was a real bug, now fixed in 587ded83a.

install-job is a pre-install,pre-upgrade helm hook, so it only runs on deploy or upgrade. ./dev-deploy.sh restart console is a plain kubectl rollout restart and never triggers it — so /cache/workspace kept whatever source was last copied. The rsync loop this PR removes had been papering over that, and dropping it meant a restart would silently rebuild stale code. Worse, the README section I added tells people to use exactly that command after editing.

The init container now syncs webapps, libs and types from /project before building — the same reason bulker's init container does cp -r /project/bulker/* /src/ before it compiles. Dotfiles are excluded, so the cached .next survives and a stale host build is never copied over it.

Verified on minikube rather than reasoned about: wrote a marker file into webapps/console, ran ./dev-deploy.sh restart console, and read it back from /cache/workspace/webapps/console in the new pod.

@jitsu-code-review jitsu-code-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the console Helm deployment change and its developer documentation, including the rendered template and the production build/start flow. I also checked the existing review thread; the source re-sync in the build init container addresses that concern. No additional actionable findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant