Skip to content

feat: measure Schaffa usage and catch errors with self-hosted Umami and GlitchTip - #14

Merged
beastyrabbit merged 11 commits into
mainfrom
feat/umami-glitchtip
Oct 21, 2026
Merged

beastyrabbit merged 11 commits into
mainfrom
feat/umami-glitchtip

Conversation

@beastyrabbit

@beastyrabbit beastyrabbit commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

Until now Schaffa had no idea how its landing page, account flow, or published content was used, and server or browser errors only showed up if someone happened to read the container logs. This PR connects Schaffa to the homelab's self-hosted Umami (analytics) and GlitchTip (error tracking). Analytics only run after a visitor opts in through a new bilingual cookie banner.

What changes for visitors

  • Consent banner (DE/EN) on Schaffa's own public pages: landing, /skills, account and sign-in, and the Impressum and privacy pages. It is self-hosted vanilla-cookieconsent v3 with no CDN, and follows <html lang>. The two categories are necessary and analytics, and "Analyse akzeptieren" / "Nur notwendige" carry equal weight. A Cookie-Einstellungen / Cookie settings link in the shared legal footer reopens it at any time. Before a choice nothing is stored. After a choice only cc_cookie holds the decision, for six months. Withdrawing consent reloads the page with a GET, so a token form is never resubmitted.
  • Umami, after opt-in only: pageviews, web vitals (data-performance), outbound links, and custom events (sign-in start, CTA clicks, quickstart and skill-install copies, token copies). It also records session replay and heatmaps with strict masking: every input, the account workspace and token reveals are blocked. Replay never loads on /account*, on pages that show a secret, or on URLs with a query or hash. Search params, hashes, titles and referrers are stripped before sending. There is no identify.
  • Admin (/admin) gets GlitchTip only: no banner, no Umami.
  • Privacy policy: one new section, Reichweitenmessung und Fehlerüberwachung / Analytics and error monitoring, is added to the existing bilingual page from feat: publish Impressum and privacy policy on schaffa.dev #12. The cookies section names cc_cookie while keeping "without sign-in and without a banner choice nothing is stored" true.
Banner, desktop (EN) Banner, mobile (DE) Preferences (DE) New privacy section (EN)
English consent banner on the privacy page at 1280 px German consent banner at 390 px German cookie settings dialog Analytics and error monitoring section plus the updated cookies section

Published content: counted on the server, never instrumented

Pages, files, guides and presentations under /p, /f and /g belong to the people who upload them. Schaffa promises byte-identical HTML under strict, partly sandboxed CSPs. Injecting a tracker or banner there would break that promise and track readers of third-party content. So nothing is injected, and replay and heatmaps never run on it.

Views are counted server-side instead. After a successful GET, Schaffa sends a fire-and-forget Umami event content-view with the public path and kind. It forwards no visitor IP, User-Agent, referrer or title, and stores nothing on the device, so no consent is needed (Art. 6(1)(f)). Obvious bots are skipped. Guide .json/.md exports and the interactive-page warning screen are not counted. The transport has a 1.5 s timeout, at most 8 requests in flight, no retries, and it drains response bodies, so a slow Umami can never delay serving. The same transport records API outcomes without personal data: content published, deleted or rejected by a scan, uploads rejected by quota, lockdown or rate limit, signup/login success or failure, and tokens created or revoked.

GlitchTip

  • Server: @sentry/node is preloaded through node --import, with the Fastify integration. Only unexpected 5xx errors are reported; deliberate operational states such as storage_quota or interactive_disabled are not. console.error is captured too.
  • Browser: a vendored @sentry/browser bundle is lazy-loaded on idle. A tiny early queue catches errors and unhandled rejections that happen before it loads. User-facing error paths report unexpected exceptions.
  • Privacy: SDK 11 dataCollection disables every personal-data category. Headers, cookies, bodies, query strings, sfa_/sus_ tokens, Bearer values, JWTs and e-mail addresses are removed before sending. No replay and no browser storage, so it runs as technically necessary.
  • Release metadata: environment and release = schaffa@<package version>. Traces are sampled at 10 %, with propagation same-origin only.
  • Test route: admins can trigger POST /admin/api/telemetry-test, which throws "Schaffa GlitchTip test error — safe to ignore".
  • Source maps:
    • The Docker build injects debug IDs and removes the .js.map files from the image, so maps are never shipped or served.
    • On a v* tag, a new sourcemaps job runs after publish. It rebuilds dist deterministically, checks that the debug IDs match the exact GHCR image it just pushed, and uploads to https://glitchtip.heerlab.com with the Infisical-synced GLITCHTIP_AUTH_TOKEN secret.
    • First-party browser scripts are served unminified, so they need no maps.
  • CSP: only the CSPs of Schaffa's own UI gain the Umami and GlitchTip origins, and only when they are configured. User-content CSPs are unchanged.

Everything is off unless UMAMI_URL/UMAMI_WEBSITE_ID and/or GLITCHTIP_DSN are set. That keeps local dev, tests and preview builds out, and self-hosted instances stay telemetry-free by default. PostHog was never used in the app, so there is nothing to remove.

Environments and configuration

Schaffa has one deployment (production), so it gets one Umami website (Schaffa) and one GlitchTip project (beasty-rabbit/schaffa). The public client config goes into the kub-homelab HelmRelease in a separate PR. The upload token lives only in Infisical and the synced repository secret.

Verification

  • Full check: pnpm check passes: typecheck, Biome, 174 tests and gitleaks, with 6 platform-only skips. pnpm build passes.
  • Image: docker build passes. The image contains no *.js.map and keeps the debug IDs. The container starts and /healthz returns 200. A local rebuild produced the same debug IDs as the image.
  • Browser tests run against the built app with stubbed Umami and GlitchTip endpoints. They check that:
    • no Umami request is made before consent, requests start after accepting, and they stop after withdrawal;
    • a thrown error reaches the stubbed envelope endpoint;
    • no cookie or storage exists before a choice, and only cc_cookie after one;
    • a German page gets the German banner;
    • published /p/ HTML stays byte-identical with an unchanged CSP.
  • Unit tests cover the collector (fake fetch), scrubbing, config parsing, CSP composition, the legal sections and footer behaviour on non-schaffa.dev hosts.
  • Live checks: a test event sent with the server User-Agent was accepted and recorded by Umami, so it passes the bot filter.
  • Independent review: a delivery review by Claude Opus 5.5 ran three rounds. All findings were fixed: a POST resubmit on consent withdrawal, ignored span hooks, expected errors being reported, double-counted views, undrained sockets, the CSP changing while telemetry was off, an English-only banner, and a 404 privacy link on self-hosted instances. The final round had no findings.

TODOs and open points

  • Legal text needs human review. It is based on the shared templates and is not legal advice.
  • Retention (user decision, same for all sites): Umami analytics data 25 months, Umami session replays and heatmaps 30 days, GlitchTip error events 90 days. A nightly kub-homelab job enforces the Umami periods, and GlitchTip is pinned to 90 days. The privacy section states exactly these periods in DE and EN.
  • Replay and heatmaps: Umami's recorder config route (/api/websites/<id>/recorder) is now public. Replay and heatmaps get verified live after the release.

Somewhere a GPU is overheating so I don't have to think.

Powered by human calories and mass GPU cycles.
Powered by human calories and mass GPU cycles.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T19:45:47.593826Z 2212667 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Powered by human calories and mass GPU cycles.
Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

The new head fixes the CI audit failure with a scoped brace-expansion override and resolves the 21 SonarQube quality-gate issues; behaviour is unchanged. It also updates the privacy retention periods to the ones the kub-homelab deletion job enforces: Umami analytics 25 months, replays and heatmaps 30 days, GlitchTip 90 days.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c1c6cd14f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/telemetry.ts Resolved
Comment thread src/telemetry.ts Resolved
Comment thread public/scripts/telemetry.js Resolved
Comment thread src/scan-worker.ts Resolved
Comment thread public/scripts/telemetry.js Resolved
Comment thread src/telemetry-scrub.ts Resolved
Comment thread src/instrumentation.ts Resolved
Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 7268749 fixes all seven findings from the previous review.

Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 8375037 adds one privacy-text change: the API documentation at /api (self-hosted Scalar) stores the selected code-example language in local storage, and Scalar has no option to turn that off, so the cookies section now says so.

@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8375037c40

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/instrumentation.ts Resolved
Comment thread src/telemetry-scrub.ts Resolved
Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 28f5590 fixes both findings from the previous review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 28f559040b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/telemetry-scrub.ts Resolved
Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head f44b411 replaces the email allowlist with conservative whole-token redaction.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f44b411ce3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/telemetry-scrub.ts Resolved
Comment thread public/scripts/telemetry.js Resolved
Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 306824a fixes both findings from the previous review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 306824a856

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/telemetry-scrub.ts Resolved
Comment thread public/scripts/telemetry.js Resolved
Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 3b33f85 fixes both findings from the previous review.

Powered by human calories and mass GPU cycles.
@beastyrabbit

Copy link
Copy Markdown
Owner Author

Note

🤖 Claude Opus 5.5 responding on behalf of beastyrabbit

@codex review

New head 2212667 removes the duplicated browser scrubber, which pushed SonarQube new-code duplication to 3.6%. Server and browser now share one implementation (src/telemetry-scrub-core.ts), bundled for the browser by scripts/build-telemetry.mjs. The previous head 3b33f85 contains the fixes for both findings from the last review.

@sonarqube-skyway-gmbh

Copy link
Copy Markdown

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 22126671b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@beastyrabbit
beastyrabbit merged commit 5951896 into main Oct 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant