[WIP, NO MERGE]: Added recurring job to sync automation runs to Tinybird

[troyciesco]

ref https://linear.app/ghost/issue/NY-1559/ghost-backend-send-events-to-traffic-analytics-service-with-recurring

This cannot be merged as-is because it includes database migrations, so it will need to be broken up further before review.

This probably will actually replace its parent pr #30441 instead of getting merged into it, but i wanted to take it one step at a time and leave the stack we were working from intact for now.

• Replaces the boot.js proof of concept with a tinybird-sync service scheduled through the jobs service every five minutes, with a per-process random offset
• Adds a tinybird_syncs table storing a per-table watermark (last_synced_updated_at), plus updated_at indexes on automation_runs and automation_run_steps so each run only scans recently changed rows
• Sends only the columns the Tinybird materialized views read; member id, member email, and lock bookkeeping never leave MySQL
• Reads rows in keyset-paginated batches and splits requests by UTF-8 byte size under Tinybird's Events API limit
• Uses wait=true so Tinybird acknowledges committed rows, fails the job on quarantined rows or HTTP errors without advancing the watermark, and aborts stalled requests after a timeout
• Holds back rows updated within the last minute so a still-open transaction cannot be skipped by the watermark
• Syncs both tables concurrently and guards against overlapping runs
• Adds a backgroundJobs.tinybirdSync config flag to disable the job
• Reverts the WIP changes that read stats from Tinybird; that work belongs to NY-1560
• Covers the sync logic with unit tests against an in-memory SQLite database and a fake Tinybird Express server

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

• ⚠️ Tested performance on staging database servers, as performance on local machines is not comparable to a production environment
• Satisfies idempotency requirement (both up() and down())
• Does not reference models
• Filename is in the correct format (and correctly ordered)
• Targets the next minor version
• All code paths have appropriate log messages
• Uses the correct utils
• Contains a minimal changeset
• Does not mix DDL/DML operations

Schema changes

• Both schema change and related migration have been implemented
• For index changes: has been performance tested for large tables
• For new tables/columns: fields use the appropriate predefined field lengths
• For new tables/columns: field names follow the appropriate conventions
• Does not drop a non-alpha table outside of a major version

Data changes

• Mass updates/inserts are batched appropriately
• Does not loop over large tables/datasets
• Defends against missing or invalid data
• For settings updates: follows the appropriate guidelines

[nx-cloud]

🤖 Nx Cloud AI Fix

Ensure the fix-ci command is configured to always run in your CI pipeline to get automatic fixes in future runs. For more information, please see https://nx.dev/ci/features/self-healing-ci

---

View your CI Pipeline Execution ↗ for commit 06b12e5

Command	Status	Duration	Result
nx run ghost:test:ci:integration	❌ Failed	4m 20s	View ↗
nx run ghost:test:integration	❌ Failed	2m 24s	View ↗
nx run @tryghost/admin:test:acceptance	✅ Succeeded	7m 59s	View ↗
nx run ghost:test:ci:e2e	✅ Succeeded	3m 59s	View ↗
nx run ghost:test:legacy	✅ Succeeded	3m 9s	View ↗
nx run ghost:test:e2e	✅ Succeeded	2m 8s	View ↗
nx run ghost-monorepo:lint:boundaries	✅ Succeeded	24s	View ↗
nx run-many -t test:unit -p @tryghost/admin,@tr...	✅ Succeeded	42s	View ↗
Additional runs (6)	✅ Succeeded	...	View ↗

💡 Dealing with memory or CPU issues? See memory and CPU details with the resource usage add-on ↗.

---

☁️ Nx Cloud last updated this comment at 2026-09-02 19:56:41 UTC