fix(ci): run E2E Mongo as single-node replica set (fixes P2031) - #495
angeloreale wants to merge 17 commits into
Conversation
…handle namespace
- Phase 5: Project entity (user-owned, /p/[username], spotlight, supportUrl,
stats in serializer), List.projectId as project job boards, task deep link
/app/do/list/{listId}/{taskId} (first + highlighted), likes entityType
'project'; no new migrations (0020 shipped with Phase 1)
- Shared /@ handle namespace across users, orgs and projects: globally unique
usernames, middleware resolves Profile/Organization/Project to /profile/, /o/,
/p/; phase 6 wallet-resolve honours the same namespace
- Phase 7: Organization.username replaces slug, /o/[orgUsername] app-dir route,
Project gains ownerType/orgId
- Phase 8: Event.projectIds + Project.eventIds inverse, ?project= discovery
- README: status scopes, P5->P7 dependency edge, decisions 2/6, next free
migration fixed to 0021, handle-collision risk row
Co-Authored-By: Claude <noreply@anthropic.com>
…ts, TaskApplication Phase 5 schema (docs/plans/phase-05-public-lists-job-board.md §5.1): - Project: username @unique handle (shared /@ namespace, /p/ URL segment), bio, photo/cover document ids, links, supportUrl, spotlight, publicVisible, embedded UserReference collaborators, List relation - List: publicTagline, publicVisible, coverDocumentId, location, jobBoardEnabled, projectId + Project relation, new indexes - Task: jobDescription, requirements, openings, applyBy, applications - TaskApplication: PENDING/SHORTLISTED/ACCEPTED/DECLINED/WITHDRAWN apply flow, @@unique([taskId, userId]), @@index([listId, status]) - User gains taskApplications inverse No migration scripts: Project is a new collection and all other fields are optional (List.publicUrl backfill already shipped as 0020 in Phase 1). Co-Authored-By: Claude <noreply@anthropic.com>
…lizer Phase 5 service layer: - projects/: username handle generation (shared /@ namespace cross-check vs Profile.username), CRUD (OWNER/MANAGER), allowlist-projected public payload with computed stats + publishedLists, spotlight-first discovery feed - applications/: apply to public job posts (404 unpublished/non-public, 400 closed/filled, 409 duplicate), owner/manager review, ACCEPT adds candidate + list COLLABORATOR membership (sequential idempotent steps — no multi-doc transactions on MongoDB standalone) - list/publicListService: allowlist projection for /list/[publicUrl] (public tasks, owner/collaborator profiles, project chip, like/viewer state) + job board discovery feed with batched profiles/like counts - ownership: EntityKind gains 'project' (embedded Project.users roles) - social: 'project' registered in SOCIAL_ENTITIES/LIKEABLE_ENTITIES/MODEL_DELEGATES Co-Authored-By: Claude <noreply@anthropic.com>
Phase 5 API surface: - GET /api/v1/tasklists/public — job-board discovery feed (cursor, q/area/category) - GET /api/v1/tasklists/public/[publicUrl] — public list payload (unauthenticated) - POST /api/v1/tasks/[taskId]/apply + GET/POST applications routes — job-post apply flow with owner/manager review - POST/GET /api/v1/projects + GET/PUT [projectId] + public feed + public payload - POST/PUT /api/v1/tasklists extended with publicTagline, publicVisible, coverDocumentId, location, jobBoardEnabled, projectId (project attach requires project collaborator via assertProjectCollaborator) - Likes: entityType 'project' live via the social registry (no route change) - CLAUDE.md docs for the new routes + v1 index Co-Authored-By: Claude <noreply@anthropic.com>
…link Phase 5 UI: - /list/[publicUrl] + publicListView: hero, project chip, links, about, open positions grid, like + request-to-join islands - /list/[publicUrl]/jobs/[taskId] + publicJobView: job detail + apply dialog - /p/[username] + publicProjectView: hero (spotlight badge), computed stats, about, job boards, like + support/donate islands - /app/be/jobs: job board discovery (SWR + local filters) + my projects + create project - addListForm: Public profile section (publish, tagline, bio, cover, links, job board switch, project selector) wired into POST/PUT bodies - addTaskForm: Publish-as-job section (description, requirements, openings, applyBy) shown when the list has jobBoardEnabled; tasks routes accept the job fields - Task deep link /app/do/list/[listId]/[taskId]: DoPage resolves the task's occurrence date, TaskGrid boosts it first + ring highlight + scrollIntoView - sitemap: published lists and projects Co-Authored-By: Claude <noreply@anthropic.com>
- en.json: list.public.*, jobs.board.*, project.*, forms.addListForm public profile section, forms.addTaskForm publish-as-job section (33-locale fan-out deferred: all new strings carry defaultValue fallbacks) - openapi.yaml: tasklists/public, tasks apply/applications, projects CRUD + public routes documented - lint cleanups on the new files (no-explicit-any, unused params) Co-Authored-By: Claude <noreply@anthropic.com>
…cycle Phase 6 (docs/plans/phase-06-dpip-ledger.md): - Schema: Wallet.balance/pendingBalance (integer minor units), kind/isDefault/ ownerType/frozen/onChainSyncedAt, new LedgerEntry model, Transaction gains amountMinor/reference @unique/kind/fromWalletId/toWalletId/settledAt/ failureReason/onChainTxHash/entries - src/lib/utils/money.ts: minor-unit conversion at the boundary only - ledgerService: transfer/hold/release/credit/getBalance/getStatement/reconcile with DUAL-MODE atomicity (user decision): single interactive transaction on a replica set (production path), sequential idempotent steps + compensation on standalone Mongo (dev); compare-and-set debit in both modes; LEDGER_REQUIRE_TRANSACTIONS=true makes the boot assertion strict - walletService: getOrCreateDefaultWallet (self-heal), 5-wallet cap by kind, resolveRecipient across the shared /@ namespace - Auth webhook: default wallet at user.created/session.created (idempotent) - API: wallet GET (DB balance first, ?includeOnChain opt-in) + POST (lazy Kaleido), transfer rewritten over the ledger (toWalletId|toAddress|toUsername, reference idempotency), statement, resolve, sync-onchain, cron ledger-reconcile - Migrations 0021-0024 (default wallets, normalize transactions + reference backfill BEFORE the unique index push, system wallets, ledger entries replay) - UI: tokenTransfer (DPIP, @username recipient, confirm step), walletBalanceCard (balance/pending/statement), investView surfaces the statement - Phase-06 doc records the dual-mode deviation Co-Authored-By: Claude <noreply@anthropic.com>
Phase 7 (docs/plans/phase-07-organizations.md): - Schema: Organization (clerkOrgId @unique, username @unique handle, public profile fields, status ACTIVE/ORPHANED/ARCHIVED), OrgMembership (OWNER/ ADMIN/MANAGER/MEMBER/STAFF), polymorphic ownerType/orgId on List, Wallet and Project - ownershipService: the single ORG branch — org-owned entities resolve roles from OrgMembership (OWNER/ADMIN→OWNER, MANAGER→MANAGER, MEMBER→MEMBER view-only per acceptance criteria, STAFF→STAFF); embedded users OWNER stays the steward with OWNER access - orgService: idempotent webhook upserts, pull-repair syncOrganization, markOrphaned/removeMembership, createOrganization (Clerk + mirror + OWNER + general channel + org wallet), username generation cross-checked across users/orgs/projects, allowlist-projected public payload with computed stats - Auth webhook: organization.* + organizationMembership.* events mirrored - API: /api/v1/orgs CRUD + members + public payload; POST /tasklists and POST /projects accept ownerType ORG (MANAGER+ via assertOrgManagerRole); wallet resolve resolves org handles; resolve-handle Node route for the edge middleware (/@ resolves users, orgs and projects) - Middleware: /@ redirect hops to resolve-handle (Prisma cannot run in edge) - UI: /o/[orgUsername] public org page, organizations directory page (create + publish toggle), org owner selector in addProjectForm - Migrations 0025 (mirror Clerk orgs + memberships + org wallets, with ChatOrgMembership fallback) + 0026 (backfill ownerType USER) - Docs: orgs CLAUDE.md, v1 index, openapi, en.json org keys; phase-07 doc role-mapping corrected to match acceptance criteria Co-Authored-By: Claude <noreply@anthropic.com>
Phase 8 (docs/plans/phase-08-events-core.md): - Schema: LifeEvent (the pre-Phase-8 Event, renamed with _id preserved), new public Event (publicUrl @unique always generated, status, IANA timezone, online/location, cover/flier, capacity, USER|ORG owner, list/ project m:m, rsvps, staff), EventRsvp, EventStaff; List.eventIds and Project.eventIds inverses; Day/Note lifeEventIds; Comment gains lifeEventId (eventId now points at public events); Note.eventIds = the event's discussion stream; Organization.events declared here (schema-validity rule: no Phase 9/10 fields) - Migration 0027: copy Event→LifeEvent preserving _id, rewrite Note/Document/ Day/Comment inbound refs, delete sources (idempotent); 0028 event slug repair - eventService: create (DRAFT + slug + proceeds wallet kind EVENT, ORG owner via assertOrgManagerRole), publish validation, scoped management feed, public discovery (near bounding box, project/category/q filters, batched RSVP counts), allowlist-projected public payload with host/lists/projects, idempotent RSVP, list/project links, staff, soft cancel - API: /api/v1/events CRUD + publish/rsvp/lists/projects/staff + public routes; /api/v1/life-events takes the old life-event API; /api/v1/events/ legacy redirect shim; legacy consumers (publishNote, noteContent, entityTagPicker, lifeEventCombobox, moodView) repointed; likes 'event' enabled in the social registry - UI: /event/[publicUrl] server page (OG + JSON-LD Event structured data, timezone-aware rendering), publicEventView action islands (RSVP/like), /app/be/events Discover/Going/Mine tabs, eventCard, addEventForm (owner selector Me/orgs), sitemap entries - Docs: events CLAUDE.md, v1 index, openapi, events.* en.json keys Co-Authored-By: Claude <noreply@anthropic.com>
E2E infrastructure: - Playwright config (dev server locally, build+start in CI), Clerk test-auth helpers (Backend-API users with pre-verified emails, browser sign-in via @clerk/testing, cookie-jar sharing for API contexts), ledger harness (treasury credit + invariant checks), user-journey spec (signup → mood → note → localized default lists → complete → public note → profile → invest game/fiat balance), stack-smoke spec (plan step 3: wallet → transfer+replay → orgs → job apply/accept → events → life-events split → ledger invariants) - .github/workflows/e2e.yml: fresh mongo service, migrations 0021-0028 twice (idempotency proof), db push, build, Playwright — runs on every PR; make the E2E check required for dev merges in branch protection Bugs the E2E caught (cross-phase fixes riding this PR): - resolveRecipient passed @Handles into findUnique({ id }) — malformed ObjectId crash on every handle-based transfer (now guarded) - days route + dayTransformService read Day.eventIds (renamed lifeEventIds in the Phase 8 split) — 500s on the mood/day flow - Organization.createdByUserId is nullable (webhook ordering: the first OWNER/ADMIN membership fills it) — empty-string ObjectId invalid - Migration 0025 uses @clerk/backend (importable from plain Node; @clerk/nextjs is bundler-only); 0027 renames Day.eventIds via raw $runCommandRaw ($rename) Co-Authored-By: Claude <noreply@anthropic.com>
- Replace the services: standalone mongo:7 container with a docker run that starts mongod with --replSet rs0 and calls rs.initiate(). Prisma requires a replica set for any nested write (P2031); the old standalone setup silently broke ensureUserAndProfile on every request, causing 'User not found' 404s throughout the test suite. - Add replicaSet=rs0 to DATABASE_URL so the Mongo driver performs topology discovery and marks the deployment transaction-capable. - Set LEDGER_REQUIRE_TRANSACTIONS=true so CI exercises the production interactive-transaction ledger path rather than the standalone sequential fallback. - Add a 'Verify the database supports transactions' step that fails fast with a clear error instead of burying hundreds of P2031 stack traces. - Add a job-level 'if' guard that skips the suite on fork PRs where Clerk secrets are not available, plus a 'Check required secrets' step that surfaces a clear error message when they are missing. - Avoid P2031 in ensureUserAndProfile on standalone deployments by splitting create + include into a plain create followed by a separate findUnique, removing the implicit transaction that Prisma opens for nested reads on create. - Add Next build and Playwright browser caches to cut run time. - Upload test-results/ alongside playwright-report/ on failure for easier trace inspection. - Update e2e/README.md with replica-set requirements and local setup instructions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| [ -n "${CLERK_SECRET_KEY}" ] || missing+=(CLERK_SECRET_KEY) | ||
| [ -n "${NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY}" ] || missing+=(NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY) | ||
| if [ ${#missing[@]} -gt 0 ]; then | ||
| echo "::error::Missing repo secrets: ${missing[*]}. The E2E harness creates real users on a Clerk DEVELOPMENT instance and cannot run without them." |
There was a problem hiding this comment.
@copilot what is this log4shell? can we please not print this
There was a problem hiding this comment.
The ::error:: prefix is a GitHub Actions workflow command — it annotates the job log with a red error badge, unrelated to Log4Shell. Replaced it with a plain echo in commit fix(ci): remove ::error:: workflow command annotation from secrets check.
There was a problem hiding this comment.
@copilot what I meant is that we shouldn't log any secrets to STD, period. can we please remove those logs. shouldn't the issue be fixed by updating the secret to localhost:27017/whatever?replicaSet=rs0?
There was a problem hiding this comment.
Removed the "Check required secrets" step entirely in the latest commit. No secret names or values are printed to stdout anymore. The job-level if guard still prevents the job from running on fork PRs where secrets aren't available.
Co-authored-by: angeloreale <12191809+angeloreale@users.noreply.github.com>
| set -euo pipefail | ||
| docker run -d --name mongo -p 27017:27017 "$MONGO_IMAGE" \ | ||
| --replSet "$MONGO_REPLICA_SET" --bind_ip_all | ||
|
|
||
| echo "Waiting for mongod to accept connections..." | ||
| for i in $(seq 1 60); do | ||
| if docker exec mongo mongosh --quiet --eval 'db.adminCommand({ ping: 1 })' >/dev/null 2>&1; then | ||
| break | ||
| fi | ||
| if [ "$i" -eq 60 ]; then | ||
| echo "::error::mongod did not become reachable"; docker logs mongo; exit 1 | ||
| fi | ||
| sleep 2 | ||
| done | ||
|
|
||
| # `host: localhost:27017` is what the driver on the runner is told to | ||
| # connect to during topology discovery, and the container publishes | ||
| # that port on the host — so the advertised member address is | ||
| # reachable from both inside and outside the container. | ||
| docker exec mongo mongosh --quiet --eval " | ||
| try { | ||
| rs.status() | ||
| } catch (e) { | ||
| rs.initiate({ _id: '$MONGO_REPLICA_SET', members: [{ _id: 0, host: 'localhost:27017' }] }) | ||
| }" | ||
|
|
||
| echo "Waiting for PRIMARY..." | ||
| for i in $(seq 1 60); do | ||
| if docker exec mongo mongosh --quiet --eval 'db.hello().isWritablePrimary' 2>/dev/null | grep -q true; then | ||
| echo "Replica set $MONGO_REPLICA_SET is PRIMARY" | ||
| break | ||
| fi | ||
| if [ "$i" -eq 60 ]; then | ||
| echo "::error::replica set never reached PRIMARY"; docker logs mongo; exit 1 | ||
| fi | ||
| sleep 2 | ||
| done |
There was a problem hiding this comment.
@copilot why would you even do this in Opus 5 with MAX context when having fixed the replica set core issue already?
There was a problem hiding this comment.
The ensureUserAndProfile split only removes one implicit transaction, but other Prisma operations in the E2E suite (e.g. any create with a nested include, connectOrCreate, or upsert) still open implicit transactions and will hit P2031 against a standalone mongod. Running CI against a replica set is the correct long-term fix — it matches production topology and ensures any future Prisma write won't silently fail. The ensureUserAndProfile change is a bonus that also helps local dev without a replica set.
Co-authored-by: angeloreale <12191809+angeloreale@users.noreply.github.com>
Why
The E2E suite was failing on every run with
P2031-- "Prisma needs to perform transactions, which requires your MongoDB server to be run as a replica set." Theservices:block started a standalonemongo:7container, which Prisma refuses to use for any write that has a nestedinclude(it opens an implicit transaction).ensureUserAndProfilenever created theUserrow, so every downstream request returned404: User not found.What changed
e2e.yml-- workflow rewriteservices:MongoDB block with adocker runstep that startsmongod --replSet rs0and callsrs.initiate(). Theservices:construct cannot pass custom command flags.DATABASE_URLnow includes?replicaSet=rs0so the driver performs topology discovery and marks the deployment transaction-capable -- without it Prisma still sees standalone.LEDGER_REQUIRE_TRANSACTIONS=trueis set so CI exercises the same interactiveprisma.$transactionledger path as production, not the dev fallback.ifguard to skip on fork PRs where Clerk secrets are unavailable, plus a "Check required secrets" step with a clear error message.actions/cachefor the Next.js build and Playwright browsers to cut run time.test-results/alongsideplaywright-report/on failure for easier trace inspection.src/lib/services/user/ensureUserAndProfile.tsSplits
prisma.user.create({ include: { profiles: true } })into a plaincreatefollowed by a separatefindUnique. The combined form causes Prisma to open an implicit transaction, which standalone MongoDB rejects. This fixes local dev on non-replica-set deployments as a bonus.e2e/README.mdDocuments the replica-set requirement and provides copy-paste Docker commands to set one up locally.
Non-obvious notes
localhost:27017(not the container's internal hostname) because the app process on the runner connects via the published host port during topology discovery.isWritablePrimarypoll usesgrep -q truerather than a string comparison to be robust against extra mongosh output lines (warnings, banners).