Phase 5: Cutover — Fargate becomes production, legacy SSH workflows disabled

[Bahugunajii]

Summary

Phase 5 (cutover) of the EC2 → ECS Fargate migration. Three coordinated changes that take Fargate from "validated and idle" to "the actual deploy target":

1. Flip new workflow triggers from workflow_dispatch only to auto-trigger
2. Rename legacy SSH workflows to .yml.disabled
3. Switch prod logger to console-only (deferred from Phase 0 User API init #2)

This PR is stacked on top of #500 (Phase 4) so the workflow trigger edits target the new files. Once Phase 4 merges to main, the GitHub PR base will resolve to main automatically.

⚠️ Pre-merge requirements

Before this PR merges, all of these must be true:

• PRs Phase 0: Pre-migration code changes for ECS Fargate #497, Phase 1: Terraform infrastructure for ECS Fargate #498, Phase 2: Dockerfile and .dockerignore for Fargate runtime #499, Phase 4: ECS Fargate deploy workflows (workflow_dispatch only) #500 all merged to main
• terraform apply has run for both environments (staging and prod)
• DB_SERVICE_AWS_ACCESS_KEY_ID / DB_SERVICE_AWS_SECRET_ACCESS_KEY set in repo secrets (from terraform apply outputs with create_ci_user = true)
• First image manually pushed to both ECR repos (gh workflow run staging_deploy_ecs.yml and gh workflow run production_deploy_ecs.yml once each)
• Staging Fargate validated end-to-end (health check, CSV imports, Google Sheets, LiveView WebSocket — full checklist in design doc section 5.2)
• Prod Fargate validated end-to-end
• Both ALB target groups report healthy

If any of these aren't done, don't merge. The previous PRs are reversible by themselves; this one is the point where deploys flip over.

Change details

1. Workflow trigger flip

File	Before	After
staging_deploy_ecs.yml	workflow_dispatch only	pull_request: branches: [main] + workflow_dispatch
production_deploy_ecs.yml	workflow_dispatch only	push: branches: [release] + workflow_dispatch

Mirrors the legacy workflows' triggers exactly, so the muscle memory ("PR to main = staging deploy", "push to release = prod deploy") still holds.

2. Legacy SSH workflow renames

.github/workflows/staging_deploy.yml      → staging_deploy.yml.disabled
.github/workflows/production_deploy.yml   → production_deploy.yml.disabled
.github/workflows/run_migrations.yml      → run_migrations.yml.disabled

GitHub Actions only reads .yml/.yaml files, so these are inert. Files kept (not deleted) so the rollback is git mv X.yml.disabled X.yml.

ci.yml (lint/dialyzer/credo, no deploy) stays unchanged.

3. Prod logger to console-only

In config/prod.exs:

# Override the global :console + LoggerFileBackend setup so prod only writes
# to stdout. On Fargate, the awslogs driver captures stdout/stderr to
# CloudWatch Logs automatically; the file backend would just write to the
# task's ephemeral filesystem and disappear when the task is replaced.
config :logger, backends: [:console]

The global config/config.exs ships both :console and LoggerFileBackend. On EC2 the file backend wrote to logs/info.log, which a CloudWatch agent shipped. On Fargate the file would disappear on every task replacement — and the awslogs driver is already capturing stdout. This change was deliberately deferred from Phase 0 because flipping it earlier would have broken EC2's log shipping during the migration.

Rollback procedure (during the 1-week soak)

If Fargate misbehaves and you need to put EC2 back in the path:

1. DNS: update the Route53 A-record for db.avantifellows.org (and/or staging-db.avantifellows.org) from the ALB alias to the EC2 instance's public IP. (Or destroy the Route53 record from Terraform and let the old DNS-by-IP setup take over — depends on what was there before.)
2. Workflows: open a revert-of-this-PR or just rename:

   git mv .github/workflows/production_deploy.yml.disabled \
          .github/workflows/production_deploy.yml
   git mv .github/workflows/staging_deploy.yml.disabled \
          .github/workflows/staging_deploy.yml
   git mv .github/workflows/run_migrations.yml.disabled \
          .github/workflows/run_migrations.yml

   And revert the _ecs.yml trigger changes back to workflow_dispatch-only so they don't fight the SSH workflows.
3. Logger: revert the config/prod.exs change so the next EC2 deploy gets the file backend back.
4. EC2 keeps running the pre-cutover code, so its logs/info.log continues being written and shipped by the existing CloudWatch agent.

The EC2 instance is the safety net; it stays running until step 12 of the design doc's order of operations ("After 1+ week stable: decommission EC2, delete old workflows").

What's NOT in this PR

These are operational steps, not code:

• Actually running terraform apply for the first time
• Hand-firing the first ECS deploys
• Validating the staging/prod Fargate environments
• Updating GitHub Secrets with the CI user credentials

These are also deferred to a future cleanup PR (after the 1-week soak):

• Deleting the *.yml.disabled files
• Removing the LoggerFileBackend dep from mix.exs (only used by the now-disabled file backend in EC2)
• Decommissioning the EC2 instances themselves (out-of-band AWS console / CLI operation)

Test plan

• actionlint passes on both modified workflow files
• Legacy renames are detected as renames by git (100% similarity), not deletes + adds
• Logger override is in config/prod.exs so dev (config/dev.exs) is unaffected
• (Manual, post-merge) After a PR is opened against main, staging_deploy_ecs.yml runs automatically and staging_deploy.yml.disabled does NOT
• (Manual) After a push to release, production_deploy_ecs.yml runs automatically and production_deploy.yml.disabled does NOT
• (Manual) CloudWatch Logs /ecs/db-service-prod receives the application logs (no file backend writes)

🤖 Generated with Claude Code

[deepansh96]

Review Comments

1. Image tag should match checked-out ref

   Small consistency improvement: these workflows accept a git_ref input and check out that ref, but the Docker image tag is based on ${{ github.sha }}.

   Even if we normally deploy the default branch/release branch, it would be cleaner if the image tag came from the commit that was actually checked out and built. That makes the workflow easier to reason about and keeps the image tag, source commit, and ECS task definition aligned.

   Recommended fix: resolve the checked-out commit with git rev-parse HEAD after checkout and use that value for the Docker tag/ECS image instead of ${{ github.sha }}.

2. Confirm production DB SSL verification plan

   Question on the prod DB SSL config: we are using SSL, but certificate verification is disabled (verify_none).

   Is this just for cutover, or are we planning to add proper certificate verification later?

   If we are keeping SSL for prod, we should also verify the DB certificate instead of leaving verify_none long term.

3. Keep Dockerfile port in sync with ECS

   Small cleanup: the Dockerfile still exposes port 4000, but ECS runs the app on 8080.

   This is not blocking because ECS passes PORT=8080, and the task/container mapping also uses 8080. But the Dockerfile comment/config now points to the old port, which can confuse future debugging.

   Recommended fix: update the Dockerfile EXPOSE line to match the ECS port, or remove it if we do not want Dockerfile metadata to imply a runtime port.

4. Commit Terraform provider lockfile

   Terraform provider lockfile should probably be committed.

   Right now .terraform.lock.hcl is ignored. That means different machines can resolve different provider versions over time, even if the Terraform code has not changed.

   Recommended fix: stop ignoring .terraform.lock.hcl and commit the Terraform lockfile so staging/prod applies use the same provider versions.

5. Enable ECS deployment rollback

   ECS service should have automatic rollback enabled.

   In terraform/ecs.tf, the service has normal rolling deployment settings:

   • deployment_minimum_healthy_percent = 100
   • deployment_maximum_percent = 200
   • health_check_grace_period_seconds = 120

   But there is no deployment_circuit_breaker block. So if a bad task revision is deployed and the new tasks cannot become healthy behind the ALB, ECS can keep trying the bad revision instead of automatically rolling back to the last working one.

   This matters more here because CI registers new task definition revisions during deploy, and Terraform has ignore_changes = [task_definition, desired_count]. So Terraform also will not automatically correct the service back to the previous task definition on the next apply.

   Recommended fix: add ECS deployment circuit breaker with rollback enabled:

   deployment_circuit_breaker {
     enable   = true
     rollback = true
   }

6. Scope down Cloudflare credentials

   Cloudflare credentials should be scoped down before prod cutover.

   Terraform currently uses Cloudflare email + api_key, which is likely a Global API Key. Recommended fix: use a zone/DNS-scoped API token instead, so the blast radius is limited to the DNS changes this Terraform actually needs.

7. Set ECS deploy workflow AWS secrets

   Reminder before cutover: the ECS deploy workflows use DB_SERVICE_AWS_ACCESS_KEY_ID and DB_SERVICE_AWS_SECRET_ACCESS_KEY.

   These secrets do not seem to be set currently, so the ECS deploy workflows will fail at the AWS credential step unless they are added before cutover.

8. Add tfvars example

   Can we add a terraform/terraform.tfvars.example file?

   Right now the repo has environment-specific tfvars files for staging/prod, but no example showing the full set of required variables. A checked-in example would make it easier to review what needs to be configured without opening the environment files.

   Suggested content can use placeholder values only, especially for secrets.

9. Restrict production ECS deploy ref

   Prod ECS deploy should only allow the release branch.

   The production workflow currently has a manual ref input, so it can be run against any branch/SHA. For staging that flexibility is useful, but for prod the blast radius is higher.

   Recommended fix: restrict the production ECS deploy workflow so normal prod deploys can only run from release.

10. Align ECS Exec config and IAM permissions

    ECS Exec is enabled on the service, but the task role does not include the SSM permissions ECS Exec needs.

    terraform/ecs.tf has enable_execute_command = true. But in terraform/iam.tf, the task role policy currently only covers the CSV imports S3 bucket.

    If ECS Exec is expected to work after cutover, the task role should include the required ssmmessages permissions. Otherwise, we should disable enable_execute_command until we actually want to support it.

11. Scope security group lookup to VPC

    Security group lookup in the ECS workflows should be scoped to the VPC.

    Both ECS deploy workflows resolve the task security group only by name:

    aws ec2 describe-security-groups \
      --filters "Name=group-name,Values=${SECURITY_GROUP_NAME}"

    Security group names are only unique within a VPC, not across the whole AWS account/region. If another VPC has a security group with the same name, this lookup can pick the wrong one.

    Recommended fix: add a VPC filter to the lookup, or pass the exact security group ID from Terraform/GitHub variables.

12. Narrow CI deploy IAM permissions

    CI deploy IAM permissions should be narrowed before prod cutover.

    In terraform/iam.tf, the CI user policy allows ECR push/pull and ECS deploy actions with resources = ["*"]. Some read/auth actions may need broad scope, but the mutating actions should be limited where possible.

    The main risky ones are things like ecs:UpdateService, ecs:RunTask, ecs:StopTask, ecs:RegisterTaskDefinition, and ECR image push actions. With *, the same GitHub secret could affect ECS/ECR resources outside this service if the key is leaked or misused.

    Recommended fix: scope the CI policy to the db-service ECR repositories, ECS cluster/service/task definitions, and only the roles it needs to pass.

13. Add Terraform wrapper commands per environment

    Terraform env commands should be wrapped so staging/prod state does not get mixed up.

    The same terraform/ folder is used for both staging and prod. The difference is decided by two things:

    • backend state key, passed during terraform init
    • tfvars file, passed during terraform plan/apply

    The state key decides which remote state file Terraform reads/writes, for example staging state vs prod state. The tfvars file decides which environment values are used.

    The risky part is that terraform init -backend-config=... is sticky in the local .terraform/ directory. So a local checkout can stay initialized against staging state, and someone can later accidentally run a prod plan/apply with prod tfvars against the staging state file.

    That can make Terraform compare the wrong state with the wrong environment values, which is a high-risk footgun for infra changes.

    Recommended fix: add small wrapper scripts or Makefile targets like tf-init-staging, tf-plan-staging, tf-init-prod, tf-plan-prod, etc. so the backend key and tfvars file are always paired correctly.