Skip to content

common.sh: caller-set DPLA_DATA, I3_CONF and SPARK_MASTER win over .env - #857

Merged
zspieges merged 1 commit into
mainfrom
fix_env_set_up
Nov 19, 2026
Merged

zspieges merged 1 commit into
mainfrom
fix_env_set_up

Conversation

@zspieges

@zspieges zspieges commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

init_paths in scripts/common.sh sources $I3_HOME/.env with set -a, so .env overrides anything already in the environment. The comment above init_paths says these paths "can all be overridden with environment variables", but in practice they can't: DPLA_DATA=/x ./scripts/s3-sync.sh hub still syncs from .env's DPLA_DATA.

Visible symptom: scripts/tests/test-hub-alias-mock.sh fails on any machine with a repo .env that sets DPLA_DATA. The test sets DPLA_DATA to a temp directory, .env replaces it, and s3-sync.sh syncs from the real data directory (against the stubbed aws), so the expected-call assertion fails. It passes on CI only because CI has no .env.

Change

  • init_paths saves DPLA_DATA, I3_CONF and SPARK_MASTER if the caller set them (non-empty) and puts them back after loading .env.
  • Everything else in .env still wins as before. This is on purpose, so .env keeps control of JAVA_HOME (project rule 1: the build relies on it for Java 11+) and of tokens, and a JAVA_HOME from a login shell or SSM bash -lc can't swap the JDK.
  • scripts/SCRIPTS.md documents the precedence.

Callers whose behaviour changes

  • Tests that set DPLA_DATA (test-hub-alias-mock.sh, test-nara-gate.sh, test-nara-force-sync.sh, the Community Webs export tests) now get the value they set, whether or not a local .env exists.
  • Orchestrator (scheduler/orchestrator/hub_processor.py:570) sets SPARK_MASTER per hub. A SPARK_MASTER in .env used to override it; now the orchestrator's value is used.
  • Monthly batch (ingest_pipeline_python/helper_scripts/run_monthly_batch.py:266) exports I3_CONF=/home/ec2-user/ingestion3-conf/i3.conf before each ingest.sh. That export now wins over the box .env's I3_CONF, if .env sets one.

Nothing in .github/ sets these three variables.

Tests

New test_env_precedence in scripts/tests/test-scripts.sh (runs in --quick too):

  • caller-set DPLA_DATA (path with a space), I3_CONF and SPARK_MASTER beat .env;

  • .env fills them in when the caller sets nothing;

  • .env still wins for other variables (SLACK_CHANNEL).

  • ./scripts/tests/test-scripts.sh --quick

  • ./scripts/tests/test-hub-alias-mock.sh passes locally with a repo .env that sets DPLA_DATA

  • Box .env checked: I3_CONF is unset or /home/ec2-user/ingestion3-conf/i3.conf

🤖 Generated with Claude Code

This pull request changes how scripts/common.sh loads $I3_HOME/.env during init_paths. Caller-set values now take precedence over .env for three variables: DPLA_DATA, I3_CONF, and SPARK_MASTER.

Behavior change:

  • Before .env loads, init_paths saves each of the three variables that is non-empty in the caller environment.
  • init_paths loads .env as before.
  • init_paths restores the saved values. A caller value replaces the .env value.
  • If the caller sets none of the three variables, the .env values apply.
  • All other .env variables keep their existing precedence. This includes JAVA_HOME and SLACK_CHANNEL. .env still overrides caller values for these variables.

Files changed (3 files, +51/-1):

  • scripts/common.sh (+14/-0): Adds the save and restore logic around the .env load.
  • scripts/tests/test-scripts.sh (+36/-0): Adds test_env_precedence. The test creates a temporary .env and checks:
    • Caller values win for DPLA_DATA, I3_CONF, and SPARK_MASTER. One case uses a DPLA_DATA path that contains a space.
    • .env values apply when the caller sets none of them.
    • .env overrides a caller-set SLACK_CHANNEL.
      The test removes its temporary directory. main runs this test after test_common_sh.
  • scripts/SCRIPTS.md (+1/-1): Documents the precedence rule and the exception list.

Effects on callers, as stated by the author:

  • Tests, the hub orchestrator's per-hub SPARK_MASTER, and the monthly batch's exported I3_CONF use the caller value even when .env sets the same variable.

Operational notes:

  • No environment variables or AWS Secrets Manager keys are added, removed, or renamed.
  • No database migration is included.
  • No changes to CodePipeline, CodeBuild, ECS task definitions, or IAM policies.
  • No public API or response shape changes.
  • Security: the change affects only which value a shell script uses. It does not change credential handling. .env still supplies secrets such as SLACK_BOT_TOKEN.

Verification status:

  • The PR description leaves these checks unchecked: ./scripts/tests/test-scripts.sh --quick, ./scripts/tests/test-hub-alias-mock.sh with a repo .env that sets DPLA_DATA, and the box .env check for I3_CONF.
  • No test or CI results were available for this review.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

init_paths now preserves non-empty caller values for DPLA_DATA, I3_CONF, and SPARK_MASTER when it loads .env. Documentation and tests describe and verify this behavior, including .env precedence for other variables.

Changes

Environment Variable Precedence

Layer / File(s) Summary
Preserve caller values during .env loading
scripts/common.sh, scripts/SCRIPTS.md, scripts/tests/test-scripts.sh
init_paths restores non-empty caller values for DPLA_DATA, I3_CONF, and SPARK_MASTER after loading .env. The documentation describes this precedence. Tests check caller values, .env defaults, and .env precedence for SLACK_CHANNEL.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: dominicbm

Merge Risk: Minimal · up to 3abe9

Caller-set DPLA_DATA, I3_CONF, and SPARK_MASTER values now take precedence over .env, as documented, and a new test covers this. No merge-blocking issue was found. The author should still run the unchecked validation steps before relying on it.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: in scripts/common.sh, caller-set DPLA_DATA, I3_CONF and SPARK_MASTER now take precedence over values loaded from .env. The raw summary confirms init_paths saves and re…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: in scripts/common.sh, caller-set DPLA_DATA, I3_CONF and SPARK_MASTER now take precedence over values loaded from .env. The raw summary confirms init_paths saves and restores these caller values after loading .env, and the PR objectives state the same behavior. The title is specific and matches the changeset.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 991a8ec6-d04d-49dc-a542-fe15e4b3ce4b
Commits

Reviewing files that changed from the base of the PR and between 572b612 and 3abe93e.

Files selected for processing: 3

  • scripts/SCRIPTS.md
  • scripts/common.sh
  • scripts/tests/test-scripts.sh

Limit details: You’ve used the included review currently available.

@zspieges
zspieges merged commit b1f36b4 into main Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant