Repository navigation
common.sh: caller-set DPLA_DATA, I3_CONF and SPARK_MASTER win over .env - #857
Merged
Merged
Conversation
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
init_pathsinscripts/common.shsources$I3_HOME/.envwithset -a, so.envoverrides anything already in the environment. The comment aboveinit_pathssays these paths "can all be overridden with environment variables", but in practice they can't:DPLA_DATA=/x ./scripts/s3-sync.sh hubstill syncs from.env'sDPLA_DATA.Visible symptom:
scripts/tests/test-hub-alias-mock.shfails on any machine with a repo.envthat setsDPLA_DATA. The test setsDPLA_DATAto a temp directory,.envreplaces it, ands3-sync.shsyncs from the real data directory (against the stubbedaws), so the expected-call assertion fails. It passes on CI only because CI has no.env.Change
init_pathssavesDPLA_DATA,I3_CONFandSPARK_MASTERif the caller set them (non-empty) and puts them back after loading.env..envstill wins as before. This is on purpose, so.envkeeps control ofJAVA_HOME(project rule 1: the build relies on it for Java 11+) and of tokens, and aJAVA_HOMEfrom a login shell or SSMbash -lccan't swap the JDK.scripts/SCRIPTS.mddocuments the precedence.Callers whose behaviour changes
DPLA_DATA(test-hub-alias-mock.sh,test-nara-gate.sh,test-nara-force-sync.sh, the Community Webs export tests) now get the value they set, whether or not a local.envexists.scheduler/orchestrator/hub_processor.py:570) setsSPARK_MASTERper hub. ASPARK_MASTERin.envused to override it; now the orchestrator's value is used.ingest_pipeline_python/helper_scripts/run_monthly_batch.py:266) exportsI3_CONF=/home/ec2-user/ingestion3-conf/i3.confbefore eachingest.sh. That export now wins over the box.env'sI3_CONF, if.envsets one.Nothing in
.github/sets these three variables.Tests
New
test_env_precedenceinscripts/tests/test-scripts.sh(runs in--quicktoo):caller-set
DPLA_DATA(path with a space),I3_CONFandSPARK_MASTERbeat.env;.envfills them in when the caller sets nothing;.envstill wins for other variables (SLACK_CHANNEL)../scripts/tests/test-scripts.sh --quick./scripts/tests/test-hub-alias-mock.shpasses locally with a repo.envthat setsDPLA_DATABox
.envchecked:I3_CONFis unset or/home/ec2-user/ingestion3-conf/i3.conf🤖 Generated with Claude Code
This pull request changes how
scripts/common.shloads$I3_HOME/.envduringinit_paths. Caller-set values now take precedence over.envfor three variables:DPLA_DATA,I3_CONF, andSPARK_MASTER.Behavior change:
.envloads,init_pathssaves each of the three variables that is non-empty in the caller environment.init_pathsloads.envas before.init_pathsrestores the saved values. A caller value replaces the.envvalue..envvalues apply..envvariables keep their existing precedence. This includesJAVA_HOMEandSLACK_CHANNEL..envstill overrides caller values for these variables.Files changed (3 files, +51/-1):
scripts/common.sh(+14/-0): Adds the save and restore logic around the.envload.scripts/tests/test-scripts.sh(+36/-0): Addstest_env_precedence. The test creates a temporary.envand checks:DPLA_DATA,I3_CONF, andSPARK_MASTER. One case uses aDPLA_DATApath that contains a space..envvalues apply when the caller sets none of them..envoverrides a caller-setSLACK_CHANNEL.The test removes its temporary directory.
mainruns this test aftertest_common_sh.scripts/SCRIPTS.md(+1/-1): Documents the precedence rule and the exception list.Effects on callers, as stated by the author:
SPARK_MASTER, and the monthly batch's exportedI3_CONFuse the caller value even when.envsets the same variable.Operational notes:
.envstill supplies secrets such asSLACK_BOT_TOKEN.Verification status:
./scripts/tests/test-scripts.sh --quick,./scripts/tests/test-hub-alias-mock.shwith a repo.envthat setsDPLA_DATA, and the box.envcheck forI3_CONF.