-
Notifications
You must be signed in to change notification settings - Fork 51
ci: install upstream ddtrace wheel during PR-triggered unit tests #813
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+166
−75
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Unless explicitly stated otherwise all files in this repository are licensed | ||
| # under the Apache License Version 2.0. | ||
| # This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
|
|
||
| # Shared helpers for rewriting the ddtrace dependency in pyproject.toml. | ||
| # Sourced by scripts/build_layers.sh and scripts/setup_python_env.sh, so the | ||
| # layer build and the unit-test/lint/publish jobs use the same env-var | ||
| # contract and resolve the dep in a single pip pass. | ||
| # | ||
| # Env-var contract (highest precedence first): | ||
| # DD_TRACE_COMMIT Specific dd-trace-py commit SHA from GitHub. | ||
| # DD_TRACE_COMMIT_BRANCH dd-trace-py branch name from GitHub. | ||
| # DD_TRACE_WHEEL Path to a pre-built ddtrace .whl file. | ||
| # UPSTREAM_PIPELINE_ID GitLab pipeline ID from dd-trace-py. Looks up the | ||
| # matching wheel from S3, trying the smaller | ||
| # serverless build first then falling back to the | ||
| # standard manylinux2014 build. | ||
| # | ||
| # When none of these are set, spec_ddtrace_dep is a no-op. | ||
| # | ||
| # When UPSTREAM_PIPELINE_ID is set, also requires: | ||
| # PYTHON_VERSION e.g. "3.12" (used to build the cpXY platform tag) | ||
| # ARCH "amd64" (default) or "arm64" | ||
|
|
||
| # Replace the ddtrace dependency block in pyproject.toml. | ||
| # Usage: replace_ddtrace_dep "ddtrace = { ... }" | ||
| replace_ddtrace_dep() { | ||
| echo "Replacing ddtrace dep with: $1" | ||
| perl -i -0777 -pe "s|ddtrace = \[[^\]]*\]|$1|gs" pyproject.toml | ||
| } | ||
|
|
||
| # Search S3 for a wheel matching basename + index, then rewrite the ddtrace | ||
| # dep to point at the downloaded file. Globals required: | ||
| # S3_BASE, PY_TAG, PLATFORM | ||
| # Returns 0 on success, 1 if no matching wheel was found at the index. | ||
| _search_and_spec_s3_wheel() { | ||
| local basename=$1 | ||
| local index=$2 | ||
| local search_pattern="${basename}-[^\"]*${PY_TAG}[^\"]*${PLATFORM}[^\"]*\.whl" | ||
| local index_url="${S3_BASE}/index-${index}.html" | ||
| echo "Searching for wheel ${search_pattern} in ${index_url}" | ||
| local wheel_file | ||
| wheel_file=$(curl -sSfL "${index_url}" | grep -o "${search_pattern}" | head -n 1 || true) | ||
| if [ -z "$wheel_file" ]; then | ||
| return 1 | ||
| fi | ||
| curl -sSfL "${S3_BASE}/${wheel_file}" -o "${wheel_file}" | ||
| echo "Using S3 wheel: ${wheel_file}" | ||
| replace_ddtrace_dep "${basename} = { file = \"${wheel_file}\" }" | ||
| } | ||
|
|
||
| # Rewrite pyproject.toml's ddtrace dep based on the env-var precedence above. | ||
| # No-op if no override env var is set. Returns non-zero if UPSTREAM_PIPELINE_ID | ||
| # is set but no matching S3 wheel is found. | ||
| spec_ddtrace_dep() { | ||
| if [ -n "${DD_TRACE_COMMIT:-}" ]; then | ||
| replace_ddtrace_dep "ddtrace = { git = \"https://github.com/DataDog/dd-trace-py.git\", rev = \"${DD_TRACE_COMMIT}\" }" | ||
| elif [ -n "${DD_TRACE_COMMIT_BRANCH:-}" ]; then | ||
| replace_ddtrace_dep "ddtrace = { git = \"https://github.com/DataDog/dd-trace-py.git\", branch = \"${DD_TRACE_COMMIT_BRANCH}\" }" | ||
| elif [ -n "${DD_TRACE_WHEEL:-}" ]; then | ||
| local basename | ||
| basename=$(sed 's/^.*\///' <<< "${DD_TRACE_WHEEL%%-*}") | ||
| replace_ddtrace_dep "${basename} = { file = \"${DD_TRACE_WHEEL}\" }" | ||
| elif [ -n "${UPSTREAM_PIPELINE_ID:-}" ]; then | ||
| if [ -z "${PYTHON_VERSION:-}" ]; then | ||
| echo "ERROR: PYTHON_VERSION must be set when UPSTREAM_PIPELINE_ID is set" >&2 | ||
| return 1 | ||
| fi | ||
| S3_BASE="https://dd-trace-py-builds.s3.amazonaws.com/${UPSTREAM_PIPELINE_ID}" | ||
| PY_TAG="cp$(echo "$PYTHON_VERSION" | tr -d '.')" | ||
| if [ "${ARCH:-amd64}" = "amd64" ]; then | ||
| PLATFORM="manylinux2014_x86_64" | ||
| else | ||
| PLATFORM="manylinux2014_aarch64" | ||
| fi | ||
| _search_and_spec_s3_wheel "ddtrace_serverless" "serverless" \ | ||
| || _search_and_spec_s3_wheel "ddtrace" "manylinux2014" \ | ||
| || { echo "ERROR: No matching ddtrace wheel for ${PY_TAG} ${PLATFORM} in pipeline ${UPSTREAM_PIPELINE_ID}" >&2; return 1; } | ||
| fi | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Unless explicitly stated otherwise all files in this repository are licensed | ||
| # under the Apache License Version 2.0. | ||
| # This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
|
|
||
| # Sets up the Python environment for the lint, unit-test, and publish-pypi | ||
| # CI jobs (and for local repro of those flows). Replaces the inline | ||
| # .python-before-script anchor previously defined in | ||
| # ci/input_files/build.yaml.tpl. | ||
| # | ||
| # Steps: | ||
| # 1. (Optional) Rewrite pyproject.toml's ddtrace dep based on the | ||
| # env-var contract documented in scripts/_spec_ddtrace_dep.sh | ||
| # (DD_TRACE_COMMIT / DD_TRACE_COMMIT_BRANCH / DD_TRACE_WHEEL / | ||
| # UPSTREAM_PIPELINE_ID). When dd-trace-py's CI triggers this repo's | ||
| # pipeline it sets UPSTREAM_PIPELINE_ID, so the unit-test job | ||
| # exercises the PR's wheel rather than the released ddtrace. | ||
| # 2. Create and activate a virtualenv ("venv/"). | ||
| # 3. Install lambda-python's runtime + dev dependencies. pip resolves the | ||
| # whole graph in one pass against the (possibly rewritten) pyproject.toml, | ||
| # so any version conflicts surface as install errors instead of | ||
| # runtime surprises. | ||
| # 4. Install poetry. | ||
| # | ||
| # Same dep-resolution path as scripts/build_layers.sh — both source | ||
| # scripts/_spec_ddtrace_dep.sh. | ||
| # | ||
| # DD_TRACE_COMMIT / DD_TRACE_COMMIT_BRANCH build ddtrace from source, which | ||
| # requires cargo, cmake, and a C/C++ toolchain — not present in the slim | ||
| # Python runner images. They are intended for local repro / git-bisect | ||
| # workflows. The dd-trace-py CI trigger uses UPSTREAM_PIPELINE_ID. | ||
| # | ||
| # Venv contract: this script sources venv/bin/activate inside its own | ||
| # subshell, so the activation does NOT persist into the calling job. Calling | ||
| # jobs must `source venv/bin/activate` themselves before running their | ||
| # command (matching the existing pattern in build.yaml.tpl). | ||
| # | ||
| # Environment variables: | ||
| # PYTHON_VERSION Python minor version (e.g. 3.12 or just 12). Required | ||
| # when the UPSTREAM_PIPELINE_ID branch is taken. | ||
|
|
||
| set -e | ||
|
|
||
| # Normalize Python version shorthand (e.g. 12 -> 3.12, 3.12 -> 3.12) | ||
| if [ -n "${PYTHON_VERSION:-}" ]; then | ||
| if [[ "$PYTHON_VERSION" =~ ^[0-9]+$ ]]; then | ||
| PYTHON_VERSION="3.${PYTHON_VERSION}" | ||
| fi | ||
| fi | ||
|
|
||
| # Backup pyproject.toml so the rewrite doesn't persist across runs (matters | ||
| # for local invocations; CI runners are ephemeral but cheap to be tidy). | ||
| cp pyproject.toml pyproject.toml.bak | ||
| cleanup() { | ||
| mv pyproject.toml.bak pyproject.toml 2>/dev/null || true | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| source "$(dirname "$0")/_spec_ddtrace_dep.sh" | ||
| spec_ddtrace_dep | ||
|
|
||
| pip install virtualenv | ||
| virtualenv venv | ||
| source venv/bin/activate | ||
| pip install .[dev] | ||
| pip install poetry | ||
|
|
||
| python -c "import ddtrace; print('ddtrace version:', ddtrace.__version__)" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just for clarification, which case is this covering?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure, but it was in the previous version of this logic when it was still in the
build_layer.shscript.