-
Notifications
You must be signed in to change notification settings - Fork 2
Ops 5780 cap 1 ci to build and publish arm based docker images #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
5b3b8eb
180b46f
739ec19
99268b0
f687190
33834a2
006f490
c51c6aa
e52024d
4ddb8ef
12393d2
e69797f
23d8f3f
d1327e1
06b66db
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -24,19 +24,34 @@ on: | |||||
| description: "TriggeredBy: (default manual) Indicates what triggered this workflow. Set to 'ubuntu-refresh' to suppress Slack notifications." | ||||||
| default: "manual" | ||||||
| type: string | ||||||
|
|
||||||
| permissions: | ||||||
| contents: write | ||||||
| id-token: write | ||||||
|
|
||||||
| jobs: | ||||||
| Staging_Build_And_Push_Base_Image_From_Version: | ||||||
| runs-on: [mend-self-hosted, profile=developer-platform-xlarge-od] | ||||||
| runs-on: [mend-self-hosted, "profile=${{ matrix.runner_profile }}"] | ||||||
| strategy: | ||||||
| fail-fast: false | ||||||
| matrix: | ||||||
| include: | ||||||
| - arch: amd64 | ||||||
| runner_profile: developer-platform-xlarge-od | ||||||
| tag_suffix: "" | ||||||
| platform_info: "" | ||||||
| handle_git: false | ||||||
| - arch: arm64 | ||||||
| runner_profile: renovate-arm-od | ||||||
| tag_suffix: "-arm64" | ||||||
| platform_info: "ARM64" | ||||||
| handle_git: false | ||||||
|
|
||||||
| steps: | ||||||
| - name: Check out code | ||||||
| uses: actions/checkout@v3 | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🌐 Web query:
💡 Result:
What’s recommended (2024–2025 and now):
Sources: 🌐 Web query:
💡 Result:
Sources Citations:
Upgrade - uses: actions/checkout@v3
+ uses: actions/checkout@v4📝 Committable suggestion
Suggested change
🧰 Tools🪛 actionlint (1.7.11)[error] 52-52: the runner of "actions/checkout@v3" action is too old to run on GitHub Actions. update the action's version to fix this issue (action) 🤖 Prompt for AI Agents |
||||||
| with: | ||||||
| ref: "main" | ||||||
| ref: ${{ github.ref }} | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. make sure it's main |
||||||
| fetch-depth: 0 | ||||||
|
|
||||||
| - name: Configure AWS credentials | ||||||
|
|
@@ -72,26 +87,27 @@ jobs: | |||||
| with: | ||||||
| username: ${{ secrets.DOCKERHUB_USERNAME }} | ||||||
| password: ${{ secrets.DOCKERHUB_TOKEN }} | ||||||
|
|
||||||
| - name: Build Docker Images | ||||||
| - name: Build Docker Images (${{ matrix.arch }}) | ||||||
| env: | ||||||
| ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }} | ||||||
| run: | | ||||||
| echo "🔨 Starting Docker build process..." | ||||||
| ./bin/build.sh "${{ github.event.inputs.Tag }}" "$ECR_REGISTRY" "${{ github.event.inputs.CopyVersionScript }}" | ||||||
| echo "✅ Docker build completed successfully" | ||||||
| echo "🔨 Starting ${{ matrix.arch }} Docker build process..." | ||||||
| ./bin/build.sh "${{ github.event.inputs.Tag }}" "$ECR_REGISTRY" "${{ github.event.inputs.CopyVersionScript }}" "${{ matrix.tag_suffix }}" | ||||||
| echo "✅ ${{ matrix.arch }} Docker build completed successfully" | ||||||
| shell: bash | ||||||
|
|
||||||
| - name: Publish to ECR | ||||||
| - name: Publish to ECR (${{ matrix.arch }}) | ||||||
| env: | ||||||
| ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }} | ||||||
| run: | | ||||||
| echo "📤 Starting ECR publish process..." | ||||||
| ./bin/publish.sh "${{ github.event.inputs.Tag }}" "$ECR_REGISTRY" | ||||||
| echo "✅ ECR publish completed successfully" | ||||||
| echo "📤 Starting ${{ matrix.arch }} ECR publish process..." | ||||||
| ./bin/publish.sh "${{ github.event.inputs.Tag }}" "$ECR_REGISTRY" "${{ matrix.tag_suffix }}" | ||||||
| echo "✅ ${{ matrix.arch }} ECR publish completed successfully" | ||||||
| shell: bash | ||||||
|
|
||||||
| - name: Branch, Commit, Push and Tag Changes | ||||||
| if: matrix.handle_git == true | ||||||
| env: | ||||||
| GH_TOKEN: ${{ github.token }} | ||||||
| run: | | ||||||
|
|
@@ -138,11 +154,11 @@ jobs: | |||||
| git push | ||||||
| shell: bash | ||||||
|
|
||||||
| - name: Notify Slack - Images Ready | ||||||
| if: always() && github.event.inputs.TriggeredBy != 'ubuntu-refresh' | ||||||
| env: | ||||||
| ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }} | ||||||
| SLACK_WEBHOOK_URL: ${{ secrets.STG_SLACK_WEBHOOK_URL }} | ||||||
| run: | | ||||||
| echo "📤 Sending Slack notification..." | ||||||
| ./bin/notify-slack.sh "${{ github.event.inputs.Tag }}" "${{ job.status }}" "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" | ||||||
| # - name: Notify Slack - Images Ready (${{ matrix.arch }}) | ||||||
| # if: always() && github.event.inputs.TriggeredBy != 'ubuntu-refresh' | ||||||
| # env: | ||||||
| # ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }} | ||||||
| # SLACK_WEBHOOK_URL: ${{ secrets.STG_SLACK_WEBHOOK_URL }} | ||||||
| # run: | | ||||||
| # echo "📤 Sending Slack notification for ${{ matrix.arch }}..." | ||||||
| # ./bin/notify-slack.sh "${{ github.event.inputs.Tag }}" "${{ job.status }}" "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" "${{ matrix.platform_info }}" | ||||||
|
Comment on lines
+157
to
+164
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Slack notification is fully commented out — no build status notifications will be sent. The previous workflow notified Slack on every run (with 🤖 Prompt for AI Agents
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why not? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
That said, there's a likely valid reason here: with the new matrix strategy (amd64 + arm64), the notification step would fire twice per workflow trigger — once for each arch — which could be noisy or confusing. That's a reasonable motivation to temporarily suppress it while figuring out the right approach (e.g., send one aggregated notification, or only notify from the The flag is simply a reminder that commenting it out entirely means no failure alerts either, which is a regression in observability. If this is intentional for the WIP phase, a short comment like 🧠 Learnings used |
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,7 @@ set -e | |
| RELEASE=$1 | ||
| REGISTRY_PREFIX=$2 | ||
| COPY_VERSIONS_JSON=$3 | ||
| TAG_SUFFIX=$4 # Optional: "-arm64" or empty for AMD64 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. probably need to update based on main |
||
|
|
||
|
|
||
| if [ -z "$RELEASE" ]; then | ||
|
|
@@ -32,41 +33,41 @@ echo "Building images with registry prefix: ${REGISTRY_PREFIX}" | |
| docker pull ubuntu:24.04 | ||
|
|
||
|
|
||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-scanner-sast:${RELEASE} -f repo-integrations/scanner/DockerfileSast . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-controller:${RELEASE} -f repo-integrations/controller/Dockerfile . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-remediate:${RELEASE} -f repo-integrations/remediate/Dockerfile . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE} -f repo-integrations/scanner/Dockerfile . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}-full -f repo-integrations/scanner/Dockerfile.full . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-scanner-sast:${RELEASE}${TAG_SUFFIX} -f repo-integrations/scanner/DockerfileSast . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-controller:${RELEASE}${TAG_SUFFIX} -f repo-integrations/controller/Dockerfile . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-remediate:${RELEASE}${TAG_SUFFIX} -f repo-integrations/remediate/Dockerfile . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}${TAG_SUFFIX} -f repo-integrations/scanner/Dockerfile . | ||
| docker build --no-cache -t ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}${TAG_SUFFIX}-full -f repo-integrations/scanner/Dockerfile.full . | ||
|
|
||
|
|
||
| #Validate built images successfully created | ||
| echo "🔍 Validating built images..." | ||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-scanner-sast:${RELEASE} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-scanner-sast:${RELEASE} was not built successfully" | ||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-scanner-sast:${RELEASE}${TAG_SUFFIX} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-scanner-sast:${RELEASE}${TAG_SUFFIX} was not built successfully" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it might be already part of the release tag make sure to take main, if not then looks fine |
||
| exit 1 | ||
| fi | ||
| echo "✅ SAST scanner image validated" | ||
|
|
||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-controller:${RELEASE} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-controller:${RELEASE} was not built successfully" | ||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-controller:${RELEASE}${TAG_SUFFIX} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-controller:${RELEASE}${TAG_SUFFIX} was not built successfully" | ||
| exit 1 | ||
| fi | ||
| echo "✅ Controller image validated" | ||
|
|
||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-remediate:${RELEASE} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-remediate:${RELEASE} was not built successfully" | ||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-remediate:${RELEASE}${TAG_SUFFIX} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-remediate:${RELEASE}${TAG_SUFFIX} was not built successfully" | ||
| exit 1 | ||
| fi | ||
| echo "✅ Remediate image validated" | ||
|
|
||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE} was not built successfully" | ||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}${TAG_SUFFIX} 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}${TAG_SUFFIX} was not built successfully" | ||
| exit 1 | ||
| fi | ||
| echo "✅ SCA scanner image validated" | ||
|
|
||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}-full 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}-full was not built successfully" | ||
| if [ -z "$(docker images -q ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}${TAG_SUFFIX}-full 2> /dev/null)" ]; then | ||
| echo "❌ ${REGISTRY_PREFIX}/base-repo-scanner:${RELEASE}${TAG_SUFFIX}-full was not built successfully" | ||
| exit 1 | ||
| fi | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -122,7 +122,7 @@ if ! grep -q "# END OF BASE IMAGE" repo-integrations/scanner/Dockerfile.full; th | |
| exit 1 | ||
| fi | ||
|
|
||
| apply_dockerfile_modifications "repo-integrations/scanner/Dockerfile.full" "scanner" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it might be wrong - both files using scanner
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see you changed the txt files . not sure why, it would break updates for both type of files. |
||
| apply_dockerfile_modifications "repo-integrations/scanner/Dockerfile.full" "scanner-full" | ||
| if [ "$COPY_VERSIONS_JSON" = true ]; then | ||
| append_scanner_script_support "repo-integrations/scanner/Dockerfile.full" | ||
| fi | ||
|
|
@@ -139,6 +139,9 @@ if [ ! -f $remediateDockerfile ]; then | |
| fi | ||
|
|
||
| sed '/# END OF BASE IMAGE/ q' $remediateDockerfile > repo-integrations/remediate/Dockerfile | ||
|
|
||
| apply_dockerfile_modifications "repo-integrations/remediate/Dockerfile" "remediate" | ||
|
|
||
| # ================================ | ||
| # SUMMARY OUTPUT | ||
| # ================================ | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -35,11 +35,13 @@ modify_dockerfile() { | |||||
| case "$action" in | ||||||
| "REMOVE") | ||||||
| # Remove lines matching pattern | ||||||
| sed -i.bak "/${pattern}/d" "$temp_file" | ||||||
| # Use @ as delimiter to avoid conflicts with / in paths and | in pipes | ||||||
| sed -i.bak "\\@${pattern}@d" "$temp_file" | ||||||
| ;; | ||||||
| "COMMENT") | ||||||
| # Comment out lines matching pattern - simple single line | ||||||
| sed -i.bak "s/^\(.*${pattern}.*\)$/# \1/" "$temp_file" | ||||||
| # Use @ as delimiter to avoid conflicts with / in paths and | in pipes | ||||||
| sed -i.bak "s@^\(.*${pattern}.*\)$@# \1@" "$temp_file" | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The string
Fix by using mixed quoting so 🐛 Proposed fix- sed -i.bak "s@^\(.*${pattern}.*\)$@# \1@" "$temp_file"
+ sed -i.bak 's@^\(.*'"${pattern}"'.*\)$@# \1@' "$temp_file"📝 Committable suggestion
Suggested change
🧰 Tools🪛 Shellcheck (0.11.0)[error] 44-44: Argument mixes string and array. Use * or separate argument. (SC2145) 🤖 Prompt for AI Agents
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure why changing this - need to be careful not to break existing comments. Once you take the main, you will see a few more was added. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The original code using sed -i.bak "s/^\(.*${pattern}.*\)$/# \1/" "$temp_file"There's no The new code that changed to sed -i.bak "s@^\(.*${pattern}.*\)$@# \1@" "$temp_file"Inside a double-quoted string, bash sees The fix with mixed quoting keeps the sed -i.bak 's@^\(.*'"${pattern}"'.*\)$@# \1@' "$temp_file"Here 🧠 Learnings used |
||||||
| ;; | ||||||
| "COMMENT_BLOCK") | ||||||
| # Comment out multi-line blocks - handle both pattern in RUN line and in preceding comments | ||||||
|
|
@@ -128,17 +130,20 @@ EOF | |||||
| ;; | ||||||
| "ADD_AFTER") | ||||||
| # Add replacement after line matching pattern | ||||||
| sed -i.bak "/${pattern}/a\\ | ||||||
| # Use @ as delimiter to avoid conflicts with / in paths and | in pipes | ||||||
| sed -i.bak "\\@${pattern}@a\\ | ||||||
| ${replacement}" "$temp_file" | ||||||
| ;; | ||||||
| "ADD_BEFORE") | ||||||
| # Add replacement before line matching pattern | ||||||
| sed -i.bak "/${pattern}/i\\ | ||||||
| # Use @ as delimiter to avoid conflicts with / in paths and | in pipes | ||||||
| sed -i.bak "\\@${pattern}@i\\ | ||||||
| ${replacement}" "$temp_file" | ||||||
| ;; | ||||||
| "REPLACE") | ||||||
| # Replace entire line matching pattern | ||||||
| sed -i.bak "s/.*${pattern}.*/${replacement}/" "$temp_file" | ||||||
| # Use @ as delimiter to avoid conflicts with / in paths and | in pipes | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same as above |
||||||
| sed -i.bak "s@.*${pattern}.*@${replacement}@" "$temp_file" | ||||||
| ;; | ||||||
| *) | ||||||
| echo " Warning: Unknown action '$action', skipping" | ||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| # Configuration file for remediate Dockerfile modifications | ||
| # Format: ACTION:PATTERN:REPLACEMENT | ||
|
|
||
| # No changes required for remediate (as per your requirement) | ||
| # You can add modifications here if needed in the future | ||
| # Fix ARM64 compatibility: Remove amd64-specific base image reference | ||
| # This makes the base image multi-arch compatible (pulls correct arch automatically) | ||
| REPLACE:BASE_IMAGE=amd64:ARG BASE_IMAGE=ubuntu:24.04 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| # Configuration file for scanner Dockerfile.full modifications | ||
| # Format: ACTION:PATTERN:REPLACEMENT | ||
|
|
||
| # Comment out the entire mono installation block | ||
| COMMENT_BLOCK:Install Mono | ||
|
|
||
| # Comment out the entire nuget installation block | ||
| COMMENT_BLOCK:Install Nuget CLI | ||
|
|
||
| # Comment out only the specific nuget install line - precise and safe | ||
| COMMENT:apt-get install.*nuget | ||
|
|
||
| # ARM64 compatibility fixes (only for Dockerfile.full) | ||
|
|
||
| # Fix libssl package to use architecture-aware download | ||
| # Comment out both lines of the AMD64-specific block and add a new arch-aware RUN command | ||
| COMMENT:libssl1.1_1.1.1f-1ubuntu2.24_amd64.deb | ||
| COMMENT: dpkg -i /tmp/libssl1.1.deb | ||
| ADD_AFTER: dpkg -i /tmp/libssl1.1.deb:RUN ARCH=$(uname -m); ARCH_SUFFIX=${ARCH/x86_64/amd64}; ARCH_SUFFIX=${ARCH_SUFFIX/aarch64/arm64} && curl -o /tmp/libssl1.1.deb "https://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2.24_${ARCH_SUFFIX}.deb" && dpkg -i /tmp/libssl1.1.deb | ||
|
|
||
| # Fix Miniconda installer to use architecture-aware download | ||
| # Comment out the x86_64-specific line and add a new arch-aware one | ||
| COMMENT:Miniconda3-py39_24.1.2-0-Linux-x86_64.sh | ||
| ADD_AFTER:Miniconda3-py39_24.1.2-0-Linux-x86_64.sh: wget https://repo.anaconda.com/miniconda/Miniconda3-py39_24.1.2-0-Linux-$(uname -m).sh -O ${USER_HOME}/miniconda3/miniconda.sh && \\ |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
handle_git: falsefor both matrix entries fully disables theBranch, Commit, Push and Tag Changesstep.Both
amd64andarm64sethandle_git: false, so theif: matrix.handle_git == trueguard on Line 113 is never satisfied. Release branch creation, tagging, and merge-to-develop will never execute. Before this PR those operations ran unconditionally; this is a functional regression.If the intent is that only one architecture performs git operations (as is typical — git work should not be duplicated per-arch), the
amd64entry should havehandle_git: true:- arch: amd64 runner_profile: developer-platform-xlarge-od tag_suffix: "" platform_info: "" - handle_git: false + handle_git: true🤖 Prompt for AI Agents
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds about right
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nadav-t, glad we're aligned! Here's the fix to apply — only theamd64entry should drive git operations so the release branch, tag, and merge-to-develop run exactly once:- arch: amd64 runner_profile: developer-platform-xlarge-od tag_suffix: "" platform_info: "" - handle_git: false + handle_git: true - arch: arm64 runner_profile: renovate-arm-od tag_suffix: "-arm64" platform_info: "ARM64" handle_git: falseThis ensures the
Branch, Commit, Push and Tag Changesstep (guarded byif: matrix.handle_git == true) executes exactly once — on theamd64runner — restoring the pre-PR behaviour.🧠 Learnings used