Skip to content

S3 upload failure with new version 4.1.100 #467

Closed
Closed
S3 upload failure with new version 4.1.100#467
Labels
bug

Description

@lukey78
lukey78
opened on Mar 17, 2026

Summary

With the latest image update to v4.1.100 our backups in the cluster fail when moving the backups to an S3 storage. This still works with stacks running a previous version.

Docker Hash not working:
sha256:761d8613c2c5921ea71f91408eadfe2ad43294c0731abc5b5e0cfc241696e08a

Logs

DEBUG_MODE=TRUE backup-now

026-03-17.09:45:09 [DEBUG] /etc/services.available/dbbackup-01/run ** [01-db__appDb] Skipping SSL verification for HTTPS S3 Hosts
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-01/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ s3_ssl=--no-verify-ssl
+ [[ -n os.alp2.flow.swiss ]]
+ PARAM_AWS_ENDPOINT_URL=' --endpoint-url https://os.alp2.flow.swiss'
+ silent aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz --no-verify-ssl
+ '[' TRUE = true ']'
+ '[' '' = true ']'
+ '[' debug = debug ']'
+ aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload failed: ./mysql_appDb_db_20260317-094508.sql.gz to s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz SSL validation failed for https://os.alp2.flow.swiss/d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz EOF occurred in violation of protocol (_ssl.c:2427)
+ move_exit_code=1
+ '[' MD5 '!=' none ']'
+ silent run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5 s3://d5f9e-db-backup-test/cdw-dev/ --no-verify-ssl
+ '[' TRUE = true ']'
+ '[' '' = true ']'
+ '[' debug = debug ']'
+ run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5 s3://d5f9e-db-backup-test/cdw-dev/ --no-verify-ssl
+ sudo -Eu dbbackup aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5 s3://d5f9e-db-backup-test/cdw-dev/ --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload failed: ./mysql_appDb_db_20260317-094508.sql.gz.md5 to s3://d5f9e-db-backup-test/cdw-dev/mysql_appDb_db_20260317-094508.sql.gz.md5 An error occurred (ValidationError) when calling the PutObject operation: The input fails to satisfy the constraints specified by an AWS service.
+ run_as_user rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5
+ sudo -Eu dbbackup rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz.md5
+ run_as_user rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ run_as_user rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/01_dbbackup.JMhYOE/mysql_appDb_db_20260317-094508.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ check_exit_code move mysql_appDb_db_20260317-094508.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${1}" in
+ write_log debug 'Move exit Code is 0'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:45:17 [DEBUG] /etc/services.available/dbbackup-01/run ** [01-db__appDb] Move exit Code is 0
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-01/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${move_exit_code}" in
+ write_log error 'Moving of backup '\''mysql_appDb_db_20260317-094508.sql.gz'\'' reported errors'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:45:17 [ERROR] /etc/services.available/dbbackup-01/run ** [01-db__appDb] Moving of backup 'mysql_appDb_db_20260317-094508.sql.gz' reported errors

In contrast, with an older version (in this case 4.1.21) this parts looks like this:

2026-03-17.09:50:46 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Moving backup to S3 Bucket
[...]
+ write_log debug 'Skipping SSL verification for HTTPS S3 Hosts'
2026-03-17.09:50:46 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Skipping SSL verification for HTTPS S3 Hosts
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-02/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ s3_ssl=--no-verify-ssl
+ [[ -n os.alp2.flow.swiss ]]
+ PARAM_AWS_ENDPOINT_URL=' --endpoint-url https://os.alp2.flow.swiss'
+ silent aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz --no-verify-ssl
+ '[' true = true ']'
+ aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload: ./pgsql_app_en101b-db_20260317-095045.sql.gz to s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz
+ move_exit_code=0
+ '[' MD5 '!=' none ']'
+ silent run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5 s3://d5f9e-db-backup-test/even-dev/ --no-verify-ssl
+ '[' true = true ']'
+ run_as_user aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5 s3://d5f9e-db-backup-test/even-dev/ --no-verify-ssl
+ sudo -Eu dbbackup aws --endpoint-url https://os.alp2.flow.swiss s3 cp /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5 s3://d5f9e-db-backup-test/even-dev/ --no-verify-ssl
/usr/lib/python3.12/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'os.alp2.flow.swiss'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
upload: ./pgsql_app_en101b-db_20260317-095045.sql.gz.md5 to s3://d5f9e-db-backup-test/even-dev/pgsql_app_en101b-db_20260317-095045.sql.gz.md5
+ run_as_user rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5
+ sudo -Eu dbbackup rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz.md5
+ run_as_user rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ run_as_user rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ sudo -Eu dbbackup rm -rf /tmp/backups/02_dbbackup.bgbeVY/pgsql_app_en101b-db_20260317-095045.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ check_exit_code move pgsql_app_en101b-db_20260317-095045.sql.gz
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${1}" in
+ write_log debug 'Move exit Code is 0'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:50:48 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Move exit Code is 0
+ output_on
+ '[' true = true ']'
+ case "$(basename "$0")" in
++ basename /etc/services.available/dbbackup-02/run
+ case "$PWD" in
+ set -x
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ case "${move_exit_code}" in
+ write_log debug 'Moving of backup '\''pgsql_app_en101b-db_20260317-095045.sql.gz'\'' completed successfully'
+ var_true ''
+ '[' '' = true ']'
+ '[' '' = yes ']'
+ output_off
+ '[' true = true ']'
+ set +x
2026-03-17.09:50:48 [DEBUG] /etc/services.available/dbbackup-02/run ** [02-en101b-db__app] Moving of backup 'pgsql_app_en101b-db_20260317-095045.sql.gz' completed successfully
```

Activity

  1. lukey78
    added
    bug
    on Mar 17, 2026

  2. NeckBeardPrince commented on Aug 20, 2026

    @NeckBeardPrince
    NeckBeardPrince
    on Aug 20, 2026

    Same issue on 4.9.0 how has this not been addressed yet?!

  3. NeckBeardPrince commented on Aug 20, 2026

    @NeckBeardPrince
    NeckBeardPrince
    on Aug 20, 2026

    Wait... by default you set --no-verify-ssl. That would skip SSL verification for https://s3.amazonaws.com.

  4. nfrastack
    locked and limited conversation to collaborators on Sep 4, 2026
  5. tiredofit
    converted this issue into a discussion #503 on Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions