feat: add monitoring to cronjobs

[MarceloRobert]

Changes

• Added healthcheck_id parameter to relevant commands used in the cronjobs
• Added map from healthcheck ids to healthcheck.io uuid, the ids reflect the defined cronjobs and are set via environment_variable
• Added wrapper to run functions with healthcheck.io pings for start, success and failure
• Applied healthcheck changes to main cronjob execution

How to test

For this PR, the ideal way to test would be with a healthcheck account and uuid so you can monitor the pings. Follow the instructions there to create an account and a check.

• In the code, set one of the cronjobs to notifications --action=fake-report and set a small interval, such as */2 * * * * (run every 2 minutes). That is:

(
    "*/2 * * * *",
    "django.core.management.call_command",
    [
        "notifications",
        "--monitoring-id=notifications_new_issues",
        "--action=fake_report",
    ],
),

• You won't need any other cronjob for testing, but if you want to make more than one check you're welcome
• Check that the cronjobs are not going to send email reports accidentally
• Get your healthcheck id and use it in the cronjobs
• Start the cronjobs either via docker compose up backend --build or locally with poetry run ./manage.py crontab add (docker is the recommended way)
• In the healthcheck.io dashboard, see the details of your healthcheck and monitor the received pings

The cronjobs should even if the ping fails or the monitoring_id is empty

Required production changes

• Create 6 different checks for the 6 monitored cronjobs
• Set up on Discord a desired webhook (could reuse the already-present ones)
• Set up in the checks the integration to webhook, POSTing to the discord webhook when a check goes down. Example of body:

{
  "avatar_url": "https://avatars.githubusercontent.com/u/11725450?s=200&v=4",
  "username": "KernelCI Healthcheck Notifications",
  "content": "Healthcheck.io Alert: $NAME service went down!"
}

Also set the request headers to Content-Type: application/json

• Disable the default email notification integration in the checks

Closes #1792

[bhcopeland]

Awesome to see healthchecks.io being used. Have you setup an account? We need to make sure one global KCI account is used with me/denys (@nuclearcat) (and you guys on it). Lets talk about this in tomorrows meeting

[MarceloRobert]

Awesome to see healthchecks.io being used. Have you setup an account? We need to make sure one global KCI account is used with me/denys (@ nuclearcat) (and you guys on it). Lets talk about this in tomorrows meeting

Yeah, I have setup a personal account so I can test this later. Since this change won't stop the cronjobs that don't have monitoring we can go checking the accounts without blocking the PR.

[alanpeixinho]

nit: we can use urllib.parse.urljoin

[MarceloRobert]

urljoin actually was a bit worse because if you join url/middle with end, it will result in url/end instead of url/middle/end. Formatting the strings seems to be safe for this operation

[MarceloRobert]

@bhcopeland all set up and tested, this is now ready for review. If you want to move forward with the healthcheck.io integration, I left some instructions on what to do.

cc @nuclearcat

[nuclearcat]

I will proceed tomorrow with that, thank you very much!

[gustavobtflores]

LGTM, just update the docs to reflect the variables set on .env.example

[gustavobtflores]

these seem different from the ones in .env.example

[MarceloRobert]

I shortened the names in settings but forgot about them here, fixed.

[dede999]

The success path special-cases status != "success" to avoid appending a
subpath, making the URL-building logic fragile. healthcheck.io also accepts
/{uuid}/0 for success, but more importantly, this branching makes the code
harder to follow and test.

Consider extracting a helper:

    def _build_status_url(base: str, status: str) -> str:
        if status == "success":
            return base
        return f"{base.rstrip('/')}/{status}"

[MarceloRobert]

I included the status into the _resolve_monitoring_url function

[dede999]

nit: out() is used for both info and error messages. The rest of the
backend uses logging.getLogger(__name__) with appropriate levels. Consider:

logger = logging.getLogger(__name__)
# then: logger.info(...), logger.warning(...), logger.error(...)

[MarceloRobert]

I think I was having problems with a simple logger before, but I replaced out for log_message and it seems to be ok now

[dede999]

nit: HEALTHCHECK_BASE_URL is a module-level constant but is patched directly
in tests via @patch(... "HEALTHCHECK_BASE_URL" ...). This couples tests to
internals and makes it harder to change the constant's location later.

Consider moving it to settings.py alongside HEALTHCHECK_MONITORING_PATH_MAP,
which would make _resolve_monitoring_url purely functional and easier to test

[MarceloRobert]

Done

[alanpeixinho]

Got the pings.