fix: sort aggregation values to avoid deadlocks in process_pending

[gustavobtflores]

Description

Fixes a PostgreSQL deadlock in the tree_listing table caused by concurrent updates from the ingester and aggregation processor. When two transactions locked overlapping rows in different orders, PostgreSQL detected a deadlock and killed one transaction. This fix ensures all transactions acquire row locks in a deterministic order by sorting values by their lock-key columns before executing executemany().

Why the deadlock happens

Two concurrent processes update the tree_listing table inside separate transactions:

1. The ingester runs aggregate_checkouts_and_pendings() inside transaction.atomic(), which calls update_tree_listing(), an INSERT ... ON CONFLICT DO UPDATE on tree_listing.

2. The aggregation processor runs _process_tree_listing() inside transaction.atomic() — a plain UPDATE tree_listing via executemany().

Both processes use cursor.executemany() which sends one statement per row. PostgreSQL acquires row-level locks incrementally as each statement executes. When two transactions lock overlapping rows in different orders, PostgreSQL detects a deadlock and kills one of them.

The iteration order came from dict.values() (insertion order in Python 3.7+), which depends on the arrival order of tests/builds, not on any stable sort key. So two concurrent batches that touched overlapping tree_listing rows could easily lock them in opposite order, causing deadlocks.

By sorting both writers by the same lock-key columns in the same deterministic order, all transactions now acquire row locks consistently, making deadlocks theoretically impossible.

Changes

• Sort checkout_values in update_tree_listing() by the unique constraint columns (origin, tree_name, git_repository_url, git_repository_branch, git_commit_hash) before the INSERT ... ON CONFLICT DO UPDATE
• Sort values in _process_tree_listing() (process_pending_aggregations.py) by the UPDATE WHERE clause columns (origin, tree_name, git_repository_branch, git_repository_url, git_commit_hash) before executing the UPDATE

How to test

1. Run the backend test suite: docker compose exec test_backend poetry run pytest
2. Verify no regressions in tree listing aggregation tests

[Copilot]

Pull request overview

Fixes PostgreSQL deadlocks on tree_listing caused by concurrent executemany() updates acquiring row locks in inconsistent orders, by sorting batch value lists before executing SQL.

Changes:

• Sorts _process_tree_listing() UPDATE batches before cursor.executemany() to enforce a deterministic lock acquisition order.
• Sorts update_tree_listing() UPSERT batches before cursor.executemany() to enforce a deterministic lock acquisition order.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
backend/kernelCI_app/management/commands/process_pending_aggregations.py	Sorts tree_listing UPDATE batch values prior to executemany() to reduce deadlock risk.
backend/kernelCI_app/management/commands/helpers/aggregation_helpers.py	Sorts tree_listing UPSERT batch values prior to executemany() to reduce deadlock risk.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MarceloRobert]

LGTM

[MarceloRobert]

nit: avoid using Any

[alanpeixinho]

LGTM