rq: add a garbage collector to the worker

[efahl]

Implement a maintenance hook on the standard redis queue worker to do garbage collection on expired builds. When a result expires from the queue, its data will be removed from the public/store/ directory at the regular maintenance interval (default is every 600 seconds).

[efahl]

I've been running this for about 9 months now, no issues on my local server.

I test it out by setting the ttl values way down in api.py and watching the logs on the server after I queue up a bunch of builds.

+    failure_ttl: str = "3m"
+    result_ttl: str = "15m"

[efahl]

But, are we running out of space due to old podman/docker containers hanging around, too?

Maybe an hourly cron job that does this?

podman container prune --force && podman image prune --force

[codecov]

Codecov Report

Attention: Patch coverage is 43.58974% with 22 lines in your changes missing coverage. Please review.

Project coverage is 88.79%. Comparing base (5e65dec) to head (49ba426).
Report is 241 commits behind head on main.

Files with missing lines	Patch %	Lines
asu/rq.py	42.10%	22 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1370      +/-   ##
==========================================
+ Coverage   80.75%   88.79%   +8.03%     
==========================================
  Files          15       15              
  Lines         977     1258     +281     
==========================================
+ Hits          789     1117     +328     
+ Misses        188      141      -47     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[aparcar]

this change seems unrelated?

[efahl]

It's so the other contents of the init file are not deleted.

[Copilot]

Pull Request Overview

This PR adds a garbage collector to the redis queue worker by implementing a custom GCWorker, which periodically deletes expired builds from the public store directory.

• Updated the podman-compose.yml to launch the worker with garbage collection enabled
• Added a GCWorker class in asu/rq.py to perform periodic clean-ups
• Updated version handling and registration in asu/init.py and the GitHub publish workflow

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
podman-compose.yml	Added scheduler and worker-class arguments to enable GC functionality
asu/rq.py	Introduced GCWorker with a clean_store method for deleting expired builds
asu/init.py	Registered GCWorker, ensuring it is exported
.github/workflows/publish.yml	Updated version assignment mechanism using sed for improved consistency

[Copilot]

The comment contains a spelling error ('insignficant'). It should be corrected to 'insignificant'.

Suggested change

	insignficant, likely due to `stat` calls being the bottleneck.
	(Just for comparison, tests against store mounted on a fast SSD
	insignificant, likely due to `stat` calls being the bottleneck.

[efahl]

Just been playing with the podman API, this might be added to that worker, too. It appears that pruning only affects unused containers/images, so you can safely just call these and reclaim the orphaned space.

$ python -c 'from asu.util import get_podman; print(get_podman().containers.prune())'
{'ContainersDeleted': [], 'SpaceReclaimed': 0}

$ python -c 'from asu.util import get_podman; print(get_podman().images.prune())'
{'ImagesDeleted': [{'Deleted': '4a2e6bf10ec5d4c70ff5b8598193285b6d090d47ad737bfe322c0be298e1e6d1', 'Untagged': ''}], 'SpaceReclaimed': 927715421}

[aparcar]

Right now this is my crontab, however we could also use the GC worker:

@daily cd /srv && find ./store  -ctime +1 -exec rm -rf {} \;
@daily podman volume prune -af