Possible provider cleanup bug on Provider 0.12.0.

[Christoph203]

Dashboard showed 0 active leases, but Kubernetes still had 3 tenant namespaces running GPU miner workloads. Provider inventory showed 3 of 4 GPUs allocated.

Provider logs repeatedly showed:

lease is out of funds
sending withdraw
payment closed
failed to do lease withdrawal

for lease IDs 27025137, 27025330 and 27025339.

The miner deployments continued running until I manually deleted them. They were NOT recreated by the provider, suggesting the leases were no longer actively managed.

After deleting the namespaces, all GPUs were immediately released.

Provider info:
chris@akash:~$ sudo kubectl describe pod -n akash-services akash-provider-0 | grep Image:
Image: ghcr.io/akash-network/provider:0.12.0 Image: ghcr.io/akash-network/provider:0.12.0

Provider-Wallet: akash1pvskd94jtkph9vgqpf6ecg2as8pvck9w5a7wga
domain: provider.cmolls.de

deployments before cleanup.log
ProviderLog before cleanup.log
pods before cleanup.log

pods after cleanup.log
moreProviderLogs.log
HelmInfo.log

OutOfBalanceSearch.log

Search_JobID_27025137.log
Search_JobID_27025339.log
Search_JobID_27025330.log

Please note the Power-drop on June 5th which indicates one job was deleted. The power difference is exactly 1 RTX3090 needs.

[cmartins88]

Hi @Christoph203 — this looks like the same bug as #515 ("Escrow-depletion lease closes are silently dropped"). Your symptoms are the textbook signature.

What's happening: your leases ran out of escrow (lease is out of funds / payment closed / failed to do lease withdrawal). Leases depleted that way are closed in the escrow module's EndBlocker — there's no transaction, so the EventLeaseClosed is emitted as a block-level event (FinalizeBlockEvents).

The provider's chain-event publisher (chain-sdk go/util/events/publish.go, the version pinned by provider 0.12.0) only iterates blkResults.TxsResults in processBlock and never reads FinalizeBlockEvents — so that close event is silently dropped and never reaches the provider's event bus. Cluster teardown (cluster/service.go) is 100% driven by that bus event with no on-chain reconciliation fallback, so teardownLease is never called: the namespace + manifest CR persist forever and the GPUs stay allocated even though the dashboard shows 0 active leases. That's exactly why kubectl delete namespace immediately freed your GPUs and the workloads weren't recreated — the provider had stopped managing them. Full source-level trace is in #515.

This is now at least the third independent provider hitting it (two in #515 + yours), and it went fleet-wide after AEP-76, so it's not specific to your config.

In the meantime, you don't have to hunt these down by hand. There's a community "dangling deployments" janitor script that reaps exactly this — stale tenant namespaces / manifest CRs / providerhosts / orphaned bids for leases no longer active on-chain:

https://gist.githubusercontent.com/Zblocker64/6da5f4833289270450260d360922cb11/raw/873fcc96934d499b10b5e2889829da11f726cdbd/cleanup_provider.sh

We run it on a guarded daily cron with a flock + timeout and a circuit breaker that skips if the chain query comes back bad, so a transient RPC failure can never trigger a mass delete. Happy to share how we've got it wired up if it's useful — should stop the slow capacity bleed until the upstream fix in #515 lands.

[Christoph203]

Hey Carlos!
Thanks a lot for this detailed response.
I am "happy" to see I'm not the only one having this issue and the root-cause is already identified and is about to be fixed.

I just read the info provider version 0.14.0 is released. Is the fix already included there?

[cmartins88]

Unfortunately no - 0.14.0 does not fix it. I checked the source rather than the release notes, since the notes mention a "resource reclamation" feature that sounds related but isn't. For now I'd keep running the janitor cron I linked above - we have all our providers on 0.14.0 and still need it.

[Christoph203]

Thanks for checking!

What do I need to do with the cleanup script? Just add a cronjob or do I need to set any variables etc.?

Thanks!

[cmartins88]

Hi @Christoph203 just run it once by hand without -y parameter. After that, schedule a cronjob like every 3h (for a busy provider), and use the -y for non-interactive. That's it.