Swamp should support "Scheduled"-type status for execution #428
Description
Users need the ability to on-hold specific workflows, methods and sub-contents of workflows.
The common thread is for the thought and use cases is - separate the intent to act from the act itself.
In regulated or high-consequence environments, you want a reviewable staging area between "I know what needs to happen" and "do it now." The queue gives you that gap for review, collaboration, approval, and coordinated timing. Some real-world examples of where this is may be valuable are shown below:
Planned maintenance windows — You prep a database migration and an app deployment during the day, queue both, then execute them together at 2am during the change window. Nobody is scrambling to kick things off at 2am — it's already staged and reviewed.
# During the day — prep and queue
swamp workflow queue db-migration
swamp workflow queue app-deployment
# Review what's staged
swamp queue list
# Execute together when window has started/all user traffic drained
swamp queue execute --all
Employee onboarding — New hire starts Monday. On Wednesday prior, you queue: create AD account, assign GitHub org access, create Slack account, add to VPN group. On Monday morning, execute --all. If the start date slips, purge --all if they never start.
# Wednesday — queue all the provisioning
swamp method queue ad-account create --input user=jsmith
swamp method queue github-org add-member --input user=jsmith --input org=acme
swamp method queue slack create-account --input user=jsmith
swamp method queue vpn add-to-group --input user=jsmith --input group=engineering
# Review
swamp queue list
# Monday morning — go
swamp queue execute --all
# OR if start date slips and they never join
swamp queue purge --all
Infrastructure decommissioning — You're sunsetting an old environment. Queue the teardown of compute, then networking, then DNS. A senior engineer reviews the queue, confirms it looks right, then releases it at a time that's suitable to them, or releases the parts of the scheduled work that are relevant at that time. Nobody accidentally nukes a live subnet.
# Queue the teardown in order of intent
swamp workflow queue teardown-compute
swamp workflow queue teardown-networking
swamp workflow queue teardown-dns
# Senior engineer reviews
swamp queue list
# Execute selectively, one at a time
swamp queue execute --workflow teardown-compute
# verify it's clean...
swamp queue execute --workflow teardown-networking
# verify...
swamp queue execute --workflow teardown-dns
# OR if they decide a piece isn't ready
swamp queue purge --workflow teardown-dns
Blue/green deployment quality gate — Your workflow has: deploy green environment, run smoke tests, swap load balancer. You put the swap on hold. Smoke tests pass, but you also want a product manager to eyeball the green environment before it goes live. They confirm, you release the swap. If they spot something wrong, you purge and tear down green.
# Run deploy + smoke tests immediately (not queued)
swamp workflow run deploy-green
swamp workflow run smoke-tests --input env=green
# Queue the swap separately
swamp method queue load-balancer swap --input target=green
# PM eyeballs green, gives thumbs up
swamp queue execute --method swap
# OR PM spots a problem — purge the swap and tear down
swamp queue purge --method swap
swamp workflow run teardown-green
This could be worked with a centralised queueing system, that manages the state of the work for anything that isn't fire-and-forget.
The Queue
There should be a new type of interaction that is via a queue, the queue's job is to manage workflow and method run requests so they can be executed later by either the same user, CI or another user manually.
An example of how to add a workflow run to queue:
swamp workflow queue <workflow-name>
For a method:
swamp method queue <model> <method name>
For these cases, swamp will add a new line item to the queue, to show that there is work that can be released. The queue can be purged or executed by the user either in bulk or individually:
swamp queue purge --all will purge the entire queue
swamp queue purge --workflows will purge the queue of all workflows
swamp queue purge --methods will purge the queue of all methods
swamp queue pure --workflow <workflow name> will purge the queue of a single workflow called
swamp queue purge --method <method name> will purge the queue of a single method called
Each workflow and method invocation can only be added to the queue once and there can be no relationships between queued members. E.g. queue item A must finish before queue item B. All those relationships should be handled by the workflow engine itself. I.e. if you need that dependency, wrap the two jobs in a new workflow and map the dependency that way, then add the wrapper to the queue.
If you need to add the same method to the queue twice, either vary them by input, e.g.
swamp method queue <model> <method> --input app=binary and then
swamp method queue <model> <method> --input app=docker
Or wrap both the method executions in a workflow, like the approach in the section above.
To execute items off the queue
swamp queue execute --all will execute the entire queue
swamp queue execute --workflows will execute the queue of all workflows
swamp queue execute --methods will execute the queue of all methods
swamp queue execute --workflow <workflow name> will execute the queue of a single workflow called
swamp queue execute --method <method name> will execute the queue of a single method called
As some examples of the interactions
Workflow jobs
As some examples:
Workflow A
|____________ Job 1 - build
|____________ Job 2 - ship-binary (put on hold)
When you workflow run it, it runs 1, tells you 2 is on-hold and you need to run swamp workflow release ship-binary which will release the second half of the workflow.
Method Runs
You should be able to do the same with methods
swamp method schedule <model> <method>
e.g.
swamp method schedule binary release --input version=v1.0.0