slack: schedule messages instead of dropping them on rate limit#332
slack: schedule messages instead of dropping them on rate limit#332nazriz wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves Slack alert delivery during burst conditions by falling back from chat.postMessage to chat.scheduleMessage when Slack returns HTTP 429, and adds new runtime/STL wrappers to support cancelling scheduled messages and retrieving message history (with metadata) for correlation.
Changes:
- Add a
postMessage429 fallback that schedules messages into the earliest available 5‑minute window (with retries forrestricted_too_manyand schedule API 429s). - Add runtime API endpoints for
chat.deleteScheduledMessageandconversations.history. - Extend the Plaid STL Slack module with new types and helper functions for scheduled delivery responses, scheduled-message deletion, and history retrieval.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| runtime/plaid/src/functions/api.rs | Registers new Slack host functions for scheduled-message deletion and conversations history. |
| runtime/plaid/src/apis/slack/api.rs | Implements 429→schedule fallback, adds Slack API request builders for the new endpoints, and includes unit tests for scheduling helpers. |
| runtime/plaid-stl/src/slack/mod.rs | Adds new STL request/response types and wrappers for scheduled-message deletion and conversations history, plus a new post helper that can represent scheduled delivery. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
obelisk
left a comment
There was a problem hiding this comment.
I think this scheduling should be done at the rule level letting us more quickly respond to changes in the Slack API and keeping the runtime simpler.
ebfda19 to
72ac4bf
Compare
72ac4bf to
62f8ec7
Compare
|
I believe the check failed due to network error, job needs to be re-run, but I am unable to do so. |
Problem
During alert bursts the notifier can exceed Slack’s per-channel chat.postMessage rate limit (~1 message/sec). Slack responds with HTTP 429, and these responses are currently treated as runtime errors, causing alerts to be dropped.
Solution
Expose Slack’s scheduled message APIs and introduce an optional rate-limit mode where
postMessagereturns a structured ratelimited response instead of failing.This allows the PagerDuty Slack notifier to own the retry policy by scheduling messages into future delivery windows, retrying if a scheduling window is full, while keeping the runtime Slack-agnostic.
Tested with a simulated 300-message alert storm resulted in zero dropped alerts and no runtime error logs.