Skip to content

slack: schedule messages instead of dropping them on rate limit#332

Open
nazriz wants to merge 1 commit into
obelisk:mainfrom
nazriz:slack-schedule-on-ratelimit
Open

slack: schedule messages instead of dropping them on rate limit#332
nazriz wants to merge 1 commit into
obelisk:mainfrom
nazriz:slack-schedule-on-ratelimit

Conversation

@nazriz

@nazriz nazriz commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Problem

During alert bursts the notifier can exceed Slack’s per-channel chat.postMessage rate limit (~1 message/sec). Slack responds with HTTP 429, and these responses are currently treated as runtime errors, causing alerts to be dropped.

Solution

Expose Slack’s scheduled message APIs and introduce an optional rate-limit mode where postMessage returns a structured ratelimited response instead of failing.

This allows the PagerDuty Slack notifier to own the retry policy by scheduling messages into future delivery windows, retrying if a scheduling window is full, while keeping the runtime Slack-agnostic.

Tested with a simulated 300-message alert storm resulted in zero dropped alerts and no runtime error logs.

Copilot AI review requested due to automatic review settings July 2, 2026 03:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Slack alert delivery during burst conditions by falling back from chat.postMessage to chat.scheduleMessage when Slack returns HTTP 429, and adds new runtime/STL wrappers to support cancelling scheduled messages and retrieving message history (with metadata) for correlation.

Changes:

  • Add a postMessage 429 fallback that schedules messages into the earliest available 5‑minute window (with retries for restricted_too_many and schedule API 429s).
  • Add runtime API endpoints for chat.deleteScheduledMessage and conversations.history.
  • Extend the Plaid STL Slack module with new types and helper functions for scheduled delivery responses, scheduled-message deletion, and history retrieval.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File Description
runtime/plaid/src/functions/api.rs Registers new Slack host functions for scheduled-message deletion and conversations history.
runtime/plaid/src/apis/slack/api.rs Implements 429→schedule fallback, adds Slack API request builders for the new endpoints, and includes unit tests for scheduling helpers.
runtime/plaid-stl/src/slack/mod.rs Adds new STL request/response types and wrappers for scheduled-message deletion and conversations history, plus a new post helper that can represent scheduled delivery.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread runtime/plaid/src/apis/slack/api.rs Outdated
Comment thread runtime/plaid/src/apis/slack/api.rs Outdated
Comment thread runtime/plaid/src/apis/slack/api.rs
Comment thread runtime/plaid/src/apis/slack/api.rs Outdated
Comment thread runtime/plaid/src/apis/slack/api.rs Outdated
Copilot AI review requested due to automatic review settings July 2, 2026 04:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread runtime/plaid/src/apis/slack/api.rs
Comment thread runtime/plaid/src/apis/slack/api.rs Outdated
Comment thread runtime/plaid/src/apis/slack/api.rs Outdated

@obelisk obelisk left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this scheduling should be done at the rule level letting us more quickly respond to changes in the Slack API and keeping the runtime simpler.

Copilot AI review requested due to automatic review settings July 3, 2026 05:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread runtime/plaid/src/apis/slack/api.rs
Comment thread runtime/plaid/src/apis/slack/api.rs
Comment thread runtime/plaid-stl/src/slack/mod.rs Outdated
Copilot AI review requested due to automatic review settings July 3, 2026 05:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread runtime/plaid-stl/src/slack/mod.rs Outdated
Comment thread runtime/plaid/src/apis/slack/api.rs
@nazriz nazriz force-pushed the slack-schedule-on-ratelimit branch from ebfda19 to 72ac4bf Compare July 3, 2026 05:39
@nazriz nazriz force-pushed the slack-schedule-on-ratelimit branch from 72ac4bf to 62f8ec7 Compare July 3, 2026 05:49
@nazriz

nazriz commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

I believe the check failed due to network error, job needs to be re-run, but I am unable to do so.

@nazriz nazriz requested a review from obelisk July 3, 2026 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants