Skip to content

Add DeadLetterPublisher for publishing poisoned messages to a DLQ topic - #5089

Open
mauriceyap wants to merge 1 commit into
dlq-ingester-3from
dlq-ingester-4
Open

Add DeadLetterPublisher for publishing poisoned messages to a DLQ topic#5089
mauriceyap wants to merge 1 commit into
dlq-ingester-3from
dlq-ingester-4

Conversation

@mauriceyap

Copy link
Copy Markdown
Collaborator

A new self-contained pulsarutils.DeadLetterPublisher which wraps a Pulsar producer, publishes a payload plus DeadLetterMetadata (original topic, subscription, attempt count, last error, message IDs) as message properties. Also adds MessageIdsToStrings, a small helper for rendering []pulsar.MessageID as strings for serialization.

(Not wired into anything yet, so
this PR is inert with respect to running behaviour — pure new capability, reviewable purely on its own correctness - producer options, timeout handling, error wrapping).

@greptile-apps

greptile-apps Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds a reusable Pulsar dead-letter publishing capability.

  • Introduces DeadLetterPublisher with configurable compression and send timeout.
  • Encodes original-message context as Pulsar message properties.
  • Adds a helper for rendering Pulsar message IDs as strings.
  • Documents the shared pulsarutils package and its publishers.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains within the scope of the available previous review threads.

Important Files Changed

Filename Overview
internal/common/pulsarutils/dlq_publisher.go Adds the dead-letter metadata model and a self-contained asynchronous Pulsar publisher with timeout and close handling.
internal/common/pulsarutils/dlq.go Adds a helper that converts Pulsar message IDs to their string representations.
internal/common/pulsarutils/doc.go Adds package documentation covering Pulsar construction and publishing helpers.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant DLP as DeadLetterPublisher
    participant Producer as Pulsar Producer
    participant DLQ as Dead-letter Topic
    Caller->>DLP: Publish(ctx, payload, metadata)
    DLP->>DLP: Create timeout context
    DLP->>DLP: Convert metadata to properties
    DLP->>Producer: SendAsync(payload, properties)
    Producer->>DLQ: Publish message
    Producer-->>DLP: Completion callback
    DLP-->>Caller: Success or wrapped error
Loading

Reviews (5): Last reviewed commit: "Add DeadLetterPublisher for publishing p..." | Re-trigger Greptile

@datadog-armadaproject

datadog-armadaproject Bot commented Aug 3, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 1 Pipeline job failed

CI | All jobs succeeded   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c6a844a | Docs | Datadog PR Page | Give us feedback!

@mergify

mergify Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

nikola-jokic
nikola-jokic previously approved these changes Aug 5, 2026
A new self-contained `pulsarutils.DeadLetterPublisher` which wraps a Pulsar producer, publishes a
payload plus `DeadLetterMetadata` (original topic, subscription, attempt count, last error,
message IDs) as message properties. Also adds `MessageIdsToStrings`, a small helper for
rendering `[]pulsar.MessageID` as strings for serialization.

(Not wired into anything yet, so
this PR is inert with respect to running behaviour — pure new capability, reviewable purely on
its own correctness - producer options, timeout handling, error wrapping).

Signed-off-by: Maurice Yap <mauriceyap@hotmail.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants