Enable dead-letter topics and jittered backoff for all three ingesters

[mauriceyap]

Turns the feature on: sets deadLetterTopic, deadLetterMaxAttempts, maxBackoffTime, backoffRandomizationFactor, and backoffMultiplier in both the production config templates and local-dev configs for the event, lookout, and scheduler ingesters. Each ingester's Validate() now also calls Pulsar.Validate().

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

Greptile Summary

The PR enables bounded, jittered retry behavior and dead-letter publishing for all three ingesters, and ensures each ingester validates its nested Pulsar configuration.

• Adds retry limits, jitter parameters, multipliers, and dead-letter settings to production and local-development configurations.
• Extends Event, Lookout, and Scheduler ingester validation to include Pulsar.Validate().
• Removes an obsolete Lookout ingester configuration key.

Confidence Score: 5/5

The PR appears safe to merge.

The previously reported invalid shared validation fixture now leaves DeadLetterMaxAttempts unset, which satisfies the nested validation contract, and no blocking failure remains.

Important Files Changed

Filename	Overview
internal/eventingester/configuration/validation.go	Adds nested Pulsar backoff validation to Event ingester configuration validation.
internal/lookoutingester/configuration/validation.go	Adds nested Pulsar backoff validation to Lookout ingester configuration validation.
internal/scheduleringester/config.go	Adds nested Pulsar backoff validation to Scheduler ingester configuration validation.
config/eventingester/config.yaml	Enables bounded jittered retries and dead-letter publishing for the Event ingester.
config/lookoutingester/config.yaml	Enables bounded jittered retries and dead-letter publishing while removing an ignored legacy key.
config/scheduleringester/config.yaml	Enables bounded jittered retries and a scheduler-specific dead-letter topic.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  P[Pulsar event topic] --> E[Event ingester]
  P --> L[Lookout ingester]
  P --> S[Scheduler ingester]
  E -->|bounded jittered retries exhausted| ED[events-dead-letter]
  L -->|bounded jittered retries exhausted| ED
  S -->|bounded jittered retries exhausted| SD[scheduler-events-dead-letter]

Loading

_{Reviews (7): Last reviewed commit: "ugh" | Re-trigger Greptile}

[datadog-armadaproject]

⚠️ Warnings

🚦 1 Pipeline job failed

CI | All jobs succeeded     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b5ef745 | Docs | Datadog PR Page | Give us feedback!}