feat(connectors): add S3 sink connector

[atharvalade]

Which issue does this PR close?

Closes #2956

Rationale

Iggy lacks a native way to write stream messages to Amazon S3 and S3-compatible stores (MinIO, Cloudflare R2, Backblaze B2, DigitalOcean Spaces). This is a frequently requested capability for data lake ingestion and long-term archival pipelines.

What changed?

There was no connector for persisting Iggy messages to object storage. Users had to build custom consumers and upload logic to get data into S3.

This PR adds a new iggy_connector_s3_sink crate that implements the Sink trait. It buffers messages in-memory per stream/topic/partition, rotates files by size or message count, renders S3 keys from a configurable path template ({stream}/{topic}/{date}/{hour}/...), and uploads with retry + exponential backoff. Supports json_lines, json_array, and raw output formats with optional Iggy metadata and header embedding. Uses rust-s3 (already in workspace) with path-style addressing auto-enabled for custom endpoints.

Key implementation details:

• 6 source modules: lib.rs (config + entry point), client.rs (S3 client init + bucket verification), buffer.rs (in-memory accumulation + rotation logic), formatter.rs (JSON/raw output + metadata/header inclusion), path.rs (template engine for S3 keys with offset-based filenames), sink.rs (Sink trait: open/consume/close lifecycle)
• 36 unit tests covering config deserialization, buffer rotation, path template rendering, all output formats, credential validation, and edge cases
• CI integration: added to _build_rust_artifacts.yml and edge-release.yml for cdylib plugin builds and release notes
• Error handling: warnings logged on invalid config fallbacks, explicit buffer management on upload failure, close() warns if S3 client was never initialized
• End-to-end tested locally with MinIO in Docker, Iggy server, CLI producer, and connector runtime — verified messages flow from Iggy stream into S3 bucket as properly formatted JSON

Local Execution

• Passed
• Pre-commit hooks ran
• Full CI checklist passed locally:
  • cargo fmt --check -- pass
  • cargo clippy --tests -D warnings -- pass (zero warnings)
  • cargo test -p iggy_connector_s3_sink -- 36/36 pass
  • markdownlint --check -- pass
  • trailing-whitespace -- pass
  • trailing-newline -- pass
  • license-headers -- pass

AI Usage

1. Opus 4.6
2. used for scaffolding boilerplate and initial file structure, all logic was reviewed and iterated manually
3. Verified through full local compilation, 36 unit tests, clippy with -D warnings, and end-to-end testing with MinIO Docker + Iggy server + CLI producer + connector runtime
4. Yes

---

Here are all the relevant screenshots:

• MinIO Docker container running and accessible at localhost:9000
• MinIO web console showing the created iggy-test bucket
• Iggy server started with root credentials configured
• Iggy CLI creating stream application_logs and topic api_requests
• Iggy CLI sending test messages to the topic
• Connector runtime loading the S3 sink plugin and connecting to MinIO
• Connector runtime consuming messages and uploading to S3
• MinIO console showing the uploaded .jsonl file in the correct path structure (application_logs/api_requests/{date}/{hour}/)
• Contents of the uploaded file showing properly formatted JSON lines with metadata (offset, timestamp, stream, topic, partition_id, payload)
• All 36 unit tests passing
• cargo clippy --tests -D warnings passing with zero warnings

[codecov]

Codecov Report

❌ Patch coverage is 76.99594% with 170 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.72%. Comparing base (fcc5a44) to head (96ed8d1).

Files with missing lines	Patch %	Lines
core/connectors/sinks/s3_sink/src/sink.rs	0.00%	73 Missing ⚠️
core/connectors/sinks/s3_sink/src/client.rs	59.37%	39 Missing ⚠️
core/connectors/sinks/s3_sink/src/lib.rs	80.66%	35 Missing ⚠️
core/connectors/sinks/s3_sink/src/formatter.rs	89.94%	18 Missing and 2 partials ⚠️
core/connectors/sinks/s3_sink/src/buffer.rs	97.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3103      +/-   ##
============================================
- Coverage     72.74%   72.72%   -0.02%     
  Complexity      943      943              
============================================
  Files          1117     1123       +6     
  Lines         96342    97081     +739     
  Branches      73543    74298     +755     
============================================
+ Hits          70083    70607     +524     
- Misses        23715    23903     +188     
- Partials       2544     2571      +27     

Components	Coverage Δ
Rust Core	73.50% <76.99%> (-0.01%)	⬇️
Java SDK	62.30% <ø> (ø)
C# SDK	69.10% <ø> (-0.31%)	⬇️
Python SDK	81.43% <ø> (ø)
Node SDK	91.40% <ø> (ø)
Go SDK	38.97% <ø> (ø)

Files with missing lines	Coverage Δ
core/connectors/sinks/s3_sink/src/path.rs	100.00% <100.00%> (ø)
core/connectors/sinks/s3_sink/src/buffer.rs	97.00% <97.00%> (ø)
core/connectors/sinks/s3_sink/src/formatter.rs	89.94% <89.94%> (ø)
core/connectors/sinks/s3_sink/src/lib.rs	80.66% <80.66%> (ø)
core/connectors/sinks/s3_sink/src/client.rs	59.37% <59.37%> (ø)
core/connectors/sinks/s3_sink/src/sink.rs	0.00% <0.00%> (ø)

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.