Skip to content

feat: kafka driver change#958

Merged
hash-data merged 35 commits into
stagingfrom
feat/kafka-driver-change
Jun 19, 2026
Merged

feat: kafka driver change#958
hash-data merged 35 commits into
stagingfrom
feat/kafka-driver-change

Conversation

@saksham-datazip

@saksham-datazip saksham-datazip commented May 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Migrated the Kafka consumer implementation from Segment Kafka-Go to Franz-Go to leverage improved consumer group management, rebalance handling, and lifecycle APIs provided by Franz-Go.

As part of this migration:

  • Added support for static membership using instance.id to improve retry and reconnect behavior. This helps avoid unnecessary rebalances during transient failures or consumer restarts when the same instance rejoins the group.

  • Implemented rebalance detection using Franz-Go consumer group callbacks along with generation ID tracking stored in consumer metadata. During sync execution, the active generation is continuously validated against the latest assigned generation to detect stale consumers or lost partition ownership.

  • Added graceful shutdown handling on successful rebalance detection. Instead of continuing to process records with outdated assignments, the sync now exits cleanly to prevent duplicate processing and stale partition consumption during consumer group transitions.

Fixes #794

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Tested retry scenarios by intentionally returning errors from StreamChanges. Verified that retries worked correctly and did not trigger unnecessary consumer group rebalances due to static membership support using instance.id.
  • Stopped the sync during active processing and triggered a consumer group rebalance. Verified that the sync detected the rebalance successfully and exited gracefully without processing duplicate data.

Screenshots or Recordings

N/A

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

@saksham-datazip saksham-datazip changed the title Feat/kafka driver change feat/kafka driver change May 20, 2026
@saksham-datazip saksham-datazip changed the title feat/kafka driver change feat/ kafka driver change May 20, 2026
@saksham-datazip saksham-datazip changed the title feat/ kafka driver change feat: kafka driver change May 20, 2026
vikaxsh
vikaxsh reviewed Jun 8, 2026
View reviewed changes
Comment thread pkg/kafka/reader.go
Comment thread drivers/kafka/internal/cdc.go
hash-data
hash-data reviewed Jun 12, 2026
View reviewed changes

@hash-data hash-data left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice Code

Comment thread pkg/kafka/reader.go Outdated
Comment thread drivers/kafka/internal/cdc.go
Comment thread drivers/kafka/internal/cdc.go Outdated
Comment thread pkg/kafka/balancer.go
Comment thread pkg/kafka/balancer.go Outdated
Comment thread pkg/kafka/reader.go
Comment thread pkg/kafka/reader.go Outdated
Comment thread drivers/kafka/internal/cdc.go Outdated
Comment thread drivers/kafka/internal/cdc.go Outdated
Comment thread drivers/kafka/internal/cdc.go
@saksham-datazip

saksham-datazip commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Nice Code

Thnks

@hash-data hash-data temporarily deployed to integration_tests June 19, 2026 06:32 — with GitHub Actions Inactive
@hash-data hash-data temporarily deployed to integration_tests June 19, 2026 06:32 — with GitHub Actions Inactive
hash-data
hash-data reviewed Jun 19, 2026
View reviewed changes
Comment thread drivers/kafka/internal/kafka.go
hash-data
hash-data reviewed Jun 19, 2026
View reviewed changes
Comment thread pkg/kafka/reader.go
ImDoubD-datazip
ImDoubD-datazip reviewed Jun 19, 2026
View reviewed changes
Comment thread pkg/kafka/reader.go Outdated
ImDoubD-datazip
ImDoubD-datazip approved these changes Jun 19, 2026
View reviewed changes

@ImDoubD-datazip ImDoubD-datazip left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hash-data hash-data hash-data left review comments

@vikaxsh vikaxsh vikaxsh left review comments

@ImDoubD-datazip ImDoubD-datazip ImDoubD-datazip approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@saksham-datazip @hash-data @vikaxsh @ImDoubD-datazip