Skip to content

Use different strategies for the key and the value in the same topic #573

@vascoferraz

Description

@vascoferraz

Hi everyone :)

I read the documentation and I did not find any way to use different strategies, that is, TopicNameStrategy, RecordNameStrategy or TopicRecordNameStrategy, for the key and the value in the same topic.

I also tried the following configurations but them failed while deploying:

First attempt - Add two topics entries (one for the value and another for the key)

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalArgumentException: com.purbon.kafka.topology.exceptions.ValidationException: Missing required value.schema.file on schemas for topic xxx-customer-v1

Second attempt: Duplicate the topic entry and set RecordNameStrategy for the value (first entry) and TopicNameStrategy for the key (second entry).

context: "XX"
source: "yyyyy"
projects:
  - name: "my-project"
    topics:
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "RecordNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
      - name: "xxx-customer-v1"
        dataType: "avro"
        subject.name.strategy: "TopicNameStrategy"
        schemas:
          - key.schema.file: "schemas/xxx-customer-v1-key.avsc"
            key.record.type: "java.lang.String"
            value.schema.file: "schemas/xxx-new_customer-v1-value.avsc"
            value.record.type: "com.xxx.customers.NewCustomer"
        config:
          replication.factor: "3"
          num.partitions: "3"
          compression.type: "gzip"

Exception in thread "main" java.lang.IllegalStateException: Duplicate key xxx-customer-v1

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions