Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add support for KEY_VALUE Schema #164

Open
oneebhkhan opened this issue Sep 25, 2023 · 0 comments
Open

[FEATURE] Add support for KEY_VALUE Schema #164

oneebhkhan opened this issue Sep 25, 2023 · 0 comments
Labels
type/feature Indicates new functionality

Comments

@oneebhkhan
Copy link

I am streaming CDC messages from PostGRES using the PostGRES Debezium Connector.

The topic has the following schema

"type": "KEY_VALUE",
    "timestamp": 1695369844027,
    "properties": {
      "key.schema.name": "KafKaJson",
      "key.schema.properties": "{\"__AVRO_READ_OFFSET__\":\"0\"}",
      "key.schema.type": "JSON",
      "kv.encoding.type": "SEPARATED",
      "value.schema.name": "KafKaJson",
      "value.schema.properties": "{\"__AVRO_READ_OFFSET__\":\"0\"}",
      "value.schema.type": "JSON"

I am successfully able to consume the messages on the Pulsar topic, however when I try to consume it using the Spark connector, I get the following error:
We do not support KEY_VALUE currently

I would like to request for this connector to support the KEY_VALUE schema so that Im properly correctly able to consume the messages.

As a stop-gap solution, I have enabled the allowDifferentTopicSchemas flag - which allows me to consume the messages since the Topic schema are not taken into account.

@oneebhkhan oneebhkhan added the type/feature Indicates new functionality label Sep 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/feature Indicates new functionality
Projects
None yet
Development

No branches or pull requests

1 participant