Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better support for Confluent's kafka AvroSerializer/AvroDeserializer #405

Open
robbertvanwaveren opened this issue Sep 1, 2023 · 2 comments
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@robbertvanwaveren
Copy link

Is your feature request related to a problem? Please describe.

The confluent AvroSerializer/AvroDeserializer classes require both a schema as well as an object to_dict and from_dict to object function. Using the following code I can successfully use the avro_schema, asdict and parse_obj functions from the generated code from this library to fill those requirements:

    avro_serializer = AvroSerializer(registry_client, msg.avro_schema(), lambda obj, sc: obj.asdict(standardize_factory=standardize_custom_type))
    avro_deserializer = AvroDeserializer(registry_client, msg.avro_schema(), lambda dict, sc: Message.parse_obj(dict))

However its still a bit clunky as the functions do not exactly match. Espescially the current implementation of the to_dict is close but not exactly usable as it does not fill the standardize_factory as above.

Note that successful integration is key to usage of these models in combination with an enterprise setting that relies of a schema-registry and confluents avro encoding header to support schema evolution across different implementation languages.

Describe the solution you'd like

I'd like the generated classes to come with class-level methods matching the provided lambdas, or if that is not feasible at least include the standardize_factory=standardize_custom_type) in the call to asdict from the object's to_dict by default.

Describe alternatives you've considered

Additional context

@marcosschroh
Copy link
Owner

@robbertvanwaveren

Note that successful integration is key to usage of these models in combination with an enterprise setting that relies of a schema-registry and confluents avro encoding header to support schema evolution across different implementation languages.

I agree but in this case the confluent python ecosystem is not stable at all so I would not go to far in the integration. However, I think we can include standardize_factory: Optional[Callable[..., Any] = None in the to_dict method . PR is welcome.

PS: Documentation and examples would be also nice to have.

@marcosschroh
Copy link
Owner

Another solution for this could be to have a mixings.Confluent that could add extra functionality for this use case

@marcosschroh marcosschroh added the good first issue Good for newcomers label Sep 28, 2023
@marcosschroh marcosschroh added the enhancement New feature or request label Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants