Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding an Avro backend #386

Open
wants to merge 39 commits into
base: master
Choose a base branch
from
Open

Adding an Avro backend #386

wants to merge 39 commits into from

Conversation

ahasha
Copy link

@ahasha ahasha commented Dec 18, 2015

This PR will fix #370

I'm pretty sure there are edge cases that I haven't thought through, and there are a number of avro types that are not supported (i.e. ['null', 'error_union', 'bytes', 'enum', 'request', 'error', 'fixed']).

However, I figured it is now complete enough to share and get feedback.

@ahasha
Copy link
Author

ahasha commented Dec 18, 2015

Uh oh, it looks like the avro package itself is not Python 3 compatible.

E   _pytest.config.ConftestImportFailure: (local('/home/travis/build/blaze/odo/odo/backends/tests/conftest.py'), (<class 'SyntaxError'>, SyntaxError('invalid syntax', ('/home/travis/miniconda/envs/odo/lib/python3.4/site-packages/avro/schema.py', 340, 23, '      except Exception, e:\n')), <traceback object at 0x7f1ba8e62888>))

@cpcloud cpcloud added this to the 0.4.1 milestone Dec 18, 2015
@@ -31,6 +31,7 @@ requirements:

test:
requires:
- avro
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this package isn't python 3 compatible, specify it with a conda selector like this:

- avro  # [py27]

@ahasha
Copy link
Author

ahasha commented Dec 18, 2015

Thanks for the near-instantaneous code review!

@cpcloud
Copy link
Member

cpcloud commented Dec 18, 2015

No, thank you for the PR!

reader = self.reader
return schema.parse(self.reader.meta['avro.schema']) if reader else None

uri = property(lambda self: self._uri)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason not to just make self._uri be self.uri and be a public attribute?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same with self._codec and self._schema

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uri, schema, and codec feel like properties that should be immutable. If a user changed them, you would need to refresh any open readers and writers. It feels more natural to expect the user to create a new AVRO object rather than reusing an existing one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, no biggie

return schema_from_dict(self.reader.schema) if reader else None

path = property(lambda self: self._path)
codec = property(lambda self: self._codec)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make these proper data attributes, could you please define a set callable that simply raises AttributeError? See here:

https://docs.python.org/3/howto/descriptor.html#descriptor-protocol

Copy link
Author

@ahasha ahasha Oct 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure the property function here achieves what you're describing. Would you like me to add a test verifying it, or refactor it into a more standard method format with the @property decorator? I'll just add the test for now.

@kwmsmith
Copy link
Member

@ahasha OK, first iteration done.

@ahasha
Copy link
Author

ahasha commented Aug 18, 2016

@kwmsmith -- thanks for the code review! I don't have bandwidth to respond to it immediately, but I will make time over the next week or two (hopefully). At any rate, I do plan to get back to this.

@kwmsmith
Copy link
Member

Thanks to you for being willing to pick this up again.

@kwmsmith
Copy link
Member

kwmsmith commented Oct 9, 2016

@ahasha Thanks for picking this back up -- I'll wait to review / merge this PR until you give me an "all finished".

@majidaldo
Copy link

bump

@ahasha
Copy link
Author

ahasha commented Aug 9, 2017

Hey there. Sorry, but priorities shifted over time and I won't be able to continue work on this PR.

(It would make me happy if someone cared enough to finish it and get it merged, though!)

Copy link
Author

@ahasha ahasha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Releasing my partial review response from back in October

@majidaldo
Copy link

can we conclude this?
what i care about:

  • is it using fastavro
  • don't care about python 2
  • just care about 'record' datatype

i might be able to work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a backend for apache Avro
7 participants