Add logging

[kylebarron]

See https://pyo3.rs/v0.22.5/ecosystem/logging, https://github.com/vorner/pyo3-log.

It looks like upstream is already logging: https://github.com/apache/arrow-rs/blob/2b373f850c5ac3d8ba6a1a0dfb98f5711f5318c8/object_store/src/aws/builder.rs#L939

[kylebarron]

This is more complex because arrow-rs uses the tracing crate: vorner/pyo3-log#22

[jessekrubin]

Hello @kylebarron (et al),

I have been wanting to create some sort of tracing wrapper/library w/ pyo3 for some time (as I don't like python's logging lib) and would be down to team up on that if you are interested! I think more than anything it requires some discussion/thought/design.

PS: I’ve been working on wrapping/shimming the reqwest library (which you can find here) and was happy to find that I arrived at a similar method for wrapping and handling a response body (using a bytes wrapper type and IntoPyObject); I did make some changes to my impl after I found obstore—hope that’s alright! Are you planning to publish the pyo3-bytes crate?

PPS: I work in geospatial computation/visualization (which it seems you do as well!) and would love to collaborate on some pyo3 projects.

[kylebarron]

👋

I have been wanting to create some sort of tracing wrapper/library w/ pyo3 for some time (as I don't like python's logging lib) and would be down to team up on that if you are interested! I think more than anything it requires some discussion/thought/design.

I don't think at this point logging is a priority for me. I dug into tracing a little bit last week while reading up on #105 (see #122 ) but I learned of another approach that's simpler to implement. Tracing seems non-trivial to implement bindings for.

PS: I’ve been working on wrapping/shimming the reqwest library (which you can find here)

First, that looks like an absolutely massive API surface you're shooting for. I can't imagine that would be easy to maintain, and I imagine discoverability would be hard for end users, too, to know which Python modules are stable enough to depend on.

I arrived at a similar method for wrapping and handling a response body (using a bytes wrapper type and IntoPyObject)

In obstore we don't always materialize a response body into a single buffer. We hold a generic response type and then support async or sync iteration over the stream of chunks, or you can materialize it into one chunk.

I see in ryo3-bytes you have a tiny shim around Bytes, but you don't implement conversions from Python. And you only support copying memory between Rust and Python, while here we support zero-copy conversions via the buffer protocol.

Are you planning to publish the pyo3-bytes crate?

It is published: https://docs.rs/pyo3-bytes/latest/pyo3_bytes/

would love to collaborate on some pyo3 projects.

I'm always open to contributions! Obstore is getting ready for an 0.3 release, but if you're interested there are some other issues you could tackle, for example #89

[jessekrubin]

First, that looks like an absolutely massive API surface you're shooting for. I can't imagine that would be easy to maintain, and I imagine discoverability would be hard for end users, too, to know which Python modules are stable enough to depend on.

True. I am having fun working on this in my spare time (when I am not working for a living); the hope is that people aside from just me will contribute to the project.

In obstore we don't always materialize a response body into a single buffer. We hold a generic response type and then support async or sync iteration over the stream of chunks, or you can materialize it into one chunk.

I see in ryo3-bytes you have a tiny shim around Bytes, but you don't implement conversions from Python. And you only support copying memory between Rust and Python, while here we support zero-copy conversions via the buffer protocol.

Firstly, very cool.

I figured IntoPyObject trick out primarily for parsing and returning json via the jiter python crate.

Does this mean you are able to create a python-bytes object without copying? or do you mean it's zero-copy in that you return a pyo3-bytes object?

I'm always open to contributions! Obstore is getting ready for an 0.3 release, but if you're interested there are some other issues you could tackle, for example #89

I will take a look when I have a spare minute! Have any priorities?

[kylebarron]

Does this mean you are able to create a python-bytes object without copying? or do you mean it's zero-copy in that you return a pyo3-bytes object?

No it's not a concrete bytes type. It's an object that implements the Python buffer protocol. So for example you can create numpy arrays that view onto rust memory directly, without having to make a copy.

[kylebarron]

I'm always open to contributions! Obstore is getting ready for an 0.3 release, but if you're interested there are some other issues you could tackle, for example #89

I will take a look when I have a spare minute! Have any priorities?

I think I implemented #89. There isn't a lot more obvious work to do here; #105 may be the next simplest piece of work.