Development towards 0.2.0 (#2)

* Reimplement type mapping into enum Signed-off-by: Andrej Orsula <[email protected]> * Add typy mapping for generated class bindings Signed-off-by: Andrej Orsula <[email protected]> * Improve type mapping and use `call_method0()/1()` where appropriate Signed-off-by: Andrej Orsula <[email protected]> * Add support for tuple and (frozen)set types Signed-off-by: Andrej Orsula <[email protected]> * Update project status Signed-off-by: Andrej Orsula <[email protected]> * Bump to 0.2.0 Signed-off-by: Andrej Orsula <[email protected]> --------- Signed-off-by: Andrej Orsula <[email protected]>
AndrejOrsula · Jan 23, 2024 · d2a7382 · d2a7382
1 parent 42d9bbb
commit d2a7382
Show file tree

Hide file tree

Showing 13 changed files with 1,661 additions and 750 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -20,12 +20,12 @@ license = "MIT OR Apache-2.0"
 readme = "README.md"
 repository = "https://github.com/AndrejOrsula/pyo3_bindgen"
 rust-version = "1.70"
-version = "0.1.1"
+version = "0.2.0"
 
 [workspace.dependencies]
-pyo3_bindgen = { path = "pyo3_bindgen", version = "0.1.1" }
-pyo3_bindgen_engine = { path = "pyo3_bindgen_engine", version = "0.1.1" }
-pyo3_bindgen_macros = { path = "pyo3_bindgen_macros", version = "0.1.1" }
+pyo3_bindgen = { path = "pyo3_bindgen", version = "0.2.0" }
+pyo3_bindgen_engine = { path = "pyo3_bindgen_engine", version = "0.2.0" }
+pyo3_bindgen_macros = { path = "pyo3_bindgen_macros", version = "0.2.0" }
 
 assert_cmd = { version = "2" }
 clap = { version = "4.4", features = ["derive"] }

diff --git a/README.md b/README.md
@@ -115,15 +115,8 @@ fn main() {
 Afterwards, include the generated bindings anywhere in your crate.
 
 ```rs
-#[allow(
-    clippy::all,
-    non_camel_case_types,
-    non_snake_case,
-    non_upper_case_globals
-)]
-pub mod target_module {
-    include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
-}
+include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
+pub use target_module::*;
 ```
 
 ### <a href="#-option-2-cli-tool"><img src="https://www.svgrepo.com/show/353478/bash-icon.svg" width="16" height="16"></a> Option 2: CLI tool
@@ -155,26 +148,20 @@ pyo3_bindgen = { version = "0.1", features = ["macros"] }
 Then, you can call the `import_python!` macro anywhere in your crate.
 
 ```rs
-#[allow(
-    clippy::all,
-    non_camel_case_types,
-    non_snake_case,
-    non_upper_case_globals
-)]
-pub mod target_module {
-    pyo3_bindgen::import_python!("target_module");
-}
+pyo3_bindgen::import_python!("target_module");
+pub use target_module::*;
 ```
 
 ## Status
 
 This project is in early development, and as such, the API of the generated bindings is not yet stable.
 
-- Not all Python types are mapped to their Rust equivalents yet. Especially the support for mapping types of module-wide classes for which bindings are generated is also still missing. For this reason, a lot of boilerplate might be currently required when using the generated bindings (e.g. `let typed_value: target_module::Class = any_value.extract()?;`).
-- The binding generation is primarily designed to be used inside build scripts or via procedural macros. Therefore, the performance of the codegen process is [benchmarked](./pyo3_bindgen_engine/benches/bindgen.rs) to understand the potential impact on build times. Surprisingly, even the initial unoptimized version of the engine is able to process the entire `numpy` 1.26.3 in ~300 ms on a *modern* laptop while generating 166k lines of formatted Rust code (line count includes documentation). Adding more features might increase this time, but there is also plenty of room for optimization in the current naive implementation.
+- Not all Python types are mapped to their Rust equivalents yet. For this reason, some additional typecasting might be currently required when using the generated bindings (e.g. `let typed_value: target_module::Class = any_value.extract()?;`).
+- The binding generation is primarily designed to be used inside build scripts or via procedural macros. Therefore, the performance of the codegen process is [benchmarked](./pyo3_bindgen_engine/benches/bindgen.rs) to understand the potential impact on build times. Although there is currently plenty of room for optimization in the current naive implementation, even the largest modules are processed in less than a second on a *modern* laptop.
 - The generation of bindings should never panic as long as the target Python module can be successfully imported. If it does, it is a bug resulting from an unexpected edge-case Python module structure or an unforeseen combination of enabled PyO3 features.
+- However, the generated bindings might not directly compile in some specific cases. Currently, there are two known issue; bindings will contain duplicate function definitions if present in the original code, and function parameters might use the same name as a class defined in the same scope (allowed in Python but not in Rust). If you encounter any other issues, consider manually rewriting the problematic parts of the bindings.
 - Although implemented, the procedural macros might not work in all cases - especially when some PyO3 features are enabled. In most cases, PyO3 fails to import the target Python module when used from within a `proc_macro` crate. Therefore, it is recommended to use build scripts instead for now.
-- The code will be refactored and cleaned up in the upcoming releases. The current implementation is a result of a very quick prototype that was built to test the feasibility of the idea.
+- The code will be refactored and cleaned up in the upcoming releases. The current implementation is a result of a very quick prototype that was built to test the feasibility of the idea. For example, configurability of the generated bindings is planned (e.g. allowlist/ignorelist of attributes). Furthermore, automatic generation of dependent Python modules will be considered in order to provide a more complete typing experience.
 
 Please [report](https://github.com/AndrejOrsula/pyo3_bindgen/issues/new) any issues that you might encounter. Contributions are more than welcome! If you are looking for a place to start, consider searching for `TODO` comments in the codebase.
 

diff --git a/pyo3_bindgen/src/lib.rs b/pyo3_bindgen/src/lib.rs
@@ -32,15 +32,8 @@
 //! Afterwards, include the generated bindings anywhere in your crate.
 //!
 //! ```rs
-//! #[allow(
-//!     clippy::all,
-//!     non_camel_case_types,
-//!     non_snake_case,
-//!     non_upper_case_globals
-//! )]
-//! pub mod target_module {
-//!     include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
-//! }
+//! include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
+//! pub use target_module::*;
 //! ```
 //!
 //! ### <a href="#-option-2-cli-tool"><img src="https://www.svgrepo.com/show/353478/bash-icon.svg" width="16" height="16"></a> Option 2: CLI tool
@@ -72,15 +65,8 @@
 //! Then, you can call the `import_python!` macro anywhere in your crate.
 //!
 //! ```rs
-//! #[allow(
-//!     clippy::all,
-//!     non_camel_case_types,
-//!     non_snake_case,
-//!     non_upper_case_globals
-//! )]
-//! pub mod target_module {
-//!     pyo3_bindgen::import_python!("target_module");
-//! }
+//! pyo3_bindgen::import_python!("target_module");
+//! pub use target_module::*;
 //! ```
 
 pub use pyo3_bindgen_engine::{

diff --git a/pyo3_bindgen_engine/src/bindgen.rs b/pyo3_bindgen_engine/src/bindgen.rs
@@ -10,6 +10,15 @@ pub use class::bind_class;
 pub use function::bind_function;
 pub use module::{bind_module, bind_reexport};
 
+// TODO: Refactor everything into a large configurable struct that keeps track of all the
+//       important information needed to properly generate the bindings
+//  - Use builder pattern for the configuration of the struct
+//  - Keep track of all the types/classes that have been generated
+//  - Keep track of all imports to understand where each type is coming from
+//  - Keep track of all the external types that are used as parameters/return types and consider generating bindings for them as well
+
+// TODO: Ensure there are no duplicate entries in the generated code
+
 /// Generate Rust bindings to a Python module specified by its name. Generating bindings to
 /// submodules such as `os.path` is also supported as long as the module can be directly imported
 /// from the Python interpreter via `import os.path`.
@@ -75,7 +84,21 @@ pub fn generate_bindings_for_module(
     py: pyo3::Python,
     module: &pyo3::types::PyModule,
 ) -> Result<proc_macro2::TokenStream, pyo3::PyErr> {
-    bind_module(py, module, module, &mut std::collections::HashSet::new())
+    let all_types = module::collect_types_of_module(
+        py,
+        module,
+        module,
+        &mut std::collections::HashSet::new(),
+        &mut std::collections::HashSet::default(),
+    )?;
+
+    bind_module(
+        py,
+        module,
+        module,
+        &mut std::collections::HashSet::new(),
+        &all_types,
+    )
 }
 
 /// Generate Rust bindings to a Python module specified by its `source_code`. The module will be

diff --git a/pyo3_bindgen_engine/src/bindgen/attribute.rs b/pyo3_bindgen_engine/src/bindgen/attribute.rs
@@ -1,13 +1,15 @@
-use crate::types::map_attr_type;
+use crate::types::Type;
 
 /// Generate Rust bindings to a Python attribute. The attribute can be a standalone
 /// attribute or a property of a class.
-pub fn bind_attribute(
+pub fn bind_attribute<S: ::std::hash::BuildHasher + Default>(
     py: pyo3::Python,
-    module_name: Option<&str>,
+    module_name: &str,
+    is_class: bool,
     name: &str,
     attr: &pyo3::PyAny,
     attr_type: &pyo3::PyAny,
+    all_types: &std::collections::HashSet<String, S>,
 ) -> Result<proc_macro2::TokenStream, pyo3::PyErr> {
     let mut token_stream = proc_macro2::TokenStream::new();
 
@@ -72,29 +74,29 @@ pub fn bind_attribute(
     };
     let setter_ident = quote::format_ident!("set_{}", name);
 
-    let getter_type = map_attr_type(getter_type, true)?;
-    let setter_type = map_attr_type(setter_type, false)?;
+    let getter_type = Type::try_from(getter_type)?.into_rs_owned(module_name, all_types);
+    let setter_type = Type::try_from(setter_type)?.into_rs_borrowed(module_name, all_types);
 
-    if let Some(module_name) = module_name {
+    if is_class {
         token_stream.extend(quote::quote! {
             #[doc = #getter_doc]
             pub fn #getter_ident<'py>(
+                &'py self,
                 py: ::pyo3::marker::Python<'py>,
             ) -> ::pyo3::PyResult<#getter_type> {
-                py.import(::pyo3::intern!(py, #module_name))?
-                .getattr(::pyo3::intern!(py, #name))?
+                self.getattr(::pyo3::intern!(py, #name))?
                 .extract()
             }
         });
         if has_setter {
             token_stream.extend(quote::quote! {
                 #[doc = #setter_doc]
                 pub fn #setter_ident<'py>(
+                    &'py self,
                     py: ::pyo3::marker::Python<'py>,
                     value: #setter_type,
                 ) -> ::pyo3::PyResult<()> {
-                    py.import(::pyo3::intern!(py, #module_name))?
-                    .setattr(::pyo3::intern!(py, #name), value)?;
+                    self.setattr(::pyo3::intern!(py, #name), value)?;
                     Ok(())
                 }
             });
@@ -103,10 +105,9 @@ pub fn bind_attribute(
         token_stream.extend(quote::quote! {
             #[doc = #getter_doc]
             pub fn #getter_ident<'py>(
-                &'py self,
                 py: ::pyo3::marker::Python<'py>,
             ) -> ::pyo3::PyResult<#getter_type> {
-                self.as_ref(py)
+                py.import(::pyo3::intern!(py, #module_name))?
                 .getattr(::pyo3::intern!(py, #name))?
                 .extract()
             }
@@ -115,12 +116,11 @@ pub fn bind_attribute(
             token_stream.extend(quote::quote! {
                 #[doc = #setter_doc]
                 pub fn #setter_ident<'py>(
-                    &'py mut self,
                     py: ::pyo3::marker::Python<'py>,
                     value: #setter_type,
                 ) -> ::pyo3::PyResult<()> {
-                    self.as_ref(py)
-                        .setattr(::pyo3::intern!(py, #name), value)?;
+                    py.import(::pyo3::intern!(py, #module_name))?
+                    .setattr(::pyo3::intern!(py, #name), value)?;
                     Ok(())
                 }
             });