Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/initial bifrost2 #1

Merged
merged 48 commits into from
Jul 11, 2024
Merged

Feature/initial bifrost2 #1

merged 48 commits into from
Jul 11, 2024

Conversation

wiwichips
Copy link
Collaborator

@wiwichips wiwichips commented Jun 25, 2024

Initial Bifrost 2 Structure

This PR contains the base set up for Bifrost 2 (the new Python SDK for DCP). It outlines the file structure of the module, and the dev tools used (poetry for Python management and testing, pminit for npm install for instance).

File structure

bifrost2/
├── dcp
│   ├── api
│   │   ├── compute_for.py
│   │   ├── __init__.py
│   │   ├── job.py
│   │   └── result_handle.py
│   ├── dry
│   │   ├── aio.py
│   │   ├── class_manager.py
│   │   ├── class_registry.py
│   │   └── __init__.py
│   ├── initialization.py
│   ├── __init__.py
│   └── js
│       ├── __init__.py
│       ├── load_dcp_client.py
│       ├── node_modules/
│       │   └── dcp-client -> (dcp-client via npm)
│       ├── package.json
│       ├── package-lock.json
│       └── utils.py
├── example.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
    └── (test files here)

Module logic is in dcp/, tests are in tests/...

dcp.api

  • bifrost 2 specific hard coded api features
  • for instance: this is where job.wait() and compute_for live

dcp.dry

  • generics used for programmaticly generated structures
  • for instance, the class maker functionality lives here for auto generating Bifrost2 versions of js/pm classes

dcp.js

  • JavaScript code specific to Bifrost2 lives here
  • for instance, the node modules required by Bifrost 2 such as dcp-client

(note: the file structure is subject to change and may have changed by the time you review this PR)

Module initialization algorithm

When dcp.init() is called a module system is programatically generated which matches the JavaScript CJS modules. At the same time, any classes, or functions come across are automatically converted to more Pythonic structures.

List of things this PR does

  • Transforms the CJS JS module we get from PythonMonkey requiring dcp-client into a Python module
  • Wrap raw PythonMonkey JSProxy so they abide by the Bifrost 2 Spec
  • Pythonic asynchronous behaviour
  • Initial setup for additional api to be hooked in easily
  • Tests

Bad name, but I like the concept. Each time a file changes, it re-runs
allowing you to see when tests fail. More interestingly is that unlike
using watch for accomplish this, you can also debug your code with pdb
Wraps functions which return awaitables for use in a blocking fashion
while still providing access to them via an aio submodule of each
class.
Each Bifrost Python class will now have a static method to get the
corresponding JS Class' constructor! Useful for comparisons among
other things.
As well as some minor refactors...
This sets up a simple algorithm for setting up a bifrost dcp
module system out of the cjs dcp-client modules.
dcp/__init__.py Outdated Show resolved Hide resolved
@wiwichips
Copy link
Collaborator Author

internal implementation details are leaked when one imports dcp, need to limit that

@wiwichips wiwichips self-assigned this Jun 26, 2024
@wiwichips
Copy link
Collaborator Author

bfclass class maker is ugly and should be cleaned up

And then await it only if the value it returns is awaitable...
dcp/__init__.py Outdated Show resolved Hide resolved
so it will be easier to change the arch to use a single event loop...
This is a temporary ugly patch to accomplish this, it should be
refactored into something cleaner later...
dcp/__init__.py Outdated Show resolved Hide resolved
dcp/dry/bfclass.py Outdated Show resolved Hide resolved
dcp/dry/fn.py Outdated Show resolved Hide resolved
- pass js instance to class directly
- don't bother with aio_ctor or aio_methods
- simplify code / make code more pythonic
At some point it didn't make sense to store the event loop in aio,
but now it could make sense... maybe elsewhere would be better but
maybe not! Also I initially had aio be fn (for managing functions)
so I had a comment TODO for checking return values. Now that is no
longer needed
@wiwichips wiwichips force-pushed the feature/initial-bifrost2 branch 3 times, most recently from 9ce96ae to 016823e Compare July 8, 2024 14:57
For instance, dcp.compute.for is "hidden", - it doesn't
show up as own property. We'll still want people to be able
to use it... Well not "for" since that's a syntax error but
"do"!
dcp/initialization.py Outdated Show resolved Hide resolved
@wiwichips
Copy link
Collaborator Author

Merging now since this is the initial PR which will enable others to contribute.

@wiwichips wiwichips changed the title Draft: Feature/initial bifrost2 Feature/initial bifrost2 Jul 11, 2024
@wiwichips wiwichips merged commit f258aef into main Jul 11, 2024
4 checks passed
Severn-J-Lortie pushed a commit that referenced this pull request Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants