A ready to use example: generate polymer and run simulation #5
Conversation
src/molexp/cmdline.py
Outdated
There was a problem hiding this comment.
You want to use CMDLineExecutionManager as the execution manager -- like I'm doing here https://github.com/DAGWorks-Inc/hamilton/blob/d3c08a5a65413f11609917a452680d8624f52563/examples/cmdline_orchestrator/run.py. And I suggest using multithreading rather then multiprocessing.
Right now it relies on a tag cmdline to be on the function. We could instead set a property on the decorated functions or something to check for. But this will enable things to run in parallel, because they will be sent over to the remote executor to run.
1. Moves function outside, and makes proper use of @resolve 2. Changes local task executor to be synchronous - this should be fine to do. The tasks are then executed via the remote one. This at least didn't seem to break when I changed it.
1. need to submit each node to remote. extended class there. 2. multiprocessing still causes SERDE issues. Multithreading seems to work here.
Changes how the project module is run
example/cmdline/run.py
Outdated
| from dagworks import adapters | ||
|
|
||
| tracker = adapters.DAGWorksTracker( | ||
| username=USER_NAME, | ||
| api_key=API_KEY, | ||
| project_id=int(PROJECT_ID), | ||
| dag_name="toy-cmdline-dag", | ||
| tags={"env": "local"}, # , "TODO": "add_more_tags_to_find_your_run_later"}, | ||
| ) | ||
| hamilton_adapters.append(tracker) |
There was a problem hiding this comment.
You can change this to be the open source one -- e.g. add sf-hamilton-sdk, and sf-hamilton-ui to the additional dependencies in the pyproject.toml.
| @@ -8,9 +8,7 @@ readme = "README.md" | |||
| requires-python = ">3.8" | |||
| version = "0.1.0" | |||
| dependencies = [ | |||
| "sf-hamilton[visualization]", | |||
| "tqdm", | |||
There was a problem hiding this comment.
I think you still need tqdm.
| "sf-hamilton[visualization]", | ||
| "tqdm", | ||
| "dagworks-sdk", | ||
| "sf-hamilton" | ||
| ] | ||
| [project.optional-dependencies] | ||
| dev = [ |
There was a problem hiding this comment.
can add optional dependencies like I mentioned above.
No description provided.