Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow support #82

Merged
merged 36 commits into from
Sep 18, 2024
Merged

Workflow support #82

merged 36 commits into from
Sep 18, 2024

Conversation

koparasy
Copy link
Member

  • Implements orchestrator to push sub-selection jobs and training jobs using flux
  • Extends stager to receive model update notifications and push them to the user provided filter mechanism
  • Initial support for a workflow description language
  • Adds new example of binomial options, the benchmark can scale the time on a per element basis.
  • Adds a training script (Thank you @ZwFink) for binomial

@koparasy koparasy requested a review from lpottier August 12, 2024 17:07
@koparasy koparasy mentioned this pull request Aug 12, 2024
Closed
@koparasy koparasy marked this pull request as ready for review August 12, 2024 17:08
koparasy and others added 22 commits August 12, 2024 10:16
@koparasy
Add working correct version of binomial options
2780a46
@koparasy
Add missing files, clean interface
2bb2c30
@koparasy
Add AMS support for binomial options
9ab2c98
@koparasy
Enable MPI for Binomial with weak scaling
4744f76
@koparasy
Add CMake for examples
af41a4d
@koparasy
srand seed based on rank id
d1c41a7
@koparasy
Add missing files
a0837c8
@koparasy
Implements sequential workflow including staging
d32d391
@koparasy
FS stager now can delete files from source directory
d8afdb9
@koparasy
Seperates 'domain_name' from 'dbLabel'
6d16b08
@koparasy
Fixes kosh store to group ensembles based on domain name
bd123be
@koparasy
Register data per domain into kosh
973fe94
@koparasy
End to end sequential workflow
b703808
@koparasy
Sub-select and train script
e23e2ce
@koparasy
ams flux executor
e799dd0
@koparasy
Initial extensions for orchestrator
a13a77f
@koparasy
Working orchestrator
d9186c2
@koparasy
Adding orchestrator
e3066dc
@lpottier @koparasy
Added fake rmq publish option
4b48bd2 
Signed-off-by: Loic Pottier <[email protected]>
@lpottier @koparasy
typo
6deecb0 
Signed-off-by: Loic Pottier <[email protected]>
@lpottier @koparasy
Fix bug
5dcbe64 
Signed-off-by: Loic Pottier <[email protected]>
@koparasy
Pushes model updates to the stager
1c9f28c
@koparasy
Copy link
Member Author

@lpottier this is 'ready' for review. I will start cleaning up the code and adding comments etc. Keep in mind that this is the first implementation of the end-to-end so the process is likely to be sub-optimal.

lpottier
lpottier approved these changes Sep 6, 2024
View reviewed changes
Copy link
Collaborator

@lpottier lpottier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very big PR, looks good to me! Thanks @koparasy

examples/tools/train.py Outdated Show resolved Hide resolved
src/AMSWorkflow/ams/ams_flux.py Outdated Show resolved Hide resolved
@koparasy koparasy merged commit c15e5a7 into develop Sep 18, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lpottier lpottier lpottier approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@koparasy @lpottier