Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial structure for out of the application workflow orchestration #5

Draft
wants to merge 9 commits into
base: develop
Choose a base branch
from

Commits on Aug 28, 2023

  1. Configuration menu
    Copy the full SHA
    00df733 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2023

  1. Add Python Applications of the AMS-Orchestrator

    * Add wrapper to start flux with proxy
    * Add infrastructure of connecting to and receiving from RMQ
    * Add initial daemon implementation
    milroy authored Aug 29, 2023
    Configuration menu
    Copy the full SHA
    1d08ff3 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. Configuration menu
    Copy the full SHA
    c4d323f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5f9da01 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Fixed several problems in AMS packages and added a AMSBroker to send …

    …messages to a RabbitMQ queue
    
    Signed-off-by: Loic Pottier <[email protected]>
    lpottier committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    44c15e6 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Added initial scripts to launch AMS workflow (#4)

    * Added initial script to bootstrap Flux on CORAL/IBM machine
    * Added support Slurm based system, tested with flux-core 0.49 on Ruby/Lassen
    * Added script to launch AMS miniapp with Flux
    * Reverted script to support older version of flux, Lassen bootstrap only works wityh Flux<= 0.45 (tested with 0.45)
    * Added scripts to add secrets on OC
    * Added new scripts to launch the entire AMS workflow
    * Upgrade all scripts, they are now fully functional (main script communicates with AMS daemon via RMQ)
    
    ---------
    
    Signed-off-by: Loic Pottier <[email protected]>
    lpottier authored Sep 7, 2023
    Configuration menu
    Copy the full SHA
    2d93aec View commit details
    Browse the repository at this point in the history
  2. Minor fix, removed absolute path

    Signed-off-by: Loic Pottier <[email protected]>
    lpottier committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    d883442 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. Added database backend for CSV

    Signed-off-by: Loic Pottier <[email protected]>
    lpottier committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    184f53c View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. This commit addresses multiple problems in the broker part of AMS (#9)

    * This commit addresses multiple problems in the broker part of AMS
    - we are not sending input/output as encoded string anymore, we send binary blobs
    - base64 has been removed
    - a bug has been fixed with (very) old libevent version (<= 2.0.21-stable)
    - offloading inputs/outputs to the thread managing RMQ is now much faster
    
    * Moved to ResourceManager, created AMSMessage structures, moved to smart pointers.
    
    * Complete re-design of the RabbitMQ backend
    
    * Removed EventBuffer, removed pthread and signals. Big cleanup of the code.
    
    * Added documentation and new AMSMsgHeader class + moved from memcpy to ResourceManager::copy
    lpottier authored Nov 7, 2023
    Configuration menu
    Copy the full SHA
    61bfb06 View commit details
    Browse the repository at this point in the history