Skip to content
This repository has been archived by the owner on Oct 10, 2023. It is now read-only.

[WIP] New node API: Venv support #59

Merged
merged 99 commits into from
Aug 1, 2023
Merged

[WIP] New node API: Venv support #59

merged 99 commits into from
Aug 1, 2023

Commits on Jun 21, 2023

  1. Don't load parameter manifest for param types

    @smahmed776
    smahmed776 committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    3662162 View commit details
    Browse the repository at this point in the history

  2. some formatting

    @smahmed776
    smahmed776 committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    b3108f0 View commit details
    Browse the repository at this point in the history

  3. format codes

    @smahmed776
    smahmed776 committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    de8d1aa View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Add inherited datacontainer types

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    8d9e948 View commit details
    Browse the repository at this point in the history

  2. Update decorator to accept deps kwarg

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    1b2a5cb View commit details
    Browse the repository at this point in the history

  3. Forgot to remove comment

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    4593b5c View commit details
    Browse the repository at this point in the history

  4. Bump version

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    7493007 View commit details
    Browse the repository at this point in the history

  5. Export DataContainer types

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    15a5932 View commit details
    Browse the repository at this point in the history

  6. Fix decorator default param

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    0c82c42 View commit details
    Browse the repository at this point in the history

  7. Small fixes

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    925c0be View commit details
    Browse the repository at this point in the history

  8. Merge branch 'main' into jeff-autogen-manifest

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    d1e98ae View commit details
    Browse the repository at this point in the history

  9. Fix

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    7070fe4 View commit details
    Browse the repository at this point in the history

  10. Format

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    9e4110f View commit details
    Browse the repository at this point in the history

  11. Fix type errors

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    8204dd4 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'generate-command-section' into jeff-autogen-manifest

    @39bytes
    39bytes committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    849c543 View commit details
    Browse the repository at this point in the history

  13. attempt 3 times for fetching previous job result

    @smahmed776
    smahmed776 committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    0fb9d14 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into generate-command-section

    @smahmed776
    smahmed776 committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    51cbfe9 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2023

  1. Add node type to decorator

    @39bytes
    39bytes committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    e8189ab View commit details
    Browse the repository at this point in the history

  2. Let studio backend decide what to do when directions are not specified

    @39bytes
    39bytes committed Jun 28, 2023
    Configuration menu
    Copy the full SHA
    f6714c5 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. LOOP and CONDITIONAL working (#48)

    @smahmed776
    smahmed776 authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    dde6c93 View commit details
    Browse the repository at this point in the history

  2. Support multi inputs (#49)

    @39bytes
    39bytes authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    38cee77 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. Add method for parsing all list types (#51)

    @smahmed776
    smahmed776 authored Jul 6, 2023
    Configuration menu
    Copy the full SHA
    62dae20 View commit details
    Browse the repository at this point in the history

  2. migrate node_sdk and dao to flojoy package

    @izi-on
    izi-on committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    e3e8587 View commit details
    Browse the repository at this point in the history

  3. migrate job service

    @izi-on
    izi-on committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    877f489 View commit details
    Browse the repository at this point in the history

  4. fix imports

    @izi-on
    izi-on committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    47ea45f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'migrate-node-sdk' into generate-command-section 

    needs to be compatible with new node sdk created by Syed
    @izi-on
    izi-on committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    e05918c View commit details
    Browse the repository at this point in the history

  6. reduce waiting time for fetching job input

    @izi-on
    izi-on committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    5ccc983 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. added extra param for all dc types

    @youngsun4786
    youngsun4786 committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    9e91000 View commit details
    Browse the repository at this point in the history

  2. fixed default input

    @youngsun4786
    youngsun4786 committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    0b53327 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #53 from flojoy-io/extra-parameter-nick 

    added extra param for all dc types
    @smahmed776
    smahmed776 authored Jul 7, 2023
    Configuration menu
    Copy the full SHA
    040c019 View commit details
    Browse the repository at this point in the history

  4. checkpoint

    @izi-on
    izi-on committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    584d052 View commit details
    Browse the repository at this point in the history

  5. add missing locks

    @izi-on
    izi-on committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    b7f7ae2 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2023

  1. rewrite job service

    @izi-on
    izi-on committed Jul 8, 2023
    Configuration menu
    Copy the full SHA
    e09355b View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2023

  1. no redis version

    @izi-on
    izi-on committed Jul 9, 2023
    Configuration menu
    Copy the full SHA
    6ca4410 View commit details
    Browse the repository at this point in the history

  2. removed some useless print statements

    @izi-on
    izi-on committed Jul 9, 2023
    Configuration menu
    Copy the full SHA
    ec3a3d9 View commit details
    Browse the repository at this point in the history

  3. small tweaks to memory

    @izi-on
    izi-on committed Jul 9, 2023
    Configuration menu
    Copy the full SHA
    58a9231 View commit details
    Browse the repository at this point in the history

  4. more adjustments to memory

    @izi-on
    izi-on committed Jul 9, 2023
    Configuration menu
    Copy the full SHA
    4bfd3bb View commit details
    Browse the repository at this point in the history

  5. raise exception when no object found from datastore

    @izi-on
    izi-on committed Jul 9, 2023
    Configuration menu
    Copy the full SHA
    1534bec View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. fix return type of get_job_function

    @izi-on
    izi-on committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    6dd7826 View commit details
    Browse the repository at this point in the history

  2. get rid of redis entirely

    @izi-on
    izi-on committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    e59c8e8 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. Merge branch 'generate-command-section' into new-node-api

    @39bytes
    39bytes committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    327f5e0 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. export smallmemory from package and update some functions

    @smahmed776
    smahmed776 committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    2b1e23a View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2023

  1. node init implemented

    @izi-on
    izi-on committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    4200414 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. add @run_in_venv decorator

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    b24f52f View commit details
    Browse the repository at this point in the history

  2. add cloudpickle to reqs

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    66e55d6 View commit details
    Browse the repository at this point in the history

  3. implement node init

    @izi-on
    izi-on committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    74a67f0 View commit details
    Browse the repository at this point in the history

  4. pin flojoy and cloudpickle versions to those of parent process

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    873fa81 View commit details
    Browse the repository at this point in the history

  5. delete venv on failed install

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    8cf6136 View commit details
    Browse the repository at this point in the history

  6. merge new-node api

    @izi-on
    izi-on committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    6cf4b56 View commit details
    Browse the repository at this point in the history

  7. add multiplatform CI for flojoy_node_env

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    1961168 View commit details
    Browse the repository at this point in the history

  8. fix CI pytest

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    e8381fb View commit details
    Browse the repository at this point in the history

  9. fix on.push.branches

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    b4cb69b View commit details
    Browse the repository at this point in the history

  10. fix wf name

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    255bd61 View commit details
    Browse the repository at this point in the history

  11. update test_flojoy_node_env.py

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    86201cd View commit details
    Browse the repository at this point in the history

  12. empty commit to trigger wf

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    c84c628 View commit details
    Browse the repository at this point in the history

  13. fix ci

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    32d987c View commit details
    Browse the repository at this point in the history

  14. fix ci

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    dcc7f59 View commit details
    Browse the repository at this point in the history

  15. fix tempfile.gettempdir patch

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    624658d View commit details
    Browse the repository at this point in the history

  16. fix tempfile.gettempdir patch

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    27a6158 View commit details
    Browse the repository at this point in the history

  17. fix test

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    d75f94c View commit details
    Browse the repository at this point in the history

  18. add start_method override to MultiprocessingExecutableContextManager

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    bb03b53 View commit details
    Browse the repository at this point in the history

  19. force set_start_method

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    4b6c930 View commit details
    Browse the repository at this point in the history

  20. add support for win32 python executable path

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    6fd4d26 View commit details
    Browse the repository at this point in the history

  21. fix mock_tempdir for windows

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    e30f1a2 View commit details
    Browse the repository at this point in the history

  22. add 2 more tests

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    356178f View commit details
    Browse the repository at this point in the history

  23. add --runslow option for pytest

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    18fd20a View commit details
    Browse the repository at this point in the history

  24. fix automated test

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    f9356a7 View commit details
    Browse the repository at this point in the history

  25. merge new-node-api in current

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    4914fc5 View commit details
    Browse the repository at this point in the history

  26. merge new-node-api in current

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    6786755 View commit details
    Browse the repository at this point in the history

  27. cache in FLOJOY_CACHE_DIR

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    4c29ed4 View commit details
    Browse the repository at this point in the history

  28. add docstr to run_in_venv

    @Roulbac
    Roulbac committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    a2a41c6 View commit details
    Browse the repository at this point in the history

  29. Merge pull request #56 from flojoy-io/reda-pip-env 

    Add `@run_in_venv` decorator, allow to run functions in separate virtual environments
    @jackparmer
    jackparmer authored Jul 14, 2023
    Configuration menu
    Copy the full SHA
    dedb2b8 View commit details
    Browse the repository at this point in the history

  30. Update flojoy_node_venv.py

    @Roulbac
    Roulbac authored Jul 14, 2023
    Configuration menu
    Copy the full SHA
    c77b704 View commit details
    Browse the repository at this point in the history

  31. publish from current branch

    @smahmed776
    smahmed776 committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    4f2e3ea View commit details
    Browse the repository at this point in the history

  32. Merge branch 'new-node-api' of github.com:flojoy-io/python into new-n… 

    …ode-api
    @smahmed776
    smahmed776 committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    b0a033e View commit details
    Browse the repository at this point in the history

  33. add vector type here

    @39bytes
    39bytes committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    07c68c0 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2023

  1. add flojoy hf_hub_download

    @Roulbac
    Roulbac committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    6c55750 View commit details
    Browse the repository at this point in the history

  2. change default cache path

    @Roulbac
    Roulbac committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    08748c7 View commit details
    Browse the repository at this point in the history

  3. bump pypi version to dev6

    @Roulbac
    Roulbac committed Jul 15, 2023
    Configuration menu
    Copy the full SHA
    9f59c5c View commit details
    Browse the repository at this point in the history

  4. Merge pull request #60 from flojoy-io/reda-hf-hub-download 

    Consolidate HuggingFace caching in `~/.flojoy/cache/hf_hub`
    @Roulbac
    Roulbac authored Jul 15, 2023
    Configuration menu
    Copy the full SHA
    41d79df View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2023

  1. include typestub in pypi dist and clean __init__.pyi

    @Roulbac
    Roulbac committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    763abf0 View commit details
    Browse the repository at this point in the history

  2. up version to dev7

    @Roulbac
    Roulbac committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    7014d3c View commit details
    Browse the repository at this point in the history

  3. fix stubs

    @Roulbac
    Roulbac committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    31c3094 View commit details
    Browse the repository at this point in the history

  4. add missing imports to pyi

    @Roulbac
    Roulbac committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    fe69544 View commit details
    Browse the repository at this point in the history

  5. add snapshot_download

    @Roulbac
    Roulbac committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    b6a150b View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into new-node-api

    @smahmed776
    smahmed776 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    e6e6321 View commit details
    Browse the repository at this point in the history

  7. bump version to dev11

    @smahmed776
    smahmed776 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    9f8525f View commit details
    Browse the repository at this point in the history

  8. add surface case to plotly utils

    @smahmed776
    smahmed776 committed Jul 17, 2023
    Configuration menu
    Copy the full SHA
    7cd5d54 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. add error handing from run_in_venv

    @Roulbac
    Roulbac committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    126b1d1 View commit details
    Browse the repository at this point in the history

  2. fix typo

    @Roulbac
    Roulbac committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    87fb27f View commit details
    Browse the repository at this point in the history

  3. add hfhub to reqs

    @Roulbac
    Roulbac committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    9e45426 View commit details
    Browse the repository at this point in the history

  4. install flojoy in automated test

    @Roulbac
    Roulbac committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    e0ed1dd View commit details
    Browse the repository at this point in the history

  5. move flytekit to 1.6.2

    @Roulbac
    Roulbac committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    4d08e7b View commit details
    Browse the repository at this point in the history

  6. omit flyte test because of a bug

    @Roulbac
    Roulbac committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    260f422 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. Merge pull request #61 from flojoy-io/reda-fix-run-in-venv-hang 

    Prevent `@run_in_venv` from hanging if throws an error, pass child process exception
    @Roulbac
    Roulbac authored Jul 19, 2023
    Configuration menu
    Copy the full SHA
    d428115 View commit details
    Browse the repository at this point in the history

  2. Update setup.py

    @Roulbac
    Roulbac authored Jul 19, 2023
    Configuration menu
    Copy the full SHA
    d06e291 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. replace run_in_venv recv_bytes by recv

    @Roulbac
    Roulbac committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    a31b83c View commit details
    Browse the repository at this point in the history

  2. align run_in_venv pipe IO to read/send bytes

    @Roulbac
    Roulbac committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    e492415 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Add new field in result object called "text_blob" for TextBlob DC type ( 

    #64)

* bytes type and base64 encoder

* fix support for old node params typing

* text_blob dtype

* add bytes class

* add textblob class support

* offline mode for precompilation

* merge node-init into main

* get rid of useless var

* add lock for read operations in dao

* fix/hacky invalid value plotly error

* style

* option to toggle print

* switch to logging instead

* style

* add new field in result object called text_blob

* return text_blob for both text_blob and bytes type

---------

Co-authored-by: Vinicius <[email protected]>
Co-authored-by: Luiz Tauffer <[email protected]>
Co-authored-by: Hristo <[email protected]>
Co-authored-by: Joey Yu <[email protected]>
    @smahmed776 @vinicvaz
    @luiztauffer @izi-on @itsjoeoui
    5 people authored Aug 1, 2023
    Configuration menu
    Copy the full SHA
    0604f4c View commit details
    Browse the repository at this point in the history