Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(datasets): Improved Dependency Management for Spark-based Datasets #911

Merged
merged 30 commits into from
Nov 14, 2024
Merged

feat(datasets): Improved Dependency Management for Spark-based Datasets #911

merged 30 commits into from
Nov 14, 2024

Commits on Oct 26, 2024

  1. added the skeleton for the utils sub pkg 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    e4a9e47 View commit details
    Browse the repository at this point in the history

  2. moved the utility funcs from spark_dataset to relevant modules in _utils 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    1ab51ad View commit details
    Browse the repository at this point in the history

  3. updated the use of utility funcs in spark_dataset 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    0187cf1 View commit details
    Browse the repository at this point in the history

  4. fixed import in databricks_utils 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    84675bf View commit details
    Browse the repository at this point in the history

  5. renamed _strip_dbfs_prefix to strip_dbfs_prefix 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    ae3829d View commit details
    Browse the repository at this point in the history

  6. updated the other modules that import from spark_dataset to use _utils 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    c5bb7c5 View commit details
    Browse the repository at this point in the history

  7. updated the use of strip_dbfs_prefix in spark_dataset 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    48df490 View commit details
    Browse the repository at this point in the history

  8. fixed lint issues 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    f562190 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2024

  1. removed the base deps for spark, pandas and delta from databricks dat… 

    …asets

Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 27, 2024
    Configuration menu
    Copy the full SHA
    0a89638 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. Merge branch 'main' into feature/improve_spark_dependencies

    @MinuraPunchihewa
    MinuraPunchihewa authored Oct 30, 2024
    Configuration menu
    Copy the full SHA
    72e1925 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Merge branch 'main' into feature/improve_spark_dependencies

    @noklam
    noklam authored Oct 31, 2024
    Configuration menu
    Copy the full SHA
    406adfa View commit details
    Browse the repository at this point in the history

  2. moved the file based utility funcs to databricks_utils 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    8f7842f View commit details
    Browse the repository at this point in the history

  3. fixed the imports of the file based utility funcs 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    6e50319 View commit details
    Browse the repository at this point in the history

  4. fixed lint issues 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    63834a9 View commit details
    Browse the repository at this point in the history

  5. fixed the use of _get_spark() in tests 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    6df7741 View commit details
    Browse the repository at this point in the history

  6. fixed uses of databricks utils in tests 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    80f9fb5 View commit details
    Browse the repository at this point in the history

  7. fixed more tests 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    03da501 View commit details
    Browse the repository at this point in the history

  8. fixed more lint issues 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    0534907 View commit details
    Browse the repository at this point in the history

  9. fixed more tests 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    65d1c30 View commit details
    Browse the repository at this point in the history

  10. fixed more tests 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    f7f0b5e View commit details
    Browse the repository at this point in the history

  11. improved type hints for spark & databricks utility funcs 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    608d2a2 View commit details
    Browse the repository at this point in the history

  12. fixed more lint issues 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    e28cdf1 View commit details
    Browse the repository at this point in the history

  13. further improved type hints for utility funcs 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    781a7ef View commit details
    Browse the repository at this point in the history

  14. fixed a couple of incorrect type hints 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    768e8ae View commit details
    Browse the repository at this point in the history

  15. fixed several incorrect type hints 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    c44e32a View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Merge branch 'main' into feature/improve_spark_dependencies

    @noklam
    noklam authored Nov 7, 2024
    Configuration menu
    Copy the full SHA
    4b31967 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2024

  1. Merge branch 'main' into feature/improve_spark_dependencies

    @merelcht
    merelcht authored Nov 13, 2024
    Configuration menu
    Copy the full SHA
    4c427cc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into feature/improve_spark_dependencies

    @merelcht
    merelcht authored Nov 13, 2024
    Configuration menu
    Copy the full SHA
    7334e1b View commit details
    Browse the repository at this point in the history

  3. updated the release notes 

    Signed-off-by: Minura Punchihewa <[email protected]>
    @MinuraPunchihewa
    MinuraPunchihewa committed Nov 13, 2024
    Configuration menu
    Copy the full SHA
    d7a385d View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2024

  1. Reorder release notes 

    Signed-off-by: Merel Theisen <[email protected]>
    @merelcht
    merelcht committed Nov 14, 2024
    Configuration menu
    Copy the full SHA
    0f65bdb View commit details
    Browse the repository at this point in the history