Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix/list dir filter ext #93

Merged
merged 15 commits into from
Dec 5, 2023
Merged

Fix/list dir filter ext #93

merged 15 commits into from
Dec 5, 2023

Conversation

ankeko
Copy link
Collaborator

@ankeko ankeko commented Dec 1, 2023 •
edited
Loading

📥 Pull Request Description

The filter_ext filtering was performed before the files in the subfolders were extracted (recursive logic).

👀 Affected Areas

Every list_dir, where the filter_ext is used.

📝 Checklist

Please make sure you've completed the following tasks before submitting this pull request:

  • Pre-commit hooks were executed
  • Changes have been reviewed by at least one other developer
  • Tests have been added or updated to cover the changes (only necessary if the changes affect the executable code)
  • All tests ran successfully
  • All merge conflicts are resolved
  • Documentation has been updated to reflect the changes
  • Any necessary migrations have been run

ankeko and others added 10 commits September 27, 2023 16:19
@ankeko
fix: remove job flag to prevent multiple mlflow runs
8c8b78f
@dstalzjohn
fix: remove job flag to prevent multiple mlflow runs (#82)
31b3875 
## 📥 Pull Request Description

This pull request fixes a bug, which created two experiment runs out of
one dagster pipeline run. This is a bug, caused by the dagster
itegration of MLFlow. The implementation in niceML was correct. When the
bug of this package is fixed in the future, niceML may has to be updated
as well.
With this fix, niceML, dagster and MLFlow work and only one MLFlow
experiment run is created per dagster pipeline run.

## 👀 Affected Areas

Dagster Pipeline
MLFlow integration

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed
- [x] Changes have been reviewed by at least one other developer
- [ ] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run
@dstalzjohn
fix: Replace lambda class_extractor in DirClsDataInfoListing with…
98f25df 
… a private function because lambdas are not pickable (#85)

## 📥 Pull Request Description

Replace lambda `class_extractor` in `DirClsDataInfoListing` with a private function because lambdas are not pickable

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed
- [x] Changes have been reviewed by at least one other developer
- [ ] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run
@aiakide
Merge branch 'main' into develop
de2bc1f
@ankeko
fix: Generate mkdocs graphs with multiple dependencies (#86)
61488b2 
## 📥 Pull Request Description

Implemented option to generate pipeline graphs of dagster jobs with
multiple op dependencies.

## 👀 Affected Areas

Documentation

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed
- [x] Changes have been reviewed by at least one other developer
- [ ] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run
@ankeko @dstalzjohn
fix: Reload finished experiments in dashboard (#87)
aafe59a 
## 📥 Pull Request Description

If the dashboard was started during a running experiment, the experiment
was visible in the dashboard but with incomplete information. Even after
the experiment was finished, the Cache was not updated, so the
experiment would still be incomplete in the dashboard.

This issue was fixed by checking for the latest modification in the
experiment files and updating the cache, if it was outdated.

Additionaly, the test pipeline was fixed, by defining the execution
sequence of tests.

## 👀 Affected Areas

- dashboard
- test pipeline

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed
- [x] Changes have been reviewed by at least one other developer
- [x] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run

---------

Co-authored-by: Denis Stalz-John <[email protected]>
@aiakide
Merge branch 'main' into develop
2161bdf
@ankeko
feat: write and read json options in experiment context (#90)
1aa8bd5 
## 📥 Pull Request Description

Added write and read options for json files to ExperimentContext.
This enables the user to write out additional Information, for example
target classes.

## 👀 Affected Areas

Trainings Pipelines

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed
- [x] Changes have been reviewed by at least one other developer
- [x] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run
@dstalzjohn @aiakide
fix: pillow vulnerability and other + formatting (#89)
9c62f3b 
## 📥 Pull Request Description

Fix vulnerabilities and run some pre-commit hooks for reformatting
etc...

## 📝 Checklist

Please make sure you've completed the following tasks before submitting
this pull request:

- [x] Pre-commit hooks were executed (not all green)
- [x] Changes have been reviewed by at least one other developer
- [ ] Tests have been added or updated to cover the changes (only
necessary if the changes affect the executable code)
- [x] All tests ran successfully
- [x] All merge conflicts are resolved
- [ ] Documentation has been updated to reflect the changes
- [ ] Any necessary migrations have been run

---------

Co-authored-by: Nils <[email protected]>
@ankeko
fix: filter_ext in list_dir
dd99ba8
@ankeko ankeko added 🪲 bug Related to bugs or unexpected behavior 🧪 testing Related to testing or test cases labels Dec 1, 2023
@ankeko ankeko requested review from dstalzjohn and aiakide December 1, 2023 12:39
@aiakide
Copy link
Collaborator

aiakide commented Dec 1, 2023

Linked issue: #91

@ankeko ankeko removed the request for review from aiakide December 5, 2023 08:44
@ankeko ankeko merged commit d8171fb into main Dec 5, 2023
6 checks passed
@ankeko ankeko deleted the fix/list_dir_filter_ext branch December 5, 2023 12:39
@aiakide aiakide mentioned this pull request Dec 18, 2023
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dstalzjohn dstalzjohn dstalzjohn approved these changes

Assignees
No one assigned
Labels
🪲 bug Related to bugs or unexpected behavior 🧪 testing Related to testing or test cases
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ankeko @aiakide @dstalzjohn