Dev (#203)

* added essential rows to config (#187)

* Added dash user interface (#188)

* Added dash user interface

* Fixed bugs with dui

* Fixed bugs with dui

---------

Co-authored-by: ibragimmergaliev <[email protected]>

* Added augmentations for segmentation of NDVI images (#192)

* added augmentations

* added ndvi preprocessing

* added test

* refactoring

* Added Segmentation Losses (#191)

* added segmentation losses and update package

* added tests

* Feature/train arable model (#189)

* added parameters folder names

* interim

* interim

* bugfix

* add learning rate monitor

* add default values to optimizer

* updated readme

* bugfix

* made lr monitor as a default callback

* added ndvi preprocessing

* removed files

* refactored files

* minor refactorings

---------

Co-authored-by: InnopolisUniversity <[email protected]>

* Feature/add schedulers (#169)

* interim

* added lr logging

* interim changes

* added essential attributes

* interim

* updated configs

* added a function for the config folder retrieval

* removed redundant lines

* added learning rate monitor callback

* refactored the file

* added test for scheduler config files

* removed redundant lines

* minor refactoring

* removed redundant lines

* added auto scheduler

* updated scheduler configs

* added scripts

* refactored optimizer class

* made scheduler initialization without try except

* changed scheduler object initialization

* updated

* updated cfg

* fixed cfg

* removed cfg

* updated cfg

* updated step scheduler cfg

* update func

* updated scheduler cfgs

* updated shell scripts

* updated

* change order (#194)

* Added fix gor config savings

* Bugfix (#196)

* change order

* minor refactoring

* updated configs

* bugfixes

* updated shell scripts

* Bugfix (#199)

* change order

* minor refactoring

* updated configs

* bugfixes

* updated shell scripts

* Update tests thresholds

* uploaded data to s3 and updated config

* fixed bug in infer.py

* renamed config

* renamed dataset config

* renamed model config name

* updated shell script

* added error handling in function

* added ability to make inference on untrained model

* important changes

* added aeroflot inference data

* added datasets for inference

* fix import errors

* updated folder structure

* added default values

---------

Co-authored-by: KGallyamov <[email protected]>

* Bugfix (#201)

* change order

* minor refactoring

* updated configs

* bugfixes

* updated shell scripts

* Update tests thresholds

* uploaded data to s3 and updated config

* fixed bug in infer.py

* renamed config

* renamed dataset config

* renamed model config name

* updated shell script

* added error handling in function

* added ability to make inference on untrained model

* important changes

* added aeroflot inference data

* added datasets for inference

* fix import errors

* updated folder structure

* added default values

* convert script to cpu

---------

Co-authored-by: KGallyamov <[email protected]>

* Bugfix (#202)

* change order

* minor refactoring

* updated configs

* bugfixes

* updated shell scripts

* Update tests thresholds

* uploaded data to s3 and updated config

* fixed bug in infer.py

* renamed config

* renamed dataset config

* renamed model config name

* updated shell script

* added error handling in function

* added ability to make inference on untrained model

* important changes

* added aeroflot inference data

* added datasets for inference

* fix import errors

* updated folder structure

* added default values

* convert script to cpu

* edited dataset configs

---------

Co-authored-by: KGallyamov <[email protected]>

* Delete innofw/utils/data_utils/dataset_prep/prep_gis.py

* updated scripts according to pmi

---------

Co-authored-by: InnopolisUniversity <[email protected]>
Co-authored-by: ibragimmergaliev <[email protected]>
Co-authored-by: KGallyamov <[email protected]>

config/augmentations_train/arable-ndvi-min.yaml

@@ -0,0 +1,15 @@
+defaults:
+    - /augmentations/preprocessing:
+    - /augmentations/combined:
+        - none
+    - /augmentations/position:
+        - random_resized_crop
+    - /augmentations/color:
+    - /augmentations/postprocessing:
+
+
+augmentations:
+    position:
+        random_crop:
+            height: 224
+            width: 224

config/augmentations_val/arable-ndvi-min.yaml

@@ -0,0 +1,15 @@
+defaults:
+    - /augmentations/preprocessing:
+    - /augmentations/combined:
+        - none
+    - /augmentations/position:
+        - random_resized_crop
+    - /augmentations/color:
+    - /augmentations/postprocessing:
+
+
+augmentations:
+    position:
+        random_crop:
+            height: 224
+            width: 224

config/datasets/detection/air_data.yaml

@@ -12,8 +12,8 @@ _target_: innofw.core.integrations.ultralytics.datamodule.UltralyticsDataModuleA
 train:
 test:
 infer: 
-  source: https://api.blackhole.ai.innopolis.university/public-datasets/testing/air_data/infer.zip
-  target: ./data/air_data_s3/
+  source: https://api.blackhole.ai.innopolis.university/public-datasets/air_data/train.zip
+  target: ./data/air_data/
 
 num_workers: 8
 

config/datasets/detection/auto.yaml

@@ -12,8 +12,8 @@ _target_: innofw.core.integrations.ultralytics.datamodule.UltralyticsDataModuleA
 train:
 test:
 infer: 
-  source: https://api.blackhole.ai.innopolis.university/public-datasets/testing/auto_data/infer.zip
-  target: ./data/auto_data_s3/
+  source: https://api.blackhole.ai.innopolis.university/public-datasets/auto/train.zip
+  target: ./data/auto/
 
 num_workers: 8
 

config/datasets/detection/wares_sort.yaml

@@ -12,8 +12,8 @@ _target_: innofw.core.integrations.ultralytics.datamodule.UltralyticsDataModuleA
 train:
 test:
 infer: 
-  source: https://api.blackhole.ai.innopolis.university/public-datasets/testing/sort_data/infer.zip
-  target: ./data/sort_data_s3/
+  source: https://api.blackhole.ai.innopolis.university/public-datasets/wares_sort/train.zip
+  target: ./data/wares_sort/
 
 num_workers: 8
 

config/experiments/semantic-segmentation/arable-bin-seg-ndvi/fields/baseline.yaml

@@ -2,16 +2,16 @@
 defaults:
   - _self_
   - override /datasets: semantic-segmentation/arable-bin-seg-ndvi/fields/tile-512-90-days
-  - override /augmentations_train: arable-ndvi-strong
-  - override /augmentations_val: arable-ndvi-strong
-  - override /models: semantic-segmentation/deeplabv3plus
+  - override /augmentations_train: arable-ndvi-min
+  - override /augmentations_val: arable-ndvi-min
+  - override /models: semantic-segmentation/unet
   - override /losses: semantic-segmentation/dice
   - override /optimizers: adam
   - override /schedulers: cosine_annealing
   - override /loggers:
 
 datasets:
-  num_workers: 32
+  num_workers: 4
 
 models:
   in_channels: 3

config/schedulers/cosine_annealing.yaml

@@ -1,3 +1,3 @@
 description: Cosine Annealing
 _target_: torch.optim.lr_scheduler.CosineAnnealingLR
-T_max: 20
+T_max: 3

examples/cosine_annealing_ls_scheduler.sh

@@ -1 +1 @@
-python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=cosine_annealing epochs=100 optimizers.lr=0.001
+python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=cosine_annealing epochs=20 optimizers.lr=0.001 +trainer.overfit_batches=1

examples/exponential_lr_scheduler.sh

@@ -1 +1 @@
-python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=exponential epochs=10 optimizers.lr=0.001
+python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=exponential epochs=10 optimizers.lr=0.001 +trainer.overfit_batches=1

examples/lambda_lr_shcheduler.sh

@@ -1 +1 @@
-python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=lambda epochs=50 optimizers.lr=0.001
+python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=lambda epochs=20 optimizers.lr=0.001 +trainer.overfit_batches=1

examples/multistep_lr_scheduler.sh

@@ -1 +1 @@
-python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=multistep optimizers.lr=0.001 epochs=50
+python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=multistep optimizers.lr=0.001 epochs=20 +trainer.overfit_batches=1

examples/step_lr_scheduler.sh

@@ -1 +1 @@
-python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=step epochs=18 optimizers.lr=0.001
+python train.py experiments=semantic-segmentation/arable-bin-seg-ndvi/fields/baseline schedulers=step epochs=20 optimizers.lr=0.001 +trainer.overfit_batches=1