Getting a list of configurations for the same group in the defaults list (wihtout overriding) #2243

YerePhy · 2022-05-27T09:48:38Z

YerePhy
May 27, 2022

Hi,

I would like to have a dict like the following in my main script:

cfg = {
  "{key}": {"value"},
  .
  .
  .
  "trainer": {
    "callbacks": [ModelSummary(), ModelCheckpoint(), ...],
    "devices": 2, 
    "accelerator": "gpu"  
    }
}

I have the following folder structure for the configs:

train/
├── config.yaml
├── datamodule
│   ├── monai_holdout_datamodule.yaml
│   ├── patch_func
│   │   └── label.yaml
│   ├── splitter
│   │   └── hold_out.yaml
│   ├── test_stage_transformations
│   │   └── dummy.yaml
│   └── train_stage_transformations
│       └── dummy.yaml
├── lightningmodule
│   ├── architecture
│   │   └── unet.yaml
│   ├── loss
│   │   └── generalized_dice.yaml
│   ├── metrics
│   │   └── dice.yaml
│   ├── optimizer_configuration
│   │   └── adam.yaml
│   ├── scheduler_configuration
│   │   └── step_lr.yaml
│   └── unet_like.yaml
└── trainer
    ├── callbacks
    │   ├── model_checkpoint.yaml
    │   └── model_summary.yaml
    └── standard.yaml

Im trying to merge the callbacks in a list inside of standard.yaml:

defaults: 
  - _self_
  - callbacks/model_summary
  - callbacks/model_checkpoint

Then including standard.yaml as a configuration for trainer group in config.yaml:

defaults:
  - _self_
  - datamodule/monai_holdout_datamodule
  - lightningmodule/unet_like
  - trainer/standard
datamodule_type: monai_hold_out
lightningmodule_type: unet_like

However this is not working, any proposal?

Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 86, in _call_target
    return _target_(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'max_depth'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 26, in <module>
    main()
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/main.py", line 48, in decorated_main
    _run_hydra(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 12, in main
    cfg = hydra.utils.instantiate(cfg, _recursive_=True)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 218, in instantiate
    return instantiate_node(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 348, in instantiate_node
    cfg[key] = instantiate_node(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 348, in instantiate_node
    cfg[key] = instantiate_node(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 331, in instantiate_node
    return _call_target(_target_, partial, args, kwargs, full_key)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 91, in _call_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error in call to target 'pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint':
TypeError("__init__() got an unexpected keyword argument 'max_depth'")
full_key: trainer.callbacks

Answered by jieru-hu

Jun 17, 2022

hi @YerePhy - could you try to create a smaller repro that only involves config that causing issues? that'd be easier for us to repro on our side and debug together.

View full answer

Jasha10 · 2022-05-31T15:35:13Z

Jasha10
May 31, 2022
Collaborator

@YerePhy please see my comment here for a recommendation on how to compose a list of callbacks.

6 replies

Jasha10 Jun 1, 2022
Collaborator

Please print the composed config and make sure that _metrics_dict is showing up in the right place:

python my_app.py --cfg=job

YerePhy Jun 1, 2022
Author

Sorry, but the project is changing fast...

I've applied your proposal and a new error resolving the interpolation is arising, the merged configuration seems OK now but it looks like there is a new problem with the resolvers...

Tree:

./configs/
├── hydra
│   └── job_logging
│       └── custom.yaml
└── train
    ├── brats2021.yaml
    ├── datamodule
    │   ├── brats2021.yaml
    │   ├── patch_fn
    │   │   └── label_classed.yaml
    │   ├── split_fn
    │   │   └── hold_out.yaml
    │   ├── test_stage_transformations
    │   │   └── brats2021.yaml
    │   └── train_stage_transformations
    │       └── brats2021.yaml
    ├── lightningmodule
    │   ├── architecture
    │   │   └── unet.yaml
    │   ├── brats2021.yaml
    │   ├── loss
    │   │   └── generalized_dice.yaml
    │   ├── loss_evaluator
    │   │   └── simple_output_loss_evaluator.yaml
    │   ├── metrics
    │   │   └── dice.yaml
    │   ├── metrics_evaluator
    │   │   ├── brats_2021_metrics_evaluator.yaml
    │   │   └── simple_output_metrics_evaluator.yaml
    │   ├── optimizer_configuration
    │   │   └── adam.yaml
    │   ├── patch_predictor
    │   │   └── patch_predictor_simple_output.yaml
    │   ├── postprocessing
    │   │   └── brats2021.yaml
    │   ├── postprocessing_metrics
    │   │   └── brats2021.yaml
    │   └── scheduler_configuration
    │       └── step_lr.yaml
    └── trainer
        ├── brats2021.yaml
        └── logger
            └── wandb.yaml

Then:

# /configs/train/lightningmodule/metrics/dice.yaml
_target_: monai.metrics.DiceMetric
include_background: True
reduction: mean_batch

And:

# /configs/train/lightningmodule/brats2021.yaml
defaults:
  - architecture: unet
  - optimizer_configuration: adam
  - scheduler_configuration: step_lr
  - loss: generalized_dice
  - postprocessing: brats2021
  - loss_evaluator: simple_output_loss_evaluator 
  - metrics_evaluator: brats_2021_metrics_evaluator
  - postprocessing_metrics: brats2021
  - patch_predictor: patch_predictor_simple_output
  - metrics@_metrics_dict.dice: dice
  - _self_
type: semantic_segmentator
metrics: ${oc.dict.values:_metrics_dict}
optimizer: Adam
scheduler: StepLR

And finally:

# /configs/train/brats2021.yaml
defaults:
  - datamodule: brats2021
  - lightningmodule: brats2021
  - trainer: brats2021
  - override hydra/job_logging: custom
  - override lightningmodule/scheduler_configuration: null
  - _self_
hydra: 
  searchpath:
    - file://${oc.env:HOME}/Repositories/brain_tumor/src/configs
lightningmodule:
  scheduler: null
datamodule:
  batch_size: 1

With main script train.py:

import hydra
import pytorch_lightning as pl
from omegaconf import DictConfig, OmegaConf
from utils import datamodule_factory, lightningmodule_factory

@hydra.main(config_path="./configs/train", config_name="brats2021")
def main(cfg: DictConfig):
    """
    Perform training.
    """
    OmegaConf.register_new_resolver("get_method", hydra.utils.get_method)
    OmegaConf.resolve(cfg) # I think that the error is coming from here
    del cfg._metrics_dict
    cfg = hydra.utils.instantiate(cfg, _recursive_=True)
    datamodule_type = cfg["datamodule"].pop("type")
    lightningmodule_type = cfg["lightningmodule"].pop("type")
    datamodule = datamodule_factory(datamodule_type)(**cfg["datamodule"])
    lightningmodule = lightningmodule_factory(lightningmodule_type)(
        **cfg["lightningmodule"]
    )
    trainer = pl.Trainer(**cfg["trainer"])
    trainer.fit(model=lightningmodule, datamodule=datamodule)

When I run python train.py --cfg=job, I get:

datamodule:
  split_fn:
    _target_: functools.partial
    _args_:
    - ${get_method:monai.data.utils.partition_dataset}
    ratios:
    - 0.7
    - 0.15
    - 0.15
    num_partitions: 3
  patch_fn:
    _target_: monai.transforms.RandCropByLabelClassesd
    keys:
    - INPUT
    - GROUND_TRUTH
    spatial_size:
    - 64
    - 64
    - 1
    label_key: GROUND_TRUTH
    image_key: INPUT
    ratios:
    - 3
    - 3
    - 6
    num_samples: 4
  train_stage_transformations:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.LoadImaged
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.AddChanneld
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.ConcatItemsd
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      name: INPUT
    - _target_: monai.transforms.ConvertToMultiChannelBasedOnBratsClassesd
      keys:
      - GROUND_TRUTH
    - _target_: monai.transforms.BorderPadd
      keys:
      - INPUT
      - GROUND_TRUTH
      spatial_border: 64
  test_stage_transformations:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.LoadImaged
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.AddChanneld
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.ConcatItemsd
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      name: INPUT
    - _target_: monai.transforms.ConvertToMultiChannelBasedOnBratsClassesd
      keys:
      - GROUND_TRUTH
  type: monai_hold_out
  dataset_path: /home/yere/Repositories/brain_tumor/datasets/dataset_concat
  modalities:
  - T1
  - T1CE
  - T2
  - FLAIR
  - GROUND_TRUTH
  ext: .nii.gz
  batch_size: 1
  num_workers: 1
  load_subset: 0.01
lightningmodule:
  architecture:
    _target_: monai.networks.nets.UNet
    spatial_dims: 2
    in_channels: 4
    out_channels: 3
    channels:
    - 64
    - 128
    - 256
    - 512
    strides:
    - 2
    - 2
    - 2
    kernel_size: 3
    up_kernel_size: 3
    num_res_units: 4
    act: PRELU
    norm: INSTANCE
  optimizer_configuration:
    lr: 0.001
    betas:
    - 0.9
    - 0.999
  loss:
    _target_: monai.losses.GeneralizedDiceLoss
    include_background: false
    to_onehot_y: false
    sigmoid: true
    softmax: false
    reduction: mean
  postprocessing:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.Activations
      softmax: false
      sigmoid: true
    - _target_: monai.transforms.AsDiscrete
      argmax: false
      to_onehot: None
      threshold: 0.5
      rounding: None
  loss_evaluator:
    _target_: functools.partial
    _args_:
    - ${get_method:package.lightning_modules.utils.simple_output_loss_evaluator}
  metrics_evaluator:
    _target_: functools.partial
    _args_:
    - ${get_method:package.lightning_modules.utils.brats_2021_metrics_evaluator}
  postprocessing_metrics:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.Activations
      softmax: false
      sigmoid: true
    - _target_: monai.transforms.AsDiscrete
      argmax: null
      to_onehot: null
      threshold: 0.5
      rounding: null
  patch_predictor:
    _target_: functools.partial
    _args_:
    - ${get_method:package.lightning_modules.utils.patch_predictor_simple_output}
  _metrics_dict:
    dice:
      _target_: monai.metrics.DiceMetric
      include_background: true
      reduction: mean_batch
  type: semantic_segmentator
  metrics: ${oc.dict.values:_metrics_dict}
  optimizer: Adam
  scheduler: null
trainer:
  callbacks:
  - _target_: pytorch_lightning.callbacks.ModelCheckpoint
    verbose: true
  limit_train_batches: 2
  max_epochs: 3

And the error when i run train.py:

Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 25, in 
    main()
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/main.py", line 48, in decorated_main
    _run_hydra(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 378, in 
    lambda: hydra.run(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 12, in main
    OmegaConf.resolve(cfg)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 792, in resolve
    omegaconf._impl._resolve(cfg)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 44, in _resolve
    _resolve_container_value(cfg, i)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 23, in _resolve_container_value
    node._set_value(resolved._value())
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/nodes.py", line 44, in _set_value
    self._val = self.validate_and_convert(value)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/nodes.py", line 57, in validate_and_convert
    return self._validate_and_convert_impl(value)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/nodes.py", line 133, in _validate_and_convert_impl
    raise UnsupportedValueType(
omegaconf.errors.UnsupportedValueType: Value 'function' is not a supported primitive type

YerePhy Jun 2, 2022
Author

@Jasha10, what about deleting this? I would re-open it when the structure of my project is more stable, or you think that we shouldn't?

Jasha10 Jun 3, 2022
Collaborator

@Jasha10, what about deleting this? I would re-open it when the structure of my project is more stable, or you think that we shouldn't?

Either way is fine :P

omegaconf.errors.UnsupportedValueType: Value 'function' is not a supported primitive type

I think the problem is with get_method. Resolving the config causes e.g. ${get_method:monai.data.utils.partition_dataset} to be replaced with the actual monai.data.utils.partition_dataset function, which leads to the Value 'function' is not a supported error.

What if you use this resolver instead of get_method?

OmegaConf.register_new_resolver("wrap_method", lambda dotpath: {'_target_': dotpath, '_partial_': True})

(This would mean replacing ${get_method:monai.data.utils.partition_dataset} with ${wrap_method:monai.data.utils.partition_dataset} in your config).

YerePhy Jun 16, 2022
Author

Hi again @Jasha10, sorry for the delay...

I've registered the new resolver and make the changes in the config files (replace get_method by wrap_method). Then I've followed your comment on how to compose a list of callbacks. However, it does not work yet:

Output of python myapp.py --cfg=job:

datamodule:
  split_fn:
    _target_: functools.partial
    _args_:
    - ${wrap_method:monai.data.utils.partition_dataset}
    ratios:
    - 0.7
    - 0.15
    - 0.15
    num_partitions: 3
  patch_fn:
    _target_: monai.transforms.RandCropByLabelClassesd
    keys:
    - INPUT
    - GROUND_TRUTH
    spatial_size:
    - 92
    - 92
    - 92
    label_key: GROUND_TRUTH
    image_key: INPUT
    ratios:
    - 3
    - 3
    - 6
    num_samples: 4
  train_stage_transforms:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.LoadImaged
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.AddChanneld
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.ConcatItemsd
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      name: INPUT
    - _target_: monai.transforms.ConvertToMultiChannelBasedOnBratsClassesd
      keys:
      - GROUND_TRUTH
    - _target_: monai.transforms.Resized
      keys:
      - INPUT
      - GROUND_TRUTH
      mode:
      - area
      - nearest
      spatial_size:
      - 192
      - 192
      - 192
    - _target_: monai.transforms.BorderPadd
      keys:
      - INPUT
      - GROUND_TRUTH
      spatial_border: 64
    - _target_: monai.transforms.CastToTyped
      keys:
      - GROUND_TRUTH
      dtype: int
  test_stage_transforms:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.LoadImaged
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.AddChanneld
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      - GROUND_TRUTH
    - _target_: monai.transforms.ConcatItemsd
      keys:
      - T1
      - T1CE
      - T2
      - FLAIR
      name: INPUT
    - _target_: monai.transforms.Resized
      keys:
      - INPUT
      - GROUND_TRUTH
      mode:
      - area
      - nearest
      spatial_size:
      - 192
      - 192
      - 192
    - _target_: monai.transforms.MapLabelValued
      keys:
      - GROUND_TRUTH
      orig_labels:
      - 0
      - 1
      - 2
      - 4
      target_labels:
      - 0
      - 1
      - 2
      - 3
    - _target_: monai.transforms.AsDiscreted
      keys:
      - GROUND_TRUTH
      to_onehot: 4
  patch_fn/spatial_size:
  - 72
  - 72
  - 72
  type: monai_hold_out
  dataset_path: /home/yere/Repositories/brain_tumor/datasets/dataset_concat
  modalities:
  - T1
  - T1CE
  - T2
  - FLAIR
  - GROUND_TRUTH
  ext: .nii.gz
  batch_size: 1
  num_workers: 2
  load_subset: null
  shuffle: false
lightningmodule:
  architecture:
    _target_: monai.networks.nets.UNet
    spatial_dims: 3
    in_channels: 4
    out_channels: 3
    channels:
    - 64
    - 64
    - 64
    - 64
    strides:
    - 2
    - 2
    - 2
    kernel_size: 3
    up_kernel_size: 3
    num_res_units: 2
    act: PRELU
    norm: null
  optimizer_configuration:
    lr: 0.001
  scheduler_configuration:
    step_size: 3
    gamma: 0.1
  postprocessing:
    _target_: monai.transforms.Compose
    transforms:
    - _target_: monai.transforms.Activations
      softmax: false
      sigmoid: true
    - _target_: monai.transforms.ToNumpy
    - _target_: package.transforms.nvidia_brats2021_connected_components.NVIDIABraTS2021ConnectedComponents
      et: 2
      wt: 0
      tc: 1
      et_th: 0.4
      tc_th: 0.4
      wt_th: 0.45
    - _target_: monai.transforms.AddChannel
    - _target_: monai.transforms.MapLabelValue
      orig_labels:
      - 0
      - 1
      - 2
      - 4
      target_labels:
      - 0
      - 1
      - 2
      - 3
    - _target_: monai.transforms.AsDiscrete
      to_onehot: 4
  loss:
    _target_: monai.losses.DiceFocalLoss
    sigmoid: true
    softmax: false
    include_background: true
    reduction: mean
  metrics_evaluator:
    _target_: package.evaluators.simple.SimpleMetricsEvaluator
    metrics:
      DICE:
        _target_: monai.metrics.DiceMetric
        reduction: mean_batch
      HAUSDORFF:
        _target_: monai.metrics.HausdorffDistanceMetric
        reduction: mean_batch
    before_computation_transforms:
      _target_: monai.transforms.Compose
      transforms:
      - _target_: monai.transforms.Activations
        softmax: false
        sigmoid: true
      - _target_: monai.transforms.AsDiscrete
        argmax: null
        to_onehot: null
        threshold: 0.5
        rounding: null
  patch_predictor:
    _target_: functools.partial
    _args_:
    - ${wrap_method:package.lightning_modules.utils.simple_output_patch_predictor}
  type: semantic_segmentator
  optimizer: Adam
  scheduler: StepLR
  sw_kwargs_test:
    roi_size:
    - 96
    - 96
    - 96
    overlap: 0.25
trainer:
  logger:
    _target_: pytorch_lightning.loggers.WandbLogger
    name: test_run
    save_dir: ${oc.env:HOME}/Repositories/brain_tumor/logs
    project: ONCO-BRAIN
    entity: qubionets
  _callback_dict:
    wandb_overlapping_masks_3d:
      _target_: package.callbacks.wandb_overlapping_masks_3d.WANDBOverlappingMasks3D
      input_aliases:
        0: T1
        1: T1CE
        2: T2
        3: FLAIR
      masks_aliases:
        0: TC
        1: ET
        2: WT
      batch_idxs:
      - 0
      - 1
      output_postprocessing:
        _target_: monai.transforms.Compose
        transforms:
        - _target_: monai.transforms.Activations
          sigmoid: true
        - _target_: monai.transforms.AsDiscrete
          threshold: 0.5
      input_postprocessing:
        _target_: monai.transforms.Compose
        transforms:
        - _target_: monai.transforms.ScaleIntensity
          minv: 0
          maxv: 1
      ground_truth_postprocessing: null
      n_slices: 2
      slices_dim: -1
      principal_channel_gt: 0
    save_test_results:
      _target_: package.callbacks.save_test_results.SaveTestResults
      batch_idxs:
      - 0
      - 1
      metadata_key: GROUND_TRUTH_meta_dict
      save_dir: /home/yere/Repositories/brain_tumor/test_results
      input_channels_aliases:
        0: T1
        1: T1CE
        2: T2
        3: FLAIR
  callbacks: ${oc.dict.values:_callback_dict}
  limit_train_batches: 1
  limit_val_batches: 1
  limit_test_batches: 1.0
  max_epochs: 2
  log_every_n_steps: 1
  gpus: 1

Main script:

import hydra
import pytorch_lightning as pl
from omegaconf import DictConfig, OmegaConf
from utils import datamodule_factory, lightningmodule_factory

@hydra.main(config_path="./configs/experiment", config_name="config_3D_test")
def main(cfg: DictConfig):
    """
    Perform training.
    """
    OmegaConf.register_new_resolver("wrap_method", lambda dotpath: {'_target_': dotpath, '_partial_': True})
    OmegaConf.resolve(cfg)
    del cfg._callback_dict
    cfg = hydra.utils.instantiate(cfg, _recursive_=True)
    datamodule_type = cfg["datamodule"].pop("type")
    lightningmodule_type = cfg["lightningmodule"].pop("type")
    datamodule = datamodule_factory(datamodule_type)(**cfg["datamodule"])
    datamodule.setup()
    lightningmodule = lightningmodule_factory(lightningmodule_type)(
        **cfg["lightningmodule"]
    )
    trainer = pl.Trainer(**cfg["trainer"])
    cfg["trainer"]["logger"].log_hyperparams(cfg)
    trainer.fit(model=lightningmodule, datamodule=datamodule)

main()

Ouptut:

The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="./configs/experiment", config_name="config_3D_test")
/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
  ret = run_job(
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 29, in 
    main()
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/utils.py", line 453, in 
    lambda: hydra.run(
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 14, in main
    OmegaConf.resolve(cfg)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 768, in resolve
    omegaconf._impl._resolve(cfg)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 44, in _resolve
    _resolve_container_value(cfg, i)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/_impl.py", line 23, in _resolve_container_value
    node._set_value(_get_value(resolved))
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/nodes.py", line 46, in _set_value
    self._val = self.validate_and_convert(value)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/nodes.py", line 76, in validate_and_convert
    return self._validate_and_convert_impl(value)
  File "/home/yere/anaconda3/envs/brain_tumor/lib/python3.8/site-packages/omegaconf/nodes.py", line 154, in _validate_and_convert_impl
    raise UnsupportedValueType(
omegaconf.errors.UnsupportedValueType: Value 'dict' is not a supported primitive type

It looks like there is something wring resolving the callaback dict...

Thanks!

jieru-hu · 2022-06-17T19:07:26Z

jieru-hu
Jun 17, 2022

hi @YerePhy - could you try to create a smaller repro that only involves config that causing issues? that'd be easier for us to repro on our side and debug together.

7 replies

YerePhy Jun 27, 2022
Author

Done sir, but then I get another error, seems to be related to the interpolation in callbacks: ${oc.dict.values:_callback_dict}:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 14, in 
    main()
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/utils.py", line 453, in 
    lambda: hydra.run(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/yere/Repositories/brain_tumor/src/train.py", line 12, in main
    cfg_instantiated = hydra.utils.instantiate(cfg, _recursive_=True)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 216, in instantiate
    OmegaConf.resolve(config)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 768, in resolve
    omegaconf._impl._resolve(cfg)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/_impl.py", line 25, in _resolve_container_value
    _resolve(node)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/_impl.py", line 14, in _resolve_container_value
    resolved = node._dereference_node()
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 246, in _dereference_node
    node = self._dereference_node_impl(throw_on_resolution_failure=True)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 277, in _dereference_node_impl
    return parent._resolve_interpolation_from_parse_tree(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 584, in _resolve_interpolation_from_parse_tree
    resolved = self.resolve_parse_tree(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 769, in resolve_parse_tree
    raise InterpolationResolutionError(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 764, in resolve_parse_tree
    return visitor.visit(parse_tree)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 206, in accept
    return visitor.visitConfigValue(self)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 101, in visitConfigValue
    return self.visit(ctx.getChild(0))
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 342, in accept
    return visitor.visitText(self)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 298, in visitText
    return self.visitInterpolation(c)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 125, in visitInterpolation
    return self.visit(ctx.getChild(0))
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar/gen/OmegaConfGrammarParser.py", line 1041, in accept
    return visitor.visitInterpolationResolver(self)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/grammar_visitor.py", line 179, in visitInterpolationResolver
    return self.resolver_interpolation_callback(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 750, in resolver_interpolation_callback
    return self._evaluate_custom_resolver(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/base.py", line 694, in _evaluate_custom_resolver
    return resolver(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 439, in resolver_wrapper
    ret = resolver(*args, **kwargs)
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/resolvers/oc/dict.py", line 30, in values
    in_dict = _get_and_validate_dict_input(
  File "/home/yere/anaconda3/envs/iq_segmentator/lib/python3.9/site-packages/omegaconf/resolvers/oc/dict.py", line 75, in _get_and_validate_dict_input
    raise ConfigKeyError(f"Key not found: '{key}'")
omegaconf.errors.InterpolationResolutionError: ConfigKeyError raised while resolving interpolation: Key not found: '_callback_dict'

Jasha10 Jun 27, 2022
Collaborator

Please try replacing callbacks: ${oc.dict.values:_callback_dict} with callbacks: ${oc.dict.values:._callback_dict}

YerePhy Jun 28, 2022
Author

Finally working, could you explain what was causing the issue? Why a simple . solves the problem?

Jasha10 Jun 28, 2022
Collaborator

Please take a look at the OmegaConf docs on Variable Interpolation.
Using a dot . makes the interpolation a "relative path". With no dot it is an "absolute path". This is like the difference between the paths "home/yere/anaconda3" and "/home/yere/anaconda3" on your computer's filesystem.

Your config looks like this:

...
trainer:
  _callback_dict:
    ...
  callbacks: ${oc.dict.values:...}

For an interpolation from trainer.callbacks to trainer._callback_dict, you either need to use an absolute path ${oc.dict.values:trainer._callback_dict} or a relative path ${oc.dict.values:._callback_dict}

YerePhy Jun 28, 2022
Author

Definitely, I should get more familiar with OmegaConf if I want to use Hydra consistently...

Thanks!

zongshun21 · 2023-04-03T03:11:37Z

zongshun21
Apr 3, 2023

train_partseg.py:45: UserWarning: The version_base parameter is not specified. Please specify a compatability version level, or None. Will assume defaults for version 1.1 @hydra.main(config_path='config', config_name='partseg') /home/wzs/anaconda3/envs/mypytorch/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'partseg': Defaults list is missing self`. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/wzs/anaconda3/envs/mypytorch/lib/python3.7/site-packages/hydra/_internal/hydra.py:127: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
configure_logging=with_log_configuration,
Error executing job with overrides: []
Traceback (most recent call last):
File "train_partseg.py", line 53, in main
print(args.pretty())
omegaconf.errors.ConfigAttributeError: Missing key pretty
full_key: pretty
object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.`

How did this problem be solved?

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting a list of configurations for the same group in the defaults list (wihtout overriding) #2243

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Getting a list of configurations for the same group in the defaults list (wihtout overriding) #2243

YerePhy May 27, 2022

Replies: 3 comments · 13 replies

Jasha10 May 31, 2022 Collaborator

Jasha10 Jun 1, 2022 Collaborator

YerePhy Jun 1, 2022 Author

YerePhy Jun 2, 2022 Author

Jasha10 Jun 3, 2022 Collaborator

YerePhy Jun 16, 2022 Author

jieru-hu Jun 17, 2022

YerePhy Jun 27, 2022 Author

Jasha10 Jun 27, 2022 Collaborator

YerePhy Jun 28, 2022 Author

Jasha10 Jun 28, 2022 Collaborator

YerePhy Jun 28, 2022 Author

zongshun21 Apr 3, 2023

YerePhy
May 27, 2022

Replies: 3 comments 13 replies

Jasha10
May 31, 2022
Collaborator

Jasha10 Jun 1, 2022
Collaborator

YerePhy Jun 1, 2022
Author

YerePhy Jun 2, 2022
Author

Jasha10 Jun 3, 2022
Collaborator

YerePhy Jun 16, 2022
Author

jieru-hu
Jun 17, 2022

YerePhy Jun 27, 2022
Author

Jasha10 Jun 27, 2022
Collaborator

YerePhy Jun 28, 2022
Author

Jasha10 Jun 28, 2022
Collaborator

YerePhy Jun 28, 2022
Author

zongshun21
Apr 3, 2023