How to configure a pipeline (composing a list using yaml)

[dagap]

I am working on a time series application and I have various preprocessing algorithms which may or may not be applied (based on the nature of the input data). So, for example, I have various sch preprocessing filters like:

configs
----> preprocessing
-----------> columnselector.yaml
-----------> columnmapper.yaml
-----------> powertransform.yaml
-----------> resampler.yaml
-----------> scaler.yaml
 etc.

Now, I want to specify a pipeline configuration where I can specify these processing steps. Buw how do I specify a list of these preprocessing steps? I tried something like:

__target__: ts.pipeline
defaults:
   preprocessing: [columnselector, columnmapper, powertransform, resampler, scaler]

However, this does not work as expected and merges everything into a single flat dictionary. In fact, config parameters with the same name get overwritten by the values of the last object in the list. What I would like to do is create a list of these objects and then be able to override them i.e. add and remove them and additionally modify/override each of their configuration parameters as well i.e. say something like:
`preprocessing.columnselector.columns="[ds, x, y]"

Been struggling with whether this is the best way to organise this configuration or if there is a more maintainable way for such a pipeline like use case? These pipelines tend to be very dynamic based on inopput data and it would be good to have some pattern for handling these.

[Jasha10]

The defaults list is good for composing dictionaries (which get merged), but it is not so good for composing lists. This is a known pain point in Hydra which will hopefully be improved in the future.
See this comment on #1939 for a suggested workaround.