Configdir render command in the next gen operators.

[4383]

Context

I open this new issue to discuss with you and track this topic in a centralized way.
Our goal is to propose a configdir render command in the next gen operators, based on oslo.config and see if this something feasible and how to implement that with k8s operators.

Description

Right now TripleO exposes hundreds of individual config parameters via the heat templates, so users configure their openstack services in the templates.

We'd prefer to not have to do all of that work to expose those config parameters, instead, users should provide config snippets, we'll write them into the configdir for each service.

Our concern is that users will get confused about why a service is behaving in a particular way, because there isn't an easy way for them to notice "oh I put that config parameter in two snippets with two different values".

So we'd like them to be able to tell the openshift operator for nova/neutron/glance/whatever "give me the fully merged config for this service" and it spits out that merged dictionary.

Observations

During my research I seen that many service operators already implement a kind of tooling to generate merged config files
Example with the ironic operator: openstack-k8s-operators/ironic-operator@54e6ac0

Indeed, if I'm right, the merge_config_dir function seems to implement more or less the same thing as the idea proposed here. This function, seems to take a dir and all the config files in this dir and seems to generate a single one config file with everything merge in it, exact?

Questions

Does that proposition of feature make sense to you?

What are the current options available to do more or less the same thing as proposed by this feature?

Where do we should store this kind of feature?

As with the merge_config_dir script do we should store a kind of entrypoint based on oslo.config on each operator or is it possible to write a new operator with this kind of common entrypoint that can act on shared directories between operators?

Let me know if you need more details.

Thanks in advance for your replies.

[4383]

@dprince, @stuggi: I addng you here to get some feedbacks about this proposal.

[gibizer]

The crudini as the tool to merge the config is problematic as it does not support the oslo.config file format it only support the original ini format. E.g. MultiOpt oslo.config fields are not handled properly by crudini. (tagging @SeanMooney here for the details)

Do you want to use the merged config only to understand the overall service config or also you want that the merged config is used as the only config file for the service itself?

[SeanMooney]

personally im against trying to do any merging of the config dynamically in pods.
if we were to implement a go function to merge the rendered go templates taking into account the speicic oslo.config format extension to the ini formant and its support for precedence then that could be acceptable.

we have started to remove all usage of the init_contianer based merging with crudini form the nova operator as it cannot be used due to our usage of multiOpt string in our configs.

there is some discussion related to this in the crudini issue tracker.
pixelb/crudini#10 (comment)

we discussed this a few times in the past but to clarify.
openstack does not use ini files for config.

it ues file in oslo.config .conf format.

in ini format

[section]
key=val1
key=val2

the value of key when parsed in val2
i.e. CONF.section.key == val2

in conf format if the type of keys was declared as MultiStrOpt

then when parsed it would CONF.section.key == ["val1", "val2"]

when i spoke to @odyssey4me about this in the past they created

https://opendev.org/openstack/ansible-config_template/commit/17330e7558dc234bf1e485d0c515167bcaf10d40

to document some of the features of ansible-config_template and how to model this.

ansible-config_template uses yaml unorderd hash sets to model MultiOpt parmaters.
technically that is lossy as MultiOpts filed are actually the ordered
but the python yaml lib and or ansible-config_template seam to preserve the order.

Right now i would consider providing a single merged config to be something we should not
do based on our past discussions. At least not in the sort term.

in the long term we could do this by implementing a oslo config parser/generator.
i looked into this to some degree before we decide not to take this path.
we can get a machine parsable scheme from Oslo. from that we could in principle generate
a type aware parser and config generator dynamically.

we could also just implement support for the .conf format without using a schema. I briefly looked at this but due to time constraints decide it was not something we could do with our current development capacity based on our internal deadlines.

the current design used for the nova operator which i had hopped would be adopted in all other operators was to rely on oslo.configs support for configuration directories instead.
https://docs.openstack.org/oslo.config/latest/configuration/options.html

When loading values from the sources defined by the following options, the precedence is as follows:

    Command Line
    Environment Variables
    Config Files from --config-dir[1](https://opendev.org/openstack/oslo.config/src/branch/master/doc/source/configuration/options.rst#fn1)
    Config Files from --config-file
    Pluggable Config Sources

If a value is specified in multiple locations, the location used will be the one higher in the list. For example, if a value is specified both on the command line and in an environment variable, the value from the command line will be the one returned.

oslo.config

    Files in a config dir are parsed in alphabetical order. Later files take precedence over earlier ones.[↩︎](https://opendev.org/openstack/oslo.config/src/branch/master/doc/source/configuration/options.rst#fnref1)

many project use multiopt so any use of config merging must fully support that
https://codesearch.opendev.org/?q=MultiStrOpt&i=nope&literal=nope&files=&excludeFiles=&repos=

[stuggi]

Sorry for the late reply here, I was out sick for most of last week.

Frist of all I think we should move this into jira where all the stuff lives. We don't use github issues.

The crudini merge is a relict from the cnosp poc we did 2 years ago or so. It was done because, at least at that time, not all services did support config dir. I am not sure if this is still the case, or all of them now support it.

Right now I think most (all?) current implemented service operators use config dir with the ordering Sean mentioned and no longer use the merge func. crudini is probably still be used in operators to set passwords passed in from the secret as env var. I think we should change this and want to move all config rendering to happen fully in go.

Whatever way we go I think it should/must be consistent throughout the operators. We should not have to check in which way an operator implemented config handling when debugging an issue. There might be exceptions if needed for non OpenStack services (galera, memcached, rabbitmq, redis).

[4383]

Hello,

Thanks everybody for your feedbacks.

@stuggi: Ok to move this to jira, however, I'm not sure in which project we should host the conversation. Please, can you indicate to me which jira project I should use to track this case. Here I used openstack-operator because I was thinking it was more generalist than the other github repos of this organization, kinda like the feature proposed here that is not specific to a service.

Also, I don't know your preferred workflow, are you used to propose blue prints or specifications or something like that before starting implementing something?

@SeanMooney: At first glance, the go command seems a good starting point and the oslo.config generator could be a long run target.

Do you want to use the merged config only to understand the overall service config or also you want that the merged config is used as the only config file for the service itself?

@gibizer: I think our goal, for now, is to offer a simple way for users to understand the overall service config.

[4383]

Let me know if the Jira ticket I created is well configured (right components, etc..) https://issues.redhat.com/browse/OSP-22500

[4383]

Let's close this PR as this topic is already tracked into jira, see my previous comment.