Support for more authentication mechanisms

[ghjklw]

Describe the feature

Thanks to the work done in #827, dbt-databricks systematically uses Databricks' databricks.sdk.core.Config to configure authentication mechanisms.

As of now, only a limited subset (PAT, OAuth U2M, Oauth M2M and Azure client/secret) of the authentication mechanisms are exposed. However the new implementation makes it really easy to extend it further based on all the options offered by Databricks' SDK for Python which follows itself Databricks unified authentication.

I am very interested in implementing it for Azure CLI and Azure Managed Identity and would love to submit a PR.

Describe alternatives you've considered

• The first alternative would be to follow the current pattern in https://github.com/databricks/dbt-databricks/blob/1.10.latest/dbt/adapters/databricks/credentials.py while trying to stick as closely as possible to Databricks unified authentication behavior/variable names.
• The second alternative would be the one I mentioned in an earlier comment to pass transparently the options to Databricks' SDK for Python in order to maximize the compatibility with all the options availables and be able to quickly deliver newer options as they're implemented. I would then make sure to not break the existing functionality by keeping the existing logic and only passing the options through to Databricks SDK if none of the existing options matched the configuration.

Who will this benefit?

Any user who wants to leverage more authentication options, and potentially increase security.
For the project I am currently working on:¨

• dbt-databricks is the last component not leveraging Azure CLI authentication in the development environment.
• It is also the only component preventing us from deactivating Databricks PAT
• In Azure, we already rely extensively on managed identity, but have to fallback to client id/secret for dbt-databricks

Such a change would therefore be a significant improvement both in terms of convenience and security.

Are you interested in contributing this feature?

Happy to implement it 😃