Merge branch '1.10.latest' of https://github.com/databricks/dbt-datab…

…ricks into bump_python_sdk_version

CHANGELOG.md

@@ -2,17 +2,32 @@
 
 ### Features
 
+- Support databricks OAuth M2M auth type. Updated OAuth readme doc with instructions.([827](https://github.com/databricks/dbt-databricks/pull/827))
+
 - Introduced use_materialization_v2 flag for gating materialization revamps. ([844](https://github.com/databricks/dbt-databricks/pull/844))
 
 ### Under the Hood
 
+- Update pinned python SDK version from 0.17.0 to 0.41.0. ([827](https://github.com/databricks/dbt-databricks/pull/827))
+
 - Implement new constraint logic for use_materialization_v2 flag ([846](https://github.com/databricks/dbt-databricks/pull/846/files)), ([876](https://github.com/databricks/dbt-databricks/pull/876))
 
-## dbt-databricks 1.9.3 (TBD)
+## dbt-databricks 1.9.5 (TBD)
 
 ### Under the Hood
 
 - Collapsing to a single connection manager (since the old one no longer works) ([910](https://github.com/databricks/dbt-databricks/pull/910))
+- Use POSIX standard when creating location for the tables ([919](https://github.com/databricks/dbt-databricks/pull/919))
+
+## dbt-databricks 1.9.4 (Jan 30, 2024)
+
+### Under the Hood
+
+- Pinned the python sql connector to 3.6.0 as a temporary measure while we investigate failure to wait for cluster start
+
+## dbt-databricks 1.9.3
+
+Yanked due to being published with the incorrect bits
 
 ## dbt-databricks 1.9.2 (Jan 21, 2024)
 

dbt/adapters/databricks/impl.py

@@ -1,4 +1,4 @@
-import os
+import posixpath
 import re
 from abc import ABC, abstractmethod
 from collections import defaultdict
@@ -226,9 +226,9 @@ def compute_external_path(
             raise DbtConfigError("location_root is required for external tables.")
         include_full_name_in_path = config.get("include_full_name_in_path", False)
         if include_full_name_in_path:
-            path = os.path.join(location_root, database, schema, identifier)
+            path = posixpath.join(location_root, database, schema, identifier)
         else:
-            path = os.path.join(location_root, identifier)
+            path = posixpath.join(location_root, identifier)
         if is_incremental:
             path = path + "_tmp"
         return path

docs/oauth.md

@@ -2,11 +2,11 @@
 
 This feature is in [Public Preview](https://docs.databricks.com/release-notes/release-types.html).
 
-Databricks DBT adapter now supports authentication via OAuth in AWS and Azure. This is a much safer method as it enables you to generate short-lived (one hour) OAuth access tokens, which eliminates the risk of accidentally exposing longer-lived tokens such as Databricks personal access tokens through version control checkins or other means. OAuth also enables better server-side session invalidation and scoping.
+## User to Machine(U2M):
 
-Once an admin correctly configured OAuth in Databricks, you can simply add the config `auth_type` and set it to `oauth`. Config `token` is no longer necessary.
+Databricks DBT adapter now supports authentication via OAuth U2M flow in all clouds. This is a much safer method as it enables you to generate short-lived (one hour) OAuth access tokens, which eliminates the risk of accidentally exposing longer-lived tokens such as Databricks personal access tokens through version control checkins or other means. OAuth also enables better server-side session invalidation and scoping.
 
-For Azure, you admin needs to create a Public AD application for dbt and provide you with its client_id.
+Simply add the config `auth_type` and set it to `oauth`. Config `token` is no longer necessary.
 
 ```YAML
 jaffle_shop:
@@ -17,16 +17,23 @@ jaffle_shop:
       catalog: <UC catalog name>
       schema: <schema name>
       auth_type: oauth  # new
-      client_id: <azure application ID> # only necessary for Azure
       type: databricks
   target: dev
 ```
 
-## Troubleshooting
+### Troubleshooting
 
 DBT expects the OAuth application to have the "All APIs" scope and redirect URL `http://localhost:8020` by default.
 
-If the OAuth application has only been configured with SQL access scopes or a custom redirect URL, you may need to update your profile accordingly:
+The default oauth app for dbt-databricks is auto-enabled in every account with expected settings, you can find it in [Account Console](https://accounts.cloud.databricks.com) > [Settings](https://accounts.cloud.databricks.com/settings) > [App Connections](https://accounts.cloud.databricks.com/settings/app-integrations) > dbt adapter for Databricks. If you cannot find it you may have disabled dbt in your account, please refer to this [guide](https://docs.databricks.com/en/integrations/enable-disable-oauth.html) to re-enable dbt as oauth app.
+
+If you encounter any issues, please refer to the [OAuth user-to-machine (U2M) authentication guide](https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.html).
+
+## Machine to Machine(M2M):
+
+Databricks DBT adapter also supports authenticate via OAuth M2M flow in all clouds.
+Simply add the config `auth_type` and set it to `oauth`. Follow this [guide](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html) to create a databricks service principal and also an OAuth secret. 
+Set `client_id` to your databricks service principal id and `client_secret` to your OAuth secret for the service principal.
 
 ```YAML
 jaffle_shop:
@@ -37,15 +44,26 @@ jaffle_shop:
       catalog: <UC catalog name>
       schema: <schema name>
       auth_type: oauth  # new
-      client_id: <azure application ID> # only necessary for Azure
-      oauth_redirect_url: https://example.com
-      oauth_scopes:
-        - sql
-        - offline_access
+      client_id: <service principal id>
+      client_secret: <oauth secret>
       type: databricks
   target: dev
 ```
 
-You can find these settings in [Account Console](https://accounts.cloud.databricks.com) > [Settings](https://accounts.cloud.databricks.com/settings) > [App Connections](https://accounts.cloud.databricks.com/settings/app-integrations) > dbt adapter for Databricks
+### Azure Service Principal
+If you are on Azure Databricks and want to use Azure Service Principal, just set `azure_client_id` to your Azure Client Id and `azure_client_secret` to your Azure Client Secret.
 
-If you encounter any issues, please refer to the [OAuth user-to-machine (U2M) authentication guide](https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.html).
+```YAML
+jaffle_shop:
+  outputs:
+    dev:
+      host: <databricks host name>
+      http_path: <http path for warehouse or cluster>
+      catalog: <UC catalog name>
+      schema: <schema name>
+      auth_type: oauth  # new
+      azure_client_id: <azure service principal id>
+      azure_client_secret: <azure service principal secret>
+      type: databricks
+  target: dev
+```

pyproject.toml

@@ -23,11 +23,11 @@ classifiers = [
 ]
 dependencies = [
     "databricks-sdk==0.41.0",
-    "databricks-sql-connector>=3.5.0, <4.0.0",
+    "databricks-sql-connector>=3.5.0, <3.7.0",
     "dbt-adapters>=1.9.0, <2.0",
     "dbt-common>=1.12.0, <2.0",
-    "dbt-core>=1.8.7, <2.0",
-    "dbt-spark>=1.8.0, <2.0",
+    "dbt-core>=1.9.0, <2.0",
+    "dbt-spark>=1.9.0, <2.0",
     "keyring>=23.13.0",
     "pydantic>=1.10.0",
 ]