Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytest discovery in VS code #1457

Open
sjraaijmakers opened this issue Nov 19, 2024 · 2 comments
Open

Pytest discovery in VS code #1457

sjraaijmakers opened this issue Nov 19, 2024 · 2 comments

Comments

@sjraaijmakers
Copy link

sjraaijmakers commented Nov 19, 2024

I am using VS code with databricks-connect v2.4.8, and I wrote some pytests. In conftest.py I created a fixture:

@pytest.fixture(scope="session")
def spark():
    spark = DatabricksSession.builder.getOrCreate()
    yield spark
    spark.stop()

Running the pytest command from within the terminal in the venv works just fine. However, when using the VS code test plugin to discover the tests I get this error:

.venv\lib\site-packages\databricks\sdk\config.py:130: in __init__
    raise ValueError(message) from e
E   ValueError: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=MY_HOST, auth_type=databricks-cli, cluster_id=MY_CLUSTER_ID

Anyone any idea on how to solve this?

@ilia-db
Copy link
Contributor

ilia-db commented Nov 19, 2024

Do you have DATABRICKS_PROFILE env var set in the terminal? This error can happen when we can't identify what profile to use for the auth. By default we look into ~/.databrickscfg file for the DEFAULT profile, and if we can't find it, then auth fails. You don't need to have DEFAULT profile, and instead specify the name directly in the code like this:

spark = DatabricksSession.builder.profile("<profile-name>").getOrCreate()

@sjraaijmakers
Copy link
Author

sjraaijmakers commented Nov 20, 2024

I do have a ~/.databrickscfg with a profile named DEFAULT:

[DEFAULT]
host=MY_HOST
auth_type=databricks-cli
cluster_id=MY_CLUSTER_ID

In the earlier provided error it does seem like Databricks can find this profile (showing MY_HOST and MY_CLUSTER_ID in the provided error). I also tried it with explicitly setting the profile to DEFAULT:

spark = DatabricksSession.builder.profile("DEFAULT").getOrCreate()

without success unfortunately.

I have the same error - terminal pytest works, while vscode plugin doesn't - when running the main_test.py file that comes with the stub when creating a new databricks project using the databricks-connect plugin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants