Default grouping for dbt sources

[fredonia88]

I upgraded to dagster==1.7.3 and dagster-dbt==0.23.3 and am now seeing my dbt sources grouped under the default asset group in the UI. The version I was on previously, dagster==1.5.13 and dagster-dbt==0.21.13, did not show my sources in the asset group section. I've tried adding necessary meta tags to the nodes themselves in the sources.yml file and to the dbt_project.yml file to group them, but its not working. How do I group my dbt sources so they aren't grouped under default? I'm using dbt-core==1.7.13. The dagster meta key does group my models correctly, but not my sources.

Here's a snippet of my dbt_project.yml file:

sources:
  some_dbt_project:
    +meta:
      dagster:
        group: sources

Here's a snippet a sources.yml file:

sources:
  - name: some_source
    description: Some source data.
    database: some_db
    schema: some_schema
    tables:
      - name: some_table_name
        config:
          meta:
            dagster:
              group: sources

After the above changes, sources still group under default:

[rexledesma]

I've tried adding necessary meta tags to the nodes themselves in the sources.yml file and to the dbt_project.yml file to group them, but its not working. How do I group my dbt sources so they aren't grouped under default?

A previous discussion answers this question. #16470 (comment)

For example, to automatically create SourceAsset's for all of your dbt sources, you could do something like:

def build_dbt_sources(manifest: Mapping[str, Any], dagster_dbt_translator: DagsterDbtTranslator) -> Sequence[SourceAsset]:
    return [
        SourceAsset(
             key=dagster_dbt_translator.get_asset_key(dbt_resource_props),
             group_name=dagster_dbt_translator.get_group_name(dbt_resource_props)
        )
        for dbt_resource_props in manifest["sources"].values()
    ]

[cmpadden]

And if you would like an example of defining upstream source assets, you can find an example here:

#21049 (comment)

This is for the dlt integration, but the same could be done for dbt.

[fredonia88]

Thanks @rexledesma and @cmpadden! That worked. Do note, the arg for SourceAsset changed from asset_key to key (at least that's what I'm seeing in dagster 1.7.3).

[rexledesma]

Updated the example to use key!

[Taytay]

This is a helpful code snippet!
It's worth noting that this will define source assets for ALL sources in your dbt manifest, whereas the default behavior will only do so for sources that are depended upon by downstream models.

This can be worked dealt with, but in my case, I had a lot of sources that were not eventually used by the models I was selecting for dagster, so this code caused a lot of new SourceAsset boxes to appear, but it was a surprise.

(It was also disappointing/surprising that my Translator didn't have "get_group_name" called on it for sources, since it does have get_asset_key_name called on sources...)