add example code for s3 io manager #11
Conversation
|
Current dependencies on/for this PR:
This comment was auto-generated by Graphite. |
|
Your pull request is automatically being deployed to Dagster Cloud.
|
my_dagster_project/repository.py
Outdated
|
|
||
| from my_dagster_project import assets | ||
|
|
||
|
|
||
| @repository | ||
| def my_dagster_project(): | ||
|
|
||
| # # To take advantage of Dagster's incremental re-execution functionality (e.g. retry from failure), |
There was a problem hiding this comment.
I think talking about re-execution is a little confusing here, because users will be able to re-execute the asset if it fails. They just won't be able to define a downstream asset that depends on this one.
There was a problem hiding this comment.
Separately, why two levels of comment indentation here?
There was a problem hiding this comment.
users will be able to re-execute the asset if it fails. They just won't be able to define a downstream asset that depends on this one.
that was why we say "incremental re-execution" here. added " When you define a downstream asset that depends on another asset, ..."
Separately, why two levels of comment indentation here?
because it differentiate the comment from code and users can uncomment the entire block -- i wonder if using triple quoted comments instead would make sense.
| # # In this example, we vary folder per branch deployment (PR number) and prod. | ||
| # # Read about best practice at https://docs.dagster.io/guides/dagster/branch_deployments | ||
| # if os.getenv("DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENT"): | ||
| # s3_prefix = f"MY_S3_PREFIX/branch_{os.getenv('DAGSTER_CLOUD_PULL_REQUEST_ID')}" |
There was a problem hiding this comment.
is it typical to capitalize S3 paths? if not, I think this should be lowercase
There was a problem hiding this comment.
the intent was to let the user replace the prefix with their own values - but guess that part was unclear. updated comments
my_dagster_project/repository.py
Outdated
| # # To take advantage of Dagster's incremental re-execution functionality (e.g. retry from failure), | ||
| # # you'll need to set up an IO manager that can move the data across runs. | ||
| # # Learn more at https://docs.dagster.io/concepts/io-management/io-managers#applying-io-managers-to-assets | ||
| # # NOTE: Uncomment the example code below to use an s3_pickle_io_manager to pass data between runs. |
There was a problem hiding this comment.
A little confusing here that there's no code directly below this line.
There was a problem hiding this comment.
reformatted the comment - hope this is more clear
includes example code to set up s3 io manager and vary prefix per branch (commented by default)
brings back the second asset