Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature offset3 #101

Merged
merged 16 commits into from
Nov 6, 2023
Merged

Feature offset3 #101

merged 16 commits into from
Nov 6, 2023

Conversation

cjthomas730
Copy link
Contributor

Updates to feature offsetting

Copy link

github-actions bot commented Nov 6, 2023

🍹 preview on cerulean-cloud-images/test

Pulumi report
Previewing update (test):
@ previewing update.........

@ previewing update...........................
pulumi:pulumi:Stack cerulean-cloud-images-test running 
docker:image:Image cerulean-cloud-images-test-cloud-run-offset-tile-image  
docker:image:Image cerulean-cloud-images-test-cloud-run-orchestrator-image  
docker:image:Image cerulean-cloud-images-test-cloud-run-tipg-image  
@ previewing update.....
gcp:container:Registry cerulean-cloud-images-test-registry  [diff: ~provider]
@ previewing update....
docker:image:Image cerulean-cloud-images-test-cloud-run-offset-tile-image  Building image '../'...
docker:image:Image cerulean-cloud-images-test-cloud-run-orchestrator-image  Building image '../'...
docker:image:Image cerulean-cloud-images-test-cloud-run-tipg-image  Building image '../'...
@ previewing update....................................................................................
docker:image:Image cerulean-cloud-images-test-cloud-run-tipg-image  sha256:cff6cf1cded99499b9be0cd4b84452d31e2aaa8ddc3c0b79a80c8bb94c7f33c9
docker:image:Image cerulean-cloud-images-test-cloud-run-tipg-image  sha256:cff6cf1cded99499b9be0cd4b84452d31e2aaa8ddc3c0b79a80c8bb94c7f33c9
docker:image:Image cerulean-cloud-images-test-cloud-run-tipg-image  Image build succeeded.
@ previewing update............................
docker:image:Image cerulean-cloud-images-test-cloud-run-orchestrator-image  sha256:b8222b49269b44055c5ae07135611dfdbdb7cefbdd829344369df7511f0bedb2
docker:image:Image cerulean-cloud-images-test-cloud-run-orchestrator-image  sha256:b8222b49269b44055c5ae07135611dfdbdb7cefbdd829344369df7511f0bedb2
docker:image:Image cerulean-cloud-images-test-cloud-run-orchestrator-image  Image build succeeded.
@ previewing update................
docker:image:Image cerulean-cloud-images-test-cloud-run-offset-tile-image  sha256:5d0e2b84d56bcdb907055d4fe27baa84d1f2bb721e53b25d8f5902d301d49551
docker:image:Image cerulean-cloud-images-test-cloud-run-offset-tile-image  sha256:5d0e2b84d56bcdb907055d4fe27baa84d1f2bb721e53b25d8f5902d301d49551
docker:image:Image cerulean-cloud-images-test-cloud-run-offset-tile-image  Image build succeeded.
pulumi:pulumi:Stack cerulean-cloud-images-test  
Resources:
5 unchanged

Copy link
Collaborator

@jonaraphael jonaraphael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! Just a sprinkling of notes. Should be ready to push with ~5 mins of correction!

@@ -44,6 +44,7 @@ def upgrade() -> None:
session = orm.Session(bind=bind)

eez = get_eez_from_url() # geojson.load(open("EEZ_and_HighSeas_20230410.json"))
eez = {"features": []} # noqa
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line needs to be deleted before Merging

@@ -36,6 +36,7 @@ def upgrade() -> None:
session = orm.Session(bind=bind)

iho = get_iho_from_url()
iho = {"features": []} # noqa
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line needs to be deleted before Merging

@@ -36,6 +36,7 @@ def upgrade() -> None:
session = orm.Session(bind=bind)

mpa = get_mpa_from_url()
mpa = {"features": []} # noqa
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line needs to be deleted before Merging

print(f"Offset 2 offset_group_bounds: {offset_2_group_bounds}")

print(
f"Original tiles are {len(base_tiles)}, {len(offset_group_bounds)}, {len(offset_2_group_bounds)}"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

offset_group_bounds and offset_2_group_bounds are the wrong variables here.
I think you want offset_tiles_bounds and offset_2_tiles_bounds

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolving this for next push

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, changes made will be reflected in the next push

@@ -407,13 +423,25 @@ async def _orchestrate(
"offset tiles",
)

offset_2_tiles_inference = await perform_inference(
offset_2_tiles_bounds,
cloud_run_inference.get_offset_tile_inference, # THIS FUNCTION NEEDS TO BE EDITTED
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment still correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, taking this out for the next push

@@ -523,7 +551,7 @@ async def _orchestrate(
async with db_client.session.begin():
orchestrator_run.success = True
orchestrator_run.inference_end_time = end_time

# XXXC >> reduce num variables to a list of featureCollection, ala the merging approach
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean up this comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do!

- offset_tile_fc: The secondary FeatureCollection to be merged with the primary.
- isolated_conf_multiplier: A multiplier for the confidence of isolated features (default is 1).
- feature_collections: A list of FeatureCollecitons to be merged, a primary and any secondary FeatureCollections
- isolated_conf_multiplier: A multiplier for the confidence of isolated features (default is 1 / len(feature_collections)).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete this line

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do!

offset_gdf = gpd.GeoDataFrame.from_features(
offset_tile_fc["features"], crs=4326
gdfs_for_processing = [
reproject_to_utm(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment about the reproject_to_utm() assumption?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do!

Copy link

github-actions bot commented Nov 6, 2023

🍹 preview on cerulean-cloud/test

Pulumi report
Previewing update (test):
@ previewing update......

@ previewing update.....
pulumi:pulumi:Stack cerulean-cloud-test running 
@ previewing update....
pulumi:providers:docker cerulean-cloud-images-test-gcr  
@ previewing update....
gcp:storage:Bucket cerulean-cloud-test-bucket-cloud-function-ais  [diff: ~provider]
gcp:cloudtasks:Queue cerulean-cloud-test-queue-cloud-tasks-ais-analysis  [diff: ~provider]
gcp:serviceAccount:Account cerulean-cloud-test-cloud-function-ais  [diff: ~provider]
gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  [diff: ~provider]
@ previewing update....
-- gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-ais delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-ais replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-ais create replacement [diff: ~detectMd5hash,name,source]
gcp:projects:IAMMember cerulean-cloud-test-cloud-function-ais-iam  [diff: ~provider]
gcp:sql:Database cerulean-cloud-test-database  [diff: ~provider]
gcp:sql:User cerulean-cloud-test-database-users  [diff: ~provider]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cloud-function-ais update [diff: ~secretEnvironmentVariables,sourceArchiveObject]
gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cloud-function-ais-invoker  [diff: ~provider]
@ previewing update....
pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
@ previewing update.............................................................................................................................................................................................
pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
@ previewing update......
docker:index:RemoteImage cerulean-cloud-images-test-remote-offset  
docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator  
aws:iam:Policy cerulean-cloud-test-lambda-titiler-policy  
docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg  
aws:s3:Bucket cerulean-cloud-test-titiler-lambda-archive  
aws:iam:Role cerulean-cloud-test-lambda-titiler-role  
aws:apigatewayv2:Api cerulean-cloud-test-lambda-titiler-api  
gcp:serviceAccount:Account cerulean-cloud-test-cloud-run-orchestrator  [diff: ~provider]
gcp:storage:Bucket cerulean-cloud-test-bucket-cloud-function-sr  [diff: ~provider]
gcp:cloudtasks:Queue cerulean-cloud-test-queue-cloud-run-orchestrator  [diff: ~provider]
gcp:serviceAccount:Account cerulean-cloud-test-cloud-function-sr  [diff: ~provider]
aws:iam:Role cerulean-cloud-test-lambda-sentinel1-iam  
~  gcp:cloudrun:Service cerulean-cloud-test-cloud-run-offset-tiles update [diff: ~metadata]
pulumi:pulumi:Stack cerulean-cloud-test running warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
pulumi:pulumi:Stack cerulean-cloud-test running warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
pulumi:pulumi:Stack cerulean-cloud-test running warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
~  gcp:cloudrun:Service cerulean-cloud-test-cloud-run-tipg update [diff: ~metadata]
aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment2  
aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-sr delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-sr replace [diff: ~name]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-sr create replacement [diff: ~name]
gcp:projects:IAMMember cerulean-cloud-test-cloud-run-orchestrator-cloudTasksEnqueuer  [diff: ~provider]
gcp:projects:IAMMember cerulean-cloud-test-cloud-run-orchestrator-cloudSqlClient  [diff: ~provider]
@ previewing update....
gcp:projects:IAMMember cerulean-cloud-test-cloud-function-sr-iam  [diff: ~provider]
-- gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-historical-run delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-historical-run replace [diff: ~name]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cloud-function-historical-run create replacement [diff: ~name]
gcp:cloudrun:IamPolicy cerulean-cloud-test-cloud-run-noauth-iam-policy-offset  [diff: ~provider]
~  gcp:cloudrun:Service cerulean-cloud-test-cloud-run-orchestrator update [diff: ~metadata,template]
gcp:cloudrun:IamPolicy cerulean-cloud-test-cloud-run-noauth-iam-policy-tipg  [diff: ~provider]
gcp:cloudrun:IamPolicy cerulean-cloud-test-cloud-run-noauth-iam-policy-orchestrator  [diff: ~provider]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cloud-function-historical-run update [diff: ~sourceArchiveObject]
~  gcp:cloudfunctions:Function cerulean-cloud-test-cloud-function-sr update [diff: ~sourceArchiveObject]
gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cloud-function-historical-run-invoker  [diff: ~provider]
aws:lambda:Function cerulean-cloud-test-lambda-sentinel1-subscription  
gcp:cloudfunctions:FunctionIamMember cerulean-cloud-test-cloud-function-sr-invoker  [diff: ~provider]
aws:sns:TopicSubscription cerulean-cloud-test-sentinel1-subscription  
aws:lambda:Permission cerulean-cloud-test-lambda-sentinel1-permission  
~  aws:s3:BucketObject cerulean-cloud-test-titiler-lambda-archive update [diff: ~source]
~  aws:lambda:Function cerulean-cloud-test-lambda-titiler-sentinel update [diff: ~sourceCodeHash]
aws:apigatewayv2:Integration cerulean-cloud-test-lambda-titiler-integration  
aws:lambda:Permission cerulean-cloud-test-lambda-titiler-permission  
aws:apigatewayv2:Route cerulean-cloud-test-lambda-titiler-route  
aws:apigatewayv2:Stage cerulean-cloud-test-lambda-titiler-stage  
pulumi:pulumi:Stack cerulean-cloud-test running Creating lambda package [running in Docker]...
pulumi:pulumi:Stack cerulean-cloud-test running Building docker image...
pulumi:pulumi:Stack cerulean-cloud-test running Copying package.zip ...
pulumi:pulumi:Stack cerulean-cloud-test running Copied package package.zip ...
pulumi:pulumi:Stack cerulean-cloud-test  9 warnings; 4 messages
Diagnostics:
pulumi:pulumi:Stack (cerulean-cloud-test):
warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: serving_state is deprecated: `serving_state` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: env_froms is deprecated: `env_from` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.
warning: working_dir is deprecated: `working_dir` is deprecated and will be removed in a future major release. This field is not supported by the Cloud Run API.

Creating lambda package [running in Docker]...
Building docker image...
Copying package.zip ...
Copied package package.zip ...

Resources:
~ 8 to update
+-3 to replace
11 changes. 39 unchanged

@cjthomas730 cjthomas730 merged commit bb33c6d into main Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants