Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest Migration - Arcgis Online Datasets #1249

Open
2 of 27 tasks
fvankrieken opened this issue Nov 15, 2024 · 0 comments · May be fixed by #1330
Open
2 of 27 tasks

Ingest Migration - Arcgis Online Datasets #1249

fvankrieken opened this issue Nov 15, 2024 · 0 comments · May be fixed by #1330
Assignees

Comments

@fvankrieken
Copy link
Contributor

fvankrieken commented Nov 15, 2024

All datasets in the arcgis feature server weekly job should be migrated.

Glancing at code.. I haven't actually defined this as a source type yet. So we need to

  • define a source for arcgis online. It will need
    • model in dcpy/models/ingest.py for this Source, as option for both Template and Config
    • case in helper in configure.py to get filename from Source
    • case in helper in configure.py to call arcgis feature server connector api to get version
    • case in helper in extract.py to call arcgis feature server connector api to get data
    • tests for above 3. There are some library tests that can be pillaged

Then, we need to migrate 21 datasets

  • dcp_cscl_commonplace
  • dcp_cscl_complex
  • dcp_waterfront_access_map_wpaa
  • dcp_waterfront_access_map_pow
  • dcp_wrp_recognized_ecological_complexes
  • dcp_wrp_special_natural_waterfront_areas
  • nysdec_freshwater_wetlands_checkzones
  • nysdec_freshwater_wetlands
  • nysdec_lands
  • nysdec_natural_heritage_communities
  • nysdec_priority_lakes
  • nysdec_priority_shorelines
  • nysdec_priority_estuaries
  • nysdec_priority_streams
  • nysdec_tidal_wetlands
  • nysparks_historicplaces
  • nysparks_parks
  • nysshpo_historic_buildings_points
  • nysshpo_historic_buildings_polygons
  • nysshpo_archaeological_buffer_areas
  • usnps_parks
@damonmcc damonmcc moved this from New to 📬 Next in Data Engineering Nov 25, 2024
@damonmcc damonmcc moved this from 📬 Next to 🏗 In progress in Data Engineering Dec 16, 2024
@damonmcc damonmcc linked a pull request Dec 16, 2024 that will close this issue
@damonmcc damonmcc moved this from 🏗 In progress to 📬 Next in Data Engineering Dec 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 📬 Next
Development

Successfully merging a pull request may close this issue.

2 participants