Summary to get Colnect working as November 2023

[dfernandezm]

Task

• Complete a working crawler with 2-3 countries
• Validate that indexing works with 2-3 countries, can be searched
• Fix the alias setup after indexing (should be removed from previous, add to new index)
• Check that all the countries can be crawled (count of banknotes matches site)
• Build Dockerfile for crawler-v2
• Move infra from crawler-v1 to v2 / extract
• Build CI for crawler-v2
• Build CI for the 3 Cloud Functions
• Collect TODOs from the above
  • Should document how to trigger all manually
  • Should fix all automation as-is
  • Get the PubSub topics and subscription deleted and recreated
  • Get the resizer Cloud Functions deleted and recreated
  • Get the GKE cluster deleted and recreated
  • Get the Indexer Functions deleted and recreated
  • Get the VPC serverless connector deleted and recreated
  • Get the Cloud Scheduler deleted and recreated
  • Remove the Terraform and use Pulumi

Description of the task to complete

Acceptance criteria

Related Tasks

• What needs to be done to complete the story as separate issues