Cost optimization w3link reads

[vasco-santos]

Motivation

Lower pricing of read costs through IPFS gateways provided (w3link + nftstorage.link). With the proposed plan, not only we target reducing costs but also get faster reads 🚀

MVP Plan 🗺️

The items below can be tackled somewhat in parallel, but we order from what we can likely land sooner to later. We also include the blockers for each item, as well as what each item unlocks from a product, performance, and cost perspective.

1. ✅ Short-term optimization of gateways being raced by relying on a tiered logic

• Tier 1: ipfs.io + Tier 2: CF + Pinata
• We might be able to remove using certain gateways altogether (to revisit this optimization as we finish things further down this list)

2. 🚧 Enable CF Cache Reserve

• Exploring using Cache Reserve to cache content on request so after the first request, we have a HTTP cached copy of the content. The cost structure of Cache Reserve is more favorable for our use case (data is often asked for repeatedly, and we move from paying per GB to per request)
• Blockers:
  • Getting access to CF Cache Reserve offering in both gateway domains
  • Logging of double hash of the content to enable purging
  • Get script tool that enables quick cache purging

3. 🚧 Read from R2 content stored with web3.storage platform

• Blockers:
  • api.web3.storage CAR files written to R2
  • api.nft.storage CAR files written to R2
  • w3up CAR files copied to R2 - attach pipeline running (needs CF Queue access)
  • add dagula-gateway to the gateways race
  • freeway support + integration with edge gateway
  • update clients to return given URLs

4. fast-ptp API

Post MVP

After MVP is in place, we unblock a set of other things we can decide on:

• Sunset superhot?
• Reimagine reads rate limits
• Programmable CF Cache Reserve with pattern optimisation using Cache control headers