Hosting solutions for Neume

[il3ven]

I am starting this thread to discuss about the best way to host neume and allocating funds for it. I think we all agree that the Gitcoin fund is the best for it.

What do we need?

• A VM to run the neume crawler with persisted storage
• Ethereum and Polygon full node
• IPFS Gateway
• Arweave Gateway

A VM to run the neume crawler with persisted storage

HiFi labs currently uses fly.io which is okay in my opinion. An instance with 2 CPU cores and 4GB RAM should cost us around $30.

I should mention if we go with a VPS like DigitalOcean then managing persistent storage is simple as it is just a linux box. The cost is the same.

Ethereum and Polygon full node

According to my calculations, we require around 100,000 requests to get the crawler upto latest block. Infura provides 25,000 daily requests in their free plan. Their next plan is for $50 which allows 100,000 requests per day. An alternative to Infura is Ankr RPC. Unlike Infura they have a pay as you use pricing system. For 100,000 requests Ankr will charge us $2. My vote is with Ankr.

IPFS Gateway

Infura's free plan provides us with 5GB of bandwidth. I don't know if 5GB is per month or for complete lifetime. Even if it is for the complete lifetime our costs shouldn't be too much. It is a dollar for 12GB of bandwidth. We shouldn't reach the 12GB limit as we only have to download metadata.

Arweave Gateway

There is no service like Infura for Arweave. We can use the public gateway which is fast enough or host our own node.

Final Estimates

$30 (fly.io) + $2 (Full node) + $1 (IPFS) = $33 per month

@reimertz Since you manage the current HiFi instance can you check if my estimates are around the same ballpark or not?

CC: @neatonk @djfnd

[neatonk]

Thanks for this very useful rationale and breakdown of costs @il3ven.

RPC: I like Ankr and agree with your reasoning.

IPFS: It's easy to run an ipfs node, so it might be worth considering that option. Also, if you dump all the CIDs from the current database, you can pin them on your node to preload them. IPFS is really slow, so this could speed up the initial run a lot.

Arweave: agree with using the public gateway. There may be a way to run a "light" node with a specific transactions/data-items, but I'm not sure and there would be little advantage since the gateways are free and seem to be pretty fast.

[il3ven]

Ok, I'll try using a self hosted IPFS node first.

[djfnd]

Great work @il3ven , thanks for pulling all this together.

Let's ratify a decision in the Open Office today and work out how we want to put it into operation.

[il3ven]

As per the recent discussions, you need to sign up for https://www.digitalocean.com/ and https://www.ankr.com/rpc/.

You can setup billing alert for digitalocean so as to not get a huge bill at the end of the month. You can also give me a role of member. This should prevent me from creating new servers.

[reimertz]

Thanks for putting this together @il3ven. Excited to see this and happy to support however I can.

Some feedback / notes;

fly.io
these seems numbers seem correct I think once could go even lower in memory if needed.

Ethereum gateway
Edit: Removed text since Cloudflare does not offer Polygon except for enterprise accounts.
HIFI does have a enteprise Cloudflare account, but we'd have to discuss if that is a solution we'd want. (I think we would be fine with it!)

One suggestion would be to continue to use this service rather than setting up new endpoints.

IPFS Gateway

Cloudflare has good offering here too, and something I think is already setup for neume. I'd have to doublecheck though.

Free
50 GB up/down included per month
$0.25 / GB

Premium ($5/month)
100 GB up/down included per month
$0.10 / GB

Arweave Gateway

What we have done on HIFI is to create a edge worker that proxy requests to arweave, cache it in the cloudflare for 10 years. It has worked really good and has resulted in very low latency for arweave-related content.

Let me know what you all think.

cc @djfnd

[djfnd]

Okey smokey!

I guess an alternative route could be that we continue as we are and just send the gitcoin money that we allocated for this ($400) to HIFI's wallet instead. Would that be an option @reimertz ?

[reimertz]

If @il3ven sees this as a solution that works as of now, I think that would be fine yes. We can discuss this today.

[il3ven]

Yes, this is fine by me. I won't be able to join today's discussion as I am travelling.