Improve LayergroupID AKA cache hit ratio

[rochoa]

Rendering a map in CARTO is a two-phase operation:

1. The client defines a set of layers with SQL queries and CartoCSS styles and sends them to the server. The server returns a set of URLs for the tiles.
2. The client uses the set of URLs to start requesting the map tiles and displaying them on the map.

The set of URLs is unique for each different set of layers. Creating the very same map –the same set of layers– results in the very same set of URLs. That's cache friendly; if nothing changes, the client will reuse most of the resources: from browser cache to CDN cached requests.

That set of layers is called, in CARTO terms, a MapConfig. But a MapConfig is, nowadays, way more than that. When we started to work on the first iteration of widgets, and later on the analyses, for Builder, we introduced those two new resources on the MapConfig. Making possible to define, along with the set of layers, a set of analyses and a set of widgets.

In contrast, the cache-friendliness of the URLs went away: now, the URLs change for every change in the MapConfig. This should not be this way: only specific changes must affect the set of URLs for each resource.

[dgaubert]

Nice summary ;) This made me think about how to improve the cache hit ratio, I have a couple of questions here:

What are the changes in MapConfig that must affect the layergroupid?

So far, I've thought that any change in MapConfig will affect the map in some way. if you change a param of an analysis the source will change so the map must change. The same for dataviews and filters. The only thing that won't change a map is to add a dataview w/o applying filters. Right?

Is there any other part of the MapConfig that won't affect the map itself?

Do we have a bare idea of how this change will affect existing maps?

If we change the way that we calculate the layergroupid, this is going to affect existing maps. isn't it?

[rochoa]

What are the changes in MapConfig that must affect the layergroupid?

This is part of the issue, understanding what affects the Layergroupid. As you are saying, adding dataviews doesn't affect it. But you can always generalize the solution for the tiles versus other resources. For instance, you can generate the hash for the MapConfig with only the layers —or in general, everything that only is understood by Windshaft— and use that hash/id for the tiles URLs, and do the same for the rest of the resources. That way, if you have the very same layers, you will have the very same Layergroupid and the URLs for the tiles will be the same.

Do we have a bare idea of how this change will affect existing maps?

It will, but do we care? I mean, there will be some maps that before were having several layergroupids during their different incarnations (i.e., when adding different dataviews), now they will share sometimes some URLs.

Another example is CARTO.JS, where you might be adding widgets dynamically during your application lifecycle. The more we reuse URLs, the better the experience will be, but also the fewer resources we will use.

[gonzaloriestra]

ClubHouse story: https://app.clubhouse.io/cartoteam/story/83658/improve-layergroupid-aka-cache-hit-ratio