Move terrain tiles to DSL Amazon storage

[almccon]

Currently the hillshade layer (that we use for Forced Migration, Overland Trails, and Canals) is broken because it's only served on HTTP, and can't be easily switched to HTTPS. We should think about just harvesting all those tiles and stashing them on an s3 bucket anyway, and just serving them from there. Since they don't go very deep into zoom, it shouldn't be a huge amount of space.

DSL is already hosting lots of tiles on s3 for mapping inequality, so adding the terrain tiles there shouldn't be a problem.

[jmadron]

What would be the best way to transfer these over? Can we do a straight transfer between buckets?

[almccon]

I did some more digging (and refreshing my memory).

Currently we're serving the hillshade tiles from URLs like this:
http://{s}.sm.mapstack.stamen.com/openterrain_2163/{z}/{x}/{y}.png ..or, a live one here: http://c.sm.mapstack.stamen.com/openterrain_2163/7/60/65.png

The reason we're serving them from mapstack is just so I could provide a stable public URL that's really just a proxy for an EC2 server running Tessera that actually serves the tiles.

Here's the direct url to that EC2 server, which would change any time the server needs to reboot: http://ec2-52-91-120-71.compute-1.amazonaws.com/richmond-terrain/#5/-13.475/4.395

The code running on that EC2 server is this: https://github.com/stamen/openterrain-map

Here's the tilemill style: https://github.com/stamen/openterrain-map/blob/master/richmond-terrain.yml

And if you look in the tilemill style, it isn't doing much, just referring to some VRTs which in turn refer to the tiles on S3.

Here's what the vrt looks like: https://github.com/stamen/openterrain-map/blob/master/SRTM-hillshade-2163-z7-local.vrt

So those are all referring to images in the s3 buckets: srtm/2163hillshade/z7/0/10.tiff
And those are all inside the data.stamen.com bucket.

HOWEVER! Those are the tiled source images (GeoTIFFs) that the openterrain code uses, but those are not the same tiles (or the same tile numbers) as normal 256x256 web map tiles. So simply grabbing those source tiles isn't very helpful to us.

So I think instead what we need to do is write a script to repeatedly hit the hillshade server and download all the relevant tiles we want in bulk.

OR: we copy over those source GeoTIFFs to the DSL's s3 bucket and the DSL could try to run the EC2 tile server themselves. The main advantage to this approach is that you only need to have the source files for z5, 6, and 7, and then the render generates the other zooms on the fly.

[almccon]

No idea if I did this right, but here's a folder of the scraped hillshade tiles. Should be easy to convert to mbtiles for serving, or just drop on s3.
hillshade2163.zip