port download code to use the batch endpoint

[missinglink]

this draft PR updates the OA downloader to use the batch.openaddresses.io endpoint.

a few things worth mentioning:

• the API and downloads require an authentication token
• each 'file' is now looked up against the OA API to get the latest 'run' (ie. latest version)
• downloads are now gzip compressed rather than zip
• the 'global downloads' remain unchanged, still pointing to the old domain, last time I looked there weren't equivalent files on the batch domain
• I haven't done any testing of the import process, only the download process so far

[missinglink]

This is ready for testing, there are some features I would have liked to add but will have to wait for a subsequent PR:

• do not decompress after downloading, instead write the .gz and do the decompression on import (saves disk space)
• source pattern matching, it would be ideal to add us/{ca,ny}/* to the config file and have the API figure it out

I still need to clarify a couple things with the OA team:

• rate-limiting, the code enforces "current policy is 10 request per minute", is this still relevant and will there be an issue with the shared community key?
• what support is there for the 'global downloads' (openaddr-collected-global.zip & openaddr-collected-global-sa.zip)?

[Joxit]

LGTM

👍 for decompression on import and source pattern matching for a next PR.

Maybe you can add a logger.error for all missing files even if errorsFatal is false. This allows you to know all the errors at once.

openaddresses/utils/download_filtered.js

Lines 24 to 32 in 60256bd

	const files = config.get('imports.openaddresses.files', []);
	const sources = await getSources(files);
	const validSources = sources.filter(source => source.url);
	// respect 'imports.openaddresses.missingFilesAreFatal' setting
	if (errorsFatal && (sources.length !== validSources.length)) {
	callback(sources.find(source => source.error)); // return first error
	return;
	}

[missinglink]

Yeah good idea about the logging, I'll go through and make sure everything is logged, even if it's at the debug log level, I also want to log the output of these child process shell commands if anything goes wrong.

I have a chat going with the OA team at the moment to clarify some of the final questions:

• rate limiting: one concurrency per-user is perfect for now, they are looking to migrate to Cloudflare+R2 so hopefully the bandwidth restrictions will be reduced/removed at some point soon.
• global downloads: there is a 'collection' named collection-global.zip which contains everything, addresses, parcels etc. I'd prefer to have a new collection with only the addresses, and in .tar.gz rather than .zip but we will see what's possible.

[vberten]

Hi @Joxit, @missinglink,

Any update on this? Is that planned to make this available in the pelias/docker repo?

Thanks!