split compress and decompress into separate packages

[alixaxel]

I'm using iltorb as part of a AWS Lambda provisioning script. Since deployment package size in serverless world is usually quite limited, Brotli is an ideal choice in terms of decompression speed and compression ratio to inflate static binaries and other large files.

iltorb seems to be the best Brotli module in NPM hands-down, but using directly without patching anything, also means the compression that is gained by Brotli is mostly lost due to the iltorb compression dependencies. Publishing a iltorb-decompress package would be super useful for my use case.

[oohnoitz]

I think the challenge here is trying to determine if it is possible to accomplish the following:

• allow us to maintain a single repository for all changes (monorepo)
• maintain a sane build system so that we can provide multiple packages (e.g. iltorb, iltorb-compress, iltorb-decompress)

Here is a quick breakdown of size:

	size	description
iltorb	~14M	contains all dependencies
iltorb.node	~900K	our current binary file (includes both enc/dec)
iltorb/enc	~772K	binary with only enc
brotli/enc	~1.2M	source for brotli/enc (not including common files)
iltorb/dec	~216K	binary with only dec
brotli/dec	~168K	source for brotli/dec (not including common files)

With some quick math, we might end up with something like with all of the dependencies installed:

	size	description
iltorb	~14.0M	stays the same
iltorb-compress	~13.7M	compression logic only (subtract dec sizes)
iltorb-decompress	~12.1M	decompression logic only (subtract enc sizes)

[oohnoitz]

I removed the node-gyp dependency since it comes bundled with npm and yarn (by using npm's bundled version). It cuts out ~9M in our package size. While the potential savings of an additional ~0.3M (for a compression-only) and ~0.9M (for a decompression-only) package might be nice, I think it'd unnecessarily complicate the build pipeline to provide/publish two additional packages.

	size	description
iltorb	~5.4M	stays the same
iltorb-compress	~5.1M	compression logic only (subtract dec sizes)
iltorb-decompress	~3.5M	decompression logic only (subtract enc sizes)

[alixaxel]

@oohnoitz I'm using a stripped-down version of iltorb-decompress on https://github.com/alixaxel/chrome-aws-lambda/tree/master/source/iltorb and it weights less than 250KB.

What accounts for the ~3.5MB you mentioned?

[oohnoitz]

It looks like our dependencies alone would account for ~2.4M, which is pretty much prebuild-install. This is used to handle the installation logic so that it would grab the proper binaries. The rest would just be the source files to compile our binary when it fails to locate one to download.

In regards to your stripped-down version, it only supports the specific AWS Lambda environment you've compiled it for. It wouldn't work across different versions of Node.js (until we switch to N-API, but that isn't supported by all active LTS versions).

[alixaxel]

Understood. Well at least iltorb is now 9MB lighter. 😃 Thanks for looking into this.

[oohnoitz]

Best outcome would be that Brotli support lands in Node.js core (nodejs/node#18964).