Reduce retention and early materialization of VirtualActionInput

[fmeum]

b82dcdc introduced a way for parameter files to be materialized lazily and in a streaming fashion. For this to be effective, the contents of VirtualActionInputs should not be retained and, ideally, never or only briefly materialized in full.

This is achieved by replacing the digest computations with a streaming approach and avoiding retention of VirtualActionInput in MerkleTrees, which may be cached. Along the way, file nodes in MerkleTrees are stored unwrapped.

[fmeum]

@justinhorvitz Context: I'm currently investigating higher memory usage with path mapping enabled and am thus planning to make it compatible with your command line preprocessing logic. While working on that, I noticed that Bazel may not make good use of this optimization as it's RE implementation may retain full param file contents and some other places would at least temporarily materialize the full file just to calculate a digest. This PR is meant to address that. Ignoring the specifics of the integration with Bazel's RE backend, which other folks can review, do you consider it reasonable?

[justinhorvitz]

Might virtualActionInput.getBytes() be called multiple times by Chunker? The approach we use for the Google-internal RE is:

• Materialize the ByteString as late as possible, using double-checked locking, storing it as a field in the "supplier"
• The "supplier" has a close() method that is called when all chunks have been sent to the CAS. Calling this nulls the ByteString field.

[justinhorvitz]

temporarily materialize the full file just to calculate a digest

This is a good catch. The google-internal RE code does not do this. Instead, it makes use of VirtualActionInput#writeTo, passing an OutputStream that computes the digest on the fly. It looks like that's the same approach you've gone with in this PR.

I left a comment about the way you're passing a supplier into Chunker. I think you have the right approach, just need to make sure that VirtualActionInput#getBytes is only called once.