Throwable indicates that the response to getMessage() may be null.

Cleaned up some links and language

Removed short circuit for executeWorkers which should never inspire
publish
Added memoized recentExecuteWorkers which will be delayed by currently
const workerSetMaxAge
Cleaned up getStorageWorkers, used grpc Deadline as premium, though
already expired is awkward

Storage removal is overzealous and will remove matching execute workers.
Received worker changes continue to only affect storage.

Adjustments will make use of options. Ensure that they have the
specified values from the commandline, and they have no checked
exception throws.

FMBs over a specified size limit (4MB in practice from grpc limits) will
be split log2n until under the limit to make requests.

Tests added to verify this. Coverage of split behavior confirmed.

Fixes buildfarm#1375

A directory which is missing during the course of validation should not
be identified as missing. This prevents a NPE and inspires validation to
emit one MISSING violation per path to a missing directory for
preconditions.

Fixes buildfarm#1374

Directories reevaluated only under the enumeration hierarchy must still
be guarded against empty child directories in their checks, and must
handle child directories missing in the index safely, with precondition
failures matching their outputs. Order is not guaranteed in precondition
output, but tests now guard this case.

Fixes buildfarm#1299

Include Action Mnemonic, Target Id, and Configuration Id in the bf-cat
output suite for RequestMetadata.

An invocation of app.run which fails for any reason in the spring
framework will exit silently. Ensure that errors are presented before
exiting the application.

* Enable custom latency buckets

* run formatter

* Remove unused load from build

* Run buildifier

* update example config with infinity bucket

---------

Co-authored-by: Trevor Hickey <[email protected]>

Avoid fileStore recalculation for entire trees to be deleted, instead
expect the callers to provide a fileStore, and that the entire tree
exists within it.

ExceutionExceptions wrap the actual exceptions of futures experienced
during putDirectory, and add no tracing capacity. Unwrap these when
thrown from failed futures.

Bleed grpc exposure into a retrier for copyExternalInput invocations,
and ensure that enough bytes have been provided from the requested blob
before returning.

The only presence of arbitrary symlinks in the CAS Filesystem is under
directories. Symlinks are explicitly identified as
non-readonly-executables. Prevent a dead symlink from throwing NSFE due
to the readonly check.

Files are delivered via readdir in utf8 encoding (on linux for xfs at
least), assume that posix will mandate this.

)

This reverts mistakenly added checks for non-null deadlines in StubInstance.

* Guard against fetchBlobFromWorker orphanization

Any exception thrown by fetchBlobFromWorker will leave the blobObserver
hanging. Ensure that the observer sees a failure and does not hang.

* Restore FMB functionality included in revert.

Handle ExecDirExceptions in InputFetcher such that the client can
observe a FAILED_PRECONDITION with PreconditionFailures that include
Violations with VIOLATION_TYPE_MISSING to inspire virtuous loop
reestablishment. A ViolationException acts as a container for this, and
can be extended to include the components of a putDirectory. Interpret
PutDirectoryExceptions with their imposed violations. Missing inputs at
the time of execute, whether linked as immediate files or through the
directory cache, will be interpreted as VIOLATION_TYPE_MISSING.

### Problem 
When workers die, their stored references are not removed from the backplane. This creates the possibility that new workers may come up with the same IP address or use an IP address previously used by another terminated host. As a result, the backplane becomes unreliable, requiring us to query each worker individually to find missing blobs. Clearly, this approach is not scalable since any problems encountered by a single worker can significantly impact the performance of the buildfarm.

### Past Work
We made code modifications for the `findMissingBlobs` function to exclusively query the backplane, prs: buildfarm#1310, buildfarm#1333, and buildfarm#1342. This update implemented the `findMissingViaBackplane` flag. However, the above issues made the `findMissingViaBackplane` flag ineffective.

### Solution
To address the issue of imposter workers, updated code to compare the start time of each worker (first_registered_at) with the insertion time of the digest. Any worker whose start time is later than the digest insertion time is considered an imposter worker. Also, the code removes imposter workers associated with the digest in the same function call.

**first_registered_at**: Added new field first_registered_at to the worker data type. This field stores the initial start time of the worker. Worker  informs the backplane about its start time, which is the same as the creation time of the cache directory (where all digests are stored) on the worker's disk.

**digest insert time**: The digest insertion time is calculated using the Time to Live (TTL) of the digest and the casExpire time. The formula for determining the digest insertion time is now() - configured casExpire + remaining ttl. In the current implementation, each worker updates the TTL of the digest upon completing the write operation. This means that the cas insert time in the backplane corresponds to the time when the last worker finished writing the digest on its disk.

### Testing
Deployed the change to our buildfarm staging, and ran full monorepo build. To make sure that the code change solve terminated worker problem, terminated bunch of workers in the middle of build. This caused temporary not_found `error`, which eventually faded away (fmb call autocorrect blob location).
<img width="1385" alt="Screenshot 2023-06-21 at 12 36 47 PM" src="https://github.com/bazelbuild/bazel-buildfarm/assets/119983081/62fcf8e0-847a-4632-b49e-cef2c17321dc">
In the above graph terminated workers during first build.


### Future Improvement
The above solution might not work if user updates `cas_expire` time between two deployments as algorithm to calculate `digest_insert_time` depends to `cas_expire` time. 

closes buildfarm#1371

When a server cannot acquire a transform token for an extended period of
time, assume that it is malfunctioning and initiate a shutdown.

Worker loss can signal cascading failure and shutdown for execute-only
peers. Ensure that a ReportResultStage seeing an SRE does not close the
stage, and that there are basic retries for remote uploads.

Works with current redis-py-cluster 2.1.3

Fixes buildfarm#1403

Prevents a hang on shutdown when redis is disconnected

Use the proper name for a status code as presented when the read will
not be retried.

Remediating https://nvd.nist.gov/vuln/detail/CVE-2023-2976

Fixes buildfarm#1421

Update platforms

* Add download metrics for buildfarm:worker

* Run formatter

* remove static

* change IO_METRIC prometheus counter to static

* Separate channel for write api

* shutdown channel on termination

* Fix deadlock by setting a timeout and removing the lock.

* remove timeout as its not necessary

* lint: delete unused import

Interactions with RemoteCasWriter which declare IOExceptions should
throw them exclusively for failures.

JCenter is retired. Pick some mirrors from
https://repo.maven.apache.org/maven2/.meta/repository-metadata.xml

Fixes buildfarm#1447

Specifying --prometheus_port for either server or worker will:
  Override the config value if delivered as int > 0, otherwise leave the
  config value alone.
Since prometheusPort as configured <= 0 disables the server, this
permits users to specify '0' to override the config with a disable
switch.

Updated quick_start docs to reflect this extra parameter.

…es (buildfarm#1444)

Co-authored-by: Trevor Hickey <[email protected]>

Support "rediss://" (extra 's', like https) to indicate Redis-SSL
connections.

For Redis client identifier, we use the `identifier` passed to RedisShardBackplane.

Specify and patch more recent rules_go and bazel_gazelle per upstream
patch. Docker images work with recent HEAD releases of bazel as a
result.

Depends on buildfarm#1453 resolution to actually build //...
Fixes buildfarm#1440

Any '.' appearance as a segment should be considered to be the current
directory in the search sequence of a working_directory path.

Method setExecuteResponseBuilder is not called in OperationContext

The cost in size for a single table bidirectional index is vast compared
to the use of 3nf integer keys. Experimental estimates offer a decrease
in file size of 90%.

…ations (buildfarm#1462)

Avoid dependency on subscriber to update state changes when removing
workers. This prevents an NPE which will occur invariably when workers
are allowably configured with subscribeToBackplane: false.

The individual metric controls for ttl are not necessary either for
performance or feature support. Use Files' attributes acquisition
mechanism for modified time.

Java logging definitions must now match java.util.logging.config.file,
update these specifications in our README.md

Use interrupt to halt the OperationQueuer.
Join on both operationQueuer and dispatchMonitor before instance stop
return.

Include Match and Report Result stages in output
Record the active operationName occupying slots in each of the stages
and present them with WorkerProfile
Avoid several unnecessary casts with interfaces for operation slot
stages.

A shard server is impractical without operation subscription, partition
subscription confirmation between servers and workers.
The failsafe execution is configuration that is likely not desired on
workers. This change removes the failsafe behavior from workers via
backplane config, and relegates the setting of failsafe boolean to
server config. If the option is restored for workers, it can be added to
worker configs so that configs may continue to be shared between workers
and servers and retain independent addressability.

Internally driven metrics and scaling controls have low, if any, usage
rates. Prometheus has largely succeeded independent publication of
metrics, and externally driven scaling is the norm. These modules have
been incomplete between cloud providers, and for the functional side of
AWS, bind us to springboot. Removing them for the sake of reduced
dependencies and complexity.

Remove this unused OperationQueue feature which provides no invocations
on any use.

This reverts commit f651cdb.

Continue the loop while we have *not* matched successfully and avoid a
confusing inversion in getMatched()

Distinguish the valid/unique/propagating methods of entry listening.

The only signal to a waiting match that will halt its current listen
loop for a valid unique operation is an interrupt.

Distinguish target param with GRPC type storage from FILESYSTEM
definition

Reinstate prior usage of LoggingMain for safe shutdown, with added
release mechanism for interrupted processes. All invoked shutdowns are
graceful, with vastly improved shutdown speed for empty workers waiting
for pipeline stages.

Tiny code cleanup

Will include operation root and inform directory cache effectiveness.

Selecting realInputDirectories by regex permits flexible patterns that
can yield drastic improvements in directory reuse for specialized
deployments. runfiles in particular are hazardous expansions of
nearly-execroot in the case of bazel.

Care must be taken to match directories exclusively.
The entire input tree is traversed for matches against expanded paths
under the root, to allow for nested selection.
Each match thus costs the number of input directories.
Counterintuitively, OutputFiles are augmented to avoid the recursive
check for OutputDirectories which only applies to actual reported
results, resulting in a path match when creating the exec root.
Regex style is java.util.Pattern, and must match the full input
directory.

This will include the path to the missed directory and the operation
which required it.

Prevent adding duplicate realInputDirectories matches

Ensure that the last leg of the execution presents a directory, rather
than the parent, per OutputDirectory's stamping.

Support a `--redis_uri` command line option for start-up.

also fixed some spelling typos.

Bumping 4.2 -> 5.3

Bump fro 0.0.6 -> 0.0.9