v1.3.0

Storage APIs GetSignedLogRoot / SetSignedLogRoot now take pointers

This at the storage layer and does not affect the log server API.
This is part of work to fix proto buffer usages where they are passed
by value or compared by generic code like reflect.DeepEquals(). Passing
them by value creates shallow copies that can share internal state. As the
generated structs contain additional exported XXX_ fields generic
comparisons using all fields can produce incorrect results.

Storage Commit takes context.Context

To support passing a context down to NodeStorage.SetLeaves, and remove various context.TODO()s,
the following functions have been modified to accept a context.Context parameter:

• storage/cache.NodeStorage.SetLeaves
• storage/cache.SetSubtreesFunc
• storage/cache.SubtreeCache.Flush
• storage.ReadonlyLogTX.Commit

Go Module Support

Go Module support has been enabled. Please use GO111MODULE=on to build Trillian.
Updating dependencies no longer requires updating the vendor directory.

TrillianMapWrite API

New API service for writing to the Trillian Map. This allows APIs such as
GetLeavesByRevisionNoProof to be removed from the read API, and these methods to
be tuned & provisioned differently for read vs write performance.

GetLeavesByRevisionNoProof API

Allow map clients to forgo fetching inclusion proofs.
This dramatically speeds things up for clients that don't need verifiability.
This situation occurs in some situation where a Trillian personality is
interacting directly with the Trillian Map.

GetMapLeafByRevision API

New GetMapLeafByRevision API for fetching a single map leaf. This allows there
to be a separate API end point for fetching a single leaf vs. the batch
GetMapLeavesByRevision API which is much slower when many leaves are requested.
This supports separate monitoring and alerting for different traffic patterns.

Add Profiling Flags to Binaries

The trillian_log_server, trillian_log_signer and trillian_map_server
binaries now have CPU and heap profiling flags. Profiling is off by default.
For more details see the
Go Blog.

Map performance tweaks

The map mode has had some performance tweaks added:

• A workaround for locking issues which affect the map when it's used in
  single-transaction mode.

Introduce BatchInclusionProof function

Added a batch version of the Merkle Tree InclusionProof function.

Updated the map RPC for getLeaves to use the new batch function to improve
efficiency.

Google Cloud Spanner support

Google Cloud Spanner is now a supported storage backend for maps.

The admin API calls to list trees backed by Cloud Spanner trees are fixed.

RPC Server Transaction Leaks Fixed

There were some cases where the Log RPC server could leak storage transactions
in error situations. These have now been fixed. If you have a custom storage
implementation review the fixes made to the MySQL Log storage to see if they
need to be applied to your code (storage/mysql/log_storage.go). The Map
server had similar issues but these were fixed without requiring changes to
storage code.

GetLatestSignedLogRoot With Consistency Proof

GetLatestSignedLogRoot in the LogServer will return a consistency proof if
first_tree_size > 0. This reduces the number of RPC calls from logClient from
2 to 1 in client.getAndVerifyLatestRoot.

Testing

Support has been added for testing against a locally running mysql docker image,
in addition to a locally running mysql instance.

Deprecated Fields Removed From SignedLogRoot Proto

Important Note: For use in Certificate Transparency this version of the
logserver binary won't work properly with an older CTFE. Make sure to update the
CTFE servers to a current version (built from a git checkout after March 20th
2019) before deploying logservers that include this change or deploy them
together with this release. Failure to do this can result in 5XX errors being
returned to clients when the old handler code tries to access fields in
responses that no longer exist.

All the fields marked as deprecated in this proto have been removed. All the
same fields are available via the TLS marshalled log root in the proto. Updating
affected code is straightforward.

Normally, clients will want to verify that the signed root is correctly signed.
This is the preferred way to interact with the root data.

There is a utility function provided that will verify the signature and unpack
the TLS data. It works well in conjunction with a LogVerifier. The public key
of the server is required.

verifier := client.NewLogVerifier(rfc6962.DefaultHasher, pk, crypto.SHA256)
root, err := crypto.VerifySignedLogRoot(verifier.PubKey, verifier.SigHash, resp.SignedLogRoot)
if err != nil {
  // Signature verified and unmarshalled correctly. The struct may now
  // be used.
  if root.TreeSize > 0 {
    // Non empty tree.
  }
}

MySQL changes

Configurable number of connections for MySQL

Two new flags have been added that limit connections to MySQL database servers:

• --mysql_max_conns - limits the total number of database connections
• --mysql_max_idle_conns - limits the number of idle database connections

By default, there is no maximum number of database connections. However, the
database server will likely impose limits on the number of connections. The
default limit on idle connections is controlled by
Go's sql package.

Enfored no concurrent use of MySQL tx

Concurrently using a single MySQL transaction can cause the driver to error
out, so we now attempt to prevent this from happening.

Removal of length limits for a tree's display_name and description

Previously, these were restricted to 20 bytes and 200 bytes respectively. These
limits have been removed. However, the underlying storage implementation may
still impose its own limitations.

Server validation of leaf hashes

The log server now checks that leaf hashes are the correct length and returns
an InvalidArgument error if they are not. Previously, GetLeavesByHash would
simply not return any matching leaves for invalid hashes, and
GetInclusionProofByHash would return a NotFound error.

Map client

A MapClient has been added to simplify interacting with
the map server.

Database Schema

This version includes a change to the MySQL and Postgres database schemas to add
an index on the SequencedLeafData table. This improves performance for
inclusion proof queries.

Deployments

The Trillian Docker images now accept GOFLAGS and GO111MODULE arguments
and set them as environment variables inside the Docker container.

The db_server Docker image
is now based on
the MySQL 5.7 image from the Google Cloud Marketplace,
rather than the official MySQL 5.7 image.
This Dockerfile supercedes Dockerfile.db, which has been removed.

There is now a mysql.cnf file
alongside the Dockerfile that makes it easy to build the image with a custom
configuration, e.g. to allow MySQL to use more memory.

The trillian-log-service and trillian-log-signer Kubernetes services will
now have load balancers configured for them that expose those services outside
of the Kubernetes cluster. This makes it easier to access their APIs. When
deployed on Google Cloud, these will be
Internal Load Balancers.
Note that this change cannot be applied to an existing deployment; delete
the existing Kubernetes services and redeploy them, otherwise you'll see an
error similar to The Service "trillian-log-service" is invalid: spec.clusterIP: Invalid value: "": field is immutable.

A working Docker Compose configuration is
now available and can be used to bring up a local Trillian deployment for
testing and experimental purposes:

docker-compose -f examples/deployment/docker-compose.yml up

Docker Compose v3.1 or higher is required.

The Terraform, Kubernetes and Docker configuration files, as well as various
scripts, all now use the same, consistently-named environment variables for
MySQL-related data (e.g. MYSQL_DATABASE). The variable names are based on
those for the
MySQL Docker image.

Docker images have been upgraded from Go 1.9 to 1.11. They now use "Distroless"
base images.

Dropped metrics

Quota metrics with specs of the form users/<user>/read and
users/<user>/write are no longer exported by the Trillian binaries (as they
lead to excessive storage requirements for Trillian metrics).

Resilience improvements in log_signer

Add timeout to sequencing loop

Added a timeout to the context in the sequencing loop, with a default of 60s.

Fix Operation Loop Hang

Resolved a bug that would hide errors and cause the OperationLoop to hang
until process exit if any error occurred.

Linting toolchain migration

gometalinter has been replaced with golangci-lint for improved performance and
Go module support.

Compact Merkle tree data structures

CompactMerkleTree has been removed from github.com/google/trillian/merkle,
and a new package github.com/google/trillian/merkle/compact was introduced. A
new powerful data structure named "compact range" has been added to that
package, and is now used throughout the repository instead of the compact tree.
It is a generalization of the previous structure, as it allows manipulating
arbitrary sub-ranges of leaves rather than only prefixes.

Storage API changes

The internal storage API is modified so that the ReadOnlyTreeTX.ReadRevision and
TreeWriter.WriteRevision entrypoints take a context.Context parameter and return
an optional error.

The SubtreeCache.GetNodeHash() method is no longer exported.

The memory storage provider has been refactored to make it more consistent with
the other storage providers.

The LogMetadata.GetUnsequencedCounts() method has been removed.

NodeReader.GetMerkleNodes now must return Node objects in the same order as
node IDs requested. Storage implementations known to us already adhere to this
requirement.

Maphammer improvements

The maphammer test tool for the experimental Trillian Map has been enhanced.

Default values changed for some signer flags

The following flags for the signer have new default values:

• --sequencer_interval: changed from 10 seconds to 100 milliseconds
• --batch_size: changed from 50 to 1000

These changes improve the signer's throughput and latency under typical
conditions.

Master election refactoring

The --resign_odds flag in logsigner is removed, in favor of a more generic
--master_hold_jitter flag. Operators using this flag are advised to set the
jitter to master_check_interval * resign_odds * 2 to achieve similar behavior.

The --master_check_interval flag is removed from logsigner.

logsigner switched to using a new master election interface contained in
util/election2 package. The interfaces in util/election are removed.

CONIKS_SHA256 hash strategy added

Support has been added for a CONIKS sparse tree hasher with SHA256 as the hash
algorithm. Set a tree's hash_strategy to CONIKS_SHA256 to use it.

Performance

The performance of SetLeaves requests on the Map has been slightly improved.
The performance of GetConsistencyProof requests has been improved when using
MySQL.

Logging

Some warning-level logging has been removed from the sequencer in favour of
returning the same information via the returned error. The caller may still
choose to log this information. This allows storage implementations that retry
transactions to suppress warnings when a transaction initially fails but a retry
succeeds.

Some incorrectly-formatted log messages have been fixed.

Documentation

API documentation in Markdown format is now available.

Other

The TimeSource type (and other time utils) moved to a separate util/clock
package, extended with a new Timer interface that allows mocking time.Timer.

The Sequencer.SignRoot() method has been removed.