With the introduction of Go modules, our projects' dependency trees have exploded in complexity. Perseus is the hero we need to battle that Kraken.
At CrowdStrike, the move to Go Modules from our existing GOPATH-mode monorepo has brought with it some
pain points, especially around tracing descendant dependencies. In the GOPATH days, when our monorepo
lived under $GOPATH/src and all engineers always had a full copy of the entire codebase, it was a
straightforward grep command to find all imports of a given package to see what other packages depend on it.
Now that we've moved the majority of our development effort to modules, neither of those two conditions
are true. Engineers are no longer required to have the code under $GOPATH/src and almost no one has
the entire codebase locally anymore. We have dozens of functional teams working on hundreds of microservices,
so most developers have pared down their local workspace to only the things they're directly working
on on a day-to-day basis.
An unfortunate side effect of this paradigm shift has been that there is no longer a direct way to see
which other modules depend on your work. Existing tools like go mod graph and go list -m all will
show you what modules you depend on - with some rough edges - and the pkg.go.dev site has an Imported By
view that shows what other packages depend on your code. The go tool won't show which things depend
on you, though. The pkg.go.dev site can only show things that it knows about, so it won't help for
private modules, and it doesn't show you which versions of those other packages depend on your code.
See CHANGELOG.md for a detailed history of changes.
Unfortunately, the go CLI commands, the pkg.go.dev site, and OSS tools like goda
and godepgraph don't quite cover the ground we need, specifically
querying for downstream dependencies. The go CLI, goda and godepgraph all do an excellent job
of surfacing up which modules your code depends on in multiple ways. The pkg.go.dev site also provides
a nice Imported By view, but it shows which packages depend on your code, not which modules, and
doesn't include the version(s) of those dependents.
For simplicity, Perseus uses a PostgreSQL database rather than an actual graph database like Neo4J, Cayley, and the like. After some initial investigation, we found that the relatively small number of entries (compared to other graph datasets) didn't warrant a specialized graph database. Additionally, our IT organization already has all of the necessary infrastructure in place to support PostgreSQL.
One potentially signficant caveat, however, is that Amazon RDS currently does not support the
pg-semver extension that we use for storing a module's semantic
version.
The perseus service is built on Connect to provide an API that supports
binary gRPC and HTTP JSON/REST requests. Both RPC bindings are provided on a single port using
Vanguard, with the JSON/REST endpoints at a nested path of /api/v1/*.
Additionally, a basic web-based user interface is available at /ui.
In addition to the interactive endpoints, the service also exports an HTTP health check at /healthz
and basic Prometheus metrics at /metrics. For debugging and troubleshooting, the service supports
retrieving Go pprof data via HTTP at /debug/pprof*.
For simplicity, we publish a pre-built Docker image (based on a scratch base) to the GitHub Container
Registry when each release is tagged. The image runs perseus server and exposes port 31138, which
you can map to whatever is appropriate for your environment. You will need to provide the URL location
of the Perseus database, along with a valid username and password to connect, via environment variables.
The default database name is "perseus", but you can override that by also providing a DB_NAME
environment variable.
> docker run --rm -e DB_ADDR=... -e DB_USER=... -e DBPASS=... -p 31138:31138 ghcr.io/crowdstrike/perseus:latest
This example uses the latest tag but you should always reference a specific version for stability.
We also generate pre-built binaries for Windows, Linux, and Mac that can be downloaded from the releases page.
The perseus binary is both the server and a CLI tool for interacting with it. The update
and query commands are the workhorses as a client. The Perseus server address can be specified
by either setting the PERSEUS_SERVER_ADDR environment variable or passing it directly to the CLI
using the --server-addr flag.
perseus update analyzes a Go module, on disk or available via public Go module proxies, and adds it
to the Perseus graph. For a module on disk that is a Git repository, the CLI will try to infer the
version by looking at the Git tags on the current commit. If there is exactly 1 module version tag,
that version will be used. If there is no semver tag or there are multiple, you will need to specify
a version.
# process the current version of the perseus module on disk
> perseus update --path ~/code/github.com/CrowdStrike/perseus
# process a specific version of example/foo
> perseus update --path ~/code/github.com/example/foo --version v1.2.3
For public modules a version must always be specified.
> perseus update --module github.com/example/foo --version v1.2.3
Once you have data in your graph, perseus query is the way to retrieve it. There are 4 available
sub-commands: list-modules, list-module-versions, ancestors, and descendants.
The first two commands return modules and versions based on glob pattern matches:
# list all modules under the github.com/example organization along with the highest version
> perseus query list-modules 'github.com/example/*' --list
Module Version
github.com/example/foo v1.2.0
github.com/example/bar v1.1.0
github.com/example/baz v1.17.23
# list all versions of github.com/example/foo using an explicit format
> perseus query list-module-versions github.com/example/foo --format 'module {{.Path}} has version {{.Version}}.'
module github.com/example/foo has version v1.2.0
module github.com/example/foo has version v1.1.0
...
The ancestors and descendants commands walk the graph to return dependency trees, either what a
specified version of a module depends on or what modules depend on it.
# show the modules that v1.2.0 of github.com/example/foo depends on as a tabular list
> perseus query ancestors github.com/example/[email protected] --list
Dependency Direct
github.com/pkg/[email protected] true
golang.org/x/[email protected] true
...
# show the modules that depend on v0.9.1 of github.com/pkg/errors as nested JSON, using jq to format it nicely
> perseus query descendants github.com/pkg/[email protected] --json | jq .
{
"module": {
"Path": "github.com/pkg/errors",
"Version": "v0.9.1"
},
"deps": [
{
"module": {
"Path": "github.com/example/foo",
"Version": "v1.2.0"
},
...
}
]
}
In addition to text-based results using --json, --list or --format, the ancestor and descendant
commands also supports outputting DOT directed graphs using the --dot flag.
# generate an SVG image of the dependency graph for the highest version of github.com/example/foo
> perseus query ancestors github.com/example/foo --dot | dot -Tsvg -o ~/foo_deps.svg
The depth of the tree can be controlled by the --max-depth flag, with a default of 4 hops.
Disclaimer: perseus is an open source project, not a CrowdStrike product. As such, it carries no
formal support, expressed or implied. The project is licensed under the MIT open source license.