Enterprise Unified Logging

This repository contains example configurations for running an Elasticsearch stack to monitor and log ShotGrid Enterprise. The visualizations and dashboards provided are examples of what could be useful for monitoring ShotGrid Enterprise, clients may choose to modify the configuration as needed.

The solution uses Fluentd as the data collector between the ShotGrid application and the Elasticsearch database.

Getting started

First, you will need to build the Fluentd image and the Kibana image using the following command.

docker-compose build

Then start Fluentd and Elasticsearch along with Kibana.

docker-compose up

Finally, you will need to change the ShotGrid application logging driver in its docker-compose.yml file from json-file to fluentd:

Remove

# json-file
logging:
  driver: "json-file"
  options:
    max-size: "2g"
    max-file: "20"

And add

# fluentd
logging:
  driver: "fluentd"
  options:
    fluentd-address: "<fluentd_server_address>:24224"
    tag: "sg.app.{{.ID}}"

How to access logs

Kibana

Logs can be access via Kibana at http://localhost:5601/

From there you can create your indexes (ex: shotgrid_logs-* already created by default) and then query Elasticsearch.

Saved Objects

Customizations (visualizations, dashbaords, etc) can be saved. Please refer to Kibana documentation on how to manage saved objects. However, once changes are made in Kibana they persist as long as the database is not removed.

Saved objects can be provisioned by default by modifying the appropriate json file in the kibana/files_docker/provision.d/ directory.

Log files

Logs are also available (not by default) in json file in the logs/ directory.

Fluentd

Configuration

All the configuration takes place in the fluentd/files_docker/fluent.conf file.

Further details on configuring fluentd is availabe in the config-file documentation.

Plugins

For Elasticsearch, we use the fluent-plugin-elasticsearch plugin.

To install additional plugin see the Dockerfile at fluentd/Dockerfile.