The KubeSphere logging system is comprised of many components that work together to achieve unified log management. Once you enable the logging module during installation, a few components will be deployed or created within the namespace kubesphere-logging-system:
- Fluent Bit: log collector deployed on each node
- Fluent-bit Operator: automates tasks for operating Fluent Bit instances
- Logsidecar Injector: injects sidecar into pods for streaming on-disk files to stdout
- Elasticsearch Cluster: log storage backend
- Curator: performs stale log cleanup
The diagram below presents the overall logging architecture:
The following sections will detail the design and functionality of each component.
There are two types of logs produced by containers:
- stdout and stderr: written to
.logfiles in the host's directory/var/log/containersby default. - application file-based logs: files on the pod's volume, and file locations may vary.
Fluent Bit serves as the logging agent. It can access host's container log directory and streams log files to stdout. In KubeSphere, we forked and modified the origin Fluent Bit project fluent/fluent-bit. The difference between fluent/fluent-bit and kubesphere/fluent-bit is that we add a reload interface to support dynamic configuration (see issue #365) so that any changes on the configuration will be detected and reloaded by the agent instead of a restart.
As shown in the diagram below, the config file watcher implements the reloading functionality and is responsible for the life cycle of fluent-bit process.
The approach to collect container logs on a pod's volume is using a log forwarder sidecar. The sidecar container forwards on-disk application logs to its own stdout. In this way, KubeSphere manages log collection in a unified manner. Besides, Logsidecar Injector comes to automate sidecar injection. It takes advantages of MutatingAdmissionWebhook to control log forwarder sidecar injection during pod creation.
Though Fluent Bit sends log to Elasticsearch by default in KubeSphere, you may change its output via editing the custom resource defined by the CustomResourceDefinition fluentbits.logging.kubesphere.io. The custom resource holds Fluent Bit input/filter/output configurations. Once changes detected, Fluent-bit operator will trigger reloading and the new configuration will take effective. The Operator pattern is the encouraged way to implement declarative configuration in Kubernetes.
Fluent Bit supports a bunch of output plugins that you can choose from to configure how logs will be stored or forwarded. On KubeSphere console, you can easily set up Elasticsearch, Kafka and Fluentd as outputs.
Take Elasticsearch output for example. The default configuration writes that indices will be created daily with the prefix ks-logstash-log. It uses builtin Elasticsearch cluster whose address is elasticsearch-logging-data.kubesphere-logging-system.svc. For more details, please explore the custom resource fluent-bit (kubectl get fluentbits.logging.kubesphere.io -n kubesphere-logging-system fluent-bit -oyaml) .
To clean up stale logs, the Elasticsearch curator cronjob performs a daily check at the cluster level. See Curator for more information.
KubeSphere wraps Elasticsearch Search and Scroll APIs to support multi-tenancy. Users can use KubeSphere console to search and analyze logs as well as exporting logs to files.