Flags:
- --add_dir_header If true, adds the file directory to the header of the log messages
- --agent-image string The image of moco-agent sidecar container
- --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
- --apiserver-qps-throttle int The maximum QPS to the API server. (default 20)
- --backup-image string The image of moco-backup container
- --cert-dir string webhook certificate directory
- --check-interval duration Interval of cluster maintenance (default 1m0s)
- --fluent-bit-image string The image of fluent-bit sidecar container
- --grpc-cert-dir string gRPC certificate directory (default "/grpc-cert")
- --health-probe-addr string Listen address for health probes (default ":8081")
- -h, --help help for moco-controller
- --leader-election-id string ID for leader election by controller-runtime (default "moco")
- --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
- --log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
- --log_file string If non-empty, use this log file (no effect when -logtostderr=true)
- --log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
- --logtostderr log to standard error instead of files (default true)
- --max-concurrent-reconciles int The maximum number of concurrent reconciles which can be run (default 8)
- --metrics-addr string Listen address for metric endpoint (default ":8080")
- --mysqld-exporter-image string The image of mysqld_exporter sidecar container
- --one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
- --pprof-addr string Listen address for pprof endpoints. pprof is disabled by default
- --skip_headers If true, avoid header prefixes in the log messages
- --skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
- --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
- -v, --v Level number for the log level verbosity
- --version version for moco-controller
- --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
- --webhook-addr string Listen address for the webhook endpoint (default ":9443")
- --zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
- --zap-encoder encoder Zap log encoding (one of 'json' or 'console')
- --zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
- --zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
- --zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
+ --add_dir_header If true, adds the file directory to the header of the log messages
+ --agent-image string The image of moco-agent sidecar container (default "ghcr.io/cybozu-go/moco-agent:0.12.1")
+ --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
+ --apiserver-qps-throttle int The maximum QPS to the API server. (default 20)
+ --backup-image string The image of moco-backup container (default "ghcr.io/cybozu-go/moco-backup:0.23.2")
+ --cert-dir string webhook certificate directory
+ --check-interval duration Interval of cluster maintenance (default 1m0s)
+ --fluent-bit-image string The image of fluent-bit sidecar container (default "ghcr.io/cybozu-go/moco/fluent-bit:3.0.2.1")
+ --grpc-cert-dir string gRPC certificate directory (default "/grpc-cert")
+ --health-probe-addr string Listen address for health probes (default ":8081")
+ -h, --help help for moco-controller
+ --leader-election-id string ID for leader election by controller-runtime (default "moco")
+ --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
+ --log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
+ --log_file string If non-empty, use this log file (no effect when -logtostderr=true)
+ --log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
+ --logtostderr log to standard error instead of files (default true)
+ --max-concurrent-reconciles int The maximum number of concurrent reconciles which can be run (default 8)
+ --metrics-addr string Listen address for metric endpoint (default ":8080")
+ --mysql-configmap-history-limit int The maximum number of MySQLConfigMap's history to be kept (default 10)
+ --mysqld-exporter-image string The image of mysqld_exporter sidecar container (default "ghcr.io/cybozu-go/moco/mysqld_exporter:0.15.1.2")
+ --one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
+ --pprof-addr string Listen address for pprof endpoints. pprof is disabled by default
+ --pvc-sync-annotation-keys strings The keys of annotations from MySQLCluster's volumeClaimTemplates to be synced to the PVC
+ --pvc-sync-label-keys strings The keys of labels from MySQLCluster's volumeClaimTemplates to be synced to the PVC
+ --skip_headers If true, avoid header prefixes in the log messages
+ --skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
+ --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=true) (default 2)
+ -v, --v Level number for the log level verbosity
+ --version version for moco-controller
+ --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
+ --webhook-addr string Listen address for the webhook endpoint (default ":9443")
+ --zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
+ --zap-encoder encoder Zap log encoding (one of 'json' or 'console')
+ --zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
+ --zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
+ --zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
Flags:
- --add_dir_header If true, adds the file directory to the header of the log messages
- --agent-image string The image of moco-agent sidecar container
- --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
- --apiserver-qps-throttle int The maximum QPS to the API server. (default 20)
- --backup-image string The image of moco-backup container
- --cert-dir string webhook certificate directory
- --check-interval duration Interval of cluster maintenance (default 1m0s)
- --fluent-bit-image string The image of fluent-bit sidecar container
- --grpc-cert-dir string gRPC certificate directory (default "/grpc-cert")
- --health-probe-addr string Listen address for health probes (default ":8081")
- -h, --help help for moco-controller
- --leader-election-id string ID for leader election by controller-runtime (default "moco")
- --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
- --log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
- --log_file string If non-empty, use this log file (no effect when -logtostderr=true)
- --log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
- --logtostderr log to standard error instead of files (default true)
- --max-concurrent-reconciles int The maximum number of concurrent reconciles which can be run (default 8)
- --metrics-addr string Listen address for metric endpoint (default ":8080")
- --mysqld-exporter-image string The image of mysqld_exporter sidecar container
- --one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
- --pprof-addr string Listen address for pprof endpoints. pprof is disabled by default
- --skip_headers If true, avoid header prefixes in the log messages
- --skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
- --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2)
- -v, --v Level number for the log level verbosity
- --version version for moco-controller
- --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
- --webhook-addr string Listen address for the webhook endpoint (default ":9443")
- --zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
- --zap-encoder encoder Zap log encoding (one of 'json' or 'console')
- --zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
- --zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
- --zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
+ --add_dir_header If true, adds the file directory to the header of the log messages
+ --agent-image string The image of moco-agent sidecar container (default "ghcr.io/cybozu-go/moco-agent:0.12.1")
+ --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true)
+ --apiserver-qps-throttle int The maximum QPS to the API server. (default 20)
+ --backup-image string The image of moco-backup container (default "ghcr.io/cybozu-go/moco-backup:0.23.2")
+ --cert-dir string webhook certificate directory
+ --check-interval duration Interval of cluster maintenance (default 1m0s)
+ --fluent-bit-image string The image of fluent-bit sidecar container (default "ghcr.io/cybozu-go/moco/fluent-bit:3.0.2.1")
+ --grpc-cert-dir string gRPC certificate directory (default "/grpc-cert")
+ --health-probe-addr string Listen address for health probes (default ":8081")
+ -h, --help help for moco-controller
+ --leader-election-id string ID for leader election by controller-runtime (default "moco")
+ --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
+ --log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true)
+ --log_file string If non-empty, use this log file (no effect when -logtostderr=true)
+ --log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
+ --logtostderr log to standard error instead of files (default true)
+ --max-concurrent-reconciles int The maximum number of concurrent reconciles which can be run (default 8)
+ --metrics-addr string Listen address for metric endpoint (default ":8080")
+ --mysql-configmap-history-limit int The maximum number of MySQLConfigMap's history to be kept (default 10)
+ --mysqld-exporter-image string The image of mysqld_exporter sidecar container (default "ghcr.io/cybozu-go/moco/mysqld_exporter:0.15.1.2")
+ --one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
+ --pprof-addr string Listen address for pprof endpoints. pprof is disabled by default
+ --pvc-sync-annotation-keys strings The keys of annotations from MySQLCluster's volumeClaimTemplates to be synced to the PVC
+ --pvc-sync-label-keys strings The keys of labels from MySQLCluster's volumeClaimTemplates to be synced to the PVC
+ --skip_headers If true, avoid header prefixes in the log messages
+ --skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true)
+ --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=true) (default 2)
+ -v, --v Level number for the log level verbosity
+ --version version for moco-controller
+ --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
+ --webhook-addr string Listen address for the webhook endpoint (default ":9443")
+ --zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error)
+ --zap-encoder encoder Zap log encoding (one of 'json' or 'console')
+ --zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity
+ --zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic').
+ --zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.
moco-backupmoco-backup command is used in ghcr.io/cybozu-go/moco-backup container.
diff --git a/searchindex.js b/searchindex.js
index 95df01b2..2c0308ff 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Object.assign(window.search, {"doc_urls":["index.html#moco-documentation","getting_started.html#getting-started","setup.html#setup","setup.html#quick-setup","setup.html#install-using-raw-manifests","setup.html#install-using-helm-chart","setup.html#customize-manifests","setup.html#next-step","helm.html#moco-helm-chart","helm.html#how-to-use-moco-helm-repository","helm.html#quick-start","helm.html#installing-cert-manager","helm.html#installing-the-chart","helm.html#values","helm.html#generate-manifests","helm.html#crd-considerations","helm.html#installing-or-updating-crds","helm.html#removing-crds","helm.html#migrate-to-v0110-or-higher","helm.html#migrate-to-v030","helm.html#release-chart","install-plugin.html#installing-kubectl-moco","install-plugin.html#installing-using-krew","install-plugin.html#installing-manually","usage.html#how-to-use-moco","usage.html#basics","usage.html#limitations","usage.html#errant-replicas","usage.html#read-only-primary","usage.html#creating-clusters","usage.html#creating-an-empty-cluster","usage.html#creating-a-cluster-that-replicates-data-from-an-external-mysqld","usage.html#bring-your-own-image","usage.html#configurations","usage.html#innodb-buffer-pool-size","usage.html#opaque-configuration","usage.html#using-the-cluster","usage.html#kubectl-moco","usage.html#mysql-users","usage.html#connecting-to-mysqld-over-network","usage.html#backup-and-restore","usage.html#object-storage-bucket","usage.html#backuppolicy","usage.html#credentials-to-access-s3-bucket","usage.html#taking-an-emergency-backup","usage.html#restore","usage.html#further-details","usage.html#deleting-the-cluster","usage.html#status-metrics-and-logs","usage.html#cluster-status","usage.html#pod-status","usage.html#metrics","usage.html#logs","usage.html#maintenance","usage.html#increasing-the-number-of-instances-in-the-cluster","usage.html#switchover","usage.html#failover","usage.html#upgrading-mysql-version","usage.html#re-initializing-an-errant-replica","usage.html#stop-clustering-and-reconciliation","usage.html#set-to-read-only","advanced.html#advanced-topics","custom-mysqld.html#building-custom-image-of-mysqld","custom-mysqld.html#dockerfile","custom-mysqld.html#how-to-build-mysqld","customize-system-container.html#customize-default-container","customize-system-container.html#system-containers","change-pvc-template.html#change-the-volumeclaimtemplates","change-pvc-template.html#metrics","change-pvc-template.html#volume-expansion","change-pvc-template.html#metrics-1","change-pvc-template.html#volume-reduction","change-pvc-template.html#1-the-user-modifies-the-specvolumeclaimtemplates-of-the-mysqlcluster-and-sets-a-smaller-volume-size","change-pvc-template.html#2-moco-updates-the-specvolumeclaimtemplates-of-the-statefulset-this-does-not-propagate-to-existing-pods-pvcs-or-pvs","change-pvc-template.html#3-the-user-manually-deletes-the-mysql-pod--pvc","change-pvc-template.html#4-wait-for-the-pod--pvc-to-be-recreated-by-the-statefulset-controller-and-for-moco-to-clone-the-data","change-pvc-template.html#5-once-the-cluster-becomes-healthy-the-user-deletes-the-next-pod-and-pvc","change-pvc-template.html#6-it-is-completed-when-all-pods-and-pvcs-are-recreated","change-pvc-template.html#references","rolling-update-strategy.html#rollingupdate-strategy","rolling-update-strategy.html#partitions","rolling-update-strategy.html#architecture","rolling-update-strategy.html#when-creating-a-statefulset","rolling-update-strategy.html#when-updating-a-statefulset","rolling-update-strategy.html#updating-partitions","rolling-update-strategy.html#forcefully-rolling-out","rolling-update-strategy.html#metrics","known_issues.html#known-issues","known_issues.html#multi-threaded-replication","crd.html#custom-resources","crd_mysqlcluster_v1beta2.html#custom-resources","crd_mysqlcluster_v1beta2.html#sub-resources","crd_backuppolicy_v1beta2.html#custom-resources","crd_backuppolicy_v1beta2.html#sub-resources","commands.html#commands","kubectl-moco.html#kubectl-moco-plugin","kubectl-moco.html#global-options","kubectl-moco.html#mysql-users","kubectl-moco.html#kubectl-moco-mysql-options-cluster_name----mysql-args","kubectl-moco.html#examples","kubectl-moco.html#kubectl-moco-credential-options-cluster_name","kubectl-moco.html#kubectl-moco-switchover-cluster_name","kubectl-moco.html#stop-or-start-clustering-and-reconciliation","kubectl-moco.html#kubectl-moco-stop-clustering-cluster_name","kubectl-moco.html#kubectl-moco-start-clustering-cluster_name","kubectl-moco.html#kubectl-moco-stop-reconciliation-cluster_name","kubectl-moco.html#kubectl-moco-start-reconciliation-cluster_name","moco-controller.html#moco-controller","moco-controller.html#environment-variables","moco-controller.html#command-line-flags","moco-backup.html#moco-backup","moco-backup.html#environment-variables","moco-backup.html#global-command-line-flags","moco-backup.html#subcommands","moco-backup.html#backup-subcommand","moco-backup.html#restore-subcommand","metrics.html#metrics","metrics.html#moco-controller","metrics.html#mysql-clusters","metrics.html#backup","metrics.html#mysql-instance","metrics.html#scrape-rules","notes.html#design-notes","design.html#design-notes","design.html#motivation","design.html#goals","design.html#non-goals","reconcile.html#how-moco-reconciles-mysqlcluster","reconcile.html#reconciler-versions","reconcile.html#the-update-policy-of-moco-agent-container","reconcile.html#clustering-related-resources","reconcile.html#statefulset","reconcile.html#when-the-statefulset-is--not--updated","reconcile.html#status-about-statefulset","reconcile.html#secrets","reconcile.html#certificate","reconcile.html#service","reconcile.html#configmap","reconcile.html#poddisruptionbudget","reconcile.html#serviceaccount","reconcile.html#backup-and-restore-related-resources","reconcile.html#cronjob","reconcile.html#job","reconcile.html#status-of-reconcliation","clustering.html#how-moco-maintains-mysql-clusters","clustering.html#terminology","clustering.html#prerequisites","clustering.html#limitations","clustering.html#possible-states","clustering.html#mysqlcluster","clustering.html#pod","clustering.html#mysql-data","clustering.html#invariants","clustering.html#the-maintenance-flow","clustering.html#gather-the-current-status","clustering.html#update-status-of-mysqlcluster","clustering.html#determine-what-moco-should-do-for-the-cluster","backup.html#backup-and-restore","backup.html#overview","backup.html#design-goals","backup.html#implementation","backup.html#backup-file-keys","backup.html#timestamps","backup.html#backup","backup.html#restore","backup.html#caveats","backup.html#considered-options","backup.html#why-do-we-use-s3-compatible-object-storage-to-store-backups","backup.html#what-object-storage-is-supported","backup.html#why-do-we-use-jobs-for-backup-and-restoration","backup.html#why-do-we-prefer-mysqlsh-to-mysqldump","backup.html#why-dont-we-do-continuous-backup","upgrading.html#upgrading-mysqld","upgrading.html#preconditions","upgrading.html#mysql-data","upgrading.html#downgrading","upgrading.html#upgrading-a-replication-setup","upgrading.html#statefulset-behavior","upgrading.html#automatic-switchover","upgrading.html#moco-implementation","upgrading.html#example","upgrading.html#limitations","upgrading.html#users-responsibility","security.html#security-considerations","security.html#grpc-api","security.html#mysql-passwords"],"index":{"documentStore":{"docInfo":{"0":{"body":10,"breadcrumbs":3,"title":2},"1":{"body":0,"breadcrumbs":4,"title":2},"10":{"body":0,"breadcrumbs":6,"title":2},"100":{"body":24,"breadcrumbs":8,"title":5},"101":{"body":5,"breadcrumbs":7,"title":4},"102":{"body":4,"breadcrumbs":7,"title":4},"103":{"body":4,"breadcrumbs":8,"title":5},"104":{"body":4,"breadcrumbs":8,"title":5},"105":{"body":4,"breadcrumbs":8,"title":5},"106":{"body":4,"breadcrumbs":8,"title":5},"107":{"body":6,"breadcrumbs":5,"title":2},"108":{"body":10,"breadcrumbs":5,"title":2},"109":{"body":342,"breadcrumbs":6,"title":3},"11":{"body":8,"breadcrumbs":7,"title":3},"110":{"body":14,"breadcrumbs":5,"title":2},"111":{"body":21,"breadcrumbs":5,"title":2},"112":{"body":47,"breadcrumbs":7,"title":4},"113":{"body":0,"breadcrumbs":4,"title":1},"114":{"body":16,"breadcrumbs":5,"title":2},"115":{"body":39,"breadcrumbs":5,"title":2},"116":{"body":9,"breadcrumbs":2,"title":1},"117":{"body":22,"breadcrumbs":3,"title":2},"118":{"body":143,"breadcrumbs":3,"title":2},"119":{"body":53,"breadcrumbs":2,"title":1},"12":{"body":51,"breadcrumbs":6,"title":2},"120":{"body":30,"breadcrumbs":3,"title":2},"121":{"body":95,"breadcrumbs":3,"title":2},"122":{"body":0,"breadcrumbs":4,"title":2},"123":{"body":10,"breadcrumbs":5,"title":2},"124":{"body":60,"breadcrumbs":4,"title":1},"125":{"body":99,"breadcrumbs":4,"title":1},"126":{"body":46,"breadcrumbs":5,"title":2},"127":{"body":43,"breadcrumbs":6,"title":3},"128":{"body":51,"breadcrumbs":5,"title":2},"129":{"body":8,"breadcrumbs":8,"title":5},"13":{"body":96,"breadcrumbs":5,"title":1},"130":{"body":13,"breadcrumbs":6,"title":3},"131":{"body":38,"breadcrumbs":4,"title":1},"132":{"body":20,"breadcrumbs":5,"title":2},"133":{"body":14,"breadcrumbs":5,"title":2},"134":{"body":21,"breadcrumbs":4,"title":1},"135":{"body":27,"breadcrumbs":4,"title":1},"136":{"body":64,"breadcrumbs":4,"title":1},"137":{"body":32,"breadcrumbs":4,"title":1},"138":{"body":25,"breadcrumbs":4,"title":1},"139":{"body":8,"breadcrumbs":4,"title":1},"14":{"body":13,"breadcrumbs":6,"title":2},"140":{"body":6,"breadcrumbs":7,"title":4},"141":{"body":9,"breadcrumbs":4,"title":1},"142":{"body":17,"breadcrumbs":4,"title":1},"143":{"body":13,"breadcrumbs":5,"title":2},"144":{"body":47,"breadcrumbs":7,"title":4},"145":{"body":67,"breadcrumbs":4,"title":1},"146":{"body":173,"breadcrumbs":4,"title":1},"147":{"body":48,"breadcrumbs":4,"title":1},"148":{"body":0,"breadcrumbs":5,"title":2},"149":{"body":190,"breadcrumbs":4,"title":1},"15":{"body":0,"breadcrumbs":6,"title":2},"150":{"body":49,"breadcrumbs":4,"title":1},"151":{"body":38,"breadcrumbs":5,"title":2},"152":{"body":19,"breadcrumbs":4,"title":1},"153":{"body":34,"breadcrumbs":5,"title":2},"154":{"body":40,"breadcrumbs":6,"title":3},"155":{"body":84,"breadcrumbs":6,"title":3},"156":{"body":244,"breadcrumbs":6,"title":3},"157":{"body":43,"breadcrumbs":6,"title":2},"158":{"body":181,"breadcrumbs":5,"title":1},"159":{"body":49,"breadcrumbs":6,"title":2},"16":{"body":13,"breadcrumbs":7,"title":3},"160":{"body":0,"breadcrumbs":5,"title":1},"161":{"body":67,"breadcrumbs":7,"title":3},"162":{"body":12,"breadcrumbs":5,"title":1},"163":{"body":391,"breadcrumbs":5,"title":1},"164":{"body":126,"breadcrumbs":5,"title":1},"165":{"body":54,"breadcrumbs":5,"title":1},"166":{"body":14,"breadcrumbs":6,"title":2},"167":{"body":23,"breadcrumbs":11,"title":7},"168":{"body":68,"breadcrumbs":7,"title":3},"169":{"body":34,"breadcrumbs":8,"title":4},"17":{"body":13,"breadcrumbs":6,"title":2},"170":{"body":52,"breadcrumbs":7,"title":3},"171":{"body":60,"breadcrumbs":7,"title":3},"172":{"body":23,"breadcrumbs":6,"title":2},"173":{"body":0,"breadcrumbs":5,"title":1},"174":{"body":36,"breadcrumbs":6,"title":2},"175":{"body":47,"breadcrumbs":5,"title":1},"176":{"body":14,"breadcrumbs":7,"title":3},"177":{"body":58,"breadcrumbs":6,"title":2},"178":{"body":11,"breadcrumbs":6,"title":2},"179":{"body":43,"breadcrumbs":6,"title":2},"18":{"body":25,"breadcrumbs":7,"title":3},"180":{"body":87,"breadcrumbs":5,"title":1},"181":{"body":32,"breadcrumbs":5,"title":1},"182":{"body":13,"breadcrumbs":6,"title":2},"183":{"body":0,"breadcrumbs":5,"title":2},"184":{"body":97,"breadcrumbs":5,"title":2},"185":{"body":25,"breadcrumbs":5,"title":2},"19":{"body":274,"breadcrumbs":6,"title":2},"2":{"body":0,"breadcrumbs":5,"title":1},"20":{"body":2,"breadcrumbs":6,"title":2},"21":{"body":17,"breadcrumbs":8,"title":3},"22":{"body":18,"breadcrumbs":8,"title":3},"23":{"body":110,"breadcrumbs":7,"title":2},"24":{"body":92,"breadcrumbs":3,"title":2},"25":{"body":68,"breadcrumbs":2,"title":1},"26":{"body":0,"breadcrumbs":2,"title":1},"27":{"body":36,"breadcrumbs":3,"title":2},"28":{"body":16,"breadcrumbs":3,"title":2},"29":{"body":0,"breadcrumbs":3,"title":2},"3":{"body":25,"breadcrumbs":6,"title":2},"30":{"body":194,"breadcrumbs":4,"title":3},"31":{"body":176,"breadcrumbs":7,"title":6},"32":{"body":15,"breadcrumbs":3,"title":2},"33":{"body":59,"breadcrumbs":2,"title":1},"34":{"body":18,"breadcrumbs":5,"title":4},"35":{"body":59,"breadcrumbs":3,"title":2},"36":{"body":0,"breadcrumbs":3,"title":2},"37":{"body":47,"breadcrumbs":3,"title":2},"38":{"body":126,"breadcrumbs":3,"title":2},"39":{"body":77,"breadcrumbs":5,"title":4},"4":{"body":8,"breadcrumbs":8,"title":4},"40":{"body":20,"breadcrumbs":3,"title":2},"41":{"body":35,"breadcrumbs":4,"title":3},"42":{"body":246,"breadcrumbs":2,"title":1},"43":{"body":46,"breadcrumbs":5,"title":4},"44":{"body":15,"breadcrumbs":4,"title":3},"45":{"body":74,"breadcrumbs":2,"title":1},"46":{"body":4,"breadcrumbs":3,"title":2},"47":{"body":17,"breadcrumbs":3,"title":2},"48":{"body":0,"breadcrumbs":4,"title":3},"49":{"body":56,"breadcrumbs":3,"title":2},"5":{"body":18,"breadcrumbs":8,"title":4},"50":{"body":55,"breadcrumbs":3,"title":2},"51":{"body":43,"breadcrumbs":2,"title":1},"52":{"body":23,"breadcrumbs":2,"title":1},"53":{"body":0,"breadcrumbs":2,"title":1},"54":{"body":29,"breadcrumbs":5,"title":4},"55":{"body":27,"breadcrumbs":2,"title":1},"56":{"body":41,"breadcrumbs":2,"title":1},"57":{"body":60,"breadcrumbs":4,"title":3},"58":{"body":42,"breadcrumbs":5,"title":4},"59":{"body":159,"breadcrumbs":4,"title":3},"6":{"body":9,"breadcrumbs":6,"title":2},"60":{"body":69,"breadcrumbs":3,"title":2},"61":{"body":0,"breadcrumbs":4,"title":2},"62":{"body":36,"breadcrumbs":8,"title":4},"63":{"body":28,"breadcrumbs":5,"title":1},"64":{"body":59,"breadcrumbs":6,"title":2},"65":{"body":53,"breadcrumbs":8,"title":3},"66":{"body":82,"breadcrumbs":7,"title":2},"67":{"body":100,"breadcrumbs":6,"title":2},"68":{"body":33,"breadcrumbs":5,"title":1},"69":{"body":79,"breadcrumbs":6,"title":2},"7":{"body":6,"breadcrumbs":6,"title":2},"70":{"body":40,"breadcrumbs":5,"title":1},"71":{"body":58,"breadcrumbs":6,"title":2},"72":{"body":40,"breadcrumbs":13,"title":9},"73":{"body":22,"breadcrumbs":14,"title":10},"74":{"body":21,"breadcrumbs":11,"title":7},"75":{"body":118,"breadcrumbs":14,"title":10},"76":{"body":29,"breadcrumbs":14,"title":10},"77":{"body":8,"breadcrumbs":9,"title":5},"78":{"body":2,"breadcrumbs":5,"title":1},"79":{"body":60,"breadcrumbs":6,"title":2},"8":{"body":0,"breadcrumbs":7,"title":3},"80":{"body":32,"breadcrumbs":5,"title":1},"81":{"body":0,"breadcrumbs":5,"title":1},"82":{"body":11,"breadcrumbs":6,"title":2},"83":{"body":38,"breadcrumbs":6,"title":2},"84":{"body":32,"breadcrumbs":6,"title":2},"85":{"body":39,"breadcrumbs":7,"title":3},"86":{"body":36,"breadcrumbs":5,"title":1},"87":{"body":8,"breadcrumbs":4,"title":2},"88":{"body":28,"breadcrumbs":5,"title":3},"89":{"body":0,"breadcrumbs":4,"title":2},"9":{"body":14,"breadcrumbs":8,"title":4},"90":{"body":1,"breadcrumbs":6,"title":2},"91":{"body":1150,"breadcrumbs":6,"title":2},"92":{"body":1,"breadcrumbs":6,"title":2},"93":{"body":494,"breadcrumbs":6,"title":2},"94":{"body":0,"breadcrumbs":2,"title":1},"95":{"body":13,"breadcrumbs":6,"title":3},"96":{"body":29,"breadcrumbs":5,"title":2},"97":{"body":25,"breadcrumbs":5,"title":2},"98":{"body":36,"breadcrumbs":10,"title":7},"99":{"body":46,"breadcrumbs":4,"title":1}},"docs":{"0":{"body":"This is the documentation site for MOCO . MOCO is a Kubernetes operator for MySQL created and maintained by Cybozu.","breadcrumbs":"MOCO » MOCO documentation","id":"0","title":"MOCO documentation"},"1":{"body":"","breadcrumbs":"Getting started » Getting started","id":"1","title":"Getting started"},"10":{"body":"","breadcrumbs":"Getting started » Helm Chart » Quick start","id":"10","title":"Quick start"},"100":{"body":"Fetch the credential information of a specified user Options Default value Description -u, --mysql-user moco-readonly Fetch the credential of the specified user --format plain Output format: plain or mycnf","breadcrumbs":"Commands » kubectl-moco » kubectl moco credential [options] CLUSTER_NAME","id":"100","title":"kubectl moco credential [options] CLUSTER_NAME"},"101":{"body":"Switch the primary instance to one of the replicas.","breadcrumbs":"Commands » kubectl-moco » kubectl moco switchover CLUSTER_NAME","id":"101","title":"kubectl moco switchover CLUSTER_NAME"},"102":{"body":"Read Stop Clustering and Reconciliation .","breadcrumbs":"Commands » kubectl-moco » Stop or start clustering and reconciliation","id":"102","title":"Stop or start clustering and reconciliation"},"103":{"body":"Stop the clustering of the specified MySQLCluster.","breadcrumbs":"Commands » kubectl-moco » kubectl moco stop clustering CLUSTER_NAME","id":"103","title":"kubectl moco stop clustering CLUSTER_NAME"},"104":{"body":"Start the clustering of the specified MySQLCluster.","breadcrumbs":"Commands » kubectl-moco » kubectl moco start clustering CLUSTER_NAME","id":"104","title":"kubectl moco start clustering CLUSTER_NAME"},"105":{"body":"Stop the reconciliation of the specified MySQLCluster.","breadcrumbs":"Commands » kubectl-moco » kubectl moco stop reconciliation CLUSTER_NAME","id":"105","title":"kubectl moco stop reconciliation CLUSTER_NAME"},"106":{"body":"Start the reconciliation of the specified MySQLCluster.","breadcrumbs":"Commands » kubectl-moco » kubectl moco start reconciliation CLUSTER_NAME","id":"106","title":"kubectl moco start reconciliation CLUSTER_NAME"},"107":{"body":"moco-controller controls MySQL clusters on Kubernetes.","breadcrumbs":"Commands » moco-controller » moco-controller","id":"107","title":"moco-controller"},"108":{"body":"Name Required Description POD_NAMESPACE Yes The namespace name where moco-controller runs.","breadcrumbs":"Commands » moco-controller » Environment variables","id":"108","title":"Environment variables"},"109":{"body":"Flags: --add_dir_header If true, adds the file directory to the header of the log messages --agent-image string The image of moco-agent sidecar container --alsologtostderr log to standard error as well as files (no effect when -logtostderr=true) --apiserver-qps-throttle int The maximum QPS to the API server. (default 20) --backup-image string The image of moco-backup container --cert-dir string webhook certificate directory --check-interval duration Interval of cluster maintenance (default 1m0s) --fluent-bit-image string The image of fluent-bit sidecar container --grpc-cert-dir string gRPC certificate directory (default \"/grpc-cert\") --health-probe-addr string Listen address for health probes (default \":8081\") -h, --help help for moco-controller --leader-election-id string ID for leader election by controller-runtime (default \"moco\") --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0) --log_dir string If non-empty, write log files in this directory (no effect when -logtostderr=true) --log_file string If non-empty, use this log file (no effect when -logtostderr=true) --log_file_max_size uint Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800) --logtostderr log to standard error instead of files (default true) --max-concurrent-reconciles int The maximum number of concurrent reconciles which can be run (default 8) --metrics-addr string Listen address for metric endpoint (default \":8080\") --mysqld-exporter-image string The image of mysqld_exporter sidecar container --one_output If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true) --pprof-addr string Listen address for pprof endpoints. pprof is disabled by default --skip_headers If true, avoid header prefixes in the log messages --skip_log_headers If true, avoid headers when opening log files (no effect when -logtostderr=true) --stderrthreshold severity logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false) (default 2) -v, --v Level number for the log level verbosity --version version for moco-controller --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging --webhook-addr string Listen address for the webhook endpoint (default \":9443\") --zap-devel Development Mode defaults(encoder=consoleEncoder,logLevel=Debug,stackTraceLevel=Warn). Production Mode defaults(encoder=jsonEncoder,logLevel=Info,stackTraceLevel=Error) --zap-encoder encoder Zap log encoding (one of 'json' or 'console') --zap-log-level level Zap Level to configure the verbosity of logging. Can be one of 'debug', 'info', 'error', or any integer value > 0 which corresponds to custom debug levels of increasing verbosity --zap-stacktrace-level level Zap Level at and above which stacktraces are captured (one of 'info', 'error', 'panic'). --zap-time-encoding time-encoding Zap time encoding (one of 'epoch', 'millis', 'nano', 'iso8601', 'rfc3339' or 'rfc3339nano'). Defaults to 'epoch'.","breadcrumbs":"Commands » moco-controller » Command line flags","id":"109","title":"Command line flags"},"11":{"body":"$ curl -fsL https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml | kubectl apply -f -","breadcrumbs":"Getting started » Helm Chart » Installing cert-manager","id":"11","title":"Installing cert-manager"},"110":{"body":"moco-backup command is used in ghcr.io/cybozu-go/moco-backup container. Normally, users need not take care of this command.","breadcrumbs":"Commands » moco-backup » moco-backup","id":"110","title":"moco-backup"},"111":{"body":"moco-backup takes configurations of S3 API from environment variables. For details, read documentation of EnvConfig in github.com/aws/aws-sdk-go-v2/config . It also requires MYSQL_PASSWORD environment variable to be set.","breadcrumbs":"Commands » moco-backup » Environment variables","id":"111","title":"Environment variables"},"112":{"body":"Global Flags: --endpoint string S3 API endpoint URL --region string AWS region --threads int The number of threads to be used (default 4) --use-path-style Use path-style S3 API --work-dir string The writable working directory (default \"/work\") --ca-cert string Path to SSL CA certificate file used in addition to system default","breadcrumbs":"Commands » moco-backup » Global command-line flags","id":"112","title":"Global command-line flags"},"113":{"body":"","breadcrumbs":"Commands » moco-backup » Subcommands","id":"113","title":"Subcommands"},"114":{"body":"Usage: moco-backup backup BUCKET NAMESPACE NAME BUCKET: The bucket name. NAMESPACE: The namespace of the MySQLCluster. NAME: The name of the MySQLCluster.","breadcrumbs":"Commands » moco-backup » backup subcommand","id":"114","title":"backup subcommand"},"115":{"body":"Usage: moco-backup restore BUCKET SOURCE_NAMESPACE SOURCE_NAME NAMESPACE NAME YYYYMMDD-hhmmss BUCKET: The bucket name. SOURCE_NAMESPACE: The source MySQLCluster's namespace. SOURCE_NAME: The source MySQLCluster's name. NAMESPACE: The target MySQLCluster's namespace. NAME: The target MySQLCluster's name. YYYYMMDD-hhmmss: The point-in-time to restore data. e.g. 20210523-150423","breadcrumbs":"Commands » moco-backup » `restore subcommand","id":"115","title":"`restore subcommand"},"116":{"body":"moco-controller MySQL clusters Backup MySQL instance Scrape rules","breadcrumbs":"Metrics » Metrics","id":"116","title":"Metrics"},"117":{"body":"moco-controller provides the following kind of metrics in Prometheus format. Aside from the standard Go runtime and process metrics , it exposes metrics related to controller-runtime , MySQL clusters, and backups.","breadcrumbs":"Metrics » moco-controller","id":"117","title":"moco-controller"},"118":{"body":"All these metrics are prefixed with moco_cluster_ and have name and namespace labels. Name Description Type checks_total The number of times MOCO checked the cluster Counter errors_total The number of times MOCO encountered errors when managing the cluster Counter available 1 if the cluster is available, 0 otherwise Gauge healthy 1 if the cluster is running without any problems, 0 otherwise Gauge switchover_total The number of times MOCO changed the live primary instance Counter failover_total The number of times MOCO changed the failed primary instance Counter replicas The number of mysqld instances in the cluster Gauge ready_replicas The number of ready mysqld Pods in the cluster Gauge current_replicas The number of current replicas Gauge updated_replicas The number of updated replicas Gauge last_partition_updated The timestamp of the last successful partition update Gauge clustering_stopped 1 if the cluster is clustering stopped, 0 otherwise Gauge reconciliation_stopped 1 if the cluster is reconciliation stopped, 0 otherwise Gauge errant_replicas The number of mysqld instances that have errant transactions Gauge processing_time_seconds The length of time in seconds processing the cluster Histogram volume_resized_total The number of successful volume resizes Counter volume_resized_errors_total The number of failed volume resizes Counter statefulset_recreate_total The number of successful StatefulSet recreates Counter statefulset_recreate_errors_total The number of failed StatefulSet recreates Counter","breadcrumbs":"Metrics » MySQL clusters","id":"118","title":"MySQL clusters"},"119":{"body":"All these metrics are prefixed with moco_backup_ and have name and namespace labels. Name Description Type timestamp The number of seconds since January 1, 1970 UTC of the last successful backup Gauge elapsed_seconds The number of seconds taken for the last backup Gauge dump_bytes The size of compressed full backup data Gauge binlog_bytes The size of compressed binlog files Gauge workdir_usage_bytes The maximum usage of the working directory Gauge warnings The number of warnings in the last successful backup Gauge","breadcrumbs":"Metrics » Backup","id":"119","title":"Backup"},"12":{"body":"NOTE: This installation method requires cert-manager to be installed beforehand. To install the chart with the release name moco using a dedicated namespace(recommended): $ helm install --create-namespace --namespace moco-system moco moco/moco Specify parameters using --set key=value[,key=value] argument to helm install. Alternatively a YAML file that specifies the values for the parameters can be provided like this: $ helm install --create-namespace --namespace moco-system moco -f values.yaml moco/moco","breadcrumbs":"Getting started » Helm Chart » Installing the Chart","id":"12","title":"Installing the Chart"},"120":{"body":"For each mysqld instance, moco-agent exposes a set of metrics. Read github.com/cybozu-go/moco-agent/blob/main/docs/metrics.md for details. Also, if you give a set of collector flag names to spec.collectors of MySQLCluster, a sidecar container running mysqld_exporter exposes the collected metrics for each mysqld instance.","breadcrumbs":"Metrics » MySQL instance","id":"120","title":"MySQL instance"},"121":{"body":"This is an example kubernetes_sd_config for Prometheus to collect all MOCO & MySQL metrics. scrape_configs:\n- job_name: 'moco-controller' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_app_kubernetes_io_component,__meta_kubernetes_pod_container_port_name] action: keep regex: moco-system;moco-controller;metrics - job_name: 'moco-agent' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name,__meta_kubernetes_pod_container_port_name,__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name] action: keep regex: mysql;agent-metrics;moco-.* - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - job_name: 'moco-mysql' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name,__meta_kubernetes_pod_container_port_name,__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name] action: keep regex: mysql;mysqld-metrics;moco-.* - source_labels: [__meta_kubernetes_namespace] action: replace target_label: namespace - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance] action: replace target_label: name - source_labels: [__meta_kubernetes_pod_label_statefulset_kubernetes_io_pod_name] action: replace target_label: index regex: .*-([0-9]) - source_labels: [__meta_kubernetes_pod_label_moco_cybozu_com_role] action: replace target_label: role The collected metrics should have these labels: namespace: MySQLCluster's metadata.namespace name: MySQLCluster's metadata.name index: The ordinal of MySQL instance Pod","breadcrumbs":"Metrics » Scrape rules","id":"121","title":"Scrape rules"},"122":{"body":"","breadcrumbs":"Design notes » Design notes","id":"122","title":"Design notes"},"123":{"body":"The purpose of this document is to describe the backgrounds and the goals of MOCO. Implementation details are described in other documents.","breadcrumbs":"Design notes » Goals » Design notes","id":"123","title":"Design notes"},"124":{"body":"We are creating our own Kubernetes operator for clustering MySQL instances for the following reasons: Firstly, our application requires strict-compatibility to the traditional MySQL. Although recent MySQL provides an advanced clustering solution called group replication that is based on Paxos , we cannot use it because of various limitations from group replication . Secondly, we want to have a Kubernetes native and the simplest operator. For example, we can use Kubernetes Service to load-balance read queries to multiple replicas. Also, we do not want to support non-GTID based replications. Lastly, none of the existing operators could satisfy our requirements.","breadcrumbs":"Design notes » Goals » Motivation","id":"124","title":"Motivation"},"125":{"body":"Manage primary-replica style clustering of MySQL instances. The primary instance is the only instance that allows writes. Replica instances replicate data from the primary and are read-only. Support replication from an external MySQL instance. Support all the four transaction isolation levels. No split-brain. Allow large transactions. Upgrade the operator without restarting MySQL Pods. Safe and automatic upgrading of MySQL version. Support automatic primary selection and switchover. Support automatic failover. Backup and restore features. Support point-in-time recovery (PiTR). Tenant users can specify the following parameters: The version of MySQL instances. The number of processor cores for each MySQL instance. The amount of memory for each MySQL instance. The amount of backing storage for each MySQL instance. The number of replicas in the MySQL cluster. Custom configuration parameters. Allow CREATE / DROP TEMPORARY TABLE during a transaction.","breadcrumbs":"Design notes » Goals » Goals","id":"125","title":"Goals"},"126":{"body":"Support for older MySQL versions (5.6, 5.7) As a late comer, we focus our development effort on the latest MySQL. This simplifies things and allows us to use advanced mechanisms such as CLONE INSTANCE. Node fencing Fencing is a technique to safely isolated a failed Node. MOCO does not rely on Node fencing as it should be done externally. We can still implement failover in a safe way by configuring semi-sync parameters appropriately.","breadcrumbs":"Design notes » Goals » Non-goals","id":"126","title":"Non-goals"},"127":{"body":"MOCO creates and updates a StatefulSet and related resources for each MySQLCluster custom resource. This document describes how and when MOCO updates them. Reconciler versions The update policy of moco-agent container Clustering related resources StatefulSet When the StatefulSet is not updated Status about StatefulSet Secrets Certificate Service ConfigMap PodDisruptionBudget ServiceAccount Backup and restore related resources CronJob Job Status of Reconcliation","breadcrumbs":"Design notes » Reconciliation » How MOCO reconciles MySQLCluster","id":"127","title":"How MOCO reconciles MySQLCluster"},"128":{"body":"MOCO's reconciliation routine should be consistent to avoid frequent updates. That said, we may need to modify the reconciliation process in the future. To avoid updating the StatefulSet, MOCO has multiple versions of reconcilers. For example, if a MySQLCluster is reconciled with version 1 of the reconciler, MOCO will keep using the version 1 reconciler to reconcile the MySQLCluster. If the user edits MySQLCluster's spec field, MOCO can reconcile the MySQLCluster with the latest reconciler, for example version 2, because the user shall be ready for mysqld restarts.","breadcrumbs":"Design notes » Reconciliation » Reconciler versions","id":"128","title":"Reconciler versions"},"129":{"body":"We shall try to avoid updating moco-agent as much as possible.","breadcrumbs":"Design notes » Reconciliation » The update policy of moco-agent container","id":"129","title":"The update policy of moco-agent container"},"13":{"body":"Key Type Default Description replicaCount number 2 Number of controller replicas. image.repository string \"ghcr.io/cybozu-go/moco\" MOCO image repository to use. image.pullPolicy string IfNotPresent MOCO image pulling policy. image.tag string {{ .Chart.AppVersion }} MOCO image tag to use. imagePullSecrets list [] Secrets for pulling MOCO image from private repository. resources object {\"requests\":{\"cpu\":\"100m\",\"memory\":\"20Mi\"}} resources used by moco-controller. crds.enabled bool true Install and update CRDs as part of the Helm chart. extraArgs list [] Additional command line flags to pass to moco-controller binary. nodeSelector object {} nodeSelector used by moco-controller. affinity object {} affinity used by moco-controller. tolerations list [] tolerations used by moco-controller. topologySpreadConstraints list [] topologySpreadConstraints used by moco-controller. priorityClassName string \"\" PriorityClass used by moco-controller.","breadcrumbs":"Getting started » Helm Chart » Values","id":"13","title":"Values"},"130":{"body":"The figure below illustrates the overview of resources related to clustering MySQL instances. Overview of clustering related resources","breadcrumbs":"Design notes » Reconciliation » Clustering related resources","id":"130","title":"Clustering related resources"},"131":{"body":"MOCO tries not to update the StatefulSet frequently. It updates the StatefulSet only when the update is a must. The conditions for StatefulSet update The StatefulSet will be updated when: Some fields under spec of MySQLCluster are modified. my.cnf for mysqld is updated. the version of the reconciler used to reconcile the StatefulSet is obsoleted. the image of moco-agent given to the controller is updated. the image of mysqld_exporter given to the controller is updated.","breadcrumbs":"Design notes » Reconciliation » StatefulSet","id":"131","title":"StatefulSet"},"132":{"body":"the image of fluent-bit given to the controller is changed. because the controller does not depend on fluent-bit. The fluent-bit sidecar container is updated only when some fields under spec of MySQLCluster are modified.","breadcrumbs":"Design notes » Reconciliation » When the StatefulSet is not updated","id":"132","title":"When the StatefulSet is not updated"},"133":{"body":"In MySQLCluster.Status.Condition, there is a condition named StatefulSetReady. This indicates the readieness of StatefulSet. The condition will be True when the rolling update of StatefulSet completely finishes.","breadcrumbs":"Design notes » Reconciliation » Status about StatefulSet","id":"133","title":"Status about StatefulSet"},"134":{"body":"MOCO generates random passwords for users that MOCO uses to access MySQL. The generated passwords are stored in two Secrets. One is in the same namespace as moco-controller, and the other is in the namespace of MySQLCluster.","breadcrumbs":"Design notes » Reconciliation » Secrets","id":"134","title":"Secrets"},"135":{"body":"MOCO creates a Certificate in the same namespace as moco-controller to issue a TLS certificate for moco-agent. After cert-manager issues a TLS certificate and creates a Secret for it, MOCO copies the Secret to the namespace of MySQLCluster. For details, read security.md .","breadcrumbs":"Design notes » Reconciliation » Certificate","id":"135","title":"Certificate"},"136":{"body":"MOCO creates three Services for each MySQLCluster, that is: A headless Service, required for every StatefulSet A Service for the primary mysqld instance A Service for replica mysql instances The Services' labels, annotations, and spec fields can be customized with MySQLCluster's spec.primaryServiceTemplate and spec.replicaServiceTemplate field. The spec.primaryServiceTemplate configures the Service for the primary mysqld instance and the spec.replicaServiceTemplate configures the Service for the replica mysqld instances. The following fields in Service spec may not be customized, though. clusterIP selector The ports field in the Service spec is also customizable. However, for the mysql and mysqlx ports, MOCO overwrites the fixed value to the port, protocol and targetPort fields.","breadcrumbs":"Design notes » Reconciliation » Service","id":"136","title":"Service"},"137":{"body":"MOCO creates and updates a ConfigMap for my.cnf. The name of this ConfigMap is calculated from the contents of my.cnf that may be changed by users. MOCO deletes old ConfigMaps of my.cnf after a new ConfigMap for my.cnf is created. If the cluster does not disable a sidecar container for slow query logs, MOCO creates a ConfigMap for the sidecar.","breadcrumbs":"Design notes » Reconciliation » ConfigMap","id":"137","title":"ConfigMap"},"138":{"body":"MOCO creates a PodDisruptionBudget for each MySQLCluster to prevent too few semi-sync replica servers. The spec.maxUnavailable value is calculated from MySQLCluster's spec.replicas as follows: `spec.maxUnavailable` = floor(`spec.replicas` / 2) If spec.replicas is 1, MOCO does not create a PDB.","breadcrumbs":"Design notes » Reconciliation » PodDisruptionBudget","id":"138","title":"PodDisruptionBudget"},"139":{"body":"MOCO creates a ServiceAccount for Pods of the StatefulSet. The ServiceAccount is not bound to any Roles/ClusterRoles.","breadcrumbs":"Design notes » Reconciliation » ServiceAccount","id":"139","title":"ServiceAccount"},"14":{"body":"You can use the helm template command to render manifests. $ helm template --namespace moco-system moco moco/moco","breadcrumbs":"Getting started » Helm Chart » Generate Manifests","id":"14","title":"Generate Manifests"},"140":{"body":"See backup.md for the overview of the backup and restoration mechanism.","breadcrumbs":"Design notes » Reconciliation » Backup and restore related resources","id":"140","title":"Backup and restore related resources"},"141":{"body":"This is the only resource created when backup is enabled for MySQLCluster. If the backup is disabled, the CronJob is deleted.","breadcrumbs":"Design notes » Reconciliation » CronJob","id":"141","title":"CronJob"},"142":{"body":"To restore data from a backup, MOCO creates a Job. MOCO deletes the Job after the Job finishes successfully. If the Job fails, MOCO leaves the Job.","breadcrumbs":"Design notes » Reconciliation » Job","id":"142","title":"Job"},"143":{"body":"In MySQLCluster.Status.Condition, there is a condition named ReconcileSuccess. This indicates the status of reconcilation. The condition will be True when the reconcile function successfully finishes.","breadcrumbs":"Design notes » Reconciliation » Status of Reconcliation","id":"143","title":"Status of Reconcliation"},"144":{"body":"For each MySQLCluster , MOCO creates and maintains a set of mysqld instances. The set contains one primary instance and may contain multiple replica instances depending on the spec.replicas value of MySQLCluster. This document describes how MOCO does this job safely. Terminology Prerequisites Limitations Possible states MySQLCluster Pod MySQL data Invariants The maintenance flow Gather the current status Update status of MySQLCluster Determine what MOCO should do for the cluster","breadcrumbs":"Design notes » Clustering » How MOCO maintains MySQL clusters","id":"144","title":"How MOCO maintains MySQL clusters"},"145":{"body":"Replication: GTID-based replication between mysqld instances. Cluster: a group of mysqld instances that replicate data between them. Primary (instance): a single source instance of mysqld in a cluster. Replica (instance): a read-only instance of mysqld that synchronizes data with the primary instance. Intermediate primary: a special primary instance that replicates data from an external mysqld. Errant transaction : a transaction that exists only on a replica instance. Errant replica: a replica instance that has errant transactions. Switchover: operation to change a live primary to a replica and promote a replica to the new primary. Failover: operation to replace a dead primary with a replica.","breadcrumbs":"Design notes » Clustering » Terminology","id":"145","title":"Terminology"},"146":{"body":"MySQLCluster allows positive odd numbers for spec.replicas value. If 1, MOCO runs a single mysqld instance without configuring replication. If 3 or greater, MOCO chooses a mysqld instance as a primary, writable instance and configures all other instances as replicas of the primary instance. status.currentPrimaryIndex in MySQLCluster is used to record the current chosen primary instance. Initially, status.currentPrimaryIndex is zero and therefore the index of the primary instance is zero. As a special case, if spec.replicationSourceSecretName is set for MySQLCluster, the primary instance is configured as a replica of an external MySQL server. In this case, the primary instance will not be writable. We call this type of primary instance intermediate primary . If spec.replicationSourceSecretName is not set, MOCO configures semisynchronous replication between the primary and replicas. Otherwise, the replication is asynchronous. For semi-synchronous replication, MOCO configures rpl_semi_sync_master_timeout long enough so that it never degrades to asynchronous replication. Likewise, MOCO configures rpl_semi_sync_master_wait_for_slave_count to (spec.replicas - 1 / 2) to make sure that at least half of replica instances have the same commit as the primary. e.g., If spec.replicas is 5, rpl_semi_sync_master_wait_for_slave_count will be set to 2. MOCO also disables relay_log_recovery because enabling it would drop the relay logs on replicas. mysqld always starts with super_read_only=1 to prevent erroneous writes, and with skip_replica_start to prevent misconfigured replication. moco-agent , a sidecar container for MOCO, initializes MySQL users and plugins. At the end of the initialization, it issues RESET MASTER | RESET BINARY LOGS AND GTIDS to clear executed GTID set . moco-agent also provides a readiness probe for mysqld container. If a replica instance does not start replication threads or is too delayed to execute transactions, the container and the Pod will be determined as unready.","breadcrumbs":"Design notes » Clustering » Prerequisites","id":"146","title":"Prerequisites"},"147":{"body":"Currently, MOCO does not re-initialize data after the primary instance fails. After failover to a replica instance, the old primary may have errant transactions because it may recover unacknowledged transactions in its binary log. This is an inevitable limitation in MySQL semi-synchronous replication. If this happens, MOCO detects the errant transaction and will not allow the old primary to rejoin the cluster as a replica. Users need to delete the volume data (PersistentVolumeClaim) and the pod of the old primary to re-initialize it.","breadcrumbs":"Design notes » Clustering » Limitations","id":"147","title":"Limitations"},"148":{"body":"","breadcrumbs":"Design notes » Clustering » Possible states","id":"148","title":"Possible states"},"149":{"body":"MySQLCluster can be one of the following states. The initial state is Cloning if spec.replicationSourceSecretName is set, or Restoring if spec.restore is set. Otherwise, the initial state is Incomplete . Note that, if the primary Pod is ready , the mysqld is assured writable. Likewise, if a replica Pod is ready, the mysqld is assured read-only and running replication threads w/o too much delay. Healthy All Pods are ready. All replicas have no errant transactions. All replicas are read-only and connected to the primary. For intermediate primary instance, the primary works as a replica for an external mysqld and is read-only. Cloning spec.replicationSourceSecretName is set. status.cloned is false. The cloning result exists and is not \"Completed\" or there is no cloning result and the instance has no data. (note: if the primary has some data and has no cloning result, the instance was used to be a replica and then promoted to the primary.) Restoring spec.restore is set. status.restoredTime is not set. Degraded The primary Pod is ready and does not lose data. For intermediate primary instance, the primary works as a replica for an external mysqld and is read-only. Half or more replicas are ready, read-only, connected to the primary, and have no errant transactions. For example, if spec.replicas is 5, two or more such replicas are needed. At least one replica has some problems. Failed The primary instance is not running or lost data. More than half of replicas are running and have data without errant transactions. For example, if spec.replicas is 5, three or more such replicas are needed. Lost The primary instance is not running or lost data. Half or more replicas are not running or lost data or have errant transactions. Incomplete None of the above states applies. MOCO can recover the cluster to Healthy from Degraded , Failed , or Incomplete if all Pods are running and there are no errant transactions . MOCO can recover the cluster to Degraded from Failed when not all Pods are running. Recovering from Failed is called failover . MOCO cannot recover the cluster from Lost . Users need to restore data from backups.","breadcrumbs":"Design notes » Clustering » MySQLCluster","id":"149","title":"MySQLCluster"},"15":{"body":"","breadcrumbs":"Getting started » Helm Chart » CRD considerations","id":"15","title":"CRD considerations"},"150":{"body":"mysqld is run as a container in a Pod. Therefore, MOCO needs to be aware of the following conditions. Missing: the Pod does not exist. Exist: the Pod exists and not Terminating or Demoting . Terminating: The Pod exists and metadata.deletionTimestamp is not null. Demoting: The Pod exists and has moco.cybozu.com/demote: true annotation. If there are missing Pods, MOCO does nothing for the MySQLCluster. If a primary instance Pod is Terminating or Demoting , MOCO controller changes the primary to one of the replica instances. This operation is called switchover .","breadcrumbs":"Design notes » Clustering » Pod","id":"150","title":"Pod"},"151":{"body":"MOCO checks replica instances whether they have errant transactions compared to the primary instance. If it detects such an instance, MOCO records the instance with MySQLCluster and excludes it from the cluster. The user needs to delete the Pod and the volume manually and let the StatefulSet controller to re-create them. After a newly initialized instance gets created, MOCO will allow it to rejoin the cluster.","breadcrumbs":"Design notes » Clustering » MySQL data","id":"151","title":"MySQL data"},"152":{"body":"By definition, the primary instance recorded in MySQLCluster has no errant transactions. It is always the single source of truth. Errant replicas are not treated as ready even if their Pod status is ready.","breadcrumbs":"Design notes » Clustering » Invariants","id":"152","title":"Invariants"},"153":{"body":"MOCO runs the following infinite loop for each MySQLCluster. It stops when MySQLCluster resource is deleted. Gather the current status Update status of MySQLCluster Determine what MOCO should do for the cluster If there is nothing to do, wait a while and go to 1 Do the determined operation then go to 1 Read the following sub-sections about 1 to 3.","breadcrumbs":"Design notes » Clustering » The maintenance flow","id":"153","title":"The maintenance flow"},"154":{"body":"MOCO gathers the information from kube-apiserver and mysqld as follows: MySQLCluster resource Pod resources If some of the Pods are missing, MOCO does nothing. mysqld SHOW REPLICAS (on the primary) SHOW REPLICA STATUS (on the replicas) Global variables such as gtid_executed or super_read_only Result of CLONE from performance_schema.clone_status table If MOCO cannot connect to an instance for a certain period, that instance is determined as failed.","breadcrumbs":"Design notes » Clustering » Gather the current status","id":"154","title":"Gather the current status"},"155":{"body":"In this phase, MOCO updates status field of MySQLCluster as follows: Determine the current MySQLCluster state. Add or update type=Initialized condition to status.conditions as True if the cluster state is not Cloning. otherwise, False. Add or update type=Available condition to status.conditions as True if the cluster state is Healthy or Degraded. otherwise, False. Add or update type=Healthy condition to status.conditions as True if the cluster state is Healthy. otherwise, False. The Reason field is set to the cluster state such as \"Failed\" or \"Incomplete\". Set the number of ready replica Pods to status.syncedReplicas. Add newly found errant replicas to status.errantReplicaList. Remove re-initialized and/or no-longer errant replicas from status.errantReplicaList Set status.errantReplicas to the length of status.errantReplicaList. Set status.cloned to true if spec.replicationSourceSecret is not nil and the state is not Cloning.","breadcrumbs":"Design notes » Clustering » Update status of MySQLCluster","id":"155","title":"Update status of MySQLCluster"},"156":{"body":"The operation depends on the current cluster state. The operation and its result are recorded as Events of MySQLCluster resource. cf. Application Introspection and Debugging Healthy If the primary instance Pod is Terminating or Demoting, switch the primary instance to another replica. Otherwise, just wait a while. The switchover is done as follows. It takes at least several seconds for a new primary to become writable. Make the primary instance super_read_only=1. Kill all existing connections except ones from localhost and ones for MOCO. Wait for a replica to catch up the executed GTID set of the primary instance. Set status.currentPrimaryIndex to the replica's index. If the old primary is Demoting, remove moco.cybozu.com/demote annotation from the Pod. Cloning Execute CLONE INSTANCE on the intermediate primary instance to clone data from an external MySQL instance. If the cloning goes successful, do the same as Intermediate case. Restoring Do nothing. Degraded First, check if the primary instance Pod is Terminating or Demoting, and if it is, do the switchover just like Healthy case. Then, do the same as Intermediate case to try to fix the problems. It is not possible to recover the cluster to Healthy if there are errant or stopped replicas, though. Failed MOCO chooses the most advanced instance as the new primary instance. The most advanced means that its retrieved GTID set is the superset of all other replicas except for those have errant transactions. To prevent accidental writes to the old primary instance (so-called split-brain), MOCO stops replication IO_THREAD for all replicas. This way, the old primary cannot get necessary acks from replicas to write further transactions. The failover is done as follows: Stop IO_THREAD on all replicas. Choose the most advanced replica as the new primary. Errant replicas recorded in MySQLCluster are excluded from the candidates. Wait for the replica to execute all retrieved GTID set. Update status.currentPrimaryIndex to the new primary's index. Lost There is nothing can be done. Intermediate On the primary that was an intermediate primary, wait for all the retrieved GTID set to be executed. Start replication between the primary and non-errant replicas. If a replication has no data, MOCO clones the primary data to the replica first. Stop replication of errant replicas. Set super_read_only=1 for replica instances that are writable. Adjust moco.cybozu.com/role label to Pods according to their roles. For errant replicas, the label is removed to prevent users from reading inconsistent data. Finally, make the primary mysqld writable if the primary is not an intermediate primary.","breadcrumbs":"Design notes » Clustering » Determine what MOCO should do for the cluster","id":"156","title":"Determine what MOCO should do for the cluster"},"157":{"body":"This document describes how MOCO takes a backup of MySQLCluster data and restores a cluster from a backup. Overview Design goals Implementation Backup file keys Timestamps Backup Restore Caveats Considered options Why do we use S3-compatible object storage to store backups? What object storage is supported? Why do we use Jobs for backup and restoration? Why do we prefer mysqlsh to mysqldump? Why don't we do continuous backup?","breadcrumbs":"Design notes » Backup and restore » Backup and restore","id":"157","title":"Backup and restore"},"158":{"body":"A MySQLCluster can be configured to take backups regularly by referencing a BackupPolicy in spec.backupPolicyName. For each MySQLCluster associated with a BackupPolicy, moco-controller creates a CronJob. The CronJob creates a Job to take a full backup periodically. The Job also takes a backup of binary logs for Point-in-Time Recovery (PiTR) . The backups are stored in a S3-compatible object storage bucket. This figure illustrates how MOCO takes a backup of a MySQLCluster. Backup moco-controller creates a CronJob and Role/RoleBinding to allow access to MySQLCluster for the Job Pod. At each configured interval, CronJob creates a Job. The Job dumps all data from a mysqld using MySQL shell's dump instance utility . The Job creates a tarball of the dumped data and put it in a bucket of S3 compatible object storage. The Job also dumps binlogs since the last backup and put it in the same bucket (with a different name, of course). The Job finally updates MySQLCluster status to record the last successful backup. To restore from a backup, users need to create a new MySQLCluster with spec.restore filled with necessary information such as the bucket name of the object storage, the object key, and so on. The next figure illustrates how MOCO restores MySQL cluster from a backup. Restore moco-controller creates a Job and Role/RoleBinding for restoration. The Job downloads a tarball of dumped files of the specified backup. The Job loads data into an empty mysqld using MySQL shell's dump loading utility . If the user wanted to restore data at a point-in-time, the Job downloads saved binlogs. The Job applies binlogs up to the specified point-in-time using mysqlbinlog . The Job finally updates MySQLCluster status to record the restoration time.","breadcrumbs":"Design notes » Backup and restore » Overview","id":"158","title":"Overview"},"159":{"body":"Must: Users must be able to configure different backup policies for each MySQLCluster. Users must be able to restore MySQL data at a point-in-time from backups. Users must be able to restore MySQL data without the original MySQLCluster resource. moco-controller must export metrics about backups. Should: Backup data should be compressed to save the storage space. Backup data should be stored in an object storage. Backups should be taken from a replica instance as much as possible. These \"should's\" are mostly in terms of money or performance.","breadcrumbs":"Design notes » Backup and restore » Design goals","id":"159","title":"Design goals"},"16":{"body":"MOCO Helm Chart installs or updates CRDs by default. If you want to manage CRDs on your own, turn off the crds.enabled parameter.","breadcrumbs":"Getting started » Helm Chart » Installing or updating CRDs","id":"16","title":"Installing or updating CRDs"},"160":{"body":"","breadcrumbs":"Design notes » Backup and restore » Implementation","id":"160","title":"Implementation"},"161":{"body":"Backup files are stored in an object storage bucket with the following keys. Key for a tarball of a fully dumped MySQL: moco/