Cloudwatch Alarms and Scaling Policies exposed to end user

[mootpt]

I began creating some Alarms, scaling policies, and what have you to ensure CircleCI workers scaled based on the threshold end users provided, but later stumbled across: https://circleci.com/docs/enterprise/cloudwatch/

It wasn't clear if CircleCI Enterprise is creating any Alarms under the hood or if it simply providing metrics in a particular namespace for monitoring health. It doesn't explicitly state that the alarms are being created, so I assume it's just the metrics (e.g ContainersAvailable).

All that said, it might be worth throwing some Alarms and scaling policies into the repo with a simple conditional for turning it on and off. Also, I would suggest exposing the threshold for said Alarms as a variable to the end user. Similarly this could be exposed for nomad cluster as well.

Something like:

resource "aws_cloudwatch_metric_alarm" "workers_out" {
  count               = "${var.enable_cw ? 1 : 0}"
  alarm_name          = "workers-scaling-out-alarm"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "ContainersAvailable"
  namespace           = "CircleCIEnterprise"
  period              = "300"
  statistic           = "Average"
  threshold           = "${var.worker_so_threshold}"

  dimensions {
    QueueName = "${aws_autoscaling_group.builder_asg.name}"
  }

  alarm_description = "This metric monitors the number containers available on the workers"
  alarm_actions     = ["${aws_autoscaling_policy.workers_out.arn}"]
}

resource "aws_autoscaling_policy" "workers_out" {
  count                  = "${var.enable_cw ? 1 : 0}"
  name                   = "workers-scaling-policy"
  scaling_adjustment     = 1
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 300
  autoscaling_group_name = "${aws_autoscaling_group.builder_asg.name}"
}

[Twang130]

Code Issues 1 Pull requests 11 Pulse
templates/ha-circle-customize.sh.tpl

#!/bin/bash
IP="$$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)"

cat << EOF > /etc/circle-installation-customizations
MONGO_BASE_URI=mongodb://circle:${mongo_password}@$${IP}:27017
export CIRCLE_SECRETS_MONGODB_MAIN_URI="$$MONGO_BASE_URI/circle_ghe"
export CIRCLE_SECRETS_MONGODB_ACTION_LOGS_URI="$$MONGO_BASE_URI/circle_ghe"
export CIRCLE_SECRETS_MONGODB_BUILD_STATE_URI="$$MONGO_BASE_URI/build_state_dev_ghe"
export CIRCLE_SECRETS_MONGODB_CONTAINERS_URI="$$MONGO_BASE_URI/containers_dev_ghe"
export CIRCLE_SECRETS_MONGODB_REMOTE_CONTAINERS_URI="$$MONGO_BASE_URI/remote_containers_dev_ghe"

Postgres DB

export CIRCLE_SECRETS_POSTGRES_MAIN_URI='postgres://circle:${postgres_password}@$${IP}:5432/circle'
export CIRCLE_SECRETS_POSTGRES_DOMAIN_URI='postgres://circle:${postgres_password}@$${IP}:5432/circle'
export CIRCLE_SECRETS_POSTGRES_BUILD_QUEUE_URI='postgres://circle:${postgres_password}@$${IP}:5432/circle'

Vault

export VAULT__SCHEME="https"
export VAULT__HOST="$${IP}"
export VAULT__PORT=8200
export VAULT__CLIENT_TOKEN=""
export VAULT__TRANSIT_MOUNT="transit"
EOF
Desktop version