Skip to content

Latest commit

 

History

History
230 lines (177 loc) · 11.5 KB

install.md

File metadata and controls

230 lines (177 loc) · 11.5 KB

Installation

This CI/CD pipeline example can be installed by running various cloud-formation scripts listed below. Note that rather than using a single script this example uses multiple scripts to establish a separation of concerns (networking, security, application) to reflect a separation of these concerns that will be in effect in most organizations. Here the separation is used to demonstrate a scalable solution to this separation of concerns. In a 'real world' secnario these cloud-formation scripts would be separated into separate git repositories. However, here they are combined for convenience.

Overall Approach

To provide scalability and consistency, all cloud-formation scripts have a set of common parameters, such as: System name, Environment name, AWS Region, and (optional) S3 Provisioning Bucket. name. Furthermore they may have specific values, such as CIDRBlock, Instance AMI and Instance Size. Those scripts run additional EC2 initialization/provisioning use an S3 bucket to share information, e.g. files to be installed on an EC2 instance,

The various cloud formation stacks follow a naming conventions that is adhered to when creating objects in the S3 bucket. The structure is used to ensure access rights are set approrpiately for the AWS IAM Policies that are generated by the security scripts.

The expected 'directory' structure is as follows:

      <bucket>/cloud-formation/<system>/<environment>/<stack>/<stack-resources>

One of the advantage of taking this structured approach (common parameters, S3 structuring) is that not only the application pipeline can be fully automated, the infrastructure can easily be automated using a development pipeline as well, building on the Automated Installation script.

Prerequisites:

  1. You need to have an AWS account with rights to create/delete/modify AWS S3, VPC, IAM, EC2, and ECS resources.
  2. Create (or import) an AWS EC2 keypair with the name <system>-shared-jenkins and save the private key. where Jenkins is the value of the parameter given to the cloud formation scripts.
  3. Create (or import) an AWS EC2 keypair with the name <system>-dev-helloworld and save the private key.
  4. Create (or import) an AWS EC2 keypair with the name <system>-test-helloworld and save the private key.
  5. Create (or import) an AWS EC2 keypair with the name <system>-prod-helloworld and save the private key.
  6. Create an AWS S3 bucket for AWS resource provisioning.

Note that:

  • The name is relevant, because the cloud-formation scripts, rather than having the named passed to it, internally builds the key pair name from the System and Environment parameters. The advantage is that this enforces a consistent naming sructure and eliminates the need for passing the key pair names.

  • The default values for is dso (default for cloud-formation parameters), so when you plan to use the default create a key pairs using dso for , e.g. dso-shared-jenkins.

    EC2 Key Pairs Example

Steps

Automated installation

  1. Install the AWS-CLI

  2. Setup programmatic access

  3. Setup an AWS profile

  4. Run install.sh

      bin/install.sh --bucket your-bucket-name --profile your-aws-profile-name

    Optionally add --system your-system-name if not using the default (i.e. dso)

  5. Continue with Jenkins Configuration

Manual Installation

Install common AWS Resources:

To install the common AWS VPC and IAM resources run the following cloud-formation scripts:

  1. Run the cloud-formation script in network/shared using the listed policy
  2. Run the cloud-formation script in security/shared using the listed policy

Install the helloworld application on EC2 instances for a dev, test, and prod environment:

Repeat these steps passing dev, test, and prod as the parameter values for the Environment parameter in the various stacks:

  1. Run the cloud-formation stack in network/hello-world using the listed policy Pass a value for parameter:

    • Environment. Either dev, test, or prod
    • ApplicationCIDRBlock. This CIDR Block must fall within the VPC CIDRBlock, e.g. 172.27.x.0/24 (where x is 10 for dev, 20 for test and 30 for prod).

  2. Run the cloud-formation stack in security/hello-world using the listed policy Pass a value for parameter:

    • Environment. Either dev, test, or prod
    • VPCCIDRBlock that matches the earlier value (see common AWS Resources).
    • PrivilegedCIDRx (for x=1..5) to enable access to resources on the private Jenkins subnet. At least one privileged address is needed to be able to obtain the initial Jenkins administrator password.

  3. Run the cloud-formation stack in helloworld/app using the listed policy Pass a value for parameter:

    • Environment. Either dev, test, or prod.
    • SecurityContext. Match the value of Environment.

    To verify that application instance is install, browse to http://<PublicIP output value of Stack>. Initially the provisioning scripts sets up a web application using the kitematic/hello-world-nginx docker image. Once the pipeline is executed this application will be automatically replaced with the actual hello world application using the [Docker image[../Dockerfile)]

Once completed you should something similar to the following in your AWS Cloud Formation console:

Cloud Formation stack

Install Jenkins on EC2 instance:

To install the the AWS resources for the Jenkins instance run the following cloud-formation scripts and subsequently configure Jenkins:

  1. Run the cloud-formation script in network/jenkins using the listed policy

  2. Run the cloud-formation script in security/jenkins using the listed policy Pass a value for parameters:

    • PrivilegedCIDRx (for x=1..5) to enable access to resources on the private Jenkins subnet. At least one privileged address is needed to be able to obtain the initial Jenkins administrator password.

  3. Run the cloud-formation script in jenkins/app using the listed policy Jenkins is configured to run inside Docker. Furthermore, in this example setup the Jenkins instance also runs SonarQube inside. Docker Pass a value to parameters:

    • ProvisioningBucket The name ofn AWS S3 Bucket you created in the prerequisites. Create the forementioned 'folder' structure in ths S3 bucket using <your bucket>/cloud-formation/<system>/shared/jenkins-app and upload file etc-init-d-jenkins.sh into this 'folder'

      S3 Example

  4. Continue with Jenkins Configuration

Configure Jenkins and the Hello World Pipeline

Once the infrastructure is created Jenkins must be configured using the following steps:

  1. Get the initial administrator password: a. When using the automated installation, the password wil be available in the stack outputs under Output JenkinsInitialPassword the value shows a json formatted value: {"<instance-id>":"<password>"} b. When using the manual installation:

    • Login to the Jenkins EC2 instance with the jenkins key ssh -i dso-shared-jenkins <jenkins-public-ip>
    • Get the key from /var/jenkins_home/secrets/initialAdminPassword

  2. Browse to Jenkins at http://<jenkins-public-ip> and use the initial password to login

  3. Choose to install the suggested plugins

  4. Configure a new admin user, i.e. do not continue with the initial admin, such that the exposed initial password is no longer valid.

  5. (Optional) Setup matrix-based security under Manage Jenkins > Configure Global Security > Access Control > Authorization:

    • Select Matric-based security
    • Add your newly create admin user and assign all privileges (or you will no longer be able to manage Jenkins) By default Jenkins Authozization is setup with Logged-in users can do anything.

  6. Next install the SSH and SSH Agent plugins under Manage Jenkins > Manage Plugins

  7. Create the following credentials under Credentials > global

    • (Optional, if repo is private) A valid Git credential (ID: *any*)
    • An SSH Username with private key for the hello-world dev app instance (ID: dso-dev-helloworld, Username: ec2-user) using the private key from the matching EC2 Keypair
    • An SSH Username with private key for the hello-world test app instance (ID: dso-test-helloworld, Username: ec2-user) using the private key from the matching EC2 Keypair
    • An SSH Username with private key for the hello-world prod app instance (ID: dso-prod-helloworld, Username: ec2-user) using the private key from the matching EC2 Keypair

    f. Add users as appropriate under Manage Jenkins > Manage Users

  8. Setup the multi-pipeline job for the Hello World Application

    a. Create the job using New Item with item name hello-world-app and the Multibranch pipeline option b. Specify a Hello World application as the Display Name (Optional) c. Add your GitHub repo as a source using the Add Source button with the GitHub option - Select the github Credentials ID you created earlier (if repo is private) - Enter your organization's name, e.g. boozallen - Select the repository, e.g. devsecops-example-helloworld - Select the Save button

    Your Jenkins pipeline should now start to discover the branches in the designated repository, and once it discovers the Jenkinsfile, start to build all branches found. If you have multiple branches and want to limit building to the master branch you can add a Filter by Name Behavior before selecting Save