Installing CloudBees Jenkins Enterprise 1.x on Amazon Web Service (AWS)


Prerequisites

Important
These requirements must be satisfied before CloudBees Jenkins Enterprise 1.x can be installed on AWS.

CloudBees Jenkins Enterprise 1.x runs on a Mesos cluster. Mesos facilities and the cje command are used to maintain, scale, and update CloudBees Jenkins Enterprise 1.x.

Controller Workstation

The CloudBees Jenkins Enterprise 1.x installer requires a Linux operating system. The machine on which the installation script runs must have the following:

  • Python 2.7

  • Network access to eventual CloudBees Jenkins Enterprise Cluster VMs

  • Disk Backup for post-installation configuration payload

Python 2.7

You can verify your version of Python by running:

$ python -V
Python 2.7.12

In addition, the following infrastructure-related pieces have to be in place before the installation starts, if you are planning to use them:

Domain Name

Work with your IT/Operations department to create a new domain name for the CloudBees Jenkins Enterprise Cluster.

NFS Server

If you are planning to use NFS as common storage, work with your IT/Operations department to create a NFS server and a mount point.

Accurate system time

The Amazon Web Services (AWS) API calls require accurate time on the calling computer. If the time is sufficiently incorrect, the AWS API calls will fail with an authentication error.

Assure your system clock is accurate by installing and configuring an NTP client on the calling computer. Refer to operating system documentation for instructions to configure your computer to use a time server.

Cluster Environment: Amazon Web Services

CloudBees Jenkins Enterprise works with all of the Amazon Elastic Compute Cloud (Amazon EC2) instance types, although some of them only work as designed in conjunction with a virtual private cloud.

AWS Credentials

There are two ways to configure an AWS environment: with or without support for EBS Storage. Regardless of which way you choose, AWS configuration requires two different sets of credentials.

First, you will need a set of credentials that will be used to setup the CloudBees Jenkins Enterprise environment. We recommend creating an IAM user with a dedicated policy that will allow for the actions needed in order to create the environment.

An example IAM policy you may use for this user is shown below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "NotAction": [
        "iam:*",
        "sts:*"
      ],
      "Resource": "*"
    }
  ]
}

Second, you will need a set of credentials that will be used for the worker process that is responsible for volume and snapshot management. There are two ways to furnish these credentials.

By default, the same set of credentials used above will also be used for this service. These credentials will be passed into the container at install time and used for the lifetime of the service. No extra configuration needs to be performed in order to use this configuration.

Using IAM Roles

Another way to achieve this is to specify an instance profile to be used by the worker instances. This is specified in the $PROJECT/cluster-init.config file in the [aws] section alongside the above credentials. The variable name is worker_instance_profile.

In order to use this instance profile, you must have created a role in the IAM settings. The role must have a trust policy defined that allows it to be launched via the EC2 service. For example:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

In addition, because the credentialed user above will start the worker instance with this role, it must explicitly have the iam:PassRole ability, which makes our initial IAM policy a bit longer:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "NotAction": [
                "iam:*",
                "sts:*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::123412341234:role/yourrole"
        }
    ]
}
Note
If you choose to use IAM Roles you will need to add an entry in your .aws/config, see AWS credential storage for more information.

Now the worker process will not have hard-coded credentials, but instead will use the EC2 metadata service to get a temporary set of credentials.

These credentials will be used to manage: * lifecycle of EBS volumes and snapshots * data on S3 buckets

Last, this worker role needs to have a policy stating what kinds of AWS operations it can perform. A minimum set of AWS operations in policy form you can apply is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:CreateTags",
                "ec2:CreateSnapshot",
                "ec2:CreateVolume",
                "ec2:AttachVolume",
                "ec2:DeleteVolume",
                "ec2:DetachVolume",
                "ec2:DeleteSnapshot",
                "ec2:DescribeSnapshots",
                "ec2:DescribeVolumes",
                "ec2:DescribeInstances",
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersionAcl",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload",
                "s3:RestoreObject"
            ],
            "Resource": [
                "arn:aws:s3:::cje-*"
            ],
            "Effect": "Allow"
        }
    ]
}

cje-* needs to be adjusted to match <cluster-name>-*.

Note
The ecr actions are only needed if an EC2 Container Registry is used.

For help managing AWS permissions refer to AWS - Manage Permissions and Policies.

AWS Credential Storage

Prior to version 1.5.0, AWS credentials were stored in cluster-init.secrets.

Credentials may continue to be stored and used from within this file. However CloudBees Jenkins Enterprise now contains support for using credentials that are stored within the conventional AWS tool credential locations. By default this file is ~/.aws/credentials, but can be overwritten with the AWS_CREDENTIAL_FILE environment variable.

In order to use credentials stored in the shared credentials file, you must specify the credential_profile in the cluster-init.config.

If you are using AWS IAM Roles

CloudBees Jenkins Enterprise allows you to use IAM Roles to control access to AWS. By configuring role assumption parameters, and specifying the role profile name in the credential_profile config entry, CloudBees Jenkins Enterprise will attempt to change roles before performing AWS operations. This can allow you to provide a shared role for CloudBees Jenkins Enterprise administration access while handing out more limited capability credentials directly to end CloudBees Jenkins Enterprise administrators.

When making use of the credential_profile option vs. using the cluster-init.secrets, the worker_instance_profile must be specified and used. System credentials will not be copied onto worker instances.

You must also add a configuration for your profile to your .aws/config file, for example here is an .aws/config file that specifies a profile with the name "my-developer-profile". Make sure the information here matches what you setup in AWS IAM for your Role & Profile.

[default]
region = us-east-1

[profile my-developer-profile]
region=us-east-1
role_arn = arn:aws:iam::9999998999:role/developer
source_profile = my-company-iam
mfa_serial = arn:aws:iam::974808099065:mfa/joesmith
role_session_name = joesmith-developer
Note
The CloudBees Jenkins Enterprise cje tool uses the Boto3 Python Library to access and control AWS. For more information on Boto’s support for AWS IAM Roles see: Boto docs: AWS Assume Role Provider. For more information on how AWS credentials must be configured for Boto see: Boto docs: Shared Credentials.

EBS Usage

If you are planning to use EBS (Elastic Block Store) as storage, the AWS credentials used for installation must also be able to create EBS resources.

AWS Components

CloudBees Jenkins Enterprise in an AWS environment uses the following Amazon Web Services:

  • EC2 (Elastic Compute Cloud) to provide resizable compute capacity

  • S3 (Simple Storage Services) to store data objects in resources called "buckets"

  • ELB (Elastic Load Balancing) to redirect data traffic as needed

  • EBS (Elastic Block Store) to provide persistent block-level storage volumes for use with Amazon EC2 instances in the AWS Cloud

  • Route 53 (domain name system) for translating Web URL names into numeric IP addresses

  • VPC (virtual private cloud) to enable provisioning of a logically isolated section of AWS

Note
Although some clusters do not use Route 53 or VPC, we recommend that configuration support those components anyway.

The recommended configuration in an AWS environment enables three different types of workers (virtual machines) as described in the following table:

General Purpose

Operations Center and Jenkins Masters run as containerized processes on general-purpose workers.

Recommended instance type: M4 2XL (32 GiB)

Dedicated ES

Elasticsearch is performed by dedicated ES workers.

Recommended instance type: R3 XL (30.5 GiB)

Executors

Jenkins agents (formerly called slaves) run on executors.

Recommended instance type: varies by intended workload

Recommended Executor instance types are as follows: - M4.2xlarge is best for basic web applications and generic workloads - C3.xlarge is best for C and C++ codebases and complex builds - i2.xlarge is best for I/O-intensive builds such as those associated with building large indexes

Deployments can be thought of as small, medium, or large depending on the number of Jenkins masters that are running (up to 3, up to 10, and up to 30, respectively).

Guidelines for resource allocation across deployments of different size are as follows (The recommended instance for all controllers is M4.large. Controllers route requests and run Apache Mesos to support resource provisioning):

Small

3 controllers and 3 General Purpose workers

Medium

3 controllers, 3 ES workers, 5 General Purpose workers

Large

3 controllers, 5 ES workers, 5 General Purpose workers, 8 Executors

Installing CloudBees Jenkins Enterprise 1.x

Use these steps to install CloudBees Jenkins Enterprise 1.x into an Amazon Web Services (AWS) cloud environment using a Linux system to run the installer.

Download the installer

Download CloudBees Jenkins Enterprise from downloads.cloudbees.com, then untar the tar.gz package and add the bin subdirectory to your PATH.

$ export INSTALLER=pse_1.11.3_linux_amd64.tar.gz
$ curl -o $INSTALLER        https://downloads.cloudbees.com/pse/latest/$INSTALLER
$ curl -o $INSTALLER.sha256 https://downloads.cloudbees.com/pse/latest/$INSTALLER.sha256
$ sha256sum -c $INSTALLER.sha256
$ tar xzvf $INSTALLER
$ export PATH=$(pwd)/pse_1.11.3/bin:$PATH
$ cje version

You should see output with version reports similar to this:

PSE Project: 5
PSE Release: 36
CloudBees PSE: 1.11.8
AWS AMI Version: 1.11.8
Build: 253
Tiger Storage: cloudbees/pse-castle:1.11.8.1
Tiger CJOC: cloudbees/cloudbees-cloud-core-oc:2.121.2.1
Tiger CJE: cloudbees/cloudbees-core-mm:2.121.2.1
Tiger Search: cloudbees/pse-elasticsearch:1.3.1
Tiger Router: cloudbees/pse-router:1.11.4
Tiger Logstash: cloudbees/pse-logstash:1.6.0
PSE SSH Gateway: cloudbees/pse-sshgateway:1.3.1
Tiger Scheduler: cloudbees/pse-palace:1.10.0
CJE Certificates: cloudbees/docker-certificates:1.2
Mesos: 0.28.2\*
zookeeper: 3.4.6\*
Marathon: 0.15.3\*
Docker: 1.13.1\*
Topbeat: 1.1.0\*
Pip: 1.5.4\*
NVMe: 0.2.19\*
Terraform: 0.11.2

If you don’t see output like this, check your PATH setting and review the command output for errors.

Create the install project

A CloudBees Jenkins Enterprise install project is a directory and collection of files that contain the configuration needed to install and configure a CloudBees Jenkins Enterprise cluster. The install project records configuration which is used during the installation and during later command line administration.

In the next steps, we’ll use the shell variable PROJECT to refer to this directory.

You can create a CloudBees Jenkins Enterprise project using a relative path. For these steps, we’ll use the location ~/cje-project:

$ export PROJECT=~/cje-project

Create the CloudBees Jenkins Enterprise project directory and its configuration files by running the init-project command as follows:

$ cje init-project $PROJECT aws

This step creates a new CloudBees Jenkins Enterprise project in $PROJECT and prepares files in the directory that will be used to install and configure CloudBees Jenkins Enterprise on AWS.

Prepare the cluster definition

A CloudBees Jenkins Enterprise installation is a collection of systems configured to provide reliable, scalable continuous delivery services. Some values in the cluster configuration must be provided by you based on your environment.

To prepare the initial cluster definition, execute the following commands:

$ cd $PROJECT
$ cje prepare cluster-init

The prepare command creates a configuration template in the cluster-init.config file. This file contains parameters for starting your CloudBees Jenkins Enterprise cluster.

Edit the file using your preferred text editor. The file provides detailed guidance on available configuration options. The next few sections highlight items you must set.

Set cluster-init.config parameters

You must set values for the parameters described in this section to successfully launch your CloudBees Jenkins Enterprise cluster. Many other parameters are available, and have usable, secure parameters pre-set in the cluster-init.config.

Configure network access

The simplest option for configuring network access is to allow access from anywhere with this setting:

tiger_admin_port_access = 0.0.0.0/0

Set cluster name

Select your cluster name. This value prefixes resource names and other tags when configuring servers and other infrastructure. Some of these resources use global identifiers (e.g., S3 buckets in AWS) and therefore this name must be unique. Choose a value that describes the CloudBees Jenkins Enterprise cluster and includes a unique identifier such as: your company name, domain name or other similar value.

This value must contain only alphanumeric, dots ('.'), dashes and underscore values and must be no longer than 20 characters.

cluster_name = <unique cluster name>

Select instance types

You must select the AWS instance types for the two virtual machine instance types that CloudBees Jenkins Enterprise creates. You must select an Amazon supported instance type.

Controller instances host cluster-level functionality such as Operations Center, analytics, and container orchestration.

The second instance type is the worker instances. Worker instances host Jenkins masters and Jenkins agents. Alternate worker types may be specified from Operations Center when adding additional worker instances.

controller_instance_type = m4.large
worker_instance_type = m4.xlarge

Select AWS region

You must select an AWS region to host the cluster. Valid values are listed in the cluster-init.config comments.

region = <aws region>

Set AWS credential profile

AWS credentials are stored and maintained by the aws command line utility. Refer to Installing the AWS Command Line Interface for installation and configuration instructions. You must configure your AWS command line credentials before applying the CloudBees Jenkins Enterprise installation.

A credential profile and a worker instance profile must be set in the cluster-init.config. The CloudBees Jenkins Enterprise installer cje uses the credential profile to add, upgrade, and remove components. Operations Center and the Managed Masters use the worker instance profile as they build and deploy software. Detailed requirements for those two profiles are available in the AWS Credentials for AWS install.

Values for credential_profile and worker_instance_profile might look like this in cluster-init.config:

credential_profile = example-com-infra-admin
worker_instance_profile = example-com-worker

Allowed profile names may be read from the AWS config file, usually stored in ~/.aws/config.

A sample section of the ~/.aws/config file might look like this:

[profile example-com-infra-admin]
region=us-east-1
role_arn = arn:aws:iam::272593129526:role/infra-admin
source_profile = example-com-iam
mfa_serial = arn:aws:iam::973709169065:mfa/my-user
role_session_name = my-user-example-com-infra-admin

Values for the ~/.aws/config file can be obtained from your AWS administrator or from the AWS console.

Verify configuration

An error in cluster-init.config will disrupt the CloudBees Jenkins Enterprise initialization process. Test your settings with cje verify before you apply them. You can identify many issues in the file before attempting initialization with the command:

$ cje verify

You should see output similar to this:

cluster-init is ready to apply

If the verification succeeds, you are ready to start the initialization. The next command creates VM instances in AWS and installs software for each instance.

Create the cluster

Create the cluster by executing the following command:

$ cje apply

cje apply can be run as many times as needed. If an error stops the installation prematurely, the installation process can be resumed by executing cje apply again.

The CloudBees Jenkins Enterprise installation project outputs progress information while it executes. cje apply may need 30 minutes or more to complete, depending on AWS responsiveness.

Retrieve cluster information

To retrieve CloudBees Jenkins Enterprise endpoints, use the command:

$ cje run display-outputs

You should see output like this:

Controllers: ec2-107-22-158-238.compute-1.amazonaws.com,ec2-34-224-167-128.compute-1.amazonaws.com,ec2-52-87-250-152.compute-1.amazonaws.com
Workers    : ec2-54-164-65-68.compute-1.amazonaws.com,ec2-54-172-53-214.compute-1.amazonaws.com,ec2-54-89-168-101.compute-1.amazonaws.com

CJOC    : http://cloudbees-trial-controller-459656157.us-east-1.elb.amazonaws.com.elb.cloudbees.net/cjoc/
Mesos   : http://mesos.cloudbees-trial-controller-459656157.us-east-1.elb.amazonaws.com.elb.cloudbees.net
Marathon: http://marathon.cloudbees-trial-controller-459656157.us-east-1.elb.amazonaws.com.elb.cloudbees.net

The CloudBees Jenkins Enterprise cluster is now available.

Your $PROJECT directory stores state and configuration information for your cluster.

You should make a copy of the entire install project directory, or store the entire directory in a source control system. As a convenience, the installation process generates a .gitignore file, ignoring specific files.

Configure domain name and HTTPS

This step configures CloudBees Jenkins Enterprise with a custom domain name and switches communication from HTTP to HTTPS.

Create DNS records

The new names of the CloudBees Jenkins Enterprise cluster need to be defined and published to Domain Name Servers (DNS).

Find load balancer name

The CloudBees Jenkins Enterprise cluster uses a load balancer for resilience and capacity management. If one of the machines providing CloudBees Jenkins Enterprise services fails or becomes unavailable, the load balancer is able to redistribute requests to other machines.

DNS maps the name of the CloudBees Jenkins Enterprise cluster to the name of the load balancer.

You can find the name of the load balancer through the Amazon Web Services console under "Load Balancers". Open the Amazon Web Services EC2 Dashboard and click the "Load Balancers" link to view the configured load balancer. Your CloudBees Jenkins Enterprise cluster will be listed with the cluster name you chose earlier.

get aws load balancer name
Create DNS alias

Create a DNS alias (CNAME record) on the DNS server for the friendly company name of the load balancer. In this example, the friendly company name was cloudbees.example.com and the DNS name of the load balancer was cd-cluster-controller-2093098324.us-east-1.elb.amazonaws.com.

It is critical that you alias your company hostname to the load balancer hostname, rather than assigning the current IP address of the load balancer to your company hostname. The specific IP address of the load balancer may change, while the AWS load balancer hostname will not change.

The alias records are configured as:

cloudbees.example.com CNAME cd-cluster-controller-2093098324.us-east-1.elb.amazonaws.com
marathon.cloudbees.example.com CNAME cd-cluster-controller-2093098324.us-east-1.elb.amazonaws.com
mesos.cloudbees.example.com CNAME cd-cluster-controller-2093098324.us-east-1.elb.amazonaws.com

Configure the alias records in Amazon Route 53 DNS with settings like these:

define cname dns record smaller

The AWS Route 53 DNS user interface allows the "A" record to be used as an alias for another hostname. That is not standard practice with other DNS providers, where the CNAME record generally defines the alias from one hostname to another. Amazon maintains the aliases inside their Route 53 DNS system rather than publishing them as CNAME records.

Create SSL certificate

An SSL certificate secures communication between web browsers and CloudBees Jenkins Enterprise. Your CloudBees Jenkins Enterprise cluster name and two other names need an SSL certificate.

Request an SSL certificate with the AWS Certificate Manager or import a certificate generated using another SSL certificate provider. A single certificate should be requested which includes your CloudBees Jenkins Enterprise cluster name and two other names, like this:

cloudbees.example.com
marathon.cloudbees.example.com
mesos.cloudbees.example.com
Create certificates using AWS Certificate Manager

After entering the domain name and the two additional names, press "Review and Request".

ssl add domain names

The "Review and Request" page will show the hostname of your CloudBees Jenkins Enterprise cluster and the additional mesos and marathon names. When you’ve confirmed they are correct, press "Confirm and request".

ssl review and request

After you "Confirm and request", the certificate request will be processed. E-mail will be sent to the registered owner of the domain to confirm ownership.

ssl validation pending

After Amazon Certificate Manager has received confirmation of ownership, the new certificate will be visible. Open the certificate manager and copy the Amazon Resource Name (ARN) for your certificate. The Amazon Resource Name of your SSL certificate will be copied into the domain-name-change.config file.

ssl with arn

Self-signed certificates / custom certificate authorities

In order to use a self-signed certificate, or one generated by a custom certificate authority, the certificate must be manually added to the cluster in order for CloudBees Jenkins Enterprise to communicate using SSL.

Register certificates inside the cluster by creating a folder named certificates. Copy the certificate files to that folder using the PEM format and .pem extension.

The certificate(s) must be registered on the machine the installer is called on. The exact procedure may differ depending on the operating system.

Prepare and apply changes

Once the SSL certificate and DNS changes are available, the CloudBees Jenkins Enterprise cluster may be updated to use them.

Prepare the configuration file for editing with the command:

$ cje prepare domain-name-change
domain-name-change is staged - review domain-name-change.config and edit as needed - then run 'cje apply' to perform the operation.

Edit the domain-name-change.config file to specify the new domain names and related parameters, including at least:

domain_name = cloudbees.example.com
protocol = https
ssl_certificate_id = arn:aws:acm:us-east-x:123456789012:certificate/cafefade-feed-aced-cede-beadeddeeded

Verify the changes with cje verify and apply the changes with cje apply.

$ cje verify
domain-name-change is verified
$ cje apply

Opening Operations Center

Your CloudBees Jenkins Enterprise installation is complete! Open Operations Center using the link from cje run display-outputs (see Retrieve cluster information) or as provided in the success message from cje init.

$ cje run display-outputs

Controllers: ec2-107-22-158-238.compute-1.amazonaws.com,ec2-34-224-167-128.compute-1.amazonaws.com,ec2-52-87-250-152.compute-1.amazonaws.com
Workers    : ec2-54-164-65-68.compute-1.amazonaws.com,ec2-54-172-53-214.compute-1.amazonaws.com,ec2-54-89-168-101.compute-1.amazonaws.com

CJOC    : https://cloudbees.example.com/cjoc/
Mesos   : https://mesos.cloudbees.example.com
Marathon: https://marathon.cloudbees.example.com

You can retrieve the Operations Center login credentials using:

$ cje run echo-secrets cjoc_username
$ cje run echo-secrets cjoc_password

After login, and completing the license page, you are directed to the Operations Center registration page to enter your name, email address, organization and click Submit.

cjoc registration

Once this completes, the Operations Center plugin install will appear in the browser window.

Operations Center will let you create new Managed Masters and administer $CloudBees Jenkins Enterprise.

Change adminstrator password

After initial login, change your administrator password to a value you’ll remember, instead of the randomly assigned password.

Be sure that you also update that administrator password in the secrets file so that future upgrades and other command line references can use the updated credentials.

What’s next

Your installation is complete. You’re ready to experience CloudBees Jenkins Enterprise 1.x. Refer to Administering CloudBees Jenkins Enterprise 1.x to configure new Managed Masters, connect to corporate authentication systems, configure permissions, and more.