Table of contents

CloudBees Jenkins Enterprise 1.x administration guide


Note
CloudBees will no longer be supporting CloudBees Jenkins Enterprise 1.x after July 30, 2020. This end-of-life announcement allows CloudBees to focus on driving new technology and product innovation for CloudBees Core. For information on moving to CloudBees Core, please refer to Migrating from CloudBees Jenkins Enterprise 1.x to CloudBees Core on modern cloud platforms which has been created to help you with the migration process. Existing customers can also contact their CSM to help ensure a smooth transition.

Operating CloudBees Jenkins Enterprise 1.x

Managing Masters

CloudBees Jenkins Enterprise enables Continuous Delivery as a Service by offering a highly elastic infrastructure that supports centralized management of teams, shareable resources across the cluster and automated management of the underlying cluster infrastructure.

With CloudBees Jenkins Enterprise’s Managed Masters, administrators can offer their teams highly available build environments with a minimum of management and configuration overhead. This section describes the CloudBees Jenkins Enterprise architecture, how administrators can scale their CloudBees Jenkins Enterprise cluster, and some upgrade and disaster recovery strategies for CloudBees Jenkins Enterprise clusters.

Individual agents can also be provisioned and controlled from CloudBees Jenkins Enterprise. Refer to Managing Agents for more information.

See the CloudBees Jenkins Enterprise reference section for more details of the CloudBees Jenkins Enterprise architecture.

CloudBees Assurance Program

CloudBees Jenkins Enterprise provide greater stability and security for Jenkins installations through the CloudBees Assurance Program. The CloudBees Assurance Program supports all Jenkins-based components, including CloudBees Jenkins Enterprise and CloudBees Jenkins Team. The CloudBees Assurance Program simplifies securing and upgrading multiple Jenkins instances. It assures compliance with CloudBees-recommended master configurations as detailed in the Beekeeper Upgrade Assistant page.

Learn more about managing upgrades and downgrades with the CloudBees Assurance Program.

Operating Managed and Client Masters

When new teams join an organization, or existing teams start a new project, CloudBees Jenkins Enterprise makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees Jenkins Enterprise, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees Jenkins Enterprise environment can handle is limited only by the capacity of the cluster.

Adding Client Masters

Occasionally administrators will need to connect existing masters to a CloudBees Jenkins Enterprise cluster, such as in the case of a team requiring a master on Windows. Existing masters that are connected to Operations Center lack key benefits of Managed Masters like high availability and automatic agent management. Whenever possible, administrators should use a Managed Master with CloudBees Jenkins Enterprise rather than connecting an existing master.

Client Masters are monitored by Operations Center just as Managed Masters are monitored. Administrators can see the status of all their Managed Masters, Team Masters, and Client Masters from the Operations Center masters page. Client Masters can receive configuration updates from Operations Center with configuration snippets. Client Masters can share agents hosted in the cluster, offloading the burden of agent management from teams.

Client Masters do not have the high availability features of Managed Masters.

Note: The existing Client Master and Operations Center must both accept JNLP requests. See Kubernetes client master configuration for more information.

To add a Client Master, log into CloudBees Jenkins Enterprise and navigate to the Operations Center dashboard. Click the New Item option in the left-hand menu and provide the following information:

  • Item name: the name of the existing Client Master to connect to the cluster

  • Select Client Master. Just as with Managed Masters, Client Masters offer some customization and configuration options on creation:

  • On-master executors: Number of builds to execute concurrently on the Client Master itself. Default setting is 2.

  • Email addresses: Contact information for the administrator responsible for maintaining the Client Master.

Once these settings are saved, Operations Center will attempt to connect the Client Master to the CloudBees Jenkins Enterprise cluster.

Verify that Operations Center and the existing Client Master can communicate with each other over both HTTP and JNLP ports. Inbound HTTP(s) traffic goes through a reverse proxy on a controller node. Host and port to use for JNLP is advertised through HTTP headers by each Jenkins master.

You can connect an existing Client Master to Operations Center by giving that Client Master the TLS certificate for Operations Center, typically through the Configure Global Security page in Operations Center. For more information, see How to programmatically connect a Client Master to CJOC.

If you are connecting multiple Client Masters to your cluster, it is a good idea to automate that task using shared configurations.

Once the Client Master is connected, administrators should configure security and access controls for the master.

Unresolved directive in operating.adoc - include::/home/ubuntu/workspace/workspace/ct-documentation_cbn-site_master/content/docs/cloudbees-documentation/admin-cje/src/main/asciidoc/../../../../operations-center-admin-guide/src/main/asciidoc/managing-masters/configuring-a-master.adoc[leveloffset=+2]

Unresolved directive in operating.adoc - include::/home/ubuntu/workspace/workspace/ct-documentation_cbn-site_master/content/docs/cloudbees-documentation/admin-cje/src/main/asciidoc/../../../../operations-center-admin-guide/src/main/asciidoc/managing-masters/configuring-plugin-catalogs.adoc[leveloffset=+2]

Unresolved directive in operating.adoc - include::/home/ubuntu/workspace/workspace/ct-documentation_cbn-site_master/content/docs/cloudbees-documentation/admin-cje/src/main/asciidoc/../../../../operations-center-admin-guide/src/main/asciidoc/managing-masters/configuring-masters-through-cli.adoc[leveloffset=+2]

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees Jenkins Enterprise administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk-upgrading Managed Masters

When a CloudBees Jenkins Enterprise cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees Jenkins Enterprise, this can be achieved by defining a cluster operation in Operations Center.

To create this task, the administrator will first need to log into Operations Center, then create a New Item of the type Cluster Operations. The administrator will then needs to select the Managed Masters cluster operation, and will then be given a set of pre-configured upgrade patterns to choose from.

The administrator will then need to specify which masters to target by using the filter Uses Docker Image and picking a Docker image used by their cluster’s masters as the upgrade target for this operation. Any masters in the cluster using the selected image will be affected by this cluster operation.

In the Steps section, the administrator should select Update Docker Image and pick the new Docker image to bulk-upgrade the targeted masters to. Next, the administrator should add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Quiet Start

There may be times during an upgrade or other maintenance when it is best to have Jenkins start, but launch no projects. For example, if an upgrade is being performed in multiple steps, the intermediate steps may not be fully configured to run projects successfully. The "quiet start plugin" can immediately place the Jenkins server in "quieting down" state on startup.

Enable "quiet start" by checking the box in Manage Jenkins  Quiet Restart. When the server is restarted, it will be in the "quieting down" state. An administrator can cancel that state using the regular UI.

Uncheck the box in Manage Jenkins  Quiet Restart when maintenance is complete. Projects will start as usual on server restart.

Reviewing Plugin Usage

Large Jenkins installations often have many plugins installed. CloudBees Jenkins Enterprise can help administrators identify plugins which may be unused or no longer useful. The CloudBees Plugin Usage Plugin helps you keep track of which plugins you are actively using and where they are used.

Plugin Usage Report

A table of installed plugins is available from Manage Jenkins  Plugin Usage, including a usage count and a list of uses of that plugin. A strikethrough font indicates disabled plugins. A progress bar runs while Jenkins scans jobs and global settings. Once the scan is complete, the Usage Count column becomes a sortable column. Click the column heading to sort the column.

usage count
Figure 1. Usage Count

The third column shows detected uses of the plugin, hyperlinked to the matching configuration screen. In the list of hyperlinks, "Jenkins" refers to global Jenkins configuration. Most other labels are names of configurable items such as jobs (using » to show folder structure where relevant); configuration associated with Jenkins users (such as credentials) will also be shown. Code packaged as a plugin but really used only as a library shared between several "real" plugins (Async Http Client Plugin, for example) will be shown as used by those plugins. »… is appended to items which have nested elements also using the plugin; for example, a native Maven project may have several modules all of which refer to the plugin.

Only a Jenkins administrator can perform a complete plugin usage analysis. Configuration read permission is required to check plugin usage.

Limitations

Beware that some plugins will not be listed in this table, because they are not directly mentioned in any stored configuration. They may affect how Jenkins runs in various ways without configuration; the Jenkins Translation Assistance Plugin is an example.

Conversely, a plugin might have some stored configuration which is of no interest or never used at all. For example, Jenkins Email Extension Plugin displays various controls in the global Configure System screen; when you click Save, Email Extension configuration is saved in hudson.plugins.emailext.ExtendedEmailPublisher.xml in your $JENKINS_HOME, even if you have made no customizations. This plugin will thus appear to be "in use" by Jenkins. Only if you have enabled the Editable Email Notification post-build action for a job will it have a usage count greater than 1, however.

Use the plugin usage table as a starting point for investigating plugins that might be unused. It is not relevant for all plugins. The table works best for checking the usage of plugins which offer concrete additions to configuration, such as build steps or publishers in jobs.

Plugins used in configuration generated by a builder (or publisher) template will not be listed (in either the template or jobs using it), since this configuration is created on the fly during a build rather than being stored permanently in the job. Configuration generated by a job (or folder) template is considered.

Adding capacity to a CloudBees Jenkins Enterprise cluster

In order to provide teams with faster build times and minimize the times builds spending waiting for resources, administrators can add capacity to their cluster by configuring more workers for their cluster.

Scaling with Workers

Within a CloudBees Jenkins Enterprise cluster, a worker is a virtual machine which is provisioned to run a specific type of workload - either Managed Masters, Jenkins agents, or Elasticsearch nodes. Workers provide the computing capacity to host and run these end-user applications. Administrators can increase their cluster’s performance and resiliency by adding more Workers and configuring more CPU and memory for Workers.

A worker can have the following responsibilities:

  • Executing builds assigned by Managed and Client Masters

  • Hosting Managed Masters

  • Hosting CloudBees Jenkins Enterprise’s persistence layer

Preparing Workers

In order to add additional workers to the CloudBees Jenkins Enterprise environment, you need to have an existing project configured and the cje tool downloaded and installed. The worker-add operation is used to set up dedicated workers.

% cje prepare worker-add
worker-add is staged - review worker-add.config and edit as needed - then run 'cje apply' to perform the operation.
%

The generated worker-add.config should then be edited to define the number of workers to add, the workload type, and a number of other platform-specific configurations as necessary.

Table 1. Workload Type
Type Description

master

(default) Jenkins masters, Operations Center, and CloudBees Jenkins Enterprise components including Elasticsearch nodes unless workers have been provisioned explicitly for Elasticsearch.

build

Worker dedicated for running Jenkins agents

elasticsearch

Worker dedicated to running Elasticsearch nodes

worker-add.config
[worker]
## Number of workers to add
count = 1
## Type of workload that the worker will handle
workload_type = master

[aws]
## The instance type of the worker
# worker_instance_type = m4.xlarge

## EBS Optimized
# Valid values: yes/no
# ebs_optimized =

## The instance root volume size in GB
# worker_volume_size = 50

## The root volume type
# worker_volume_type = gp2

## EC2 ephemeral mounts
# use_ephemeral =

## The AMI to use for the new worker
# worker_ami =

## The id of the subnet where the instance is started
# subnet_id =

## The availability zone where the instance is started
# availability_zone =

## Instance profile of the worker
# worker_instance_profile = cjppse-worker

## Save the [aws] parameter values as defaults for next worker-add operations
# save_as_default = yes
Note

On AWS, the work_volume_size defaults to 50 GB. For instances with dedicated disk space, set the proper value to use the entire disk allocated to that the instance.

On AWS, if you are going to create a build worker, consider providing a profile with limited permissions for the value worker_instance_profile as build workers do not need permissions such as create volume, delete volume,…​etc.

To make it easier to create scripts to add workers, options can be added to the cje prepare worker-add command, then the generated worker-add.config will have those fields filled in. To see all available options, add --help to the command:

cje prepare worker-add --help
usage: cje prepare [-h] [-p DIR] [--config-file CONFIG-FILE]
                   [--secrets-file SECRETS-FILE] [--aws.availability_zone VAL]
                   [--aws.resource_tags VAL] [--aws.save_as_default VAL]
                   [--aws.subnet_id VAL] [--aws.use_ephemeral VAL]
                   [--aws.worker_ami VAL] [--aws.worker_ebs_optimized VAL]
                   [--aws.worker_instance_profile VAL]
                   [--aws.worker_instance_type VAL]
                   [--aws.worker_volume_size VAL]
                   [--aws.worker_volume_type VAL] [--docker.options VAL]
                   [--worker.count VAL] [--worker.workload_type VAL]
                   worker-add

Prepares an operation.

Prepared operations must be configured and then applied using the apply command.

  worker-add            Prepare worker-add.

optional arguments:
  -h, --help            show this help message and exit
  -p DIR, --project DIR
                        Directory containing the CJE project files
  --config-file CONFIG-FILE
                        Use the specified config file
  --secrets-file SECRETS-FILE
                        Use the specified secrets file
  --aws.availability_zone VAL
                        The availability zone where the instance is started
  --aws.resource_tags VAL
                        AWS resource tags
  --aws.save_as_default VAL
                        Save the [aws] parameter values as defaults for next
                        worker-add operations
  --aws.subnet_id VAL   The id of the subnet where the instance is started
  --aws.use_ephemeral VAL
                        EC2 ephemeral mounts
  --aws.worker_ami VAL  The AMI to use for the new worker
  --aws.worker_ebs_optimized VAL
                        EBS Optimized
  --aws.worker_instance_profile VAL
                        Instance profile of the worker
  --aws.worker_instance_type VAL
                        The instance type of the worker
  --aws.worker_volume_size VAL
                        The instance root volume size in GB
  --aws.worker_volume_type VAL
                        The root volume type
  --docker.options VAL  Docker options
  --worker.count VAL    Number of workers to add
  --worker.workload_type VAL
                        Type of workload that the worker will handle
Custom AWS resource tags

During cluster-init, resource tags will be added to all AWS-created resources that support them, using resource_tags, a comma-delimited list of tags. The parameter format is TAG1=VALUE1, TAG2=VALUE2, and so on, as arguments for the expression 'resource_tags'. For example, 'resource_tags="department=engineering, owner=FrankBowman"'

This paramter applies only to AWS.

Note
Specifying resource_tags during worker-add operations will overwrite tags that were defined during cluster-init.

Deploying Workers

After the worker-add.config file has been edited to contain the appropriate configuration for the new worker(s), apply the operation with cje apply.

Once the operation completes, CloudBees Jenkins Enterprise will have added capacity to help the environment continue to scale. For more on scaling CloudBees Jenkins Enterprise within the organization, continue to the CD as a Service section.

Example cje apply
% cje apply
Using /home/cje-admin/cje_1.6.0/bin
Creating worker at /home/cje-admin/t160/.dna
Extending initialization to /home/cje-admin/cje_1.6.0/share/setup-templates/core
Creating server worker-4
Finished extension to /home/cje-admin/cje_1.6.0/share/setup-templates/core
Updating server worker-4
Updating project configuration
Reinitializing DNA project at /home/cje-admin/t160/.dna
Extending initialization to /home/cje-admin/cje_1.6.0/share/setup-templates/core
Creating credentials
Creating shared resources
Reinitializing /home/cje-admin/t160/.dna/libs
Reinitializing /home/cje-admin/t160/.dna/scripts
Updating server tiger
Updating server keypairs
Updating server controller-1
Updating server controller-2
Updating server controller-3
Updating server worker-1
Updating server worker-2
Updating server worker-3
Updating server worker-4
Updating server castle
Updating server cjoc
Updating server elasticsearch
Updating server palace
Reinitializing environment tiger
Reinitializing environment marathon
Finished extension to /home/cje-admin/cje_1.6.0/share/setup-templates/core
Reinitializing /home/cje-admin/t160/.dna/scripts
Updating server tiger
Updating server keypairs
Updating server vpc
Updating server security-groups
Updating server elasticsearch-snapshots
Updating server storage-bucket
Updating server controller-1
Updating server controller-2
Updating server controller-3
Updating server elb
Updating server elbi
Updating server elb-mesos
Updating server elbi-mesos
Updating server elb-marathon
Updating server elbi-marathon
Updating server route53
Updating server worker-1
Updating server worker-2
Updating server worker-3
Updating server worker-4
Reinitializing environment terraform
Starting new worker(s) worker-4
Enter MFA code:
[worker-4] data.null_data_source.subnets: Refreshing state...
[worker-4] data.template_cloudinit_config.config: Refreshing state...
[worker-4] data.aws_ami.pse-rhel: Refreshing state...
[worker-4] aws_instance.worker: Creating...
[worker-4] ami:                                       "" => "ami-ad6fbfbb"
[worker-4] associate_public_ip_address:               "" => "<computed>"
[worker-4] availability_zone:                         "" => "<computed>"
[worker-4] ebs_block_device.#:                        "" => "<computed>"
[worker-4] ephemeral_block_device.#:                  "" => "<computed>"
[worker-4] iam_instance_profile:                      "" => "cje-worker"
[worker-4] instance_state:                            "" => "<computed>"
[worker-4] instance_type:                             "" => "m4.xlarge"
[worker-4] key_name:                                  "" => "cje160-demo"
[worker-4] network_interface_id:                      "" => "<computed>"
[worker-4] placement_group:                           "" => "<computed>"
[worker-4] private_dns:                               "" => "<computed>"
[worker-4] private_ip:                                "" => "<computed>"
[worker-4] public_dns:                                "" => "<computed>"
[worker-4] public_ip:                                 "" => "<computed>"
[worker-4] root_block_device.#:                       "" => "1"
[worker-4] root_block_device.0.delete_on_termination: "" => "true"
[worker-4] root_block_device.0.iops:                  "" => "<computed>"
[worker-4] root_block_device.0.volume_size:           "" => "50"
[worker-4] root_block_device.0.volume_type:           "" => "gp2"
[worker-4] security_groups.#:                         "" => "<computed>"
[worker-4] source_dest_check:                         "" => "true"
[worker-4] subnet_id:                                 "" => "subnet-9d7c25c6"
[worker-4] tags.%:                                    "" => "9"
[worker-4] tags.Name:                                 "" => "cje160-demo-worker-4"
[worker-4] tags.cloudbees:pse:cluster:                "" => "cje160-demo"
[worker-4] tags.cloudbees:pse:type:                   "" => "worker"
[worker-4] tags.cloudbees:pse:version:                "" => "1.6.0-rc-1"
[worker-4] tags.cloudbees:pse:worker_type:            "" => "master"
[worker-4] tags.dna:server_id:                        "" => "cje160-demo-worker-4"
[worker-4] tags.tiger:cluster:                        "" => "cje160-demo"
[worker-4] tags.tiger:type:                           "" => "worker"
[worker-4] tags.tiger:version:                        "" => "1.6.0-rc-1"
[worker-4] tenancy:                                   "" => "<computed>"
[worker-4] user_data:                                 "" => "a4fd10ab932c6a88da2ff7f2e59fc1d712fe7a39"
[worker-4] vpc_security_group_ids.#:                  "" => "1"
[worker-4] vpc_security_group_ids.4288342008:         "" => "sg-6f818613"
[worker-4] aws_instance.worker: Still creating... (10s elapsed)
[worker-4] aws_instance.worker: Still creating... (20s elapsed)
[worker-4] aws_instance.worker: Still creating... (30s elapsed)
[worker-4] aws_instance.worker: Creation complete
[worker-4]
[worker-4] Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
[worker-4]
[worker-4] The state of your infrastructure has been saved to the path
[worker-4] below. This state is required to modify and destroy your
[worker-4] infrastructure, so keep it safe. To inspect the complete state
[worker-4] use the `terraform show` command.
[worker-4]
[worker-4] State path: terraform.tfstate
[worker-4]
[worker-4] Outputs:
[worker-4]
[worker-4] worker_hostname = ec2-52-201-196-226.compute-1.amazonaws.com
[worker-4] worker_public_host = ec2-52-201-196-226.compute-1.amazonaws.com
[worker-4] ok----------------------------------
[worker-4] Syncing servers/worker-4/config => dna-config/: -Warning: Permanently added '52.201.196.226' (ECDSA) to the list of known hosts.
[worker-4ok
[worker-4] Syncing servers/worker-4/configure-http-proxy => dna-config/: -Warning: Permanently added '52.201.196.226' (ECDSA) to the list of known hosts.
[worker-4ok
[worker-4] Syncing servers/worker-4/configure-docker => dna-config/: -Warning: Permanently added '52.201.196.226' (ECDSA) to the list of known hosts.
[worker-4ok
[worker-4] Rendering servers/worker-4/config/jce-castle-config.xml => ~/dna-config/jce-castle-config.xml:ok
[worker-4] Rendering servers/worker-4/config/log.properties => ~/dna-config/log.properties:ok
[worker-4] Rendering servers/worker-4/config/aws-credentials => ~/dna-config/aws-credentials:ok
[worker-4] Rendering servers/worker-4/config/aws-config => ~/dna-config/aws-config:ok
[worker-4] Rendering servers/worker-4/config/.docker/config.json => ~/dna-config/.docker/config.json:ok
[worker-4] Rendering servers/worker-4/config/logstash.conf => ~/dna-config/logstash.conf:ok
[worker-4] Rendering servers/worker-4/config/logstash-redhat.conf => ~/dna-config/logstash-redhat.conf:ok
[worker-4] Rendering servers/worker-4/config/topbeat.yml => ~/dna-config/topbeat.yml:ok
[worker-4] Rendering servers/worker-4/config/fetch-docker-config-cron => ~/dna-config/fetch-docker-config-cron:ok
[worker-4] Rendering servers/worker-4/config/ecr-login-cron => ~/dna-config/ecr-login-cron:ok
[worker-4] Running init on worker-4:
[worker-4] Warning: Permanently added '52.201.196.226' (ECDSA) to the list of known hosts.
[worker-4] Mesosphere registry already configured
[worker-4] Docker registry already configured
[worker-4] Topbeat registry already configured
[worker-4] Initializing packages
[worker-4] Java (java-1.8.0-openjdk-headless) already installed
[worker-4] Configuring syslog
[worker-4] Installing jq
[worker-4] Package is already installed: jq
[worker-4] Package is already installed: bind-utils
[worker-4] Installing xfsprogs
[worker-4] Package is already installed: xfsprogs
[worker-4] cje160-demo-0a3c95 was mounted successfully in /usr/share/elasticsearch/snapshot/.
[worker-4] Configure AWS Credentials
[worker-4] Not copying credentials due to specified instance profile
[worker-4] Installing pip
[worker-4] Installing python2-pip
[worker-4] Package is already installed: python2-pip
[worker-4] Configuring mesos credentials
[worker-4] Mesosphere registry already configured
[worker-4] Docker registry already configured
[worker-4] Topbeat registry already configured
[worker-4] Initializing packages
[worker-4] Java (java-1.8.0-openjdk-headless) already installed
[worker-4] Configuring syslog
[worker-4] Installing jq
[worker-4] Package is already installed: jq
[worker-4] Package is already installed: bind-utils
[worker-4] Docker already installed
[worker-4] Enabling docker service
[worker-4] Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[worker-4] firewalld is not active, not configuring it
[worker-4] Mesos already installed
[worker-4] Configuring Mesos Zookeeper access
[worker-4] Configuring Mesos slave hostname
[worker-4] Configuring Mesos slave containerizers
[worker-4] Configuring Mesos slave executor registration timeout
[worker-4] Configuring Mesos docker stop timeout 10secs
[worker-4] Configuring Mesos slave credential
[worker-4] Configuring Mesos slave attributes
[worker-4] Configuring Mesos slave logging level
[worker-4] Enabling mesos-slave service
[worker-4] Created symlink from /etc/systemd/system/multi-user.target.wants/mesos-slave.service to /usr/lib/systemd/system/mesos-slave.service.
[worker-4] Configuring Docker
[worker-4] Initializing Docker cleanup job
[worker-4] You are using pip version 8.1.2, however version 9.0.1 is available.
[worker-4] You should consider upgrading via the 'pip install --upgrade pip' command.
[worker-4] cloudbees/pse-castle:1.6.0
[worker-4] Image cloudbees/pse-castle:1.6.0 already pulled
[worker-4] Pulling Docker image: docker.cloudbees.com/unified-distribution/cje-oc@sha256:8fb0b3d4dd87482c93f9ec7874f2f3538b3fd65806308ea7f100646de6e53134
[worker-4] sha256:8fb0b3d4dd87482c93f9ec7874f2f3538b3fd65806308ea7f100646de6e53134: Pulling from unified-distribution/cje-oc
[worker-4] badd262722ac: Pull complete
[worker-4] Digest: sha256:8fb0b3d4dd87482c93f9ec7874f2f3538b3fd65806308ea7f100646de6e53134
[worker-4] Status: Downloaded newer image for docker.cloudbees.com/unified-distribution/cje-oc@sha256:8fb0b3d4dd87482c93f9ec7874f2f3538b3fd65806308ea7f100646de6e53134
[worker-4] Pulling Docker image: docker.cloudbees.com/unified-distribution/cje-mm@sha256:24a3a0ed72cbecfb0404966b51d16161c6b631ef3d8aba440b42bbb6bbcc0e61
[worker-4] sha256:24a3a0ed72cbecfb0404966b51d16161c6b631ef3d8aba440b42bbb6bbcc0e61: Pulling from unified-distribution/cje-mm
[worker-4] b93b1dad68df: Pull complete
[worker-4] Digest: sha256:24a3a0ed72cbecfb0404966b51d16161c6b631ef3d8aba440b42bbb6bbcc0e61
[worker-4] Status: Downloaded newer image for docker.cloudbees.com/unified-distribution/cje-mm@sha256:24a3a0ed72cbecfb0404966b51d16161c6b631ef3d8aba440b42bbb6bbcc0e61
[worker-4] cloudbees/pse-elasticsearch:1.3.1
[worker-4] Image cloudbees/pse-elasticsearch:1.3.1 already pulled
[worker-4] Adding jenkins user
[worker-4] Copy castle configuration
[worker-4] Installing logstash
[worker-4] cloudbees/pse-logstash:1.6.0
[worker-4] Image cloudbees/pse-logstash:1.6.0 already pulled
[worker-4] Configuring logstash service
[worker-4] Enabling cloudbees-logstash service
[worker-4] Created symlink from /etc/systemd/system/default.target.wants/cloudbees-logstash.service to /etc/systemd/system/cloudbees-logstash.service.
[worker-4] Topbeat already installed
[worker-4] Enabling topbeat service
[worker-4] Created symlink from /etc/systemd/system/multi-user.target.wants/topbeat.service to /usr/lib/systemd/system/topbeat.service.
[worker-4] Restarting logstash
[worker-4] Restarting topbeat...
[worker-4] You are using pip version 8.1.2, however version 9.0.1 is available.
[worker-4] You should consider upgrading via the 'pip install --upgrade pip' command.
Rendering /home/cje-admin/t160/.dna/servers/tiger/config/jce-castle-config.xml => /home/cje-admin/t160/.dna/servers/tiger/jce-castle-config.rendered.xml
Waiting for CJOC at http://cje160-demo-controller-1313899594.us-east-1.elb.amazonaws.com.elb.cloudbees.net/cjoc/
OK
CJOC Login successful.
Updating Servers info
Updating Castle info
Rendering /home/cje-admin/t160/.dna/servers/castle/config/jce-castle-config.xml => /home/cje-admin/t160/.dna/servers/castle/bundle/jce-castle-config.xml
Reconfiguring Marathon app /jce/castle at http://marathon.cje160-demo-controller-1313899594.us-east-1.elb.amazonaws.com.elb.cloudbees.net/v2/apps/jce/castle?force=true
Marathon ok: {"version":"2017-02-24T22:01:09.819Z","deploymentId":"5ce6b0f3-8c29-4df9-be78-fcd851fef4da"}
worker-add successfully applied
%

Configuring High Availability

On a CloudBees Jenkins Enterprise cluster, high availability is provided through an automatic failover within the cluster and requires no additional configuration by administrators.

The management layer of CloudBees Jenkins Enterprise automatically performs health checks on all managed masters in the cluster. If a health check fails, the management layer will schedule the re-deployment of that master to the cluster, potentially to a different underlying virtual machine.

When a Worker hosting a master fails and the Managed Master is rescheduled to a different Worker, the new Worker loads the dead Worker’s workspace from a Volume Service. The Volume Service regularly takes backups of the Managed Master’s workspace. The available Volume Service backend implementations are dependent on the cluster’s underlying infrastructure: NFS, EBS (AWS only), and Rsync.

Once re-deployed to a new Worker, the Managed Master will restart using its recovered workspace and will need to re-initialize before it can resume performing builds.

Managing the Cluster

Within the CloudBees Jenkins Enterprise cluster, controllers are virtual machines that host the management layer for the cluster. Controllers dictate what tasks to execute within the cluster and where to run those tasks. Controllers also service external requests and route traffic to other controllers within the cluster.

For a high availability (HA) cluster, administrators can configure multiple controllers to ensure there is no single point of failure in the cluster management layer. Administrators can also configure a load balancer to distribute traffic across the controllers.

A controller has the following responsibilities:

  • Manages the cluster resources by counting the used and available CPU, memory across the cluster, and offering them to execute tasks

  • Schedules tasks and ensures service availability within the cluster

  • Routes web traffic appropriately and if configured, terminates SSL for the cluster

Configuring Load Balancing

To balance incoming traffic between Controllers, administrators can configure load balancers to route traffic between the Controllers. The load balancers available to CloudBees Jenkins Enterprise will vary depending on which platform CloudBees Jenkins Enterprise has been deployed.

On Amazon Web Services (AWS), administrators can configure Elastic Load Balancers (ELB) to target all cluster controllers. At least two ELBs are created - one to route internal cluster traffic and one to route traffic from the outside.

On OpenStack, CloudBees Jenkins Enterprise creates and manages its own reverse proxy for load distribution, so administrators do not need to perform any additional configurations.

Managing CloudBees Jenkins Enterprise's Persistence Layer - Elasticsearch

The CloudBees Jenkins Enterprise environment includes an Elasticsearch cluster, which serves as both the persistence layer for CloudBees Jenkins Analytics and storage for the cluster’s logs.

Specific commands and configuration can be found in the Elasticsearch reference guide.

Alerting

CloudBees Jenkins Enterprise automatically monitors its infrastructure elements and all of the Jenkins masters that it manages. These alerts can be helpful in troubleshooting.

The CloudBees Jenkins Enterprise infrastructure element monitoring includes all Controller, Worker, Storage nodes (where appropriate), and Elasticsearch. CloudBees Jenkins Enterprise monitors health status for Elasticsearch. For the various infrastructure nodes, it monitors the following metrics.

  • Available disk space

  • CPU utilization for the most recent 5 minutes

  • RAM utilization for the most recent 5 minutes

If any of the data points for these metrics exceed 90% or more, a threshold which is currently immutable, CloudBees Jenkins Enterprise will emit an alert, for example:

Health checks failing: [worker-14: Disk util at 95%, worker-9: Worker down]

The following table show the possible error messages and corresponding descriptions.

Table 2. Possible Failure Messages
Messages Descriptions

Disk util at <number>%

Disk utilization reaches 90% or higher

RAM util at <number>%

RAM utilization reaches 90% or higher for five or more minutes

CPU util at <number>%

Total CPU utilization reaches 90% or higher for five or more minutes. The percent utilization is normalized to 100% across all CPU’s on the node.

Worker down

A worker node is either not reachable or its Mesos service is not operational

Controller down

A controller node is either not reachable or one or more of its core services are not operational

Mesos down

Mesos service is not operational on either a controller or worker node

Marathon down

Marathon service is not operational on a controller node

Zookeeper down

Zookeeper service is not operational on a controller node

Elasticsearch unreacheable
Timed out trying to reach Elasticsearch

Elasticsearch is experiencing connectivity difficulty

Cluster is in red status
Cluster is in yellow status

Elasticsearch is not functioning normally. Additional investigation is required

Backing Up a CloudBees Jenkins Enterprise Cluster

A CloudBees Jenkins Enterprise cluster project contains all configuration parameters and state information of the CloudBees Jenkins Enterprise Cluster. This information is needed to administer a cluster. Administrators should backup this information regularly to prevent potential information loss and loss of administrative access to the cluster.

Note
Every cluster operation can modify the cluster files. If the CloudBees Jenkins Enterprise cluster is managed by several Administrators, after performing an operation, changes should be pushed to the backup repository to make them available for others.

Administrators who create backups should take care that access to the backups are well-controlled - the cluster project and subsequent backup copy will contain sensitive information such as cloud API credentials, SSH keys and credentials to access the various CloudBees Jenkins Enterprise components (Mesos, Marathon, Elasticsearch, Operations Center, etc…​).

Preparing a CloudBees Jenkins Enterprise Cluster for Backup

The CloudBees Jenkins Enterprise CLI contains a set of commands to encrypt and decrypt a project using GPG private/public keys. To create a CloudBees Jenkins Enterprise backup, administrators will first need to ensure they have configured these keys for their cluster.

To prepare CloudBees Jenkins Enterprise for a backup, administrators will need to configure a GPG private/public key on the system used to manage the CloudBees Jenkins Enterprise cluster. See this GPG keys how-to for more information on how to generate GPG keys. Next, administrators will need to create an ACL file referencing the GPG key for encryption.

The ACL file 'secrets.acl' needs to be placed under the project directory and contain a reference to the GPG key UID. For example if the key user name is 'John Doe' and the email address of the user is "john.doe@acme.com", the secrets.acl content will be:

John Doe <john.doe@acme.com>

Backing Up CloudBees Jenkins Enterprise

To begin the backup, administrators must run the cje lock-project command and supply their GPG public key for the operation. Administrators can import all of their GPG public keys into their GPG key chain (See GPG keys how-to for more information about GPG key import) and add all key UIDs to the 'secrets.acl' (one per line) prior to locking the project. All GPG public keys must be trusted.

The cje lock-project command will encrypt the sensitive information of the project and place the project into a state that prevents further execution of any management commands. Once the project has been locked, administrators should use their preferred tool to create a backup of the CloudBees Jenkins Enterprise cluster in its current state.

For administrators using Git as their backup repository, CloudBees Jenkins Enterprise provides a pre-configured .gitignore file is provided to prevent sensitive information from being accidentally saved into a Git repository.

Once the backup is complete, the administrator must unlock the instance using the cje unlock-project command. This command decrypts the previously encrypted information and allows management commands to be executed.

Restoring a Master from a Backup

  1. The Managed Masters or Operations Center where you are performing the restore should be up and running otherwise the filesystem will not be mounted correctly.

  2. Find the worker where the Managed Masters or Operations Center is running using cje run list-applications | grep <MM-OC-NAME>.

  3. Copy the backed up Home or zip file to the /tmp folder of the worker using dna copy <BACKUP_FILE> /tmp <WORKER>.

  4. Connect to the worker using dna connect <WORKER>.

  5. Find the $JENKINS_HOME directory of the Managed Masters or Operations Center using mount | grep <MM-OC-NAME>.

  6. List the contents of the $JENKINS_HOME to ensure everything is as expected using ls -la /mnt/<MM-OC-NAME>/ID

  7. Take a backup of the current $JENKINS_HOME and put it into the /tmp folder using tar -xzf /tmp/currentBackup.tgz -C /mnt/<MM-OC-NAME>/ID <files_to_restore>

  8. Restoring:

    1. Whole $JENKINS_HOME:

      1. Remove the $JENKINS_HOME content

      2. Unzip the zip file into the $JENKINS_HOME from the /tmp folder using: tar -xzf /tmp/<BACKUP_FILE> -C /mnt/<MM-OC-NAME>/ID

      3. Ensure that the permissions are correct on the unzipped home folder

    2. Single file from the $JENKINS_HOME:

      1. Remove the file from $JENKINS_HOME.

      2. Unzip the backed up file from within the zipped up file located in the /tmp using: tar -xzf /tmp/<BACKUP_FILE> -C /mnt/<MM-OC-NAME>/ID <files_to_restore>

  9. Restart the Managed Masters or Operations Center

  10. If everything restored correctly, remove the backups from the /tmp folder using: rm /tmp/<BACKUP_FILE> /tmp/currentBackup.tgz.

Restoring workflow

Creating a 'Test' Master from a 'Production' Master Backup

  1. Ensure you have a backup of the Production master you would like to clone.

  2. In Operations Center, create a new Managed Master (or a new team master if your Production master is a team master), using the same Docker image and settings as the Production master.

  3. Once that new Test master is online and running, find the worker where the Test master is running using:

    1. cje run list-applications | grep <test-master-name>

  4. Copy the backed-up TAR or ZIP file for the Production master to the /tmp folder of the worker using:

    1. dna copy <BACKUP_FILE> /tmp <WORKER>

  5. Connect to the worker using:

    1. dna connect <WORKER>

  6. Find the $JENKINS_HOME of the Test Managed Master using:

    1. mount | grep <test-master-name>

  7. List the contents of $JENKINS_HOME to confirm everything looks correct:

    1. ls -la /mnt/<test-master-name>/ID

  8. Restore the Production master backup into a temporary directory:

    1. mkdir /tmp/prod-master-restore && cd /tmp/prod-master-restore && tar -xf /tmp/prod-master-backup.tar.gz

  9. Copy the following files from the current Test master to the /tmp/prod-master-restore directory:

    1. cd /mnt/<test-master-name>/ID && cp secret.key license.xml operations-center-client.xml /tmp/prod-master-restore/

  10. Confirm that file ownership and permissions are identical for both /mnt/<test-master-name>/ID and /tmp/prod-master-restore/.

  11. Delete the Test master’s data and move the restored Production master’s backup into place:

    1. cd /mnt/<test-master-name>/ID && rm -rf * && mv /tmp/prod-master-restore/* .

  12. Change directory to any directory outside of the /mnt/<test-master-name>/ID directory:

    1. cd ~

  13. Restart the Test Managed Master from the Operations Center web UI.

  14. If everything restored correctly then remove the backups from the /tmp folder using: rm -rf /tmp/prod-master-restore /tmp/prod-master-backup.tar.gz

Failure Recovery

Recovering from worker failures

Worker nodes constitute the engine that drives CloudBees Jenkins Enterprise, and occasionally, these worker nodes can become unproductive or unresponsive. Administrators can incorporate the following strategies to restore a worker node to full capacity.

Restarting a Worker Node

When attempting to restore a worker, administrators should first attempt a simple restart of the node. This approach is the fastest recovery strategy, and can often fix many temporary blockages.

Please follow the instructions in the CLI reference.

Replacing a Worker Node

If the worker node remains unresponsive after a restart, the administrator should simply replace the worker using the CLI.

This is a two-step process that starts with removing the worker. Once the worker has been successfully removed, the administrator can add a new worker.

Private Docker Registry

CloudBees Jenkins Enterprise allows you to use your own Private Docker Registry. The configuration of the Private Docker Registry can be done either during or after the installation of CloudBees Jenkins Enterprise.

It is recommended that you configure a Private Docker Registry as a mirror, so you can benefit from caching the images on your own local network instead of downloading them over and over again. See the official Docker documentation for more information.

Private Docker Registry After CloudBees Jenkins Enterprise Installation

The CloudBees Jenkins Enterprise CLI provides the operation docker-registry-update to configure your Docker Private Registry

$ cje prepare docker-registry-update
docker-registry-update is staged - review docker-registry-update.config and docker-registry-update.secrets and edit as needed - then run 'cje apply' to perform the operation.

docker-registry-update.config and docker-registry-update.secrets must to be configured.

In the case of docker-registry-update.config, the host and the port in which your Private Docker Registry is reachable must be specified. For example, if the Private Docker Registry is reachable in https://private-registry.example.com:5000, then the configuration would be docker_registry = private-registry.example.com:5000.

[tiger]

## Docker registry
#
# Specify the registry used to obtain CJE images.
# To use only public repositories on DockerHub, use 'public'.
# To use a private repository on DockerHub use 'docker.io'.
#
docker_registry = private-registry.example.com:5000

CloudBees Jenkins Enterprise only supports Basic Authentication, so you can only inject credentials with the style username:password. For example, for the credentials admin:admin, if we tranform them to base 64 encoded auth string, then it would be docker_registry_auth = YWRtaW46YWRtaW4=.

# Docker authorization (optional)
#
# If using a Docker registry that requires authentication, specify the
# base 64 encoded auth string used by Docker here.
#
docker_registry_auth = YWRtaW46YWRtaW4=

In case your Docker Private Registry is configured in HTTPS mode using a self-signed certificate, you will need to add the certificate to all the workers through the certificates-update operation.

$ cje prepare certificates-update
certificates-update is staged - review certificates-update.config and edit as needed - then run 'cje apply' to perform the operation.

Then, as specified in certificates-update.config, all the certificates must be in .pem format and they must be located under $CJE_PROJECT/certificates. In case there are several certificates, they must be added one by one in .pem format.

# Self-signed Certificate
#
# If the SSL certificate is self-signed or uses a custom certificate authority, it needs to be trusted by the
# workstation running the installation and must be copied to all machines in the cluster.
#
# Such certificate(s) can be dropped in certificates/ directory in the project directory.
# They will be picked up automatically and installed on all VMs/containers involved in the cluster.
#
# The certificate files must end with .pem extension.
#
# This operation only takes the certificates files as input

Once the certificates are added, complete the operation with cje apply.

Affected workers must be restarted for all these changes take effect.

$ cje prepare worker-restart

Last, go to the configuration file created by the command above and add each of the worker names (separated by comma).

[server]

## The name of the servers to restart
#
names = worker-1,worker-2,...,worker-N

Private Docker Registry in Cluster Initialization

During the cluster initialization, the configuration of the Private Docker Registry can be done in the cluster initialization files:

  • cluster-init.config

  • cluster-init.secrets

The configuration of the files is done in the same way as those in the preceding section - the only difference being that workers do not need to be restarted afterward.

cluster-init.config

# Docker authorization (optional)
#
# If using a Docker registry that requires authentication, specify the
# base 64 encoded auth string used by Docker here.
#
docker_registry_auth =
cluster-init.secrets

# Docker authorization (optional)
#
# If using a Docker registry that requires authentication, specify the
# base 64 encoded auth string used by Docker here.
#
docker_registry_auth =

Private Docker Registry Mirror

If your Private Docker Registry is working as a mirror, then the Docker Daemon in the corresponded workers needs to be configured to look for the images in your mirror instead of directly on the Internet.

The configuration of the Private Docker Registry is done at the CJE CLI project level under .dna/project.config.

In order to have the Docker daemon configured to use your private Docker registry, you will need to edit in .dna/project.config the docker_options inside each worker element, by adding --registry-mirror https://<DOCKER_PRIVATE_REGISTRY_URL>:<DOCKER_PRIVATE_REGISTRY_PORT&gt;.

[worker-2:worker]
volume_size = 50
volume_type = gp2
workload_type = master
mesos_slave_attributes = jce_cjoc:true;jce_masters:true
instance_type = m4.xlarge
ami = ami-52836d28
ebs_optimized = false
availability_zone = us-east-1d
instance_profile = cje-workers
use_ephemeral =
mesos_logging_level = WARNING
mesos_logging_retention = 30
docker_options = --log-driver=syslog --registry-mirror https://<DOCKER_REGISTRY_HOSTNAME>:<DOCKER_REGISTRY_PORT>
ssh_user = ubuntu
ssh_identity = default
image_distribution = ubuntu
resource_tags =
Warning

In case your Private Docker Registry is not secured (http), then:

  • change the --registry-mirror protocol from https to http

  • add --insecure-registry <DOCKER_REGISTRY_HOSTNAME>:<DOCKER_REGISTRY_PORT>. Note that it does NOT require any protocol (check the example below).

An example of the resulting arguments would look similar to: --registry-mirror http://mydocker-registry:5000 --insecure-registry mydocker-registry:5000.

Once .dna/project.config is updated, refresh the CloudBees Jenkins Enterprise project by executing the following command.

$ cje upgrade --config-only --force

Affected workers must be restarted for all these changes take effect.

$ cje prepare worker-restart

Last, go to the configuration file created by the command above and add each of the worker names (separated by comma).

[server]

## The name of the servers to restart
#
names = worker-1,worker-2,...,worker-N

Refreshing Operations Center and CloudBees Jenkins Enterprise master images

These instructions provide guidance on how to upgrade Docker images for Operations Center and CloudBees Jenkins Enterprise masters.

CloudBees provides monthly releases of Docker images. The image tags are provided on the release notes page.

  • The following Docker images provided by CloudBees are hosted on Docker Hub:

    • CJOC utilizes cloudbees/cje-oc:1.6.0

    • Managed Masters utilizes cloudbees/cje-mm:1.6.0

Process

  1. Updating Operations Center images.

    From the bastion host or CloudBees Jenkins Enterprise project directory location, run the following commands/perform the following:

    1. cje prepare cjoc-update

      • Follow the instructions in the console output.

    2. cjoc-update is staged; review the cjoc-update.config file and edit as needed.

    3. Run cje apply to perform the operation.

    4. Edit the cjoc-update.config file again.

    5. Set the Custom Docker image tag to the version of Operations Center.

    6. Uncomment the line docker_image and set the version as such.

      • Example: docker_image = cloudbees/cje-oc:2.138.2.2

        cjoc update
    7. Run cje verify, and if no errors, then run cje apply.

  2. Updating master images.

    • Follow the next set of instructions below or reference the full guide on upgrading managed masters here:

    • Upgrading Managed Masters

      1. From Operations Center navigate to → Manage Jenkins → Configure System

      2. Locate the section Container Master Provisioning and click Show Docker Images

      3. Click Add and fill in the input fields Name & Image Tag

      4. Name should follow the existing naming convention such as CloudBees Jenkins Enterprise 2.138.2.2

      5. Tag needs to be defined as the tag listed on Docker Hub and our release notes cloudbees/cje-mm:2.138.2.2

      6. Click Save, the updated image will be applied to any new Teams/Masters.

  3. Upgrading Teams/Masters

    1. Follow the existing documentation on upgrading masters.

    2. From the Master list the gear icon and choose configure select the image from the drop down list; click save and then restart the master to apply the updated docker image.