Table of Contents

CloudBees Core Reference Architecture - Kubernetes on AWS

Introduction

This document is designed to help you ensure that your Amazon Web Services-based Kubernetes cluster is optimally configured for running CloudBees Core in a secure and efficient way.

This document includes the following topics:

For more information on Kubernetes please refer to the official Kubernetes Documentation.

Terms and Definitions

Jenkins

Jenkins is an open-source automation server. Jenkins is an independent open-source community, to which CloudBees actively contributes. You can find more information about Jenkins and CloudBees contributions on the CloudBees site.

CloudBees Core

Commercial version of Jenkins based on Jenkins Long-Term Support (LTS) releases with frequent patches by CloudBees. CloudBees Core also provides a number of plugins that help organizations address main needs of enterprise installations: Security, Continuous Delivery, etc.

CloudBees Core Operations Center

operations console for Jenkins that allows you to manage multiple Jenkins masters. See details on the CloudBees site.

Architectural Overview

This section provides a high-level architectural overview of CloudBees Core, designed to help you understand how CloudBees Core works, how it integrates with Kubernetes, its network architecture and how Managed Masters and build agents are provisioned.

CloudBees Core is essentially a set of Docker Containers that can be deployed to run a cluster of machines within the Kubernetes container management system. Customers are expected to provision and configure their Kubernetes system before installing CloudBees Core.

CloudBees Core includes the CloudBees Core Operations Center that provisions and manages CloudBees Managed Masters and Team Masters. CloudBees Core also enables Managed Masters and Team Masters to perform dynamic provisioning of build agents via Kubernetes.

Machines and Roles

CloudBees Core is designed to run in a Kubernetes cluster. For the purposes of this section, you need to know that a Kubernetes cluster is a set of machines (virtual or bare-metal), which run Kubernetes. Some of these machines provide the Kubernetes control plane as Kubernetes Masters. They control the Containers that run on the other type of machines known as Kubernetes Nodes. The CloudBees Core containers will run on the Kubernetes Nodes.

The Kubernetes Masters provide an HTTP-based API that can be used to manage the cluster, configure it, deploy containers, etc. A command-line client called kubectl can be used to interact with Kubernetes via this API. You should refer to the Kubernetes documentation for more information on Kubernetes.

CloudBees Core Docker Containers

The Docker containers in CloudBees Core are:

  • cloudbees-cloud-core-oc: CloudBees Core Operations Center

  • cloudbees-core-mm: CloudBees Core Managed Master

The Docker containers used as Jenkins build agents are specified on a per Pipeline basis and are not included in CloudBees Core. Refer to the example Pipeline in the Agent Provisioning section for more details.

The cloudbees-cloud-core-oc, cloudbees-core-mm and build agent container images can be pulled from the public Docker Hub repository or from a private Docker Registry that you deploy and manage. If you wish to use a private registry, you will have to configure your Kubernetes cluster to do that.

CloudBees Core Kubernetes Resources

CloudBees Core is deployed to a Kubernetes cluster by applying the CloudBees Core Kubernetes configuration file, cloudbees-core.yml a YAML file that defines the set of Kubernetes resources required by CloudBees Core. This is done via a single command-line: kubectl create -f cloudbees-core.yml but to understand what happens when that command is executed, you need to understand several more Kubernetes concepts:

Kubernetes concepts

Pod

A set of Containers that share storage volumes and a network interface.

ServiceAccount

Defines an account for accessing the Kubernetes API.

Role

Defines a set of permission rules for access to the Kubernetes APIs.

RoleBinding

Binds a ServiceAccount to a Role.

ConfigMap

A directory of configuraton files made available on all Kubernetes Nodes.

StatefulSet

Managing deployment and scaling of a set of Pods.

Service

Provides access to a set of Pods at one or more TCP ports.

Ingress

Uses hostname and path of an incoming request to map the request to a specific Service.

For more details about the above concepts, refer to the Kubernetes documentation.

CloudBees Core Kubernetes resources

These are the Kubernetes resources that are created by the CloudBees Core YAML file:

ServiceAccount

jenkins - Account used to managed Jenkins Build Agents.

ServiceAccount

cjoc - Account used by Operations Center to manage Managed Masters. Role: master-management - Defines permissions needed by Operations Center to manage Jenkins Masters.

RoleBinding

cjoc - Binds the Operations Center ServiceAccount to the manage-masters Role.

RoleBinding

jenkins - Binds the jenkins ServiceAccount to the pods-all Role.

ConfigMap

cjoc-config - Defines the configuration used to start the Operations Center Java process within the Operations Center Container.

ConfigMap

cjoc-configure-jenkins-groovy - Defines location.groovy, which is executed by Operations Center on startup to set its own hostname.

ConfigMap

jenkins-agent - Defines the Bash script that is executed to start the Jenkins Agent within a Build Agent Container.

StatefulSet

cjoc - Defines a Pod for the Operations Center Container, allocates a persistent volume for its Jenkins Home directory and ensures that one such Pod is always running.

Service

cjoc - Defines a Service front-end for the Operations Center Pod with TCP ports 80 and 50000 for JNLP.

Ingress

default - Maps requests for the CloudBees Core hostname and path "/cjoc" to the Operations Center Pod.

Ingress

cjoc - Maps requests for the CloudBees Core hostname to path path "/cjoc".

When you initially deploy CloudBees Core, you will see a Operations Center Pod running and within that, a Operations Center Container. You can login to Operations Center and provision Managed Masters to run your Pipelines.

Visualizing CloudBees Core Architecture

The diagram below illustrates the CloudBees Core architecture on Kubernetes. The diagram shows three Kubernetes Master Nodes, which are the three dotted-line and overlapping rectangles on the left. The diagram also shows two Kubernetes Worker Nodes, which are the two dotted-line large rectangles in the center and right.

Here’s the key for the colors used in the diagram:

  • Green: processes which are part of Kubernetes

  • Pink: Kubernetes resources created by installing and running CloudBees Core

  • Yellow: Kubernetes resources required by CloudBees Core

Architecture diagram
Figure 1. CloudBees Core Architecture

Kubernetes Master

Running on each Kubernetes Master node, there are the Kubernetes processes that manage the cluster: the API Server, the Controller Manager and the Scheduler. In the bottom left of the diagram are resources that are created as part of the CloudBees Core installation, but that are not really tied to any one node in the system.

Kubernetes Nodes

On the Kubernetes Nodes and shown in green above is the kubelet process, which is part of Kubernetes and is responsible for communicating with the Kubernetes API Server and starting and stopping Kubernetes Pods on the Node.

On one Node, you see the Operations Center Pod which includes a Master Provisioning plugin that is responsible for starting new Master Pods. On the other node you see a Master Pod, which includes the Jenkins Kubernetes Plugin and uses that plugin to manage Jenkins Build Agents.

Each Operations Center and Master Pod has a Kubernetes Persistent Volume Claim where it stores its Jenkins Home directory. Each Persistent Volume Claim is backed by some form of storage service, such as an EBS volume on AWS or an NFS drive in an OpenShift environment. When a Master Pod is moved to a new Node, its storage volume must be detached from its old Node and then attached to the Pod’s new node.

Master Provisioning

One of the benefits of CloudBees Core is the easy provisioning of new Jenkins Managed masters from the Operations Center web interface. This feature is provided by the CloudBees Core Master Provisioning Plugin for Jenkins. When you provision a new master, you must specify the amount of memory and CPU to be allocated to the new master, and the provisioning plugin will call upon the Kubernetes API to create a master.

The diagram below shows what happens when a new Master is launched via Operations Center. First, CJOC’s Master Provisioning Kubernetes Plugin calls Kubernetes to provision a new StatefulSet to run the Managed Master Pod.

Master Provisioning Diagram
Figure 2. Master Provisioning

Agent Provisioning

Agents are created and destroyed in CloudBees Core by the Jenkins Kubernetes Plugin. A Jenkins Pipeline can specify the build agent using the standard Pipeline syntax. For example, below is a CloudBees Core Pipeline that builds and tests a Java project from a GitHub repository using a Maven and Java 8 Docker image:

Pipeline Example
podTemplate(label: 'kubernetes',
  containers: [
    containerTemplate(name: 'maven', image: 'maven:3.5.2-jdk-8-alpine', ttyEnabled: true, command: 'cat')
  ]) {
  stage('Preparation') {
    node("kubernetes") {
      container("maven") {
        git 'https://github.com/jglick/simple-maven-project-with-tests.git';
        sh "mvn -Dmaven.test.failure.ignore clean package"
        junit '**/target/surefire-reports/TEST-*.xml'
        archive 'target/*.jar'
      }
    }
  }
}

In the above example, the build agent container image is maven:3.5.2-jdk-8-alpine. It will be pulled from the Docker Registry configured for the Kubernetes cluster.

The diagram below shows how build agent provisioning works. First, when the Pipeline runs, the Kubernetes Plugin on the Managed Master calls Kubernetes to provision a new pod to run the build agent container. Second, Kubernetes launches the build agent pod to execute the Pipeline.

Agent Provisioning Diagram
Figure 3. Agent Provisioning

CloudBees Core Required Ports

CloudBees Core requires open ports:

  • 80 for http access to the web interface of Operations Center and Managed Masters

  • 443 for https access to the web interface of Operations Center and Managed Masters

  • 50000 for Java Network Launch Protocol (JNLP) access to Operations Center and Managed Masters

Refer to the Kubernetes documentation for its port requirements.

Network Encryption

Network Communication between Kubernetes clients such as kubectl, Kubernetes masters and nodes is encrypted via TLS protocol. The Kubernetes Managing TLS in a Cluster document explains how Certificates are obtained and managed by a cluster.

Communication between application containers running on a Kubernetes cluster, such as Operations Center and Managed Masters, can be encrypted as well but this requires the deployment of a Network Overlay technology such as Weave Works. End-to-end Web Browser to CloudBees Core communications can be TLS encrypted by configuring the Kubernetes Ingress that provides access to CloudBees Core to be the termination point for SSL. Network Overlay and SSL termination configuration is covered in a separate section.

High Availability

Kubernetes can be configured for high availability by using at least three Kubernetes Masters on three separate machines in different Availability Zones.

Persistence

Operations Center and Managed Masters store their data in file-system directory, known as Jenkins Home. Operations Center has its own Jenkins Home, and each Master also has one.

CloudBees Core uses a Kubernetes feature known as Persistent Volume Claims to dynamically provision persistent storage for Operations Center, each Managed Master and build agents.

Cluster Sizing and Scaling

This document provides general recommendations about sizing and scaling a Kubernetes cluster for CloudBees Core starting with some general notes about minimum requirements and ending with a table of more concrete sizing guidelines recommended by CloudBees.

General notes

When sizing and scaling a cluster you should consider the operational characteristics of Jenkins. The relevant ones are:

  • Jenkins Masters are memory and disk IOPS bound, with some CPU requirements as well. Low IOPS results into longer startup times and worse general performance. Low memory results into slow response time.

  • Build Agents requirements depend on the kind of tasks being executed on them.

Pods are defined by their CPU and memory requirement and they can’t be split across multiple hosts.

It is recommended to use hosts that are big enough so that they can host several pods (Rule of thumb : 3-5 pods per host) at the same time to maximize their actual use.

Example: You are running builds requiring 2 GB of memory each. You need configure pods to have 2 GB each for supporting such builds. The rule of thumb says you should have hosts with 6-10 GB of memory (3 x 2 - 5 x 2).

Depending on your cloud provider, it may be possible to enable auto-scaling in Kubernetes to match with the actual requirements and reduce the operational costs.

If you don’t have auto-scaling in your environment, we recommend you to plan extra capacity in order to sustain hardware failure.

Storage

Each Managed Master is provisioned on a separate Persistent Volume (PV). It is recommended to use a storage class with the most IOPS available.

The host storage is not getting used by Managed Masters but depending on the instance type you may have restrictions on the kind of block storage you can use (for example, on Azure, you need to use an instance type ending with s).

Disk space on the hosts is necessary to host docker images, containers and volumes. Build workspaces will be on host storage so there must be enough free disk space available on nodes.

CPU

CloudBees Core uses the notion of CPU defined by Kubernetes.

By default, a Managed Master requires 1 CPU. Each build agent also requires CPU, so what will determine the total CPU requirement is :

  • (mostly static) The number of Managed Masters multiplied by the number of CPU each of them requires.

  • (dynamic) The number of concurrent build agents used by the cluster multiplied by the CPU requirement of pod template. A minimum amount of 1 CPU is recommended for a pod template but you can use more cpus if parallel processing is required by the task.

Most build tasks are CPU-bound (compilation, test executions). So it is quite important when defining pod templates not to underestimate the number of cpus to allocate if you want good performance.

Memory

By default, a Managed Master requires 3 GB of RAM.

To determine the total memory requirement, take into account:

  • (mostly static) The number of Managed Masters multiplied by the amount of RAM each of them requires.

  • (dynamic) The number of concurrent build agents used by the cluster multiplied by the memory requirement of pod template

Memory also impacts performance. Not giving enough memory to a Managed Master will cause additional garbage collection and reduced performance.

Master Sizing Guidelines

Below are some more concrete sizing guidelines compiled by CloudBees Support Engineers:

Table 1. Master sizing guidelines
Requirement Baseline Rationale

Team Size

10

Most agile resources warn against going above 10 team members. Keeping the team size at 10 or below facilitates the sharing of knowledge about Jenkins and pipeline best practices.Three items

Average Weekly Users

20

Besides the team themselves, other non-team collaborators often must access the team’s Jenkins to download artifacts or otherwise collaborate with the team. This includes API clients.

Serving the Jenkins user interface impacts IO and CPU consumption and will also result in increased memory usage due to the caching of build results.

CPU Cores

4

A Jenkins of this size should have at least 4 CPU cores available.

Maximum Concurrent Builds

50

Healthy agile teams push changes multiple times per day and may have a large test suite including unit, integration and automated system tests.

We generally observe Jenkins easily handles up to 50 simultaneous builds, with some Jenkins regularly running many multiples of this number. However, poorly written or complicated pipeline code can significantly affect the performance and scalability of Jenkins since the pipeline script is compiled and executed on the master.

To increase the scalability and throughput of your Jenkins master, we recommend that Pipeline scripts and libraries be as short and simple as possible. This is the number one mistake teams make. If build logic can possibly be done in a Bash script, Makefile or other project artifact, Jenkins will be more scalable and reliable. Changes to such artifacts are also easier to test than changes to the Pipeline script

Maximum Number of Pipelines (Multi-branch projects)

75

Well-designed systems are often composed of many individual components. The microservices architecture accelerates this trend, as does the maintenance of legacy modules.

Each pipeline can have multiple branches, each with its own build history. If your team has a high number of pipeline jobs, you should consider splitting your Jenkins further.

Recommended Java Heap Size

4 GB

We regularly see Jenkins of this size performing well with 4 gigabytes of heap. This means setting the -Xmx4g as recommended in option B of our KB article on the topic.

If you observe that your Jenkins instance requires more than 8 gigabytes of heap, your Jenkins likely needs to be split further. Such high usage could be due to buggy pipelines or perhaps non-verified plugins your teams may be using.

AWS Auto-scaling groups

With a cluster set up on AWS (including EKS), it is possible to define one or several auto-scaling groups. This can be useful to assign some pods to specific nodes based on their specification.

Targeting specific nodes / segregating pods

When defining pod templates using the Jenkins Kubernetes plugin, it is possible to assign pods to nodes with particular labels.

For example, the below pipeline code would create a pod template restricted to instance type m4.2xlarge.

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')
  ],
  nodeSelector: 'beta.kubernetes.io/instance-type=m4.2xlarge') {
  node(label) {
    // some block
  }
}

Assigning pods to particular nodes is very useful if you wish to use particular instance types for certain types of workloads.

Please read Assigning Pods to Nodes to understand this feature more in details.

Warning
It is not yet possible to restrict the nodes on which masters are assigned.

Installing Kubernetes on AWS via kops

kops is a software package that helps to create and manage Kubernetes clusters on a variety of platforms and is a good choice for creating and managing a Kubernetes cluster on AWS.

The definitive source of information on kops is the kops GitHub repository and that’s where you will find instructions for installing Kubernetes via kops.

This section explains how to use kops to deploy a Kubernetes cluster that is suitable for running CloudBees Core. At a high-level these are the steps you should take:

  • Decide how to setup your Kubernetes cluster

  • Download and install kops and dependencies

  • Use kops to create your cluster

  • Setup NGINX Ingress Controller

  • Optionally enable encrypted EBS

  • Install CloudBees Core

Each of these items is explained below.

Decide how to setup your Kubernetes cluster

Before you get started you should plan how you want to setup your Kubernetes cluster.

Decide how many EC2 instances, which instance types and AMI

If you do not specify the number of nodes, instance types and which AMI to use, then kops will use default values. Currently, the default is to create a cluster with one Kubernetes Master node and two Worker Nodes. The master will use instance type m3.medium (1 CPU, 3.75 GB memory) and the workers will use t2.medium (2 CPU, 4 GB memory). All nodes will run Debian GNU/Linux 8 (Jessie). That is not a sufficiently powerful cluster for running CloudBees Core.

Do the CPU and memory math

For a production CloudBees Core cluster, you should run three Kubernetes Masters Nodes and then as many Kubernetes Worker Nodes as you need to run the Operations Center, the number of Managed Masters and Build Agents that you wish to run. Each Kubernetes Node will be an AWS EC2 instance. Refer to the above section on Sizing & Scaling and do the math to determine how many Worker Nodes you will need and what instance types you will use for them.

In the examples below we will create a cluster that can support Operations Center, two Managed Masters and thee builds that each consume 2GB of memory when running.

Here’s the math for our example memory requirements:

Operations Center             3 GiB
Managed Masters    2 x 3 GB = 6 GiB
Build Agents       3 x 2 GB = 6 GiB
                             ------
TOTAL                        15 GiB

And here are the numbers for CPUs:

Operations Center             1 vCPU
Managed Masters  2 x 1 vCPU = 2 vCPU
Build Agents     3 x 1 vCPU = 3 vCPU
                              ------
TOTAL                         6 vCPU

Based on that, in the examples, we will use two m4.xlarge(4 CPU, 16 GB memory) for worker nodes, which should exceed our requirements and give us room to grow. Master nodes don’t run containers and don’t need to be as large, so we will use m4.large for these.

Pick a suitable AMI

If you don’t want to use the default Linux, then do some research and find an AMI that is suitable for you (and works in your chosen AWS Region).

Decide which DNS to use

DNS configuration in kops can be tricky, and you have several different options from which to choose. So read the kops documentation section about how to Configure DNS very carefully.

In summary, kops can be configured to use Route 53 as the DNS for the public hostnames used to access the cluster and as the DNS used for Kubernetes internal hostnames and discovery.

kops can also be configured to skip all DNS configuration and use Gossip Protocol instead for internal discovery. If you choose to use Gossip instead of Route 53 for discovery, then you are free to use any DNS service (including Route 53) to setup the public hostnames used to access your cluster.

For some situations, the "Gossip plus your favorite DNS" option is the easiest to setup. If you to use this option, make your cluster name ends with .k8s.local; that’s will indicate to kops that you want Gossip.

Decide how to setup internal Networking

By default, kops will create a cluster that uses the Kubenetes Container Network Interface (CNI) implementation, one which does not provide a private network topology.

If you need to ensure that communication between CloudBees Core components is encrypted, then you need to pick a CNI implementation that supports private topology and specify it with the --networking option and use the --topology private option when you create your cluster. See the kops documentation on Network Topologies for details.

Now that you’ve planned-out how you will configure your cluster for CloudBees Core, you’re ready to get started with kops. The first step is a download.

Download and install kops and dependencies

Follow the instructions in the kops AWS Getting Started Tutorial to download and install kops. Read through the entire tutorial before you proceed.

Use kops to create your cluster

Follow the instructions in the kops AWS Getting Started Tutorial create your Kubernetes cluster with kops and when you get to point where you need to run the kops create cluster command, stop and make sure you are specifying the right options for CloudBees Core.

Follow the instructions to determine which options to specify in the kops create cluster command. For CloudBees Core your kops create cluster command should include the following options:

--ssh-public-key            # File name of SSH your public key
--master-count              # Number of Kubernetes Master nodes to be created (should be 3)
--node-count                # Number of Kubernetes Worker nodes to be created
--master-size               # EC2 instance type to be used for master nodes
--node-size                 # EC2 instance type to be used for worker nodes
--zones                     # Comma-separated list of availability zones
--image                     # AMI that you wish to use for EC2 instances
--authorization  RBAC       # RBAC is required

If you wish to setup a private topology, then you should also specify these options:

--topology private          # Specify private network topology
--networking weave          # Specify Weave Net networking

Example cluster creation script

The example script below shows the commands needed to create a CloudBees Core suitable Kubernetes cluster with kops. The cluster has three master nodes of type m4.large, two workers of type m4.xlarge and all nodes are running a CentOS 7 AMI. The cluster has private network topology via Weave Net and because the cluster’s name ends with .k8s.local, Gossip protocol will be used and DNS setup will be skipped.

export NAME=mycluster.k8s.local
export AWS_REGION=us-west-2
export KOPS_STATE_STORE=s3://${NAME}
export SSH_PUBKEY=~/.ssh/mykey.pem
aws s3api create-bucket --bucket $NAME --region $AWS_REGION \
    --create-bucket-configuration LocationConstraint=$AWS_REGION
aws s3api put-bucket-versioning --bucket $NAME --versioning-configuration Status=Enabled
ssh-keygen -y -f $SSH_PUBKEY > mykey.pub
kops create cluster \
    --ssh-public-key mykey.pub \
    --authorization  RBAC \
    --master-count   3 \
    --node-count     2 \
    --master-size    m4.large \
    --node-size      m4.xlarge \
    --topology       private \
    --networking     weave \
    --image          ami-02c71d7a \
    --zones          us-west-2a,us-west-2b,us-west-2c \
    --cloud-labels   owner=myname \
    $NAME
kops create secret --name $NAME sshpublickey admin -i mykey.pub
kops update cluster ${NAME} --yes

If can take some time for your cluster to start, usually around five minutes. You can use the kops validate cluster command to check and see if your cluster is ready.

Setup NGINX Ingress Controller

Once your cluster is up and running, your next step should be to install the NGINX Ingress Controller, which is required by CloudBees Core. Read the NGINX Ingress Controller Installation Guide for instructions on how to install the controller and make sure you choose the RBAC option, as your newly created Kubernetes cluster uses RBAC.

Setup DNS record for the Ingress Controller ELB

The installation of the NGINX Ingress Controller will result in the creation of an ELB and each of your Kubernetes Worker Nodes will be "in-service" on that new ELB. Use the AWS Console to find that ELB, make a note of its hostname and create a DNS record for the hostname you wish to use for CloudBees Core.

Optionally enable encrypted EBS

Next, if you want to ensure that all Jenkins Home data stored by the CloudBees Core Operations Center and Managed Masters is encypted, then you should follow the instructions in the Reference Architecture’s Enabling Storage Encryption section and enable encryption. It’s important to do this before you install CloudBees Core.

Install CloudBees Core

Now that you’ve got your Kubernetes cluster up and running with an NGINX Ingress Controller, you are ready to proceed with installation of CloudBees Core. Follow the instructions here: Installing CloudBees Core on Kubernetes

Example: AWS Network Architecture set up by kops

Kubernetes clusters can be configured in a variety of different ways. Let’s examine how the popular Kubernetes Operations (kops) tool configures Kubernetes on AWS.

In the diagram below you can see that there are two AWS Elastic Load Balancers (ELBs), each with its own AWS Security Group. There is one ELB that fronts the Kubernetes Masters and provides access to the Kubernetes API which is accessed, for example, by the kubectl command-line client. The other ELB fronts the Kubernetes Nodes and provides access to the Containers that are running on the cluster, in the case of CloudBees Core, these are Operations Center and Managed Masters.

Diagram: AWS EC2 Instances and Security Groups

Kubernetes Network Diagram

kops Network Ports

In a kops created Kubernetes cluster on AWS, Security Groups are used to control access to Kubetetes Masters and Nodes. You can see the four Security Groups created in the above diagram. These are the ports that are open for each Security Group:

API ELB Security Group

  • HTTP from anywhere

  • TCP 443 from anywhere

Ingress ELB Security Group

  • HTTP from anywhere

  • ICMP (fragmentation required, DF flag set) from anywhere

  • TCP 443 from anywhere

Master Security Group

  • SSH from anywhere

  • HTTP from anywhere

  • All from the Masters Security Group

  • Allows from the API ELB Security Group:

    • HTTP

    • TCP 443

    • TCP 4003-65536

  • Allows from the Node Security Group:

    • TCP 1-2379

    • TCP 2382-4000

    • UDP 1-65536

Node Security Group

  • All from the Node Security Group

  • All from the Masters Security Group

  • All from the Ingress ELB Security Group

Note
CloudBees does not provide support for kops. kops is discussed here as one of many alternatives for installing Kuberbetes. Refer to the kops documentation for more kops information. See the CloudBees website for CloudBees support options.

Setting up a Private and Encrypted Network

Use a Private and Encrypted Network to ensure that all network communication between the CloudBees Core Operations Center, Managed Masters and Build Agents is encrypted.

Network communication between the Kubernetes components running on Kubernetes Masters and Kubernetes nodes is encrypted using the TLS protocol. Each Kubernetes cluster includes a Certificate Authority (CA) that it uses to create and validate the TLS certificates used when Kubernetes components communicate with the Kubernetes API server. Refer to the Kubernetes documentation page Manage TLS Certificates in a Cluster for more information.

Applications deployed to a Kubernetes cluster, such as CloudBees Core, are not protected by Kubernetes built-in TLS encryption. To ensure that all network communication between application components is encrypted, configure your Kubernetes cluster to use a Network Policy Provider that supports encryption. The Kubernetes documentation Declare Network Policy page lists the providers that are supported.

Select a Network Policy Provider: Weave Net

The CloudBees Core development team has had some experiences using Weave Net with Kubernetes, but has not deployed Weave Net in production, so Weave Net is only our preliminary recommendation. We selected Weave Net for this document because of those experiences and because Weave Net uses IPSec protocol and its encryption uses the well-known NaCl library.

Weave Net is open source software licensed under the Apache Software Licence v2 and developed by Weaveworks with source code hosted at the Weave GitHub project. You can read more about the product and its requirements on the Weave Net overview page.

The Kubernetes documentation page Weave Net for Network Policy explains how to setup Weave and additional details are provided at the Weave Net documentation page Integrating Kubernetes via the Addon.

The rest of this section gives an outline of the steps you need to take to configure Weave Net to use encryption. In summary, if you provide a password-secret to Weave Net then encryption is enabled. The full documentation for this is on the Weave Net documentation page Changing Configuration Options.

Create a Kubernetes Secret to be used by Weave Net

First you need to create a Kubernetes Secret containing the password you wish to use for Weave Net. For example, you can do this via Bash. Create a weave-secret file with your password:

cat > weave-secret << EOF
s3cr3tpa55w0rd
EOF

Next, call Kubernetes to create a Secret named weave-secret:

kubectl create secret -n kube-system generic weave-secret --from-file=./weave-secret

Now we can deploy or update Weave Net to use that secret.

Deploy Weave Net with encryption enabled

The following command will deploy Weave Net to your Kubernetes cluster with encryption enabled, or will update your existing Weave Net set-up:

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')&password-secret=weave-secret"

You must wait a couple of minutes for Weave Net to start or restart, and once it is running you should be able to see it running in the kube-system namespace, for example:

$ kubectl get pods -n=kube-system | grep weave
weave-net-dqn8k                                        2/2       Running   0          2h
weave-net-lzxzt                                        2/2       Running   0          2h
weave-net-mhp2g                                        2/2       Running   0          2h

And should be able to see that the password option is set in each Pod via the kubectl describe command, for example:

$ kubectl -n=kube-system describe pods weave-net-dqn8k | grep PASS
      WEAVE_PASSWORD:  <set to the key 'weave-secret' in secret 'weave-secret'>  Optional: false

If you have problems, you may find the Weave Net Troubleshooting page helpful.

Ingress TLS Termination

Ingress TLS Termination should be used to ensure that network communication to the CloudBees Core web interfaces is encrypted from end to end.

If you would like to ensure that web browser to CloudBees Core communications is encrypted end-to-end then you will need to change the Kubernetes Ingress used by CloudBees Core to use your TLS Certificates, thereby making it the termination point for TLS.

This section provides an overview of the changes you’ll need to make, but the definitive guide to setting this up is in the Kubernetes Ingress TLS documentation.

Store your TLS Certs in a Kubernetes Secret

To make your TLS Certs available to Kubernetes, you must create a Kubernetes Secret using the Kubernetes command-line tool kubectl. For example, if your certs are in /etc/mycerts you would issue this command to create a secret named my-certs:

kubectl create secret tls my-certs \
--cert=/etc/mycerts/domain.crt --key=/etc/mycerts/privkey.pem

For more information and options see the definitive guide to secrets in the Kubernetes Secrets documentation.

Change the two CloudBees Core Ingresses to be the TLS termination point

Next, you must configure the CloudBees Core Ingress to use your TLS Certs. You do this by editing the Ingress resource named "cjoc" in the cloudbees-core.yml configuration file, uncommenting the lines around tls: and setting the correct hostname and setting the secretName to the secret that holds your TLS certificates. For example, if your CloudBees Core host name is cje.example.org and your TLS Certs secrets file is named my-certs then this is what the new CloudBees Core resource should look like.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cjoc
  annotations:
    kubernetes.io/ingress.class: "nginx"
    #nginx.ingress.kubernetes.io/ssl-redirect: "true"
    # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  tls:
  - hosts:
    - cje.example.org
    secretName: my-certs
  rules:
  - http:
      paths:
      - path: /cjoc
        backend:
          serviceName: cjoc
          servicePort: 80
    host: cje.example.org

And make the same change to the Ingress named "default".

Next you need to deploy CloudBees Core or update your existing CloudBees Core deployment to use your new Ingress definition. Either way, you need to run this command:

kubectl create -f cloudbees-core.yml

If CloudBees Core has not been deployed, then that command will deploy it. If CloudBees Core has already been deployed then that command will update the existing configuration.

Domain Name Change

  • Stop all Managed Masters/Teams Masters from Operations Center dashboard. This can be achieved either automatically with a cluster operation or manually using Managed Master/Team Master  Manage.

  • Add the new domain name by modifying the hostname in ingress/cjoc and cm/cjoc-configure-jenkins-groovy. There are two ways to do this:

    • By changing those hostname values in the cloudbees-core.yml file.

    • By editing the Operations Center ingress resource and modifying the domain name.

      $ kubectl edit ingress/cjoc

      Modify the Operations Center configuration map to change the cjoc URL.

      $ kubectl edit cm/cjoc-configure-jenkins-groovy
  • Delete the Operations Center pod waiting until is terminated.

    $ kubectl delete pod/cjoc
  • Verify that Operations Center  Manage Jenkins  Configure System  Jenkins Location  Jenkins URL has been properly updated. If it hasn’t, change to the new one and click on Save.

  • Start all Managed Masters/Team Masters from Operations Center dashboard. This can be achieved either automatically with a cluster operation or manually using Managed Master/Team Master  Manage.

The new domain name must appears in all of those resources:

$ kubectl get statefulset/<master> -o=jsonpath='{.spec.template.spec.containers[?(@.name=="jenkins")].env}'
$ kubectl get cm/cjoc-configure-jenkins-groovy -o json
$ kubectl get ingress -o wide

The domain name must be the same than what is used in the browser otherwise a default backend - 404 will be returned.

Configuring Persistent Storage

For persistence of CloudBees Core Operations Center and Managed Master data, CloudBees Core must be able to dynamically provision persistent storage. When deployed, the system will provision storage for CloudBees Core Operations Center’s JENKINS_HOME directory and whenever a new Managed Master is provisioned, CloudBees Core Operations Center will provision storage for that master’s JENKINS_HOME.

On Kubernetes, dynamic provisioning of storage is accomplished by creating a Persistent Volume Claim which will use a Storage Class to coordinate with a storage provisioner to provision that storage and make it available to CloudBees Core.

Please refer to the next section in order to set up a storage class for your environment, if applicable.

Note

A detailed explanation of Kubernetes Storage concepts is beyond the scope of this document. For additional background information, refer to the Kubernetes Dynamic Provisioning and Storage Classes blog post, Persistent Volumes and Storage Classes section of the Kubernetes documentation.

Storage Requirements

Because pipelines typically read and write many files during execution, CloudBees Core requires high-speed storage. On EKS CloudBees Core requires that you use Solid-State Disk (SSD) storage.

By default, CloudBees Core will use whatever class is configured to be the Default Storage class. There are two ways you can provide an SSD-based Storage Class for CloudBees Core:

  1. Create a new SSD-based Storage Class and make it the default. This is the easiest because you don’t have to change the CloudBees Core configuration.

  2. Create a new SSD-based Storage Class and then, before you deploy change the CloudBees Core configuration file to use the new storage class that you created.

Check the Storage Class configuration

After you create your Kubernetes cluster, get the storage classes. If the default storage class is not gp2 then you’re not using SSD storage and you will need to act.

$ kubectl get storageclass
NAME               PROVISIONER             AGE
default (default)  kubernetes.io/aws-ebs   14d

Create a new SSD-based Storage Class

To create a new SSD-based Storage Class you will have to create a YAML file specifying the class and then run a series of kubectl commands to create the class and make it the default. See Kubernetes AWS storage class documentation for more information on all supported parameters.

Create a 'gp2-storage.yaml' file with the following content for 'gp2' type with encryption enabled:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  encrypted: "true"

Create the storage class:

$ kubectl create -f gp2-storage.yaml

Making the SSD-based Storage Class the default

If you want your cluster and CloudBees Core to use your new Storage Class as the default then you will need to take a couple of additional steps:

$ kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
$ kubectl patch storageclass default -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

List the storage classes and you should see your new class is the default:

$ kubectl get sc
NAME            PROVISIONER             AGE
gp2 (default)   kubernetes.io/aws-ebs   1d
default         kubernetes.io/aws-ebs   14d

Enabling Storage Encryption

Storage encryption should be used to ensure that all CloudBees Core data is encrypted at rest. If you want to setup this you must configure storage encryption in your Kubernetes cluster before you install CloudBees Core.

This is done by configuring Kubernetes to use a default Kubernetes Storage Class that implements encryption. Refer the Kubernetes documentation for Storage Classes and your cloud provider’s documentation for more information about the available Storage Classes and how to configure them.

Configuring AWS EBS Encryption

To enable AWS encryption you must create a new Storage Class that uses the kubernetes.io/aws-ebs provisioner, enable encryption in that Storage class and then set it as the default Storage Class for your cluster.

The instructions below explain one way to this. You should refer to the Kubernetes documentation for the complete details of the AWS Storage Class.

Examine the default storage class

First, examine the existing Storage Class configuration of your cluster.

$ kubectl get storageclass
NAME            PROVISIONER             AGE
default         kubernetes.io/aws-ebs   14d
gp2 (default)   kubernetes.io/aws-ebs   1d

You should look at the existing storage class to make sure it does not already use encryption, and to verify the name of the is-default-class annotation. The official documentation says the name should be storageclass.kubernetes.io/is-default-class but as you can see below, the name used in this particular cluster is storageclass.beta.kubernetes.io/is-default-class:

$ kubectl get storageclass gp2 -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-02-13T21:37:05Z
  labels:
    k8s-addon: storage-aws.addons.k8s.io
  name: gp2
  resourceVersion: "1823083"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: 0959b194-1106-11e8-b6ad-0ea2187dbbe6
parameters:
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
Create a new encrypted Storage Class

Create a new storage class in a YAML file with contents like the below. Pick a new name; below encrypted-gp2 is used. And note that there is a new parameter encrypted: true.

If you want to specify the keys to be used to encrypt the EBS volumes created by Kubernetes for CloudBees Core, then make sure to also specify the kmsKeyId, which, according to the documentation is "the full Amazon Resource Name of the key to use when encrypting the volume. If none is supplied but encrypted is true, a key is generated by AWS. See AWS docs for valid ARN value."

Here is an example Storage Class definition that specifies encryption:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    k8s-addon: storage-aws.addons.k8s.io
  name: encrypted-gp2
parameters:
  type: gp2
  encrypted: "true"
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete

Save your Storage Class to a file named, for example, sc-new.yml.

Next, use kubectl to create that storage class.

$ kubectl create -f sc-new.yml
storageclass "encrypted-gp2" created

Look at the existing storage classes again and you should see the new one:

$ kubectl get storageclass
NAME            PROVISIONER             AGE
default         kubernetes.io/aws-ebs   14d
encrypted-gp2   kubernetes.io/aws-ebs   13s
gp2 (default)   kubernetes.io/aws-ebs   14d
Set your new Storage Class as the default

Refer to the Kubernetes documentation for changing the default storage class. In summary, you need to mark the existing default as not-default, then mark your new storage class as default. Below are the steps.

Mark the existing default storage class as not default, and be sure to use the right annotation name that we saw above:

kubectl patch storageclass gp2 \
-p '{"metadata": {"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"false"}}}'

Mark the new encrypted storage class as the default:

kubectl patch storageclass encrypted-gp2 \
-p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Now, verify that the new encrypted storage class is the default:

$ kubectl get storageclass
NAME                      PROVISIONER             AGE
default                   kubernetes.io/aws-ebs   14d
encrypted-gp2 (default)   kubernetes.io/aws-ebs   50m
gp2                       kubernetes.io/aws-ebs   14d

Now you can proceed to deploy CloudBees Core. With the encrypted storage class in place, all EBS volumes created by CloudBees Core will be encrypted.

Integrating Single Sign-on

Once your CloudBees Core cluster is up and running you can integrate it with a SAML-based single sign-on (SSO) system and configure Role Based Authentication Controls (RBAC). This is done by installing the Jenkins SAML plugin, configuring it to communicate with your IDP and configuring your IDP to communicate with CloudBees Core.

Prerequisites for this task

You will need the following items before you setup SAML based SSO and RBAC:

  • Jenkins SAML Plugin must be installed in CloudBees Core Operations Center

  • A SAML-based Identity Provider (IDP)

  • Service Provider Metadata for your IDP (an XML file provided by your IDP administrator).

  • The SAML Attribute names used by IDP for these fields (ask your IDP administrator):

    • Username

    • Email

    • Group

Note
when you make changes to the security configuration, you may lock yourself out of the system. If this happens you can recover by following the instructions in the CloudBees Knowledge Base article How do I login into Jenkins after I’ve logged myself out.

Steps to Perform

Install the SAML plugin on CloudBees Core Operations Center by:

  • Login to CloudBees Core Operations Center and select Manage Jenkins  Manage Plugins  Available

  • Enter 'SAML' in the search box

  • Click the checkbox to choose the SAML plugin

  • Press the Download now and install after restart button

  • Check the checkbox labeled Restart Jenkins when installation is complete and no jobs are running (You do not need to install the plugin on Managed Masters, just on CloudBees Core Operations Center.)

Enable and configure SAML authentication

Login to CloudBees Core Operations Center and select Manage Jenkins  Configure Global Security.

Click the Enable security checkbox and confirm there is a SAML 2.0 option in the Security Realm setting. If it is not there, then the Jenkins SAML Plugin is not installed and you need to install the SAML plugin.

Read and carefully follow the Jenkins SAML Plugin instructions.

Enter the IDP Metadata (XML data) and specify the attribute names that your IDP uses for username, email and group membership. When you are ready, click Save to store the new security settings.

Export Service Provider Metadata to your IDP

After you save your security settings, CloudBees Core Operations Center will report your Service Provider metadata (XML data). You must copy this data and give it to your IDP administrator, who will add it to the IDP configuration.

You can find the Service Provider metadata by following a link on the Configure Global Security page at the end of the SAML section. The link looks like this:

Service Provider Metadata which may be required to configure your Identity
Provider (based on last saved settings).

Login to CloudBees Core Operations Center

Once your IDP administrator confirms that your IDP Metadata has been added to the IDP, attempt to login via the Login link in CloudBees Core Operations Center.

Setup RBAC

Refer to the "Role-based matrix authorization strategy" help in Manage Jenkins  Configure Global Security to enable Role Based Access Controls (RBAC).

Cloud ready Artifact Manager for AWS

Jenkins has historically provided multiple ways to save build products, otherwise known as artifacts.

Some plugins permit you to upload artifact files to repository managers like Artifactory, and Nexus Artifact Uploader, and other plugins send artifacts to remote shared filesystems like Publish Over FTP, Publish Over CIFS and Publish Over SSH. Jenkins itself stores artifact files in the Jenkins home filesystem. In 2012, CloudBees released the Fast Archiver Plugin, which optimizes the default artifact transmission but uses the same storage location.

Unfortunately, a number of these solutions are not cloud-ready, and it is awkward and difficult to use them with CloudBees Core on modern cloud platforms. Some solutions, like S3 publisher are well suited for use in a cloud environment, but require special build steps within Pipelines.

CloudBees is developing a series of cloud-ready artifact manager plugins. The first of these is Artifact Manager on S3 plugin. This plugin permits you to archive artifacts in an S3 Bucket, where there is less need to be concerned about the disk space used by artifacts.

Easy to configure

To configure Artifact Manager on S3:

  1. Go to Manage Jenkins/Configure System.

  2. In the Artifact Managment for Builds section, select the Cloud Provider Amazon S3:

    cloud provider configured
  3. Return to Manage Jenkins/Amazon Web Services Configuration to configure your AWS credentials for access to the S3 Bucket.

  4. For your AWS credentials, use the IAM Profile configured for the Jenkins instance, or configure a regular key/secret AWS credential in Jenkins. Note that your AWS account must have permissions to access the S3 Bucket, and must be able to list, get, put, and delete objects in the S3 Bucket.

    configure credentials
  5. Save or apply the credentials configuration, and move on to configure your S3 bucket settings.

    bucket settings
  6. We recommend validating your configuration. If the validation succeeds, you’ve completed the configuration for Artifact Manager on S3.

    validation success
  7. For more details about Artifact Manager for S3, see the plugin documentation here: Artifact Manager on S3 plugin.

Uploading and downloading artifacts

The Artifact Manager on S3 plugin is compatible with both Pipeline and FreeStyle jobs. To archive, unarchive, stash or unstash, use the default Pipeline steps.

FreeStyle jobs

For FreeStyle jobs, use a post-build action of Archive the Artifacts to store your Artifacts into the S3 Bucket.

fsj step archive

To copy artifacts between projects:

  1. Make sure the Copy Artifact Plugin is installed.

  2. Use a build step to copy artifacts from the other project:

    copy artefacts

Pipeline jobs

For Pipeline jobs, use an archiveArtifacts step to archive artifacts into the S3 Bucket:

node() {
    //you build stuff
    //...
    stage('Archive') {
        archiveArtifacts "my-artifacts-pattern/*"
    }
}

To retrieve artifacts that were previously saved in the same build, use an unarchive step that retrieves the artifacts from S3 Bucket. Set the mapping parameter to a list of pairs of source-filename and destination-filename:

node() {
    //you build stuff
    //...
    stage('Unarchive') {
        unarchive mapping: ["my-artifacts-pattern/": '.']
    }
}

To save a set of files for use later in the same build (generally on another node/workspace) use a stash step to store those files on the S3 Bucket:

node() {
    //you build stuff
    //...
    stash name: 'stuff', includes: '*'
}

To retrieve files saved with a stash step, use an unstash step, which retrieves previously stashed files from the S3 Bucket and copies them to the local workspace:

node() {
    //you build stuff
    //...
    unstash 'stuff'
}

To copy artifacts between projects:

  1. Make sure the Copy Artifact Plugin is installed.

  2. Use a copyArtifacts step to copy artifacts from the other project:

    node(){
      //you build stuff
      //...
      copyArtifacts(projectName: 'downstream', selector: specific("${built.number}"))
    }

Security

Artifact Manager on S3 manages security using Jenkins permissions. This means that unless users or jobs have permission to read the job in Jenkins, the user or job cannot retrieve the download URL.

Download URLs are temporary URLs linked to the S3 Bucket, with a duration of one hour. Once that hour has expired, you’ll need to request a new temporary URL to download the artifact.

Agents use HTTPS (of the form https://my-bucket.s3.xx-xxxx-x.amazonaws.com/*) and temporary URLs to archive, unarchive, stash, unstash and copy artifacts. Agents do not have access to either the AWS credentials or the whole S3 Bucket, and are limited to get and put operations.

Performance

A major distinction between the Artifact Manager for S3 plugin and other plugins is in the load on the master and the responsiveness of the master-agent network connection. Every upload/download action is executed by the agent, which means that the master spends only the time necessary to generate the temporary URL: the remainder of the time is allocated to the agent.

The performance tests detailed below compare the CloudBees Fast Archiving Plugin and the Artifact Manager on S3 plugin.

Performance tests were executed in a Jenkins 2.121 environment running on Amazon EC2, with JENKINS_HOME configured on an EBS volume. Three different kinds of tests were executed from the GitHub repository at Performance Test, with samples taken after the tests had been running for one hour:

  • Archive/Unarchive big files: Store a 1GB file and restore it from the Artifact Manager System.

  • Archive/Unarchive small files: Store 100 small files and restore them from the Artifact Manager System. Small files are approximately 10 bytes in size, with 100 files stored and times averaged

  • Stash/Unstash on a pipeline: Execute stash and unstash steps. The Fast Archive Plugin stash/unstash operations used the default stash/unstash implementation.

As can be seen from the results, the Artifact Manager on S3 Plugin provides a measurable performance improvement on both big and small files, with the improvement measured in minutes for big files and in seconds for small files.

Artifact Manager on S3 plugin performance

Plugin link: Artifact Manager on S3

 Big Files
Time in Milliseconds Archive Unarchive

Max

48,578.00

29,899.00

Min

17,773.00

20,388.00

Avg

20,969.49

22,670.67

S3 archive big file 00
 Small Files
Time in Milliseconds Archive Unarchive

Max

2,974.00

805.00

Min

752.00

104.00

Avg

1,171.65

200.76

S3 archive small files 00
Stash
Time in Milliseconds Archive Unarchive

Max

14,902.00

9,477.00

Min

1,256.00

709.00

Avg

1,977.49

1,588.96

S3 archive stash 00

CloudBees Fast Archiving Plugin performance

Big files
Time in Milliseconds Archive Unarchive

Max

358,988.00

105,615.00

Min

110,068.00

93,193.00

Avg

277,642.22

95,771.78

fast archive big file 00
 Small Files
Time in Milliseconds Archive Unarchive

Max

1,603.00

109.00

Min

491.00

10.00

Avg

953.26

22.55

fast archive small files 00
Stash
Time in Milliseconds Archive Unarchive

Max

1,914.00

3,050.00

Min

561.00

267.00

Avg

1,075.53

976.28

fast archive stash 00