Table of contents

Administering CloudBees Jenkins Enterprise 1.x


Configuration Reference

CloudBees Jenkins Enterprise Cluster configuration is achieved by customizing two text files: cluster-init.config and cluster-init.secrets. These are text files where customization values are stored in section-based name-value pairs. For example,

  1. cluster-init.config

[tiger]
cluster_name = pse
controller_count = 3
...

Most of the attributes are common to all supported environments (AWS, OpenStack, etc…​). However, a small percentage Some of the attributes are environment-specific. While many of these attributes have reasonable default values, some need explicit initialization. There are use cases such as upgrading, managing workers, etc…​, which may require further customization of these files. Please refer to the following sections for more details.

Accessing Mesos Cluster Information

In the cluster creation output file for Operations Center, you see a link to Marathon, Mesosphere’s framework and container orchestration platform for Apache Mesos.

CloudBees Jenkins Enterprise uses Mesos to simplify running applications on a scalable cluster of servers aggregated into a single resource pool. If you access the URL for Marathon, you will be asked for credentials.

You can retrieve the Marathon and Mesos credentials using the following commands:

cje run echo-secrets mesos_web_ui_username
cje run echo-secrets mesos_web_ui_password
cje run echo-secrets marathon_username
cje run echo-secrets marathon_password

Using Jenkins SSH on Managed Masters

Some Jenkins plugins (Validated Merge plugin, Pipeline Global Library) require SSH access to Jenkins.

In the CloudBees Jenkins Enterprise environment, masters are provisioned on a dynamic host and port within the cluster. CloudBees Jenkins Enterprise provides an HTTP router in order to access Managed Masters using a fixed name. To access the SSH port of each master, you must configure your SSH client to use a proxy.

SSH Proxy Configuration for OpenSSH

Provided your domain is ci.example.com, you will need to add to ~/.ssh/config the following snippet:

  Host=*.ci.example.com
  ProxyCommand=ssh -q -p 2222 ci.example.com tunnel %h
OpenStack instructions

Beginning with version 1.1.0, the load balancer component requires a different floating IP for each port that is exposed. Therefore for SSH access, you will have to use a different IP than the one mapped to your default domain. To retrieve the IP to use for SSH access, use the command cje run resolve-attr ssh_ip and use the IP instead of ci.example.com in the ProxyCommand above.

You should end up with:

  Host=*.ci.example.com
  ProxyCommand=ssh -q -p 2222 <ssh_ip> tunnel %h
Connecting to a Master Through SSH (OpenSSH)

Provided you added your public SSH key to your user profile in the master you want to connect to, you can then use SSH commands. Please note that you will need to accept host keys for both the proxy and the master on the first connection attempt.

    $ ssh admin@cjoc.ci.example.com who-am-i
    Authenticated as: admin
    Authorities:
      authenticated

SSH Proxy Configuration for Putty

You will need to add the proxy configuration to your connection details.

Go to connection > proxy.

  • Proxy type: local

  • Proxy hostname: ci.example.com

  • Proxy port: 2222

  • Proxy command: plink ssh@%proxyhost -P %proxyport tunnel %host

Save the connection as pse (as an example).

Connecting to a Master Through SSH (Putty)

Provided you added your public SSH key to your user profile in the master you want to connect to, you can then use SSH commands. Please note that you will need to accept host keys for both the proxy and the master on the first connection attempt.

  $ plink -load pse admin@cjoc.ci.example.com who-am-i
  Authenticated as: admin
  Authorities:
    authenticated

Common Configuration Attributes

Table 1. [tiger] section
[tiger] Attribute Description

cluster_name

Name of the cluster (Required, default "pse")

This value is used to prefix resource names and tags when setting up servers and other infrastructure.

This value must contain only alphanumeric, dots, dashes and underscore values.

controller_count

Number of controllers (default 3)

Controllers manage coordination of the CloudBees Jenkins Enterprise resources. Include multiple controllers to ensure availability of the CloudBees Jenkins Enterprise cluster. When using multiple controllers, always use an odd number of controllers.

master_worker_count

Number 'master' of workers (default 2)

CloudBees Jenkins Enterprise workload, for masters, CJOC and PSE components, is handled by 'master' workers. To increase the initial capacity of your CloudBees Jenkins Enterprise cluster, increase this number. You can add more workers later with the 'worker-add' operation. We recommend a minimum of two 'master' workers.

build_worker_count

Number of 'build' workers (default 1)

CloudBees Jenkins Enterprise workload for builds is handled by 'build' workers. To increase the initial build capacity of your CloudBees Jenkins Enterprise cluster, increase this number. You can add more workers later with the 'worker-add' operation.

register_script

Operating System Registration Script

If the operating system requires a registration (e.g. RHEL), provide here the path to script handling the registration. This script will be copied to machines and executed on init and upgrades and must succeed even if the machine is already registered.

Example of script content for RHEL:

#!/bin/bash
subscription-manager register --username <rhel_username> --password <rhel_password> --auto-attach --force

unregister_script

Operating System Unregistration Script

The unregister script is run on machine shutdown. It allows to free the registration if it part of a pool. Provide here the path to a script handling machine unregistration.

Example of script content for RHEL:

#!/bin/bash
subscription-manager unregister

docker_registry

Docker registry (default public)

Specify the registry used to obtain CloudBees Jenkins Enterprise images. To use the public Docker registry, use 'public'.

http_proxy

HTTP Proxy (optional)

If servers in the CloudBees Jenkins Enterprise cluster require a web proxy to access the Internet using the HTTP protocol, specify the full URL to the proxy server including the applicable port.

http_proxy=http://webproxy.mycompany.com:8080

https_proxy

HTTPS Proxy (optional)

If servers in the CloudBees Jenkins Enterprise cluster require a web proxy to access the Internet using the HTTP protocol over SSL/TLS, specify the full URL to the proxy server including the applicable port.

https_proxy=https://webproxy.mycompany.com:8443

no_proxy

No Proxy (optional)

This variable should contain a comma-separated list of domain extensions for which http_proxy and https_proxy should not be used.

no_proxy=.internal.mycompany.com,.lab.mycompany.com

If servers in the CloudBees Jenkins Enterprise cluster don’t have any public IP, they must be accessed through a SSH proxy. The following optional section allows to configure access to a SSH proxy.

Table 2. [sshproxy] section
[sshproxy] Attribute Description

user

SSH Proxy user name (optional)

Username to log in as on the SSH proxy.

host

SSH Proxy host name (optional)

Host of the SSH proxy.

port

SSH Proxy port (optional)

Port of the SSH proxy. Defaults to 22.

identity_file

SSH Proxy identity file (optional)

Path to a private key to be used to connect to the SSH proxy.

netcat_cmd

SSH Proxy Netcat command (optional)

GNU Netcat must be installed on the machine used as SSH proxy. This allows to provide a custom path to the netcat executable. Defaults to 'nc'.

domain_name

Domain name

All the Jenkins masters (and CJOC) servers are exposed under paths for the same domain pse.example.com, e.g. pse.example.com/master-1/

The underlying infrastructure services (mesos, marathon) are exposed as subdomains (e.g. mesos.pse.example.com) and must be registered using specific CNAMEs.

If unset, the cluster will be created using a default domain name <clustername>-controller-<123456789>.<aws-region>.elb.amazonaws.com.elb.cloudbees.net

elb.cloudbees.net is a domain that resolves *.<clustername>-controller-<123456789>.<aws-region>.elb.amazonaws.com.elb.cloudbees.net to <clustername>-controller-<123456789>.<aws-region>.elb.amazonaws.com, allowing subdomains to reach marathon and mesos using the same ELB.

You can decide to provide your own domain name at cluster-init time. If you do, it is recommended that you configure the route53* options below to let CloudBees Jenkins Enterprise use Route53 to configure your domain to target the ELB created by CloudBees Jenkins Enterprise automatically.

If you do not use Route53, you will need to follow this process to complete the cluster-init:

  • apply will create the ELB, then fail because the domain name does not target the ELB

  • you need to configure your domain name to point to the created ELB

  • you need to wait for the DNS replication to occur, and also for the potential local DNS cache to expire

  • apply again to complete the cluster-init operation.

domain_separator

Domain name separator (defaults to .).

If your organization (acme.com) supports only one level of subdomain, e.g. you can create 'foo.acme.com' but not 'foo.bar.acme.com', then you can use the following configuration.

The default behavior is to create subdomains : domain_separator = . but you can set it to another character if you want.

Example:

 domain_name = pse.acme.com
 domain_separator = -

Elements of cluster will be named

mesos-pse.acme.com
marathon-pse.acme.com
pse.acme.com/cjoc/
pse.acme.com/mymaster/

protocol

The protocol to access the cluster: http or https

If https is selected, a certificate is required (see ssl_certificate_id).

Operations Center Configuration Attributes

Table 3. [cjoc] section
[cjoc] Attribute Description

memory

The container memory limit (default 1.5GB)

This value specifies the amount of memory allocated to the Operations Center container

jvm_options

The Operations Center application JVM options (default '-Xmx1024m')

This value specifies the options used by the Java Virtual Machine.

Note that if this value specifies the max heap usage of the JVM, it should be lower than the container memory limit

Docker Configuration Attributes

Table 4. [docker] section
[docker] Attribute Description

options

The Docker daemon options (default --log-driver=syslog)

This value sets the Docker daemon options. --log-driver=syslog should always be included to ensure logs are collected.

Common Troubleshooting Attributes

Note
The following debug settings are for internal use only.
Table 5. [debug] section
[debug] Attribute Description

disable_castle

Disable castle (storage agent) (default no)

disable_elasticsearch

Disable elasticsearch (default no)

disable_cjoc = no

Disable cjoc (CloudBees Jenkins Operations Center) (default no)

enable_logstash

Enable logstash (default no)

Logstash may optionally be used to capture syslog events within the CloudBees Jenkins Enterprise cluster.

force_pull_image = false

Force docker image pulls for Marathon apps (default false)

Use this value to set forcePullImage for docker containers in the Marathon applications that support this setting.

enable_hello_app

Enable the hello app (default no)

The hello app is a trivial application that tests basic app deployment to CloudBees Jenkins Enterprise. When testing support for applications, you can set this to to yes, which will generate the hello and start it as a part of the Marathon application init. You can also manually start and stop the hello application from dna.

AWS-Specific Configuration Attributes

Table 6. [aws] section
[aws] Attribute Description

region

AWS region (required)

The region in which CloudBees Jenkins Enterprise will run. This must correspond to the AWS credentials specified above.

Use one of these values:

ap-northeast-1
ap-northeast-2
ap-south-1
ap-southeast-1
ap-southeast-2
ca-central-1
eu-central-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
us-east-1
us-east-2
us-gov-west-1
us-west-1
us-west-2

resource_tags

AWS resource tags

The tags will be added to all AWS created resources that support tags resource_tags is a comma delimited list of tags. The parameter format is: TAG1=VALUE1,TAG2=VALUE2,…​.

availability_zone

AWS availability zone.

The availability zone(s) where CloudBees Jenkins Enterprise will run, separated by comma.

The number of availability zones must be either 1, or 3 for fault tolerance. If not set it will pick a random one.

image_distribution

Operating system image distribution

CloudBees Jenkins Enterprise supports several operation system distribution: rhel, centos and ubuntu.

default_ami

Custom AMI

By default, an AMI created by CloudBees will be looked up based on the selected regions and image distributions. If for some reason, you wish to provide your own AMI (it must use one of the supported distributions), you can pass its id here. The CloudBees AMIs can be looked up manually from public AMIs by using the following filters:

  • owner: 397995763618

  • name starts with "cloudbees-cje"

encrypted

Enable storage encryption (yes/no)

This setting applies to JENKINS_HOME volumes only. To use encrypted volumes for root volumes, please create an encrypted copy of CloudBees-provided AMI and pass it as default_ami (above) during this operation.

controller_instance_type

EC2 Instance type to use for controller servers (defaults to m4.large)

controller_volume_size

EC2 controller root volume size in GB (defaults to 200)

controller_volume_type

EC2 volume type to use for the controller server volume (defaults to gp2, currently the only supported volume type)

controller_ebs_optimized

Whether the instance is EBS-optimized (defaults to yes on all compatible instance types)

worker_instance_type

EC2 Instance type to use for worker servers (defaults to m4.xlarge)

worker_volume_size

EC2 worker root volume size in GB (defaults to 50)

worker_volume_type

EC2 volume type to use for the worker server volume (defaults to gp2, currently the only supported volume type)

worker_ebs_optimized

Whether the instance is EBS-optimized (defaults to yes on all compatible instance types)

worker_instance_profile

Instance profile of the worker (optional)

If set, launches the worker instance(s) with an IAM profile which will be used to grab the AWS credentials for worker operations.

storage_instance_type

EC2 Instance type to use for the storage server (defaults to m3.medium)

storage_volume_size

EC2 storage root volume size in GB (defaults to 50)

storage_volume_type

EC2 volume type to use for the storage server volume (defaults to gp2, currently the only supported volume type)

storage_ebs_optimized

Whether the instance is EBS-optimized (defaults to yes on all compatible instance types)

Cluster access restrictions

tiger_admin_port_access

Admin access from selected IPs (required)

Administrative ports are not accessible by default, but access can be granted to specific IPs and networks by using tiger_admin_port_access.

Ensure this property includes the installation machine IP, failing to do so will result in a failed installation.

In AWS or public clouds that would be your public facing IP. You can use Google by typing "what is my ip" to find public-facing IP.

In OpenStack or private clouds it would be the IP of the machine running the installation as seen by the VMs started in the cloud.

The property can contain one or more IP address ranges in CIDR notation separated by commas that are granted access to the administrative ports (for example: 192.0.2.0/24,198.51.100.1/32).

Use 0.0.0.0/0 to allow administrative connections from any IP, or 198.51.100.1/32 to allow access from only one IP (198.51.100.1).

Ports open for admin access include the ones for user access plus:

  • 22 SSH access to controllers and workers

tiger_user_port_access

User access from selected IPs (optional)

By default, applications are opened only to administrators.

By providing a value, you can restrict access to a particular set of IPs.

The property can contain one or more IP address ranges in CIDR notation separated by commas that are granted access to the administrative ports (for example: 192.0.2.0/24,198.51.100.1/32).

Ports open for user access include: * 80 http access * 443 https access (if enabled) * 2222 Jenkins SSH

VPC options

vpc_id

User-provided VPC ID (optional)

If set, CloudBees Jenkins Enterprise will create all resources within the given VPC. It is assumed that the given VPC ID is valid within the given account and region.

If unset, CloudBees Jenkins Enterprise will create a new VPC and create all resources inside it.

vpc_subnet_id

User-provided subnet ID (required if vpc_id is set)

If vpc_id above is set, it is mandatory to provide also the ID of a subnet defined inside the user-provided VPC and the availability zone it corresponds to, using the availability_zone parameter

The number of subnets must be either 1, or 3 (separated by comma) for fault tolerance.

Instances created by CloudBees Jenkins Enterprise will be launched on the provided subnet.

additional_security_group_id

User-provided security group ID

Only used if vpc_id above is defined. Adds an additional security group to all instances created in the cluster and to load balancers.

vpc_subnet

VPC subnet (default 10.16.0.0/16). Unused if vpc_id is set.

VPC and subnet CIDR block.

internal

Whether the cluster is internal only (defaults to no)

Allows to set up a CloudBees Jenkins Enterprise cluster with no public access (no public elb, no public ips) This can be enabled only if the vpc is user-provided (vpc_id and vpc_subnet_id provided above)

ssl_certificate_id

Main ELB SSL certificate id (only if protocol = https is set)

An SSL certificate can be used for SSL, generated for the main domain name, plus mesos and marathon, using the domain_separator defined above

For example if domain_separator = . (default)

cje.example.com
mesos.cje.example.com
marathon.cje.example.com

if domain_separator = -

cje.example.com
mesos-cje.example.com
marathon-cje.example.com

The certificate is defined using arn syntax

  • AWS IAM: arn:aws:iam::123456789012:certificate/some-certificate-name

  • AWS ACM: arn:aws:acm:us-east-1:123456789012:certificate/12345678-aaaa-bbbb-cccc-012345678901

route53_private_zone_name

Route53 Private Hosted Zone

A private hosted zone is used to allow loopbacks through the cluster without going through public network interfaces. By default, the cluster will create a private hosted zone matching the given domain name. However, if there is already a private hosted zone for the cluster vpc, its name should be provided below.

Example with domain_name = cje.example.com By default, a private hosted zone created by CloudBees Jenkins Enterprise would be using the name cje.example.com. If there is already a private hosted zone for 'example.com' in the cluster VPC then 'example.com' should be provided below.

use_public_route53_zone

Route53 Public Hosted Zone

If Route53 is used to manage the domain, CloudBees Jenkins Enterprise can create the records automatically. A public hosted zone for the domain provided must already exist and be properly configured

route53_zone_name

Route53 Public Hosted Zone name

By default CloudBees Jenkins Enterprise will look up a hosted zone for the provided domain name. If a hosted zone already exists for a wider domain, it can be provided here.

For example, if CloudBees Jenkins Enterprise uses the domain cje.example.com, and there is a public hosted zone for example.com, then use route53_zone_name = example.com. Otherwise CloudBees Jenkins Enterprise will look up a hosted zone for cje.example.com and will not find it.

Table 7. [storage] section
[storage] Attribute Description

type

Storage type (default ebs if permissions allow, otherwise builtin-rsync)

CloudBees Jenkins Enterprise requires a means of backing up and restoring workspace state. The storage system can be configured with various back-end types.

Types supported:

builtin-rsync

Creates a storage server within the CloudBees Jenkins Enterprise cluster that is used to backup and restore workspace files using rsync.

nfs

Uses a mounted NFS volume to store workspace files

ebs

Uses AWS elastic block store (EBS) to store and snapshot workspace files

nfs_server

NFS server (only if type = nfs)

If using the nfs storage type, you must specify a server to use.

Specify the hostname or IP address of an NFS server that is accessible to the CloudBees Jenkins Enterprise cluster.

nfs_export_dir

NFS export directory

If using the nfs storage type, specify the exported directory on the NFS server (see nfs_server above) to use for storage.

Ensure that the directory is specified exactly as defined on the NFS server. The directory should start with a forward-slash (e.g. /var/myexport).

ebs_snapshot (only if type = ebs)

EBS snapshot

If using the ebs storage type, you must specify a snapshot to be used for creating a new EBS volume.

The snapshot must be of an XFS formatted volume and available in the cluster region (see region setting above).

volume_user

Volume user (default jenkins)

Specify the user that will own the backed-up workspace files on the volume.

volume_group

Volume group (default jenkins)

Specify group that will own the backed-up workspace files on the volume.

Table 8. cluster-init.secrets.in template for AWS
Attribute Description

aws_access_key_id

AWS access key ID (required)

This is the access key ID from your AWS credentials. It must correspond to the value for secret_access_key below.

aws_secret_access_key

AWS secret access key (required)

This is the secret access key from your AWS credentials. It must correspond to the value for access_key_id (above).

aws_session_token

AWS session token (optional)

If using ephemeral credentials from Amazon STS, the session token can be supplied via this configuration option.

docker_registry_auth

Docker authorization (optional)

If using a Docker registry that requires authentication, specify the base 64 encoded auth string used by Docker here.

cjoc_username

CJOC username (optional - defaults to admin)

cjoc_password

CJOC password (optional - defaults to a generated random string)

OpenStack-Specific Attributes

Table 9. openstack section
[openstack] Attribute Description

tenant_name

OpenStack tenant name (required)

Example: tiger

auth_url

OpenStack identity service endpoint (required)

image_name

Default image name for CloudBees Jenkins Enterprise servers (required)

Example: image_name = cloudbees-tiger-ubuntu-1.0

storage_image_name

Image name override for Storage node (optional)

If not specified, then the value specified in the attribute "openstack.image_name" is used

storage_instance_type

Flavor of the storage instance (default m1.small)

If intending to use builtin-rsync, then change this value to something appropriately larger.

controller_image_name

Image name override for Controller nodes (optional)

If not specified, then the value specified in the attribute "openstack.image_name" is used

controller_instance_type

Flavor of the controller instances (default m1.large)

worker_image_name

Image name override for Worker nodes (optional)

If not specified, then the value specified in the attribute "openstack.image_name" is used

worker_instance_type

Flavor of the worker instances (default m1.large)

network_name

Network name within which Tiger servers are created (required)

floatip_network_name

Name of network that houses floating IPs to be assigned to the controller instances so they are accessible from an external network, e.g., the Internet (optional)

This network name will be used to generate one or more floating IPs to expose VMs to the outside world. If not set, only private ips from "network_name" will be assigned and CloudBees Jenkins Enterprise may not be accessible from outside the cluster, depending on your network topology.

floatip_workers

Whether to give a floating ip to workers. (default no)

By default, workers are not exposed to the outside world. Set this to 'yes' to attach workers to the floating IPs network, just like controllers. It will allow to connect external jnlp agents to the masters within the cluster.

tiger_admin_port_access

Admin access from selected IPs (required)

Administrative ports are not accessible by default, but access can be granted to specific IPs and networks by using tiger_admin_port_access.

Ensure this property includes the installation machine IP, failing to do so will result in a failed installation.

In AWS or public clouds that would be your public facing IP. You can use Google by typing "what is my ip" to find public-facing IP.

In OpenStack or private clouds it would be the IP of the machine running the installation as seen by the VMs started in the cloud.

The property can contain one or more IP address ranges in CIDR notation separated by commas that are granted access to the administrative ports (for example, 192.0.2.0/24,198.51.100.1/32).

Use 0.0.0.0/0 to allow administrative connections from any IP, or 198.51.100.1/32 to allow access from only one IP (198.51.100.1).

Table 10. storage section
[storage] Attribute Description

type count

Storage type (default builtin-rsync) Number of storage servers (default 1)

CloudBees Jenkins Enterprise requires a means of backing up and restoring workspace state. The storage system can be configured with various back-end types.

Types supported:

Types supported:

builtin-rsync

Creates a storage server within the CloudBees Jenkins Enterprise cluster that is used to backup and restore workspace files using rsync.

nfs

Uses a mounted NFS volume to store workspace files

When using rsync with a built-in storage server, set count=1. When using another backend or a provided storage server, set count=0

nfs_server

NFS server (only if type = nfs)

If using the nfs storage type, you must specify a server to use.

Specify the hostname or IP address of an NFS server that is accessible to the CloudBees Jenkins Enterprise cluster.

nfs_export_dir

NFS export directory (only if type = nfs)

If using the nfs storage type, specify the exported directory on the NFS server (see nfs_server above) to use for storage.

Ensure that the directory is specified exactly as defined on the NFS server. The directory should start with a forward-slash (e.g. /var/myexport).

volume_user

Volume user (default jenkins)

Specify the user that will own the backed-up workspace files on the volume.

volume_group

Volume group (default jenkins)

Specify group that will own the backed-up workspace files on the volume.

Table 11. cluster-init.secrets.in template for OpenStack
Attribute Description

openstack_user_name

OpenStack username (required)

openstack_password

OpenStack password (required)

docker_registry_auth

Docker authorization (optional)

If using a Docker registry that requires authentication, specify the base 64 encoded auth string used by Docker here.

cjoc_username

CJOC username (optional - defaults to admin)

cjoc_password

CJOC password (optional - defaults to a generated random string)