Table of contents

Administering CloudBees Jenkins Enterprise 2.x


Operating CloudBees Jenkins Enterprise 2.x

Caution

This guide is an old version of Operating CloudBees Jenkins Enterprise 2.x, and is superseded by Operating CloudBees Core on Modern Cloud Platforms.

Please refer to Operating CloudBees Core on Modern Cloud Platforms for updated content.

CloudBees Jenkins Enterprise enables Continuous Delivery as a Service with its

Managing masters

CloudBees Jenkins Enterprise enables Continuous Delivery as a Service by offering a highly elastic infrastructure that supports centralized management of teams, shareable resources across the cluster and automated management of the underlying cluster infrastructure.

With CloudBees Jenkins Enterprise’s Managed Masters, administrators can offer their teams highly available build environments with a minimum of management and configuration overhead. This section describes the CloudBees Jenkins Enterprise architecture, how administrators can scale their CloudBees Jenkins Enterprise cluster, and some upgrade and disaster recovery strategies for CloudBees Jenkins Enterprise clusters.

Individual agents can also be provisioned and controlled from CloudBees Jenkins Enterprise. Refer to Managing Agents for more information.

See the CloudBees Jenkins Enterprise reference section for more details of the CloudBees Jenkins Enterprise architecture.

CloudBees Assurance Program

CloudBees Jenkins Enterprise provide greater stability and security for Jenkins installations through the CloudBees Assurance Program. The CloudBees Assurance Program supports all Jenkins-based components, including CloudBees Jenkins Enterprise and CloudBees Jenkins Team. The CloudBees Assurance Program simplifies securing and upgrading multiple Jenkins instances. It assures compliance with CloudBees-recommended master configurations as detailed in the Beekeeper Upgrade Assistant page.

Learn more about managing upgrades and downgrades with the CloudBees Assurance Program.

Operating Managed and Client Masters

When new teams join an organization, or existing teams start a new project, CloudBees Jenkins Enterprise makes it easy to provision a fully managed and access controlled Jenkins master per team. In CloudBees Jenkins Enterprise, a Jenkins master is referred to as a Managed Master.

Administrators can provision Managed Masters from a standardized template or they can allow team leads to provision their own Managed Masters "on-demand." The number of masters a CloudBees Jenkins Enterprise environment can handle is limited only by the capacity of the cluster.

Adding Client Masters

Occasionally administrators will need to connect existing masters to a CloudBees Jenkins Enterprise cluster, such as in the case of a team requiring a master on Windows. Existing masters that are connected to Operations Center lack key benefits of Managed Masters like high availability and automatic agent management. Whenever possible, administrators should use a Managed Master with CloudBees Jenkins Enterprise rather than connecting an existing master.

Client Masters are monitored by Operations Center just as Managed Masters are monitored. Administrators can see the status of all their Managed Masters, Team Masters, and Client Masters from the Operations Center masters page. Client Masters can receive configuration updates from Operations Center with configuration snippets. Client Masters can share agents hosted in the cluster, offloading the burden of agent management from teams.

Client Masters do not have the high availability features of Managed Masters.

Note: The existing Client Master and Operations Center must both accept JNLP requests. See Kubernetes client master configuration for more information.

To add a Client Master, log into CloudBees Jenkins Enterprise and navigate to the Operations Center dashboard. Click the New Item option in the left-hand menu and provide the following information:

  • Item name: the name of the existing Client Master to connect to the cluster

  • Select Client Master. Just as with Managed Masters, Client Masters offer some customization and configuration options on creation:

  • On-master executors: Number of builds to execute concurrently on the Client Master itself. Default setting is 2.

  • Email addresses: Contact information for the administrator responsible for maintaining the Client Master.

Once these settings are saved, Operations Center will attempt to connect the Client Master to the CloudBees Jenkins Enterprise cluster.

Verify that Operations Center and the existing Client Master can communicate with each other over both HTTP and JNLP ports. Host and port to use for JNLP is advertised through HTTP headers by each Jenkins master.

You can connect an existing Client Master to Operations Center by giving that Client Master the TLS certificate for Operations Center, typically through the Configure Global Security page in Operations Center. For more information, see How to programmatically connect a Client Master to CJOC.

If you are connecting multiple Client Masters to your cluster, it is a good idea to automate that task using shared configurations.

Once the Client Master is connected, administrators should configure security and access controls for the master.

Configuring a Client Master

This section describes how to configure a Client Master that has already been connected to your Operations Center instance.

To access a Client Master’s configuration:

  1. Ensure you are logged in to Operations Center as a user with the Client/Managed Master > Configure permission.

  2. From the Operations Center home page/Dashboard, click to the right of your configued Client Master (avoiding its name) and choose Configure from the dropdown menu.
    Client Master dropdown menu

  3. On the resulting Client Master configuration page, you can configure the following properties:

    • Description - Enter an optional description for the Client Master.

    • Health Reporting - When this check box is selected, health-related metrics from this Client Master are collected periodically. The default data collection period is once per minute, when data consumers are present (e.g. Weather columns or CloudBees Jenkins Analytics).

    • Analytics Reporting - When this check box is selected, report events and other metrics from this Client Master for CloudBees Jenkins Analytics are collection.

    • On-master executors - Select Enforce to specify the # of executors, which periodically ensures that the number of executors on the Client Master is the value specified in this # of executors field. Allowing items (i.e. projects or jobs) to execute directly on the Client Master is a security risk since such projects/jobs could potentially access the file system and the build records of all previously run projects/jobs (which may contain sensitive information). Therefore, set this value to 0 to prevent any items from being executing directly on the Client Master.

    • Master Owner - Specify the email address/es (one per line) of the "Owner/s" to be notified whenever this Client Master goes offline or changes state.
      Note: Clicking the Advanced button opens the Delay before notification field, which allows you to specify the number of minutes (a value between 1 and 60) between notifications.

    • Plugin Catalog - Select Specify a plugin catalog for this master to choose a plugin catalog to apply to this Client Master.

Configuring Plugin Catalogs

The Beekeeper Upgrade Assistant feature of your Operations Center’s Manage Jenkins area is the main interface and entry point to the CloudBees Assurance Program.

Beekeeper Upgrade Assistant manages appropriate upgrades (and downgrades) of plugins on your Operations Center instance, in accordance with the CloudBees Assurance Program.

On Client Masters, the Beekeeper Upgrade Assistant feature is also available, and:

  • Is accessible through a Client Master’s Manage Jenkins area

  • Performs exactly the same types of management activities (as this feature does for your Operations Center instance) on their respective Client Master instances.

However, if you also need to install non-CloudBees Assurance Program plugins on your Client Masters, such as plugins which are unknown to CloudBees (perhaps developed internally within your organization) or older versions of plugins, you can configure a plugin catalog. A plugin catalog is used by the Beekeeper Upgrade Assistant to widen the acceptable scope of plugins beyond those defined by the CloudBees Assurance Program. This scope includes the ability to specify plugin dependency rules and acceptable version ranges for these plugins.

Once a plugin catalog is defined and is added to your Operations Center instance, it can then be deployed to your connected Client Master instances.

This section contains the following procedures:

Defining a Plugin Catalog

Before you can add a plugin catalog to your Operations Center instance, you must first define it.

A plugin catalog is defined in JSON file format, which in turn defines one or more plugin configurations. A plugin configuration defines the set of rules for one or more plugins (some potentially dependencies) and their versions, which can be installed to your Operations Center instance, along with the CloudBees Assurance Program-defined plugins.

To create this file, using your favorite text editor, copy and paste the unannotated empty template below into a new file and save it with an appropriate file name, similar to that of the name or displayName.

The following JSON content is an example of the contents of a plugin catalog file, whose annotations are explained underneath. An appropriate name for this plugin catalog file might be my-corporate-plugin-catalog.json. Use this information to help complete your own plugin catalog file.

{
    "type" : "plugin-catalog", (1)
    "version" : "1", (2)
    "name" : "java-web-master", (3)
    "displayName" : "Java Web Master", (4)
    "configurations" : [ (5)
        {
            "description" : "java-web-master for 2.61 and 2.73", (6)
            "prerequisites": { (7)
                "productVersion": "[2.60, 2.73]"
            },
            "includePlugins" : { (8)
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins", (9)
                    "version" : "1.8.2"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.2.1.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=", (10)
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.2.3.hpi" (11)
                }
            }
        },
        { (12)
            "description" : "java-web-master for 2.89+",
            "prerequisites": {
                "productVersion": "[2.89)"
            },
            "includePlugins" : {
                "tier3-plugin": {
                    "groupId": "com.my-company.jenkins.plugins",
                    "version" : "1.8.3"
                },
                "my-corporate-plugin": {
                    "url" : "https://download.my-company.com/jenkins/my-corporate-plugin-3.4.0.hpi"
                },
                "my-other-corporate-plugin": {
                    "sha1" : "SC/TzSN+eOrewaDZJZyzLpIQV7E=",
                    "url" : "https://download.my-company.com/jenkins/my-other-corporate-plugin-1.3.0.hpi"
                }
            }
        }
    ],
    "settings": { (13)
        "httpCredentialsProvider": { (14)
            "credentials": [ (15)
                {
                    "authenticationScope": "ANY", (16)
                    "id": "download-server-credentials" (17)
                }
            ]
        },
        "repository": { (18)
            "layout": "maven2", (19)
            "url": "https://repo.my-company.com/content/repositories/dev-connect"
        }
    }
}
  1. type (required) - must have the fixed value plugin-catalog, which defines this JSON file as a plugin catalog file.

  2. version (required) - must currently have the value 1, which defines the current metadata format for the plugin catalog.

  3. name (required) - the internal name used to identify the plugin catalog. Avoid using spaces in this name.

  4. displayName (optional) - a human-readable name for the plugin catalog. This is the name of the plugin catalog that appears in the Operations Center interface.

  5. configurations (required) - the plugin catalog can define one or more plugin configurations within this array, where each plugin configuration is represented by an individual element of this array. At least one plugin configuration is required.

  6. description - provides more context regarding the configuration itself.

  7. prerequisites (optional) - an object which allows the definition of the productVersion member, which defines a range of supported Client Master versions (using the Maven Version Range Specification) for the plugin configuration. If the prerequisites object (and hence its productVersion member) is omitted, then any Client Master version is supported.

  8. includePlugins (optional) - an object which allows you to specify the ID values of one or more plugins to include as part of this plugin configuration. For each plugin (specified by its plugin-id), specify the pair made up of the supported groupId and version for the plugin or the URL from where download it.

  9. groupId (optional) - provides the groupId for the plugin when its download will take place from a maven repository. If the groupId is not provided, this field will consider org.jenkins-ci.plugins as default value.

  10. sha1 (optional) - the SHA1 hash of the plugin that will be used to verify the downloaded plugin.

  11. url (optional) - the download URL for the plugin.

  12. (Optional) - an additional plugin configuration that provides support for different plugin versions (or plugins themselves), for different Client Master versions (specified in the prerequisites > productVersion member).

  13. settings (optional) - the plugin catalog defines all parameters used in the download of the plugins.

  14. httpCredentialsProvider (optional) - the plugin catalog defines a set of credentials using the interface CredentialsProvider.

  15. credentials (required) - the array containing the credentials. Currently must contain only one element.

  16. authenticationScope (required) - if the credentials are defined, must currently have the value ANY.

  17. id (required) - the ID of a credential that must be defined in the Client Master where the plugin catalog will be installed. This credential will be used for plugin downloads from servers which require authentication.

  18. repository (optional) - an object which allows the definition of a maven2 repository where the plugins can be download from. In case of defining a repository, every plugin that has not a specified URL will be downloaded from this location.

  19. layout (optional) - must currently have the value maven2 and if it is not defined, this value will be used as default.

{
    "type" : "plugin-catalog",
    "version" : "1",
    "name" : "",
    "displayName" : "",
    "configurations" : [
        {
            "description" : "",
            "prerequisites": {
                "productVersion": ""
            },
            "includePlugins" : {
                "replace-with-appropriate-plugin-id": {
                    "groupId": "",
                    "version" : ""
                },
                "replace-with-other-appropriate-plugin-id": {
                    "sha1" : "",
                    "url" : ""
                }
            }
        }
    ],
    "settings": {
        "httpCredentialsProvider": {
            "credentials": [
                {
                    "authenticationScope": "ANY",
                    "id": ""
                }
            ]
        },
        "repository": {
            "layout": "maven2",
            "url": ""
        }
    }
}
Adding a Plugin Catalog to Operations Center

Once you have defined your plugin catalog, you can add it to your Operations Center instance using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), cd to the directory containing the plugin catalog file you defined above.

  2. Enter the command:

    jenkins-cli plugin-catalog --put < plugin-catalog-file.json

    where plugin-catalog-file.json is the name of your plugin catalog file.
    For example, if you used the plugin catalog example above and saved it to a file named pipeline-maven.json, then you would enter the command:

    jenkins-cli plugin-catalog --put < additional-pipeline-plugins.json

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog updated correctly",
      "status": "SUCCESS"
    }

    You can now validate the plugin catalog’s suitability for deployment on a configured Client Master.

Note:

  • If the command was not successful, for example - the JSON format of the plugin file was malformed, the Jenkins CLI tool returns an error like:

    {
      "message": "Unexpected character ('}' ... [Source: ... line: 16, column: 14]",
      "status": "FAILURE"
    }
Updating a Plugin Catalog on Operations Center

The procedure of updating an existing plugin catalog on Operations Center is identical to adding the plugin catalog to Operations Center.

  1. Modify the plugin catalog file defined previously.

  2. Re-add the plugin catalog to Operations Center.

Validating a Plugin Catalog’s Suitability for a Client Master

Once you have added a plugin catalog to Operations Center, you can verify this on the Operations Center interface when validating the plugin catalog’s suitability for deployment to a configured Client Master.

This validation process verifies:

  • That the Client Master version and plugin configuration satisfies exactly one plugin catalog definition

  • That plugin dependencies are resolved

  • That credentials referenced in the plugin catalog are defined in the Client Master

  • The credential count

  • The credential scope

To validate your plugin catalog’s suitability for deployment to a Client Master:

  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and select its Specify a plugin catalog for this master check box.

  3. Select the plugin catalog from the Catalog dropdown.
    This name is the displayName specified when the plugin catalog was defined.

  4. Click the Check Validity button.
    If your plugin catalog’s definition is suitable for deployment to the Client Master, the message The catalog is compatible with the master is displayed.
    Plugin Catalog validity check

  5. ( Optional ) Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the Client Master.

Note
To validate your plugin catalog in this manner against all Client Masters that have been connected to your Operations Center instance, repeat this procedure on each Client Master.
Deploying a Plugin Catalog to a Client Master

A plugin catalog which has been added to Operations Center can be deployed to a configured Client Master (after optionally validating its suitability for deployment) using either the Operations Center interface or the Jenkins CLI tool.

Note
Be aware that only one plugin catalog can be deployed to a Client Master at a time.
Deploying a Plugin Catalog Using the Operations Center interface
  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and select its Specify a plugin catalog for this master check box.

  3. Select the plugin catalog from the Catalog dropdown.
    This name is the displayName specified when the plugin catalog was defined.

  4. Scroll down the page and click the Save or Apply button to deploy the plugin catalog to the Client Master.

  5. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Deploying a Plugin Catalog Using the Jenkins CLI Tool

This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --push plugin-catalog-name --master client-master

    where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined and client-master is the name of the Client Master configured on your Operations Center instance.
    For example, if you used the plugin catalog example above and have already added it to your Operations Center instance, then you would enter the command:

    jenkins-cli plugin-catalog --push additional-pipeline-plugins --master client-master

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog applied correctly, it will be propagated in a few seconds",
      "status": "SUCCESS"
    }
    Note
    When using this Jenkins CLI command, the --push option must always be used in conjunction with the --master option. If you only use one of these options, the Jenkins CLI tool returns an error.
  2. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Verifying a Plugin Catalog’s Deployment to a Client Master

To validate that a plugin catalog has been successfully deployed to the Client Master:

  1. Ensure you are logged in to the Client Master as a user with the Administer permission.

  2. From the Client Master home page/Dashboard, click Manage Jenkins on the left.

  3. Click Beekeeper Upgrade Assistant.

  4. Click Plugin Catalog on the left (or the section’s More Info button on the right).
    Plugin Catalog on master
    If you see the message There is a plugin catalog installed, then the plugin catalog was deployed to the Client Master and the remaining details on this page should depict the plugin catalog that had been defined and added to Operations Center, and deployed to this Client Master.

    Note
    If you only just deployed a plugin catalog to the Client Master and you see No Plugin Catalog is installed at the moment, it may take a few minutes before the Client Master registers this deployment. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.
Removing a Plugin Catalog from a Client Master

A plugin catalog that has been deployed to a Client Master can be removed from it using either the Operations Center interface or the Jenkins CLI tool.

Removing a Plugin Catalog Using the Operations Center interface
  1. Access the relevant Client Master’s configuration in Operations Center.

  2. Scoll down to the Plugin Catalog property and clear its Specify a plugin catalog for this master check box.

  3. Scroll down the page and click the Save or Apply button to confirm the removal of the plugin catalog from the Client Master.

  4. ( Optional ) Verify the plugin catalog’s removal from the Client Master.

Removing a Plugin Catalog Using the Jenkins CLI Tool

This section assumes you have configured an alias for the Jenkins CLI tool.

  1. In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --remove client-master

    where client-master is the name of the Client Master configured on your Operations Center instance.
    For example, if you have already added a plugin catalog to your Operations Center instance and deployed it to a Client Master whose name is client-master, then you would enter the command:

    jenkins-cli plugin-catalog --remove client-master

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "client-master",
      "message": "Catalog removed",
      "status": "SUCCESS"
    }
  2. ( Optional ) Verify the plugin catalog’s deployment to the Client Master.

Verifying a Plugin Catalog’s Removal from a Client Master

To validate that a plugin catalog has been successfully removed from a Client Master:

  1. Ensure you are logged in to the Client Master as a user with the Administer permission.

  2. From the Client Master home page/Dashboard, click Manage Jenkins on the left.

  3. Click Beekeeper Upgrade Assistant.

  4. Click Plugin Catalog on the left (or the section’s More Info button on the right).
    If you see the message No Plugin Catalog is installed at the moment, then the plugin catalog was removed from the Client Master.

    Note
    If you only just removed a plugin catalog from the Client Master and you see There is a plugin catalog installed, it may take a few minutes before the Client Master registers this removal. Therefore, wait a few moments and refresh this page, or revisit this procedure at a later point in time.
Removing a Plugin Catalog from Operations Center

A plugin catalog added to your Operations Center instance can be removed from it using the Jenkins CLI tool. This section assumes you have configured an alias for the Jenkins CLI tool.

Note
A plugin catalog can only be removed from your Operations Center instance when no Client Masters are using the plugin catalog.
  • In a terminal/command prompt (window), enter the command:

    jenkins-cli plugin-catalog --delete plugin-catalog-name

    where plugin-catalog-name is the name of the plugin catalog specified when the plugin catalog was defined.
    For example, if you used the plugin catalog example above, then you would enter the command:

    jenkins-cli plugin-catalog --delete additional-pipeline-plugins

    If the command was successful, the Jenkins CLI tool returns the following JSON response:

    {
      "id": "additional-pipeline-plugins",
      "message": "Catalog [additional-pipeline-plugins] successfully deleted",
      "status": "SUCCESS"
    }

Note:

  • If the Jenkins CLI tool returns an error like:

    {
      "message": "Catalog [additional-pipeline-plugins] is being used by the following master/s: [client-master]. It can not be deleted",
      "status": "FAILURE"
    }

    then the plugin catalog is still being used by at least one Client Master (in this example, a Client Master named client-master). If the plugin catalog is being used by multiple Client Masters, then each Client Master is listed in the error response. To rectify this issue, remove the plugin catalog from each Client Master listed in the error response and then try following this procedure again to remove the plugin catalog from your Operations Center instance.

Configuring Masters through CLI

Jenkins allows some operations to be invoked through CLI, some of them being useful to configure Client Masters.

But also, when we are talking about Client Masters configuration, applying the same identical configuration to each one is something desirable. Thus we need a way to gather all connected masters from the command line and then perform some operations on each one.

The list-masters CLI command on Operations Center provides information about all connected masters in JSON format, allowing you to use that information to invoke commands on each Client Master:

{
  "version": "1",
  "data": {
    "masters": [
      {
        "fullName": "my master", (1)
        "url": "http://localhost:9090/", (2)
        "status": "ONLINE" (3)
      }
    ]
  }
}
  1. fullName - the name of the Client Master, including folders.

  2. url - URL for the Client Master.

  3. status - the connection status of the Client Master. It can be 'ONLINE' or 'OFFLINE'.

Examples:

Execute a Groovy script and install a plugin

The following bash script will execute a Groovy script and install the 'beer' plugin on all the online Client Masters:

#!/usr/bin/env bash

JENKINS_CLI=jenkins-cli.jar
JENKINS_CJOC_URL=http://localhost:8080/
JENKINS_AUTH=admin:admin

if [ -z "$JENKINS_CJOC_URL" ]; then
    echo "Need to set environment variable JENKINS_CJOC_URL ({CJOC} root URL)."
    exit 1
fi

if [ -z "$JENKINS_AUTH" ]; then
    echo "Need to set environment variable JENKINS_AUTH (format: 'userId:apiToken')."
    exit 1
fi


if [ -f "$JENKINS_CLI" ]
then
	echo "Using $JENKINS_CLI."
else
	wget -O "$JENKINS_CLI" $JENKINS_CJOC_URL/jnlpJars/jenkins-cli.jar
fi

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH list-masters | jq -r '.data.masters[] | select(.status == "ONLINE") | .url' | while read url; do
	java -jar $JENKINS_CLI -s $url -auth $JENKINS_AUTH groovy = < configuration-script.groovy
	java -jar $JENKINS_CLI -s $url -auth $JENKINS_AUTH install-plugin beer
done
Push a Plugin Catalog to all Client Masters

The following bash script will add a Plugin Catalog to Operations Center and push it to all the online Client Masters:

#!/usr/bin/env bash

JENKINS_CLI=jenkins-cli.jar
JENKINS_CJOC_URL=http://localhost:8080/
JENKINS_AUTH=admin:admin

if [ -z "$JENKINS_CJOC_URL" ]; then
    echo "Need to set environment variable JENKINS_CJOC_URL ({CJOC} root URL)."
    exit 1
fi

if [ -z "$JENKINS_AUTH" ]; then
    echo "Need to set environment variable JENKINS_AUTH (format: 'userId:apiToken')."
    exit 1
fi


if [ -f "$JENKINS_CLI" ]
then
	echo "Using $JENKINS_CLI."
else
	wget -O "$JENKINS_CLI" $JENKINS_CJOC_URL/jnlpJars/jenkins-cli.jar
fi

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH  plugin-catalog --put < "java-web-catalog.json" | jq

java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH list-masters | jq -r '.data.masters[] | select(.status == "ONLINE") | .fullName' | while read masterName; do
	java -jar $JENKINS_CLI -s $JENKINS_CJOC_URL -auth $JENKINS_AUTH plugin-catalog --push "java-web" --master "$masterName" | jq
done

Upgrading Managed Masters

To update a Managed Master’s version, the administrator needs to update the version of the Docker image for the Managed Master. Once this version is updated, the Managed Master and its default plugins will be upgraded to the latest versions defined in the the image, while any non-default plugins will be left at their current versions.

master configuration overview 2

Once the Docker image definition is updated, the CloudBees Jenkins Enterprise administrator will need to restart the instance so the Managed Master can begin using its upgraded components.

action stop
action stop confirm
action start

Bulk-upgrading Managed Masters

When a CloudBees Jenkins Enterprise cluster serves many teams and contains many masters, an administrator can save time and greatly reduce the overhead of upgrading those masters by creating a repeatable task to automate this process. In CloudBees Jenkins Enterprise, this can be achieved by defining a cluster operation in Operations Center.

To create this task, the administrator will first need to log into Operations Center, then create a New Item of the type Cluster Operations. The administrator will then needs to select the Managed Masters cluster operation, and will then be given a set of pre-configured upgrade patterns to choose from.

The administrator will then need to specify which masters to target by using the filter Uses Docker Image and picking a Docker image used by their cluster’s masters as the upgrade target for this operation. Any masters in the cluster using the selected image will be affected by this cluster operation.

In the Steps section, the administrator should select Update Docker Image and pick the new Docker image to bulk-upgrade the targeted masters to. Next, the administrator should add a Reprovision step to restart the targeted masters.

Once these settings are configured, the administrator can run the cluster operation to perform a bulk upgrade of their cluster’s masters, or schedule the operation to run at a later time.

Quiet start

There may be times during an upgrade or other maintenance when it is best to have Jenkins start, but launch no projects. For example, if an upgrade is being performed in multiple steps, the intermediate steps may not be fully configured to run projects successfully. The "quiet start plugin" can immediately place the Jenkins server in "quieting down" state on startup.

Enable "quiet start" by checking the box in Manage Jenkins  Quiet Restart. When the server is restarted, it will be in the "quieting down" state. An administrator can cancel that state using the regular UI.

Uncheck the box in Manage Jenkins  Quiet Restart when maintenance is complete. Projects will start as usual on server restart.

Reviewing plugin usage

Large Jenkins installations often have many plugins installed. CloudBees Jenkins Enterprise can help administrators identify plugins which may be unused or no longer useful. The CloudBees Plugin Usage Plugin helps you keep track of which plugins you are actively using and where they are used.

Plugin usage report

A table of installed plugins is available from Manage Jenkins  Plugin Usage, including a usage count and a list of uses of that plugin. A strikethrough font indicates disabled plugins. A progress bar runs while Jenkins scans jobs and global settings. Once the scan is complete, the Usage Count column becomes a sortable column. Click the column heading to sort the column.

usage count
Figure 1. Usage Count

The third column shows detected uses of the plugin, hyperlinked to the matching configuration screen. In the list of hyperlinks, "Jenkins" refers to global Jenkins configuration. Most other labels are names of configurable items such as jobs (using » to show folder structure where relevant); configuration associated with Jenkins users (such as credentials) will also be shown. Code packaged as a plugin but really used only as a library shared between several "real" plugins (Async Http Client Plugin, for example) will be shown as used by those plugins. »… is appended to items which have nested elements also using the plugin; for example, a native Maven project may have several modules all of which refer to the plugin.

Only a Jenkins administrator can perform a complete plugin usage analysis. Configuration read permission is required to check plugin usage.

Limitations

Beware that some plugins will not be listed in this table, because they are not directly mentioned in any stored configuration. They may affect how Jenkins runs in various ways without configuration; the Jenkins Translation Assistance Plugin is an example.

Conversely, a plugin might have some stored configuration which is of no interest or never used at all. For example, Jenkins Email Extension Plugin displays various controls in the global Configure System screen; when you click Save, Email Extension configuration is saved in hudson.plugins.emailext.ExtendedEmailPublisher.xml in your $JENKINS_HOME, even if you have made no customizations. This plugin will thus appear to be "in use" by Jenkins. Only if you have enabled the Editable Email Notification post-build action for a job will it have a usage count greater than 1, however.

Use the plugin usage table as a starting point for investigating plugins that might be unused. It is not relevant for all plugins. The table works best for checking the usage of plugins which offer concrete additions to configuration, such as build steps or publishers in jobs.

Plugins used in configuration generated by a builder (or publisher) template will not be listed (in either the template or jobs using it), since this configuration is created on the fly during a build rather than being stored permanently in the job. Configuration generated by a job (or folder) template is considered.

Alerting

CloudBees Jenkins Enterprise automatically monitors its infrastructure elements and all of the Jenkins masters that it manages. These alerts can be helpful in troubleshooting.

The CloudBees Jenkins Enterprise infrastructure element monitoring includes Operations Center and Managed Masters. For the various infrastructure nodes, it monitors the following metrics.

  • Available disk space

  • CPU utilization for the most recent 5 minutes

  • RAM utilization for the most recent 5 minutes

If any of the data points for these metrics exceed 90% or more, a threshold which is currently immutable, CloudBees Jenkins Enterprise will emit an alert, for example:

Health checks failing: [worker-14: Disk util at 95%, worker-9: Worker down]

The following table show the possible error messages and corresponding descriptions.

Table 1. Possible Failure Messages
Messages Descriptions

Disk util at <number>%

Disk utilization reaches 90% or higher

RAM util at <number>%

RAM utilization reaches 90% or higher for five or more minutes

CPU util at <number>%

Total CPU utilization reaches 90% or higher for five or more minutes. The percent utilization is normalized to 100% across all CPU’s on the node.

Additional monitoring is available with the Elasticsearch Reporter plugin.