Managing CloudBees Core

Managing Your Operations Center Instance

This chapter describes how to manage your Operations Center instance.

Accessing Jenkins CLI

Jenkins CLI allows you to perform a number of activities on items (i.e. jobs or projects configured in your Operations Center instance), as well as agent- and administration-related activities on Operations Center, all from the command line.

The Operations Center’s Jenkins CLI feature (http://operations-center-url:port-number/cli) provides access to the Jenkins CLI tool (jenkins-cli.jar), which you can download from this page/URL. This page also provides detailed information about using this tool and its features.

To access the Jenkins CLI feature’s page through the Operations Center interface and download the CLI tool:

  1. Access the Manage Jenkins area.

  2. On the Manage Jenkins page, scroll down and click Jenkins CLI.

  3. On the Jenkins CLI page, click the jenkins-cli.jar link to download the CLI tool near the top of this page.

  4. Move the jenkins-cli.jar file to an appropriate location on your operation system’s file system.

Using the Jenkins CLI Tool

Once you have obtained the Jenkins CLI tool (above), to use the CLI tool:

  1. In a terminal/command prompt (window), cd to the directory containing the jenkins-cli.jar file.

  2. Enter the command:

    java -jar jenkins-cli.jar -s http://operations-center-url:port-number/ command

    where:

    • port-number is the HTTP port through which Operations Center is accessible.

    • command is the Jenkins CLI command you want to execute.

    Tip

    To obtain a list of all options (like -s above or -auth for authentication) that can be passed to the
    jenkins-cli.jar command itself, enter the command:

    java -jar jenkins-cli.jar -h

Important Examples of Usage

  • To list all available Jenkins CLI tool commands, enter the command:

    java -jar jenkins-cli.jar -s http://operations-center-url:8888/ help

    which outputs a list of commands followed by indented descriptions, like:

    add-job-to-view
      Adds jobs to view.
    ...
  • To authenticate your Jenkins user when using the Jenkins CLI tool, specify the -auth option with your username and password in the format: username:password.
    For example, enter the command:

    java -jar jenkins-cli.jar -auth username:password -s http://operations-center-url:8888/ version

    which outputs something like:

    2.107.2.2
    Note

    Many Jenkins CLI commands require an authenticated Jenkins user with appropriate permissions, such as the Administer permission.

  • To obtain more information about a particular Jenkins CLI command, specify help followed by the command itself.
    For example, to obtain more information about the plugin-catalog command, enter the command:

    java -jar jenkins-cli.jar -auth username:password -s http://operations-center-url:8888/ help plugin-catalog

    which outputs:

    java -jar jenkins-cli.jar plugin-catalog [VAL] [--delete VAL] [--master VAL] [--push VAL] [--put] [--remove VAL]
    Plugin Catalog Management
     VAL          : Use a catalog name to get its metadata content. A list of
                    catalog names is returned by calling the command without any
                    parameter.
                    plugin-catalog
                    plugin-catalog my-plugin-catalog
     --delete VAL : Use this option to delete a catalog from the available list,
                    whose name is passed as parameter.
                    --delete my-plugin-catalog
     --master VAL : Master item full name (including folders and white spaces,
                    which needs double quotes). Use it in tandem with --push to
                    select the master to push the catalog to.
                    --master my-master --push my-plugin-catalog
     --push VAL   : This option pushes a catalog to a master. Use it in tandem with
                    --master to select the master to apply the catalog to.
                    --master my-master --push my-plugin-catalog
     --put        : Create or update a catalog. It will not be applied to any
                    master, just added to the catalog management.
                    --put < my-plugin-catalog.json
     --remove VAL : Removes the current catalog from a master. The catalog will not
                    be deleted from the available list.
                    --remove my-master-with-a-plugin-catalog

Configuring an Alias for the Jenkins CLI Tool

If you find yourself using the Jenkins CLI tool on a regular basis, configure an alias to save you having to write out the entire command each time.

For instance, assuming you are using on Linux operating systems and/or a Bash [1] command language environment:

  1. cd to the directory containing the jenkins-cli.jar file.

  2. In your favorite text editor, copy and paste the following into a new "alias" file:

    #!/bin/sh
    
    java -jar jenkins-cli.jar -auth username:password -s http://operations-center-url:port-number/ "$@"
  3. Adjust the username, password, operations-center-url and port-number values as required.

  4. Save the alias file with an appropriate file name (e.g. jenkins-cli.sh).
    Note: This alias file assumes that jenkins-cli.jar is located in the same directory.

  5. Make the alias file executable (e.g. chmod a+x jenkins-cli.sh).

  6. Configure the actual alias to your alias file in your appropriate Bash configuration file (i.e .bash_profile or .bashrc), which is usually located in your home directory.
    For example, in your favorite text editor, copy and paste the following line into .bash_profile or .bashrc):

    alias jenkins-cli='/path/to/jenkins-cli.sh'
  7. Edit the /path/to value as required and save the configuration file.

  8. Reload your Bash configuration by entering the appropriate command (based on the file you edited).
    For example, enter the appropriate command:

    • source ~/.bash_profile
      or

    • source ~/.bashrc

From now on, regardless of your current directory, you can enter the command: jenkins-cli, which would actually be:

java -jar jenkins-cli.jar -auth username:password -s http://operations-center-url:port-number/

Therefore, using this alias, the important examples of usage (above) could be simplified to the following:

  • Listing all available Jenkins CLI tool commands:

    jenkins-cli help
  • Retrieving the version of your Operations Center instance:

    jenkins-cli version
  • Obtaining more information about a specific Jenkins CLI command:

    jenkins-cli help plugin-catalog
Note

The remaining Jenkins CLI examples used throughout this guide assume that this alias has been configured.

Configuring the Jenkins CLI tool to work with non-TrustStore SSL certificates

If your CloudBees Core instance has been secured with a non-TrustStore SSL certificate, then you may need to configure your local machine running the Jenkins CLI tool, to allow it to successfully communicate with CloudBees Core.

  1. Ensure that the keytool command is running on the machine running the Jenkins CLI tool.

  2. Obtain/download your CloudBees Core’s non-TrustStore SSL certificate directly from the web browser itself - naming it, for example, cloudbees-core.example.com.crt.

    In Firefox, this can be done by doing the following:

    1. Visit your CloudBees Core site.

    2. Click its padlock icon to the left of the URL field.

    3. Click the > to the right of Connection.

    4. Click More Information at the base of the resulting dialog box.

    5. In the resulting Page Info dialog box, ensure the Security tab is selected and click the View Certificate button.

    6. On the resulting Certificate Viewer dialog box, click the Details tab and click the Export button.

  3. Create a keystore and import this certificate by running the command (changing the values appropriately):

    keytool -import -noprompt -trustcacerts -alias cloudbees-core.example.com -file cloudbees-core.example.com.crt -keystore myKeystore -storepass changeme
  4. Test that the certificate configured in your keystore works and now provides your locally running Jenkins CLI tool access to your CloudBees Core instance by running this command:

    java -Djavax.net.ssl.trustStore=myKeystore -Djavax.net.ssl.trustStorePassword=changeme -jar jenkins-cli.jar -s https://cloudbees-core.example.com/cjoc/ help

    If you receive a list of available Jenkins CLI commands in both cases, then your Jenkins CLI is now working.

  5. Update the alias file/s configured for your Jenkins CLI tool to insert these additional components between java and -jar:

    -Djavax.net.ssl.trustStore=myKeystore -Djavax.net.ssl.trustStorePassword=changeme

External HTTP Endpoints

Introduction

External HTTP endpoints work with Cross Team Collaboration to enable external systems such as GitHub or Nexus to generate notification events for Pipelines.

Jenkins external notification webhooks are user-defined HTTP callbacks that can be triggered by external actions, such as pushing a new artifact to a Nexus repository. When this happens, the external system makes an HTTP request to this endpoint and passes a data payload in the body of the request. Jenkins broadcasts the request via the internal messaging API to all jobs that are configured with an event trigger.

Webhook events are typically secured with a shared secret that the sender uses to generate a unique message digest. The message digest is sent to the receiving system in an HTTP request header. Jenkins uses the digest and the shared secret to verify that the message has not been altered and that it originates from a trusted source.

In addition to using a shared secret, some webhooks support the use of IP address whitelisting, which allows an Administrator to specify a list of addresses (or address ranges) that are allowed to create a webhook events.

Note

Validating the contents of the incoming message using a shared secret known only by the sender and the receiver ensures that rogue users are not able to send "fake" events which would cause Jenkins to trigger unexpected actions.

high level
Figure 1. External HTTP Endpoints Overview

The external HTTP endpoints feature requires the installation of the following plugins on the Operations Center instance:

External HTTP Endpoints Configuration Page

External webhook endpoints are added and deleted from Manage Jenkins  Manage Notification Webhook HTTP Endpoints.

external endpoints config
Figure 2. The external HTTP endpoints configuration page
Note

The Manage Notification Webhook HTTP Endpoints option will only appear if the external-notification-plugin plugin is installed.

The configuration page also shows the details of current endpoints.

webhooks config
Figure 3. The external HTTP endpoints configuration page

The configuration screen shows all currently enabled endpoints, with the following details:

  • Name - The user friendly name given to this endpoint

  • Endpoint URL - The URL by which this endpoint can be accessed

  • Last Event - The date/time at which this endpoint was last accessed

  • Last Status - The HTTP status code associated with the last request received by this endpoint

  • Webhook Message Format - The type of message that this endpoint accepts

  • Details - Displays the full details for this endpoint

  • Delete - Deletes this endpoint.

    Note: This action cannot be undone.

Hovering over the Last Status field or clicking on the Details button shows the HTTP status message associated with the last request received by an endpoint. This can be very helpful when troubleshooting new webhook configurations.

External HTTP Notification Endpoint Types

Several endpoint types are already included. Custom endpoints can be defined by specifying the security type and relevant details. The following endpoint types are currently supported:

  • Custom JSON Webhook

  • Bitbucket Cloud

  • Bitbucket Server

  • Docker Hub

  • GitHub

  • Nexus 3

To configure a new endpoint, click Register a new endpoint.

Custom JSON Webhooks

The Custom JSON webhook allows full control of the configuration of a new webhook endpoint. This allows integrations with systems not supported with an out of the box integration. Custom JSON webhooks support two forms of security - HMAC message signatures and IP address whitelisting. At least one of these security types must be chosen, and if desired, both can be used together.

Note

Custom JSON webhooks require the use of at least one type of security - either HMAC signatures or IP address whitelists.

Custom JSON Webhooks Using HMAC Signature Validation

The custom JSON webhook allows full control of the configuration of a new webhook endpoint. This allows integrations with systems not supported with an out of the box configuration.

custom json webhook config
Figure 4. Configuring a custom JSON webhook endpoint with HMAC signature validation
  • Webhook Name - A user friendly name for this new endpoint

  • HMAC HTTP Header - The name of the HTTP request header that will contain the message digest

  • Message Signature Validation Type - The cryptographic hash type that will be used to generate the message digest

  • Secret - The shared secret that will be used to validate the authenticity of received events. Use the Generate Secret button to automatically generate a cryptographically random secret.

Refer to HMAC (hash-based message authentication code) for more information on this widely used method of verifying the integrity and authenticity of a message.

Custom JSON Webhook With Whitelists

Not all webhook providers support HMAC signature validation. For those that don’t, it is possible to provide a level of security by whitelisting the IP addresses of the system that will generate the webhook events. To do this, check the box labeled "Define a list of IP addresses that are allowed to send events to this endpoint" and enter the IP addresses (one per line) of the remote host which will be generating the webhook events. You may enter both IPv4 and IPv6 addresses, as well as address ranges using CIDR notation.

custom json webhook config none
Figure 5. Configuring a custom JSON webhook endpoint with an IP whitelist
Note

When choosing NONE as the Message Signature Validation Type the whitelist entry will automatically be shown. Although it is not recommended, you may enter 0.0.0.0/0 and ::/0 which will allow access from any host. If no addresses are specified, then access from any host will be denied.

Bitbucket Cloud

The Bitbucket Cloud webhook endpoint is pre-configured to work with Bitbucket Cloud events. Bitbucket Cloud does not support HMAC signatures, and per Atlassian’s documentation, it is recommended to secure your endpoint using an IP whitelist. The Bitbucket Cloud webhook is pre-configured with the IP addresses from Atlassian’s documentation.

bitbucket cloud webhook config
Figure 6. Configuring a Bitbucket Cloud webhook endpoint

The following fields are required to create a new Bitbucket Cloud endpoint:

  • Webhook Name - A user friendly name for this new endpoint

  • Whitelist - A list of IP addresses (or address ranges) specifying the hosts that are allowed to send Bitbucket Cloud webhook events to your Jenkins instance.

  • Secret - (Optional) A unique string which will be use to validate the authenticity of incoming webhook events. Use the Generate Secret button to automatically generate a cryptographically random secret, or leave this field blank if you do not wish to perform this validation (this is not recommended).

It is also possible to specify a shared secret which will be included in the URL that is generated for a Bitbucket Cloud webhook endpoint. Although no signature validation will be performed, incoming events that do not include this secret URL parameter will be ignored. Be sure to copy the entire url shown in the Manage Notification Webhook Endpoints list to use when configuring your Bitbucket Cloud webhook.

bitbucket cloud secret url
Figure 7. Bitbucket Cloud Webhook URL With Secret

For more information about Bitbucket Cloud webhooks, please refer to Manage Webhooks documentation provided by Atlassian.

Bitbucket Server

The Bitbucket Server webhook endpoint is pre-configured to work with Bitbucket Server events, and supports HMAC SHA256 message signatures.

bitbucket server webhook config
Figure 8. Configuring a Bitbucket Server Webhook Endpoint

The following fields are required to create a new Bitbucket Server endpoint:

  • Webhook Name - A user friendly name for this new endpoint

  • Secret - The shared secret that will be used to validate the authenticity of received events. Use the Generate Secret button to automatically generate a cryptographically random secret.

Refer to the Bitbucket Server documentation for more infomation on Bitbucket Server webhooks.

Docker Hub

The Docker Hub webhook endpoint is pre-configured to work with Docker Hub webhook events. Docker Hub does not support HMAC message signatures, so it is highly recommended to specify a secret which will be included with every webhook request that Docker Hub makes to your Jenkins instance. When a secret is configured then only requests which include this will be accepted.

docker hub webhook config
Figure 9. Configuring a Docker Hub Webhook Endpoint

The following fields are required to create a new Docker Hub endpoint:

  • Webhook Name - A user friendly name for this new endpoint

  • Secret - (Optional) A unique string which will be use to validate the authenticity of incoming webhook events. Use the Generate Secret button to automatically generate a cryptographically random secret, or leave this field blank if you do not wish to perform this validation (this is not recommended).

docker hub secret url
Figure 10. Docker Hub Webhook With Secret

Be sure to copy the entire url shown in the Manage Notification Webhook Endpoints list to use when configuring your Docker Hub webhook.

Nexus 3 Webhook

The Nexus 3 webhook endpoint allows for integration with Nexus events. Nexus webhooks are secured with a shared secret and allow you to subscribe to certain events on the Nexus repository.

nexus3 webhook config
Figure 11. Configuring a Nexus 3 webhook endpoint

The following fields are required to create a new Nexus endpoint:

  • Webhook Name - A user friendly name for this new endpoint

  • Secret - The shared secret that will be used to validate the authenticity of received events. Use the Generate Secret button to automatically generate a cryptographically random secret.

Refer to the Nexus documentation for more information on Nexus webhooks.

GitHub Webhook

The GitHub webhook endpoint integrates with GitHub events. GitHub webhooks are secured with a shared secret and allow you to subscribe to events from GitHub whenever certain actions (ie. tag/label creation, github release, etc)

github webhook config
Figure 12. Configuring a GitHub webhook endpoint

The following fields are required to create a new GitHub endpoint:

  • Webhook Name - A user friendly name for this new endpoint

  • Secret - The shared secret that will be used to validate the authenticity of received events. Use the 'Generate Secret' button to automatically generate a cryptographically random secret.

Refer to the GitHub documentation for more information on GitHub webhooks.

Integrating Webhooks with Pipeline Event Triggers

A webhook payload is used by a matching event trigger in the pipeline jobs that are interested in the message. Ensure that the following plugins are installed on the master running the pipeline:

Refer to the Cross Team Collaboration documentation to configure notifications.

When a webhook receives an event, all the Pipelines listening on events (event triggers) will be queried. The Pipelines with a trigger condition that matches the publishEvent will be executed. The trigger step uses query patterns based on jmespath syntax to match events. Triggers can be configured directly from the Pipeline job configuration page or can be written as a Pipeline step.

Example Webhook Event Trigger

This GitHub webhook example assumes you have installed the required plugins and that you have an existing GitHub account.

1 - Create a New GitHub Webhook Endpoint

From Manage Jenkins  Manage Notification Webhook Endpoints, create a new GitHub webhook with the following values:

  • Name - My First Webhook

  • Secret - NotAGoodSecret

example github webhook
Figure 13. Create a new webhook endpoint for GitHub

2 - Create a New Webhook on GitHub

Next, from the Settings tab on your GitHub repository, select Webhooks on the left hand side and click Add webhook. Fill in the form values as shown:

  • Payload URL - Paste the Jenkins webhook endpoint URL that was generated in the first step

  • Content type - Select application/json from the dropdown list

  • Secret - Enter the shared secret value (NotAGoodSecret) from the first step

  • The rest of the form values can be left at their defaults

After filling in the details, click Add webhook to save the changes.

configure github webhook
Figure 14. Create a new webhook on GitHub

3 - Validate the New Webhook Endpoint

After the webhook is defined, GitHub will send a test message to your Jenkins endpoint. Click Edit for the webhook you created and scroll to the bottom of the page. If all went well, it will show something like:

github recent deliveries
Figure 15. Validating your new endpoint
Note

If you see a red triangle instead of a green check mark, click on the message id to see more details. Make needed changes to your configuration and click on the Redeliver button to validate again.

4 - Create a Pipeline Job Triggered by the Webhook Event

Create a Pipeline that will trigger when an event is received. We will use the GitHub test message and create a job that will build when it sees the attribute "zen" with a value of "Non-blocking is better than blocking.". Create a new Pipeline job and copy the following script:

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any

    triggers {
        eventTrigger jmespathQuery("zen=='Non-blocking is better than blocking.'")
    }

    stages {
        stage('Example') {
            steps {
                sh 'echo Hello World';
            }
        }
    }
}

Save your job and then click Build Now once to ensure the event triggers are properly registered.

5 - Test!

Clicking Redeliver from the GitHub repository webhook configuration will send an event and broadcast the message to all listening jobs. The Pipeline job created in the previous step will see this event, and trigger a build!

[Pipeline] {
[Pipeline] stage
[Pipeline] { (Example)
[Pipeline] sh
[Github Trigger] Running shell script
+ echo Hello World
Hello World
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

This is a very simple example illustrating the basic steps needed to integrate external webhook events with Jenkins. For more details about writing build triggers, please refer to Cross Team Collaboration

Troubleshooting

There are a number of steps that can be helpful when troubleshooting webhooks:

  • Endpoint "Last Event"/"Last Status" fields do not show any webhook request was received

    • Verify the endpoint url is accessible outside the firewall

    • Verify that the external system that is generating webhook events is configured with the correct URL for your endpoint

  • Verify that the external system is using the expected mime-type of application/json

    • Click Details to view the error message associated with the most recent request

  • Endpoint receives a message, but the "Last Status" shows an error code

    • Click Details to view the error message associated with the most recent request

  • Endpoint successfully receives a message, but build jobs are not triggering as expected

    • Verify that Notification is enabled and configured

    • Verify that your JMESPath trigger query is correct

    • Create a new log recorder for com.cloudbees.opscenter.plugins.notification.endpoint.WebhookEndpoint and set the level to FINEST in order to view the content of the webhook message received by Jenkins. This can be helpful in creating and troubleshooting event trigger queries.

Note

The Details view can be very helpful when troubleshooting webhook problems. In addition to allowing you to verify the endpoint configuration, the details view will show additional information about the type of error you may be encountering.