Automating projects

This chapter will cover all aspects of Jenkins Pipeline, from running Pipelines to writing Pipeline code, and even extending Pipeline itself. If you’re already familiar with the concept of Jenkins Pipeline, continue on to Automating projects with a Jenkinsfile or Pipeline Development Tools.

{CJT} via Blue Ocean
Figure 1. Pipeline Runs in Blue Ocean

What is Pipeline?

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines "as code" via the Pipeline DSL. [1]

Typically, this "Pipeline as Code" would be written to a Jenkinsfile and checked into a project’s source control repository, for example:

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any (1)

    stages {
        stage('Build') { (2)
            steps { (3)
                sh 'make' (4)
            }
        }
        stage('Test'){
            steps {
                sh 'make check'
                junit 'reports/**/*.xml' (5)
            }
        }
        stage('Deploy') {
            steps {
                sh 'make publish'
            }
        }
    }
}
  1. agent indicates that Jenkins should allocate an executor and workspace for this part of the Pipeline.

  2. stage describes a stage of this Pipeline.

  3. steps describes the steps to be run in this stage

  4. sh executes the given shell command

  5. junit is a Pipeline step provided by the JUnit plugin for aggregating test reports.

Why Pipeline?

Jenkins is, fundamentally, an automation engine which supports a number of automation patterns. Pipeline adds a powerful set of automation tools onto Jenkins, supporting use cases that span from simple continuous integration to comprehensive continuous delivery pipelines. By modeling a series of related tasks, users can take advantage of the many features of Pipeline:

  • Code: Pipelines are implemented in code and typically checked into source control, giving teams the ability to edit, review, and iterate upon their delivery pipeline.

  • Durable: Pipelines can survive both planned and unplanned restarts of the Jenkins master.

  • Pausable: Pipelines can optionally stop and wait for human input or approval before continuing the Pipeline run.

  • Versatile: Pipelines support complex real-world continuous delivery requirements, including the ability to fork/join, loop, and perform work in parallel.

  • Extensible: The Pipeline plugin supports custom extensions to its DSL [1] and multiple options for integration with other plugins.

While Jenkins has always allowed rudimentary forms of chaining Freestyle Jobs together to perform sequential tasks, [2] Pipeline makes this concept a first-class citizen in Jenkins.

Building on the core Jenkins value of extensibility, Pipeline is also extensible both by users with Pipeline Shared Libraries and by plugin developers. [3]]

Pipeline Terms

Step

A single task; fundamentally steps tell Jenkins what to do. For example, to execute the shell command make use the sh step: sh 'make'. When a plugin extends the Pipeline DSL, that typically means the plugin has implemented a new step.

Node

Most work a Pipeline performs is done in the context of one or more declared node steps. Confining the work inside of a node step does two things:

  1. Schedules the steps contained within the block to run by adding an item to the Jenkins queue. As soon as an executor is free on a node, the steps will run.

  2. Creates a workspace (a directory specific to that particular Pipeline) where work can be done on files checked out from source control.

Caution
Depending on your Jenkins configuration, some workspaces may not get automatically cleaned up after a period of inactivity. See tickets and discussion linked from JENKINS-2111 for more information.
Stage

stage is a step for defining a conceptually distinct subset of the entire Pipeline, for example: "Build", "Test", and "Deploy", which is used by many plugins to visualize or present Jenkins Pipeline status/progress. [4]

Getting Started with Pipeline

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines "as code" via the Pipeline DSL. [5]

This section introduces some of the key concepts to Jenkins Pipeline and help introduce the basics of defining and working with Pipelines inside of a running Jenkins instance.

Defining a Pipeline

A basic Pipeline can be created in either of the following ways:

  • By entering a script directly in the Jenkins web UI.

  • By creating a Jenkinsfile which can be checked into a project’s source control repository.

The syntax for defining a Pipeline with either approach is the same, but while Jenkins supports entering Pipeline directly into the web UI, it’s generally considered best practice to define the Pipeline in a Jenkinsfile which Jenkins will then load directly from source control. [6]

The Jenkins project maintains a Pipeline Syntax Reference which may useful to refer to when creating a Jenkinsfile.

Defining a Pipeline in the Web UI

To create a basic Pipeline in the Jenkins web UI, follow these steps:

  • Click New Item on Jenkins home page.

Click *New Item* on the Jenkins home page
  • Enter a name for your Pipeline, select Pipeline and click OK.

Caution

{CJT} uses the name of the Pipeline to create directories on disk. Pipeline names which include spaces may uncover bugs in scripts which do not expect paths to contain spaces.

Enter a name, select *Pipeline*, and click *OK*
  • In the Script text area, enter a Pipeline and click Save.

In the *Script* text area, enter a Pipeline and click Save
  • Click Build Now to run the Pipeline.

Click *Build Now* to run the Pipeline
  • Click #1 under "Build History" and then click Console Output to see the full output from the Pipeline.

Click *Console Output* for the Pipeline

The example above shows a successful run of a basic Pipeline created in the Jenkins web UI, using two steps.

Jenkinsfile (Scripted Pipeline)
node { (1)
    echo 'Hello World' (2)
}
  1. node allocates an executor and workspace in the Jenkins environment.

  2. echo writes simple string in the Console Output.

Defining a Pipeline in SCM

Complex Pipelines are hard to write and maintain within the text area of the Pipeline configuration page. To make this easier, Pipeline can also be written in a text editor and checked into source control as a Jenkinsfile which Jenkins can load via the Pipeline Script from SCM option.

To do this, select Pipeline script from SCM when defining the Pipeline.

With the Pipeline script from SCM option selected, you do not enter any Groovy code in the Jenkins UI; you just indicate by specifying a path where in source code you want to retrieve the pipeline from. When you update the designated repository, a new build is triggered, as long as the Pipeline is configured with an SCM polling trigger.

Tip

The first line of a Jenkinsfile should be #!groovy [7] which text editors, IDEs, GitHub, etc will use to syntax highlight the Jenkinsfile properly as Groovy code.

Built-in Documentation

Pipeline ships with built-in documentation features to make it easier to create Pipelines of varying complexities. This built-in documentation is automatically generated and updated based on the plugins installed in the Jenkins instance.

The built-in documentation can be found globally at: localhost:8080/pipeline-syntax/, assuming you have a Jenkins instance running on localhost port 8080. The same documentation is also linked as Pipeline Syntax in the side-bar for any configured Pipeline project.

Pipeline Syntax in the side-bar

Snippet Generator

The built-in "Snippet Generator" utility is helpful for creating bits of code for individual steps, discovering new steps provided by plugins, or experimenting with different parameters for a particular step.

The Snippet Generator is dynamically populated with a list of the steps available to the Jenkins instance. The number of steps available is dependent on the plugins installed which explicitly expose steps for use in Pipeline.

To generate a step snippet with the Snippet Generator:

  1. Navigate to the Pipeline Syntax link (referenced above) from a configured Pipeline, or at localhost:8080/pipeline-syntax.

  2. Select the desired step in the Sample Step dropdown menu

  3. Use the dynamically populated area below the Sample Step dropdown to configure the selected step.

  4. Click Generate Pipeline Script to create a snippet of Pipeline which can be copied and pasted into a Pipeline.

Snippet Generator

To access additional information and/or documentation about the step selected, click on the help icon (indicated by the red arrow in the image above).

Global Variable Reference

In addition to the Snippet Generator, which only surfaces steps, Pipeline also provides a built-in "Global Variable Reference." Like the Snippet Generator, it is also dynamically populated by plugins. Unlike the Snippet Generator however, the Global Variable Reference only contains documentation for variables provided by Pipeline or plugins, which are available for Pipelines.

The variables provided by default in Pipeline are:

env

Environment variables accessible from Scripted Pipeline, for example: env.PATH or env.BUILD_ID. Consult the built-in Global Variable Reference for a complete, and up to date, list of environment variables available in Pipeline.

params

Exposes all parameters defined for the Pipeline as a read-only Map, for example: params.MY_PARAM_NAME.

currentBuild

May be used to discover information about the currently executing Pipeline, with properties such as currentBuild.result, currentBuild.displayName, etc. Consult the built-in Global Variable Reference for a complete, and up to date, list of properties available on currentBuild.

Further Reading

This section merely scratches the surface of what can be done with Jenkins Pipeline, but should provide enough of a foundation for you to start experimenting with a test Jenkins instance.

In the next section, Automating projects with a Jenkinsfile, more Pipeline steps will be discussed along with patterns for implementing successful, real-world, Jenkins Pipelines.

Additional Resources


With the basics out of the way, continue on to Automating projects with a Jenkinsfile.


2. Additional plugins have been used to implement complex behaviors utilizing Freestyle Jobs such as the Copy Artifact, Parameterized Trigger, and Promoted Builds plugins
3. plugin:github-organization-folder[GitHub Organization Folder plugin