On-Premise Executors

On-Premise Executors is a feature of DEV@cloud that allows customers to connect their on-premise machines (slaves) from Linux, Mac OSX or Windows using the Jenkins CLI.

There are several reasons why you might want to provide your own executor (in Jenkins terminology a slave can have multiple executors):

  • You require access to internal network resources such as private databases or source control systems (and Jenkins VPN service is not appropriate)

  • You need a specific operating system or libraries which aren’t provided on DEV@Cloud

  • You need to run performance tests on tuned hardware/software stacks

Getting Started

You need a DEV@cloud Enterprise subscription to access this feature. When subscribing or upgrading to Enterprise an input box is shown which lets the number of executors be added. This can be modified at a later date if necessary (downgrade is initiated through a support ticket today).

Next, the jenkins-cli.jar is needed. A CLI JAR of at least version 1.460 is needed; the above link currently offers the client from Jenkins 1.509.x. Also create a workspace directory on the local machine for the jobs to be executed in.

Next, you will need an SSH keypair for authentication. It is recommended that one is generated specifically for this purpose, especially if the slave will be connected from a server. If this is going to run as a system service a key without a password is required. The public key needs to be added to your user account in the CloudBees console.

We recommend that you use a system user (not one associated with a particular person) when connecting your On Premise Slaves. If you connect slaves using keys which are associated with a specific person in your company, they will stop working when that person leaves your company and loses access to your account.

Connecting the Slave

Below is the basic usage of the cli.

$ java -jar jenkins-cli.jar -s https://<myaccount>.ci.cloudbees.com -i ~/.ssh/id_rsa on-premise-executor -help:
"-help" is not a valid option
java -jar jenkins-cli.jar on-premise-executor args...
Connects a on-premise executor
 -description VAL : Description to be put on the slave
 -executors N : Number of executors
 -fsroot VAL : Directory where Jenkins places files
 -labels VAL : Whitespace-separated list of labels to be assigned for this
slave. Multiple options are allowed.
 -name VAL : Name of the slave. If unspecified, one is generated for you
 -install VAL : Install as a service; full path to ssh key is required.
 -persistent : This makes the node persistent which allows NodeDescriptors to be set.

-s specifies the address of your jenkins instance. -i is optional if the key you want to use is /.ssh/id_rsa or /.ssh/id_dsa. For other key-files a path will need to be specified. Note the SSH key must be an openssh key.

A full command is something like:

$ java -jar jenkins-cli.jar -s https://<myaccount>.ci.cloudbees.com on-premise-executor -fsroot /path/to/root -labels osx -name osx-server-2

An important piece is the label. Labels allow you to connect multiple slaves, and have jobs choose any slave that matches the label.

Installing the Slave as a Service

To install it as a service simply add -install <key> to the end of the command and it will use the configuration provided for the executor that is started by the slave.

$ java -jar jenkins-cli.jar -s https://<myaccount>.ci.cloudbees.com on-premise-executor -fsroot /path/to/root -labels osx -name osx-server-2 -install /path/to/mykey/keyfile

The recommended method to making sure that everything will work as expected is to run the command with -install first to make sure it connects.

Running the slave with systemd on linux

As most linux distros are standardising on systemd for daemonisation and process management - here is a script you can use to base your installation on. Put this in a file "/usr/lib/systemd/system/ope.service" and then you can restart it via: systemctl restart ope, systemctl status ope, etc.

Description=CloudBeees On Prem executor

ExecStart=/opt/java6/bin/java -jar /usr/share/jenkins-cli.jar -s https://cloudbees.ci.cloudbees.com on-premise-executor -fsroot /home/ec2-user/workspace -labels docker -nam\
e docker-builder-1


## To install:
## ensure all names and paths above are correct for you:
## install -D -m 644 ope.service /usr/lib/systemd/system/ope.service
## systemctl daemon-reload
## systemctl start ope
## systemctl enable ope

For this to work - you need to be using systemd, a server with ec2-user account, and also have setup your ssh key for correct access. You should run the command (java -jar …​) for the slave agent at least once in a shell OF THE USER YOU INTEND TO RUN THE SERVICE AS (ie ec2-user in this case) to ensure you have all the details right - and the master host is added to authorized_keys.

IMPORTANT - running as consistent user

This example runs as ec2-user. Note that if you change users at any point - you will need to delete the workspace directory - and start again, due to file permissions. You need to run the slave command, at all times, as the same user. Do not change it without starting again with a fresh workspace. You can also run as root (eg if you are running a docker slave - in which case you don’t need any additional user).


For installing a mac slave as on-premise executor you will need to create a launchctl plist. As an example about how to do this you can create a file called com.jenkins.ci.plist.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">

Move this file to /Library/LaunchDaemons and change the permission so it can be correctly handled by the root user.

sudo chown root com.jenkins.ci.plist
sudo chmod 644 com.jenkins.ci.plist

To install the service use:

launchctl load /Library/LaunchDaemons/com.jenkins.ci.plist
launchctl start com.jenkins.ci

In case you want to list the service and stop it you can then use:

launchctl list | grep jenkins
launchctl stop com.jenkins.ci

To troubleshooting you should check the output of /path/to/stdout.log and /path/to/error.log.

To uninstall the service run

launchctl list

and find the service. Then run

launchctl remove <service name # example - org.jenkins-ci.slave.a4ef276f>

This uninstalls the service and stops the slave.

  • Download and install Oracle JDK

  • Create a folder c:

  • Download jenkins-cli.jar from your Jenkins master in "c:"

    • See https://[myaccount].ci.cloudbees.com/cli/

    • Download https://[myaccount].ci.cloudbees.com/jnlpJars/jenkins-cli.jar

  • Copy your SSH key "my_rsa_key" in c: To generate a SSH key on Windows, you can use ssh-keygen bundled in "http://msysgit.github.io/[Git for Windows]"

  • Test the OPE connection running the command (don’t forget "-labels windows"):

    java -jar c:/jenkins/jenkins-cli.jar -s https://cyrille-leclerc.ci.cloudbees.com -i c:/jenkins/my_rsa_key on-premise-executor -fsroot c:/jenkins/ope-fs-root -labels windows -name my-windows-ope
  • Standard output should look like:

    Slave.jar version: 2.33
    This is a Windows slave
    Slave successfully connected and online
  • Verify that the Windows OPE appears on your Jenkins master dashboard


  • Create and run a test Jenkins Free Style Job with:

    • Name: "windows-job"

    • Restrict where this project can be run/Label Expression: "windows"

    • Add Build Step / Execute Windows batch command / Command : "dir"

  • The Job output should look like:

    Started by user Cyrille Le clerc
    Building remotely on my-windows-ope in workspace c:/jenkins/ope-fs-root/workspace/windows-job
    [windows-job] $ cmd /c call C:\Users\ADMINI~1\AppData\Local\Temp\2\hudson7149717920587053636.bat
    Volume in drive C has no label.
    Volume Serial Number is 5E71-6BA3
    Directory of c:\jenkins\ope-fs-root\workspace\windows-job
    04/14/2014 03:15 PM .
    04/14/2014 03:15 PM ..
    0 File(s) 0 bytes
    2 Dir(s) 12,367,929,344 bytes free
    c:\jenkins\ope-fs-root\workspace\windows-job>exit 0
    Finished: SUCCESS
  • Stop the jenkins-cli script and register the Jenkins OPE as a Windows service

    java -jar c:/jenkins/jenkins-cli.jar -i c:/jenkins/my_rsa_key -s https://<myaccount>.ci.cloudbees.com on-premise-executor -fsroot c:/jenkins/ope-fs-root -labels windows -name my-windows-ope -executors 1 -install c:/jenkins/id_rsa
  • The console output should look like Installing On-Premise Executors as a service. This can take up to 2 minutes. [ope-fs-root] $ c:-fs-root-slave.exe start Installed On-Premise Executors as a service. Note it can take a few minutes to start. Successfully started

  • In the Microsoft Management Console, you can see a "Jenkins Slave" service


  • To start/stop/uninstall the Services control panel can be used.

OPE behind a proxy

In case your OPE is running behind a proxy you will need then to add the following Java arguments to the jenkins-cli.jar.


The full jenkins-cli command should look similar to the one below.

$ java -Dhttps.proxyHost=$PROXY_HOST -Dhttps.proxyPort=$PROXY_PORT -jar jenkins-cli.jar -s https://<myaccount>.ci.cloudbees.com on-premise-executor -fsroot /path/to/root -labels osx -name osx-server-2

Configuring a Job to Run on the Provided Slave

Currently to make a job run on the provided slave, you need to select "Restrict where this project can be run" in the job configuration page, and input your label you want the job to run on.

Jobs configured to run on provided slaves will not run on the CloudBees elastic slaves, and jobs with no label set, will not run on provided slaves.

Configuring the Slave Environment (linux, OSX only)

Because of how the masters are configured at the moment, to use some tools like Maven, you may need to provide the tool in the location Jenkins expects it to be, either directly, or with a symlink. Below are the paths that are used for the JDK and Maven. You can inspect /opt on DEV@cloud slaves with a job to find other provided tools if needed.

Java parent path: /opt/jdk

Java versions under the parent path:

  • ibmjdk6

  • jdk1.5.0_22

  • jdk1.6.0_20

  • jdk1.6.0_24

  • dk1.6.0_30

  • jdk1.7.0

  • jdk1.7.0_04

Maven parent path: /opt/maven

Maven versions under the parent:

  • apache-maven-2.0.11

  • apache-maven-2.2.1

  • apache-maven-3.0.1

  • apache-maven-3.0.3

  • apache-maven-3.0.4

Known Issues

  • You must provide an absolute path for the -fsroot parameter.

  • If you receive an error such as "FATAL: Null value not allowed as an environment variable: M2_HOME", make sure that you have configured a valid installer for the Maven you are using. You can do this by navigating to /configure (Manage Jenkins → Configure System) and click "Install Automatically" underneath the Maven job you are using. You can then choose "Install from Apache" and select the appropriate version.


All users on your account are allowed to connect slaves using their registered public/private keypair.

All communication to and from your slave is secured by HTTPS.

Building Docker Images on an OPE

An OPE allows you to run what you want - this includes building Docker images. There is a quick guide here on the specifics of Docker.

Give Us Your Feedback

If you have tried (or not tried) the feature - fill in a short poll for us (takes about 30 secs).

Subscribing to On Premise Executors

To use the On Premise Executors feature you need to be subscribed to the Enterprise tier - the feature is not available in the PROFESSIONAL or STARTED accounts.

To configure the feature in the CloudBees account click on the gear icon in the upper left hand corner.


Choose "Build Executors" in the Account Tools panel.


Then enter the number of executors and click update.



I get an Exception: "Unable to connect On-Premise Executor. …​"

I get an exception "Unable to connect On-Premise Executor. There are 0 executors already connected and you tried to connect 2 but this account has a limit of 1 executors. Please purchase more if needed.”

By default, the OPE connection script try to allocate 2 executors. If you get a message …​ you tried to connect 2 but this account has a limit of 1 executors. Please purchase more if needed.”, then you must either:

  • Subscribe to more executors (See " Subscribing to On Premise Executors" above)

  • Or specify in the jenkins-cli.jar command line to use 1 executor adding the parameter "-executors 1"\{\{/code}}