Cloud ready Artifact Manager for AWS


Cloud ready Artifact Manager for AWS

Jenkins has historically provided multiple ways to save build products, otherwise known as artifacts.

Some plugins permit you to upload artifact files to repository managers like Artifactory, and Nexus Artifact Uploader, and other plugins send artifacts to remote shared filesystems like Publish Over FTP, Publish Over CIFS and Publish Over SSH. Jenkins itself stores artifact files in the Jenkins home filesystem. In 2012, CloudBees released the Fast Archiver Plugin, which optimizes the default artifact transmission but uses the same storage location.

Unfortunately, a number of these solutions are not cloud-ready, and it is awkward and difficult to use them with {CORE-CLOUD}. Some solutions, like S3 publisher are well suited for use in a cloud environment, but require special build steps within Pipelines.

CloudBees is developing a series of cloud-ready artifact manager plugins. The first of these is Artifact Manager on S3 plugin. This plugin permits you to archive artifacts in an S3 Bucket, where there is less need to be concerned about the disk space used by artifacts.

Easy to configure

To configure Artifact Manager on S3:

  1. Go to Manage Jenkins/Configure System.

  2. In the Artifact Managment for Builds section, select the Cloud Provider Amazon S3:

    cloud provider configured
  3. Return to Manage Jenkins/Amazon Web Services Configuration to configure your AWS credentials for access to the S3 Bucket.

  4. For your AWS credentials, use the IAM Profile configured for the Jenkins instance, or configure a regular key/secret AWS credential in Jenkins. Note that your AWS account must have permissions to access the S3 Bucket, and must be able to list, get, put, and delete objects in the S3 Bucket.

    configure credentials
  5. Save or apply the credentials configuration, and move on to configure your S3 bucket settings.

    bucket settings
  6. We recommend validating your configuration. If the validation succeeds, you’ve completed the configuration for Artifact Manager on S3.

    validation success
  7. For more details about Artifact Manager for S3, see the plugin documentation here: Artifact Manager on S3 plugin.

Uploading and downloading artifacts

The Artifact Manager on S3 plugin is compatible with both Pipeline and FreeStyle jobs. To archive, unarchive, stash or unstash, use the default Pipeline steps.

FreeStyle jobs

For FreeStyle jobs, use a post-build action of Archive the Artifacts to store your Artifacts into the S3 Bucket.

fsj step archive

To copy artifacts between projects:

  1. Make sure the Copy Artifact Plugin is installed.

  2. Use a build step to copy artifacts from the other project:

    copy artefacts

Pipeline jobs

For Pipeline jobs, use an archiveArtifacts step to archive artifacts into the S3 Bucket:

node() {
    //you build stuff
    //...
    stage('Archive') {
        archiveArtifacts "my-artifacts-pattern/*"
    }
}

To retrieve artifacts that were previously saved in the same build, use an unarchive step that retrieves the artifacts from S3 Bucket. Set the mapping parameter to a list of pairs of source-filename and destination-filename:

node() {
    //you build stuff
    //...
    stage('Unarchive') {
        unarchive mapping: ["my-artifacts-pattern/": '.']
    }
}

To save a set of files for use later in the same build (generally on another node/workspace) use a stash step to store those files on the S3 Bucket:

node() {
    //you build stuff
    //...
    stash name: 'stuff', includes: '*'
}

To retrieve files saved with a stash step, use an unstash step, which retrieves previously stashed files from the S3 Bucket and copies them to the local workspace:

node() {
    //you build stuff
    //...
    unstash 'stuff'
}

To copy artifacts between projects:

  1. Make sure the Copy Artifact Plugin is installed.

  2. Use a copyArtifacts step to copy artifacts from the other project:

    node(){
      //you build stuff
      //...
      copyArtifacts(projectName: 'downstream', selector: specific("${built.number}"))
    }

Security

Artifact Manager on S3 manages security using Jenkins permissions. This means that unless users or jobs have permission to read the job in Jenkins, the user or job cannot retrieve the download URL.

Download URLs are temporary URLs linked to the S3 Bucket, with a duration of one hour. Once that hour has expired, you’ll need to request a new temporary URL to download the artifact.

Agents use HTTPS (of the form https://my-bucket.s3.xx-xxxx-x.amazonaws.com/*) and temporary URLs to archive, unarchive, stash, unstash and copy artifacts. Agents do not have access to either the AWS credentials or the whole S3 Bucket, and are limited to get and put operations.

Performance

A major distinction between the Artifact Manager for S3 plugin and other plugins is in the load on the master and the responsiveness of the master-agent network connection. Every upload/download action is executed by the agent, which means that the master spends only the time necessary to generate the temporary URL: the remainder of the time is allocated to the agent.

The performance tests detailed below compare the CloudBees Fast Archiving Plugin and the Artifact Manager on S3 plugin.

Performance tests were executed in a Jenkins 2.121 environment running on Amazon EC2, with JENKINS_HOME configured on an EBS volume. Three different kinds of tests were executed from the GitHub repository at Performance Test, with samples taken after the tests had been running for one hour:

  • Archive/Unarchive big files: Store a 1GB file and restore it from the Artifact Manager System.

  • Archive/Unarchive small files: Store 100 small files and restore them from the Artifact Manager System. Small files are approximately 10 bytes in size, with 100 files stored and times averaged

  • Stash/Unstash on a pipeline: Execute stash and unstash steps. The Fast Archive Plugin stash/unstash operations used the default stash/unstash implementation.

As can be seen from the results, the Artifact Manager on S3 Plugin provides a measurable performance improvement on both big and small files, with the improvement measured in minutes for big files and in seconds for small files.

Artifact Manager on S3 plugin performance

Plugin link: Artifact Manager on S3

 Big Files
Time in Milliseconds Archive Unarchive

Max

48,578.00

29,899.00

Min

17,773.00

20,388.00

Avg

20,969.49

22,670.67

S3 archive big file 00
 Small Files
Time in Milliseconds Archive Unarchive

Max

2,974.00

805.00

Min

752.00

104.00

Avg

1,171.65

200.76

S3 archive small files 00
Stash
Time in Milliseconds Archive Unarchive

Max

14,902.00

9,477.00

Min

1,256.00

709.00

Avg

1,977.49

1,588.96

S3 archive stash 00

CloudBees Fast Archiving Plugin performance

Big files
Time in Milliseconds Archive Unarchive

Max

358,988.00

105,615.00

Min

110,068.00

93,193.00

Avg

277,642.22

95,771.78

fast archive big file 00
 Small Files
Time in Milliseconds Archive Unarchive

Max

1,603.00

109.00

Min

491.00

10.00

Avg

953.26

22.55

fast archive small files 00
Stash
Time in Milliseconds Archive Unarchive

Max

1,914.00

3,050.00

Min

561.00

267.00

Avg

1,075.53

976.28

fast archive stash 00