Pipeline Archive | Comquent GmbH, Continuous Quality in Software

In daily business you can discover a lot of pipelines in Jenkins created like that: You want to do X, look up in the Pipeline Steps Reference. Modify and fill in the parameters of the step, put it into your Jenkinsfile. You want to do Y and repeat the procedure.

A pipeline may look like that one:

timestamps {
  node {
    deleteDir()
    stage('Checkout') {
      checkout([$class: 'FancySCM', locations: [[credentialsId: 'tec_user', depthOption: 'infinity', ignoreExternalsOption: true, local: '.', remote: 'https://scm.whereever/fancyProject/.....']], workspaceUpdater: [$class: 'smartUpdater']])
      checkout([$class: 'SubversionSCM', locations: [[credentialsId: 'tec_user', depthOption: 'infinity', ignoreExternalsOption: true, local: 'anotherDir', remote: 'https://scm.whereever/fancyProject/...../anotherDir']], workspaceUpdater: [$class: 'smartUpdater']])
    }
    stage('Build Package') {
      sh 'mvn package'
    }
    stage('Deliver Artifact') {
      zip([zipFile: 'fancyProject-1.0.zip', dir: 'build', glob: "**/*.jar"])
      withCredentials([[\$class: 'UsernamePasswordMultiBinding', credentialsId: 'customerRepoTecUser', usernameVariable: 'USER', passwordVariable: 'PW']]) {
        sh "curl -v -u \\\${USER}:\\\${PW} --fail --upload-file fancyProject-1.0.zip https://internal.customer.com/repo/com/company/fancyProduct/artifacts"
      }
    }
  }
}

For the next pipeline or the next project it will be done in the same way. That’s why the amount of large and not well-arranged grows more and more in your company. Even different teams tend to have their individual style. Moreover creating pipelines is as expensive as creating your jobs as Free Style Jobs.

Refactoring in the jungle of pipelines on the build servers is troublesome: Where do we have to optimize? What does this pipeline exactly? Which side effects can spoil our efforts? Thus teams tend reluctantly to refactor.

When creating and maintaining pipelines in Jenkins in different projects and/or companies you are likely to observe the following:

• In a company only one SCM server and only one tool is used
• The same technical user is applied for most operations where required
• Only one artifact repository exists
• Common steps and code would be a help for the individual projects if they would exist

For this reasons the best practices for your pipeline steps are quite obvious:

• The checkout step can be wrapped very convieniently
• This can also be done for the upload of the artifacts whatever tool is used: Maven/Gradle/Ivy/curl/…
• The same for the download of the artifacts required for the deployment
• Helpers can be provided as common code like calling your config management system, cleaning up the workspaces after the deployment is done or has failed

Some practical useful examples:

wrappedCheckout(relativeURL, localDir, user = ‚jenkins‘)


wrappedUploadArtifact(package, repo = ‚snapshot‘, artifactDir = ‚standardDir‘)


wrappedGetArtifact(package, repo = ‚snapshot‘, artifactDir = ‚standardDir‘)


wrappedCheckout(relativeURL, localDir, user = ‚jenkins‘)

wrappedUploadArtifact(package, repo = ‚snapshot‘, artifactDir = ‚standardDir‘)

wrappedGetArtifact(package, repo = ‚snapshot‘, artifactDir = ‚standardDir‘)

wrappedGetArtifact(package, repo = ‚snapshot‘, artifactDir = ‚standardDir‘)

You can use the convention over configuration principle in your build process by forcing a fix directory for the artifacts built.

Filling configuration files required for deployment and other purposes by the values provided by a configuration management system can also easily be wrapped:

wrappedCMS(package, projectName, version)

Cleaning up of private data after a deployment for instance:

wrappedCleanup()

The code in the pipeline can be reduced enormously. The build and the deployment invocation is an exception. Even if they use the same technology the implentation tends to differ from project to project. That makes a common wrapping rather hard.

Once we have the wrappers implemented and conventions are respected the pipeline from the beginning can look like this:

timestamps {
  node {
    deleteDir()
    stage('Checkout') {
      wrappedCheckout('fancyProject/.....', '.')
      wrappedCheckout('fancyProject/...../anotherDir', 'anotherDir', 'tec_user')
    }
    stage('Build Package') {
      sh 'mvn package'
    }
    stage('Deliver Artifact') {
      wrappedUploadArtifact('com.company.fancyproject')
    }
  }
}

Put your common wrapper code to the same SCM at a particular place. It can be checked out by the pipeline itself followed by load(), or better: use the integrated Script Library in Jenkins. A practical discussion will follow.