Month: January 2013

Automatically Marking a Jenkins Builds as “Keep Forever”

This is an extension to my other post about choosing which build to deploy from a dynamic drop down. When setting up the deployment pipeline for this project we were working with a third party who’s process was to mark any builds that had been deployed to the test environment (i.e. manual QA as opposed to unit testing and automated functional tests) as “Keep Forever”. This is a built-in thing with Jenkins and fairly obviously tells Jenkins not to discard this build when it’s doing housekeeping on your build history. You can set this manually by clicking the (again obvious) “Keep Forever” button at the top of a build summary page:

Screen Shot 2013-01-31 at 13.12.48

For us, marking a build like this was really about it’s context within the deployment process and basically says “This build has passed unit tests, integration tests, automated functional tests, and has been selected by the QA team as a build to test” or slightly more succinctly: “This is a potential release candidate”. Whether you want to use the following Jenkins setup for this purpose or something else that you can think of is up to you (you might want to do the same but only mark a build when it’s deployed to the UAT environment for example), personally I wan’t to try and extent this idea of marking builds somehow and make it a bit more flexible (more on that a the bottom), but when we we’re automating and improving the CI /CD pipeline after the 3rd party left we decided to keep the processes the same for starters, so for now let’s run with this scenario for narrative purposes.

Programatically Marking a Build as Keep Forever

So first things first, as with most things Jenkins we need a plugin. This time it’s the Groovy Plugin which has a very important feature which is labeled “Execute system Groovy script”. To quote from the plugin page “The system groovy script, OTOH, runs inside the [Jenkins] master’s JVM. Thus it will have access to all the internal objects of [Jenkins], so you can use this to alter the state of [Jenkins].” This is essential for what we’re about to do and took some furious Googling to work out.

To mark a build programatically, install the plugin, add a build step and choose “Execute system Groovy script”, and then paste this code in:

import hudson.model.*

BUILD_JOB_NAME = "Build"

def getBuildVariable(String key) {
 def config = new HashMap()
 def thr = Thread.currentThread()
 def build = thr?.executable
 def buildMap = build.getBuildVariables()
 def envVarsMap = build.parent.builds[0].properties.get("envVars")
 config.putAll(envVarsMap)
 return config.get(key)
}

def getJobs() {
 def hi = hudson.model.Hudson.instance
 return hi.getItems(hudson.model.Job)
}

def getBuildJob() {
 def buildJob = null
 def jobs = getJobs()
 (jobs).each { job ->
 if (job.displayName == BUILD_JOB_NAME) {
 buildJob = job
 }
 }
 return buildJob
}

def getBuildRunToMark(hudson.model.Job job) {
 def runToMark = null
 (job.getBuilds()).each { run ->
 if (String.valueOf(run.number) == getBuildVariable("BUILD_NUMBER")) {
 runToMark = run
 }
 }
 return runToMark
}

def runToMark = getBuildRunToMark(getBuildJob())
runToMark.keepLog(true)

This is very similar to the code in the previous post, there should only be two things that you have to tweak:

  1. The same [BUILD_JOB_NAME = “Build”] line as before, just change “Build” to be whatever the name of your build job is.
  2. Near the bottom, the getBuildVariable(“BUILD_NUMBER”) method call, where BUILD_NUMBER is the name of the variable that holds the number of the build you want to mark, this is assuming that you’re using a similar parameterised build as I am with my drop-downs. If you’re not then you just need to find a way of passing in the build number as a variable, the EnvInject Plugin might help?

From memory there are two reasons why we need to run this as system Groovy script (I had to do A LOT of trial and error to get all of this drop-down build marking stuff to work so I can’t remember the exact reasons for everything):

  1. I think the top method getBuildVariable() has to run as system script otherwise you can’t get any access to the build variables, but I might be wrong, either way it’s a cut down version of the code I found on a post that saved my keyboard from a frustrated hammering.
  2. I also think the very last line, which actually marks the build, has to run as system.

Hopefully that should be all you need to mark a build programatically. As before, I couldn’t find a plugin to do this when I was experimenting but there might be one now so it might be worth a google?

Automatically Marking a Build

In our current scenario we want to mark the build once it’s been deployed to test. The simplest way to do this is to just create the build step above  in your test environment deployment job, that’s what I did at the beginning and it works fine.

I wanted to separate concerns a little bit though, so I created a separate job to do this which takes a string parameter called BUILD_NUMBER and does nothing except execute this script:

Screen Shot 2013-01-31 at 15.01.07

I then used the Parameterized Trigger Plugin at the end of the test environment deployment job to trigger the “Keep Build Forever” job on success:

Screen Shot 2013-01-31 at 15.03.07

Filtering Marked Builds for Deployment

I went a step further with this and decided that we should only be able to deploy builds to UAT and Production that have previously been deployed to test. This doesn’t mean that they’ve passed manual QA testing, but it filters the list down a bit and gave me an excuse to play around with Jenkins and Groovy a bit more (apparently I’m a glutton for punishment). Actually this is pretty simple, assuming your UAT deployment job already has a dynamic build number drop-down like the previous post.

If you do then create a new Scriptler script which I called “DynamicChoice_MarkedBuilds” and use this code:

import hudson.model.*

BUILD_JOB_NAME = "Build"

def getJobs() {
    def hi = Hudson.instance
    return hi.getItems(Job)
}

def getBuildJob() {
    def buildJob = null
    def jobs = getJobs()
    (jobs).each { job ->
        if (job.displayName == BUILD_JOB_NAME) {
            buildJob = job
        }
    }
    return buildJob
}

def getKeepForeverBuildNumbers(Job job) {
    def keepForeverBuildNumbers = []
    (job.getBuilds()).each { build ->
        if (build.isKeepLog()) {
            keepForeverBuildNumbers.add(build.number)
        }
    }
    return keepForeverBuildNumbers
}

def buildJob = getBuildJob()
    if (buildJob == null) {
    return ["No Suitable Builds Found"]
} else {
    return getKeepForeverBuildNumbers(buildJob)
}

Again BUILD_JOB_NAME should be the only thing you might have to change, and then just set you dynamic choice configuration to use this new script and you’re done!

Future Improvements

Given some more time I would have wanted to work out a simple way to share code between Scriptler scripts as a lot the code snippets in these posts have identical methods. I’d also like to find a clean way of running scripts stored in Scriptler as system scripts. Maybe these things are easy and I just completely missed it, either way I’ll come back to them when my workload allows and see what I can find.

As well as that, at the moment we’re sort of hacking a built-in Jenkins feature to use as a descriptor in the context of our deployment pipeline. This has a bad smell about it (I know some people hate the phrase “code smell” so you guys can read “feeling” instead if you want 😉 ) and it’s also really inflexible because you can only use it for one purpose, i.e. you can’t use it to mean “Potential Release Candidate” and “Business Sign-off Release” and “Latest Production Release”. It also means that if you put this step too far down the pipeline you could miss the chance to keep the build, for example if you’re using the Keep Forever tag to mean “Passed UAT” / “Production Release” and you’ve told Jenkins to only keep a certain number of builds or to delete builds over a certain age (or both) to save disk space, then you wait until the end of UAT until you mark your build but by that time the dev team has made 100 checkins triggering 100 new builds (which even a small team doing proper CI can do in a few days), your build has been tidied away. Ok so in that case you could just get some more disk space etc etc, but you get the idea.

I’m about to look into more flexible custom decoration of Jenkins build runs so if I find anything useful I’ll report back. In the meantime I hope some of this helps and you can make it work for your specific requirements.

Advertisements

Jenkins build number drop-down

As a bit of background, the basic setup I have is one build job in Jenkins that’s triggered by a GitHub checkin, and then a deployment job for each environment which in turn use Puppet to do the actual transfer of files to the respective servers and service restarts etc. What I wanted was a drop-down for each deployment job which has a list of available builds so that my entire deployment process is:

  1. Click “Build Now”
  2. Choose build to deploy from drop-down
  3. Click “Build”

(Build in this case just means running the deployment job, not actually re-building the artifacts etc)

As far as I could see there isn’t a Jenkins plugin for this and as a relative newcomer to Jenkins I think I’ll leave writing the plugin for a bit later (read “hopefully someone else who knows what they’re doing”). You will however need some plugins to do this, I can’t remember if any of these came out of the box but they’re not difficult to install:

Build number drop down:

So… step one: get a list of available builds. For this you’ll need the Dynamic Parameter Plugin, this does what it says (like most plugins) and lets you dynamically generate the options to pass as parameters to the build, it uses groovy which is lucky because that’s the way we’re going to get our list of builds.

If the plugin installed correctly you should be able to tick the “This build is parameterized” check-box at the top of the job configuration page and be able to see “Dynamic Choice Parameter (Scriptler)” in the drop-down. This is where Scriptler comes in. It offeres lots of groovy script management stuff but for our purposes the main thing is code reuse and tidiness. Because in my scenario I have a job for Dev, Test, UAT and Production I don’t really want a copy of my groovy script in each job, so I can save the script in Scriptler and reference it from all of the jobs.

You can find Scriptler in the menu on the left of the Jenkins homepage, it’s fairly self explanatory for this basic stuff so just add a new script: I set the name to be “DynamicChoice_AllBuilds” and used the same for the Id. As for the script itself you’ll need this:

import hudson.model.*

BUILD_JOB_NAME = "Build"

def getJobs() {
    def hi = Hudson.instance
    return hi.getItems(Job)
}

def getBuildJob() {
    def buildJob = null
    def jobs = getJobs()
    (jobs).each { job ->
        if (job.displayName == BUILD_JOB_NAME) {
            buildJob = job
        }
    }
    return buildJob
}

def getAllBuildNumbers(Job job) {
    def buildNumbers = []
    (job.getBuilds()).each { build ->
        buildNumbers.add(build.number)
    }
    return buildNumbers
}

def buildJob = getBuildJob()
return getAllBuildNumbers(buildJob)

Hopefully I’ve written it so that it’s decipherable but the only thing you’ll need to change is the BUILD_JOB_NAME = “Build” line, just set this to the display name of the job that you want the list of builds for, in my case it’s simply “Build”. Save the script and return to the job configuration page. In the dynamic choice parameter section you should now be able to select the new script in the “Script” drop-down.

The last thing we have to configure in this section is the “Name” attribute which has two functions: 1) It’s the label that is displayed next to the drop-down when you actually run the job. 2) It’s the name of the variable that we reference to use the build number. Therefore if you change it in this section remember to change it where it is referenced in the sections below. I chose “BUILD_NUMBER” just to keep it simple and so the dynamic parameter section looks like this:

Screen Shot 2013-01-17 at 14.02.24

If you save the job and start a build you should hopefully see something like this:

Screen Shot 2013-01-28 at 11.57.40

Doing something with it:

Ok, so we have a dynamic drop-down with all the available builds, but it doesn’t actually do anything, so now to Step 2: actually doing something useful with it!

In my setup I’m using the Copy Artifact plugin to (surprise surprise) copy all of the artifacts and archived stuff from my Build job into the workspace of my deployment job so that I can prepare it all for deployment (unzip files, rename things, flatten direcotries, etc etc) which in my case is copying a directory up to the Puppet Master.

This plugin has a lovely feature where you can specify the name of the job you want to copy from (the same name as BUILD_JOB_NAME is the grovvy script above) and also the name of a build parameter specifying which build to copy. So you would think that you just dump “BUILD_NUMBER” in the text box click save and you’re good to go! Well it turns out that this input expects a blob of XML and not just the number of the build! To get round this I used the EnvInject plugin to set another variable that contains the chosen build number wrapped in the appropriate XML (which I found from some debug output somewhere). I called my variable BUILD_NUMBER_XML so this is the variable value and what the top of my build section looks like :

BUILD_NUMBER_XML=<SpecificBuildSelector plugin="copyartifact@1.23">  <buildNumber>${BUILD_NUMBER}</buildNumber></SpecificBuildSelector>

Screen Shot 2013-01-28 at 12.01.33

And that’s it for the basic setup, you should now be able to create a Jenkins job with a dynamic build number drop-down!!

Optional:

The only plugin I haven’t mentioned is the Build Name Setter plugin. This is useful when you’re deploying builds from another job whether you’re using a drop-down or not, because you can set the name of the deployment run to include the number of the build you’re deploying, just tick the “Set Build Name” box and use something like:

Deploy Build #${ENV,var="BUILD_NUMBER"}

Screen Shot 2013-01-28 at 16.12.50