Jenkins-as-code: creating jobs from the command line during development

In the previous post in this series, I covered how to make a seed job for seed jobs via registration. In this post, I’ll cover my favorite development-time helper: running job scripts from the command line.

The problem

As our team was working on adopting this jenkins-as-code solution, I confess that local development was suboptimal. Because seed jobs pull from source control and then process job dsls, we’d have to build the dsl scripts, push to git, run the seed job, and run the built jobs, repeating as necessary till the job was working as desired. The workflow was slow enough to be annoying. Because, let’s face it, how often do you really get your automations right on the first try?

The solution

The solution ended up serendipitously coming from something Irina had built for the registration job… a way to “recover” that job on a Jenkins should something unfortunate happen to it or to easily put that job onto a new Jenkins without manually configuring it. The idea was to be able to do the following from the command line, given that a dsl script called recover_mother_job.groovy existed and whose purpose was to rebuild the mother-seed-job:

./gradlew rest -Dpattern=jobs/recover_mother_job.groovy -DbaseUrl=http://our-jenkins/ -Dusername=foo -Dpassword=bar

She found it in this bit of gold from the job-dsl folks.

Broader applicability

Eventually, it hit us that if we could use it for the mother seed job, why not for any job?

The workflow would look like this:

1. You’ve got a target Jenkins server, either local or remote
2. You’re editing your dsl script locally
3. You want to quickly build that on the target Jenkins server, without pushing to SCM yet
4. You run a single command in your terminal, then go check out the job in your target Jenkins
5. Once you’re satisfied, you push to SCM

So we copied-and-slightly-modified Matt Sheehan’s rest runner into our jenkins-automation project, and then added the rest task into our starter project, which we use for the basis of all our jenkins job repositories internally.

Try it out

Here’s what I want you to do right now:

1. Identify an existing local or remote Jenkins to run this against
2. Clone https://github.com/cfpb/jenkins-as-code-starter-project.git locally
3. cd jenkins-as-code-starter-project/
4. create a new file in the “jobs” directory called “simple.groovy” and give it these contents:

job('hello-world-dsl') {
   steps {
     shell("echo 'Hello World!'")
   }
}

Now, from that same directory, run:

./gradlew rest -DbaseUrl=http://localhost:8080 -Dpattern=jobs/simple.groovy

If you need to pass credentials, also pass

-Dusername=<username> -Dpassword=<password>

You should see output  like this:

:compileJava UP-TO-DATE
:compileGroovy
:processResources UP-TO-DATE
:classes
:rest

processing file: jenkins-as-code-starter-project/jobs/simple.groovy
Processing provided DSL script
hello-world-dsl - created

Go to your jenkins, and behold your new glorious ‘hello-world-dsl’ job.

Now, keep playing with that job in your edit. Or add more jobs to that file (or other files). Run that command over and over, and see your job updated in Jenkins. This is about as close to an “F5 Refresh” Jenkins job development workflow as I’ve found.

Warnings

A few caveats:

1. Don’t get in the bad habit of only running jobs from the terminal. Commit them to SCM and let seed jobs do the real work
2. If you’re running against a Jenkins with default CSRF crumb protection, this won’t work
3. The latest version of job-dsl-plugin has an “auto-generated dsl” feature to replace the need for manually adding new DSL APIs for plugins; this will flat-out break this method of job creation. For that reason, we have not adopted into our code any of the auto-generated bits and are sticking, for now, with either APIs that exist or with configure blocks, because that’s how important this tool has become (to me, anyway)

Next up: what about Workflow/Pipeline and Jenkinsfile?

The Workflow plugin, aka Pipelines, is standard as of Jenkins 2.0. It, also, is a groovy DSL approach to configuring automations.

In the next post, I’ll cover the differences between job-dsl and Pipelines, and how I currently see the two living together in the Jenkins ecosystem