Jenkins-as-code: comparing job-dsl and Pipelines

In the previous post in this series, I covered my favorite development-time helper: running job scripts from the command line. In this post, I’ll cover the differences between job-dsl and Pipelines, and how I currently see the two living together in the Jenkins ecosystem.

job-dsl refresher

If you’re coming into this post directly, without reading the preceding articles in the series, I strongly encourage you to start at the start and then come back. For the rest of you, a quick refresher:

job-dsl is a way of creating Jenkins jobs with code instead of the GUI. Here’s a very simple example:

When processed via a Jenkins seed job, this code will turn into the familiar Jenkins jobs you know and love.

What are Pipelines?

Jenkins Pipelines are a huge topic, way more than I am going to cover in a single blog post. So I’m going to give the 30,000 foot view, leave a whole bunch of stuff out, and I hope whet your appetite for learning more. For the impatient, skip these next few paragraphs and head straight for the Jenkins Pipeline tutorial.

At its simplest, a Pipeline is very job-dsl-ish: it lets you describe operations in code. But you have to shift your mindset quite a bit from the Freestyle jobs you know well. When configuring a Freestyle job, you have the vast array of Jenkins plugins at your fingertips in the GUI (and job-dsl) — SCM management, Build Triggers, Build Steps, Post-build actions.

Top-level view of a Freestyle job configuration
Top-level view of a Freestyle job configuration

But with Pipelines, it’s different. You get Build Triggers and your Pipeline definition. But what about the other stuff, you ask? This is where the mindshift comes in. Those things are no longer configured at the job level, but at the Pipeline level. And plugins are not automatically supported in Pipelines, so currently you get a subset of all available Jenkins functionality in Pipelines.

Top-level view of a Pipeline job configuration

Thus in practice, that means things like git repos, build steps, email, test recording/reporting, publishers, etc are all done — in text — in the Pipeline definition.

Here’s an example that ships with Jenkins:

Example Pipeline script
Example Pipeline script

This is kinda sorta like…

Probably confusing, I know. Let’s try to think of it this way: If you’ve read Jez Humble and David Farley’s Continuous Delivery, or have otherwise implemented build/deploy pipelines in Jenkins for years, you already have a solid conceptual sense of pipelines. It’s just that in Jenkins world until rather recently, you probably did this in one of two ways:

  1. Upstream / downstream jobs (possibly in combination with the Delivery Pipeline plugin); or
  2. Via the Build Flow plugin, with independent jobs being orchestrated via a simple text DSL

Either way, you probably had independent Freestyle jobs tied together somehow to make a pipeline.

Well, Jenkins Pipelines still certainly enable you to do that — and I’ll talk specifically about option #2 momentarily — but the big change here is that Pipelines enable you to do all that orchestration in a single job.

Whereas before you might have separate BuildJob, TestJob, and DeployJob tied together in one of the manners above, with Pipelines, you can do all that in a single job, using the concept of Stages to separate the discrete steps of the pipeline.

Cool! What else do I get with this?

Even with the simplest of Pipelines, you get:

  • Durability, to survive Jenkins restarts
  • Pausing for user input
  • Parallelism built in
  • Pipeline snippet generator to help you build pipelines
  • Nice visualization of your Pipelines right in the UI

But wait, there’s more

You also get Travis CI style CI via a Jenkinsfile, and Multibranch pipelines which enable pipeline configuration for different branches of a repo. From the Jenkins Pipeline tutorial: “In a multibranch pipeline configuration, Jenkins automatically discovers, manages, and executes jobs for multiple source repositories and branches.”

In addition, Jenkins Blue Ocean is shaping up to have beautiful visualizations for Pipeline jobs

Using Pipelines right now, today

Let’s say you still really like loosely coupled jobs that can run independently or together as part of a Pipeline (caveat: I have encouraged this approach for years, and it’s why I’ve long used Build Flow plugin over Build/Delivery Pipeline plugin). Right now, today, you can replace your Build Flow jobs with Pipelines.

In fact, you should do this. From the Build Flow wiki page:

A simple Pipeline script to build other jobs looks like this:

Overall, pretty similar to BuildFlow. And don’t worry, you get all the parallelism, retry, etc that you’re used to.

I am tremendously grateful for the people who’ve built and maintained the Build Flow plugin over the years. For me, it’s been a cornerstone of continuous delivery, enabling independent, reusable, loosely coupled jobs. Build Flow developers: Thank you!

But, it’s time to move on, Pipeline will replace Build Flow.

Do Pipelines replace job-dsl?

Now, to the final question: should Pipelines replace job-dsl?

I believe that’s the wrong question.

job-dsl will be complementary to Pipelines. Even if I were to stop using Freestyle jobs entirely, and build nothing but Pipelines, I’d still use job-dsl to create those jobs. In fact, if you go back to my initial post where I described the problems we were trying to solve when we adopted job-dsl, none of them are solved by Pipelines. In that respect, Pipeline is just another type of job.

A friggin’ awesome type of job, no doubt. I am incredibly excited about Pipelines and look forward to using them more. And here’s how I’ll be building those jobs, as job-dsl has full support for Pipelines:

So what is the right question?

If asking whether Pipelines replace job-dsl is the wrong question, what’s the right question?

I believe it’s:

  1. When should Pipeline replace Freestyle jobs?
  2. When should Pipeline — via Jenkinsfile — replace creating jobs directly in Jenkins (via GUI or job-dsl)?

I’m going to mostly cop out of answering those questions right now, as, for me, the answers are still evolving as I work more with Pipelines.

My initial gut reactions are:

  1. replace Build Flows, as mentioned above
  2. replace Freestyle jobs when there’s no value in running that set of jobs independently
  3. replace Freestyle jobs when you’d benefit from what Multibranch provides
  4. replace Jenkins-built jobs with Jenkinsfile when you have a TravisCI-style workflow that you want to use in Jenkins instead and you’ve seriously considered the safety and security implications for your organization (my thoughts are in the very earliest stages here)

Next up: encouraging adoption of Jenkins-as-code among teams

In the final planned post in this Jenkins-as-code series, I will address how we encouraged adoption of this approach amongst our development teams. I’ll cover where we succeeded, where we stumbled, and the work I think we still have to do.

Remaining more present by controlling information overload

A while back, I was made well aware of my, shall we say, un-presence. I’d be physically present with my family, but distracted the point of absence. Usually, it was due to a device stealing my time from them. The phone in my pocket was becoming a time-and-attention thief.

A Facebook message. A Twitter reply. An email.

I am still not great at this — my kids would probably say I’m terrible — but I’ve come a long way and thought I’d share some of my tactics for helping reduce social media and email overload.

The point of this is to show how I use friction and delay to help keep myself from being pulled out of the present moment when it comes to notifications that would otherwise constantly distract me.

In short:

  1. Using web apps, not native apps
  2. Turning off most email notifications
  3. Rolling up the email notifications that remain

Using web apps, not native apps

Heresy: when I can, I use web apps instead of native mobile apps. This is first and foremost a matter of principle: I believe in the browser as a platform and the ability to produce rich, accessible applications in standard technologies such as HTML, CSS, and JavaScript. So, in support of this principle, I will opt to use a web application over a native device application unless the native application offers compelling features that I want.

And here’s a feature I don’t want: device notifications. If I get a Facebook comment or twitter mention, I don’t care to know about it right away.

By using web apps, I am forced to go to there rather than having that information pushed to me. In other words, I use that little bit of friction to my advantage.

Disabling notifications

For nearly all social media — Twitter, Facebook, etc — I disable most email notifications. Facebook PMs and likes and comments… nope. Twitter follows… nope. I force myself to go that information; I don’t want it pushing to me.

The combination of using web apps, and disabling notifications, means that I am often hours, if not days, behind on interactions on these platforms, by choice.

Rolling up the email notifications that remain

I still get a fair amount of non-personal (i.e. person-to-person) email — newsletters, LinkedIn, GitHub, meetups, etc.

To stop those from distracting me as soon as I receive them, I use a free service called Unroll.me to manage that email. This service watches my inbox, and when it detects something that looks automated, it asks whether I want to unsubscribe, roll it up into a daily email, or keep it in the inbox.

After almost a year of using this service, I’ve unsubscribed from a lot of things I don’t care about, and rolled up many disparate communications into a daily email. I cannot recommend this service highly enough. (Caveat: it has access to all your email.)

Conclusion

How we interact with the digital world is a highly personal affair. The choices I’ve described above — to ultimately favor the present moment vs interacting with people digitally at all hours of day and night except when I explicitly choose to do so — clearly come with tradeoffs. I am slower to respond to people. I am probably not servicing certain relationships as well as I should. When I do interact on those services, it’s slightly less convenient.

But the real-time web — powerful and important as it may be — can also be a bug and not a feature, especially when it comes to interactions that, perhaps, aren’t urgent. In these cases, I give myself permission not to participate in real time.

The way I think about it is that these choices help me to favor the people I am with, right here, right now. And I cannot do that effectively if there’s a constant trickle of requests for attention coming from across the ether.

 

 

 

Jenkins-as-code: creating jobs from the command line during development

In the previous post in this series, I covered how to make a seed job for seed jobs via registration. In this post, I’ll cover my favorite development-time helper: running job scripts from the command line.

The problem

As our team was working on adopting this jenkins-as-code solution, I confess that local development was suboptimal. Because seed jobs pull from source control and then process job dsls, we’d have to build the dsl scripts, push to git, run the seed job, and run the built jobs, repeating as necessary till the job was working as desired. The workflow was slow enough to be annoying. Because, let’s face it, how often do you really get your automations right on the first try?

The solution

The solution ended up serendipitously coming from something Irina had built for the registration job… a way to “recover” that job on a Jenkins should something unfortunate happen to it or to easily put that job onto a new Jenkins without manually configuring it. The idea was to be able to do the following from the command line, given that a dsl script called recover_mother_job.groovy existed and whose purpose was to rebuild the mother-seed-job:

./gradlew rest -Dpattern=jobs/recover_mother_job.groovy -DbaseUrl=http://our-jenkins/ -Dusername=foo -Dpassword=bar

She found it in this bit of gold from the job-dsl folks.

Broader applicability

Eventually, it hit us that if we could use it for the mother seed job, why not for any job?

The workflow would look like this:

  1. You’ve got a target Jenkins server, either local or remote
  2. You’re editing your dsl script locally
  3. You want to quickly build that on the target Jenkins server, without pushing to SCM yet
  4. You run a single command in your terminal, then go check out the job in your target Jenkins
  5. Once you’re satisfied, you push to SCM

So we copied-and-slightly-modified Matt Sheehan’s rest runner into our jenkins-automation project, and then added the rest task into our starter project, which we use for the basis of all our jenkins job repositories internally.

Try it out

Here’s what I want you to do right now:

  1. Identify an existing local or remote Jenkins to run this against
  2. Clone https://github.com/cfpb/jenkins-as-code-starter-project.git locally
  3. cd jenkins-as-code-starter-project/
  4. create a new file in the “jobs” directory called “simple.groovy” and give it these contents:
job('hello-world-dsl') {
   steps {
     shell("echo 'Hello World!'")
   }
}

Now, from that same directory, run:

./gradlew rest -DbaseUrl=http://localhost:8080 -Dpattern=simple.groovy

If you need to pass credentials, also pass

-Dusername=<username> -Dpassword=<password>

You should see output  like this:

:compileJava UP-TO-DATE
:compileGroovy
:processResources UP-TO-DATE
:classes
:rest

processing file: jenkins-as-code-starter-project/jobs/simple.groovy
Processing provided DSL script
hello-world-dsl - created

Go to your jenkins, and behold your new glorious ‘hello-world-dsl’ job.

Now, keep playing with that job in your edit. Or add more jobs to that file (or other files). Run that command over and over, and see your job updated in Jenkins. This is about as close to an “F5 Refresh” Jenkins job development workflow as I’ve found.

Warnings

A few caveats:

  1. Don’t get in the bad habit of only running jobs from the terminal. Commit them to SCM and let seed jobs do the real work
  2. If you’re running against a Jenkins with default CSRF crumb protection, this won’t work
  3. The latest version of job-dsl-plugin has an “auto-generated dsl” feature to replace the need for manually adding new DSL APIs for plugins; this will flat-out break this method of job creation. For that reason, we have not adopted into our code any of the auto-generated bits and are sticking, for now, with either APIs that exist or with configure blocks, because that’s how important this tool has become (to me, anyway)

Next up: what about Workflow/Pipeline and Jenkinsfile?

The Workflow plugin, aka Pipelines, is standard as of Jenkins 2.0. It, also, is a groovy DSL approach to configuring automations.

In the next post, I’ll cover the differences between job-dsl and Pipelines, and how I currently see the two living together in the Jenkins ecosystem

Jenkins-as-code: registering jobs for automatic seed job creation

In the previous post in this series, I covered custom builders we’ve added on top of job-dsl-plugin. In this post, I’ll cover an innovative solution that’s made working with all these Jenkins-as-code repositories much easier. It centers around a seed job for seed jobs.

What are seed jobs again?

The job-dsl-plugin tutorial covers them thoroughly. In short, a seed job processes your job-dsl scripts and thus creates the Jenkins jobs from those scripts using the “Process Job DSLs” build step. You generally configure your seed jobs to listen for changes to the source code repo where you keep your jobs, such that your jobs are automatically rebuilt whenever changes occur.

The problem

I’ve mentioned “internal Jenkins-as-code repositories” several times throughout past posts in this series. Rather than have all our Jenkins jobs in a single monolithic repository, we believe that projects / project teams should be able to keep their jobs in repo locations that make the most sense to them, maybe even sitting right inside their main project repo (think: .travis.yml or Jenkinsfile, but on steroids). That does impose several burdens, however:

  1. finding those Jenkins job repositories
  2. manually configuring their seed job in all the Jenkinses in which those project’s jobs should run

In an organization of any size, discovery is a problem, and we anticipate dozens (hundreds?) of Jenkins job repositories scattered across the organization. And the whole point of jenkins-as-code is to get out of the pointy-clicky job creation business, so it violates the principle of the initiative to require manual seed job configuration.

Just for now, imagine that you have 20 projects. They all have their own jobs. You’ve tried like hell to de-snowflake them so that they’re practically automatable or templatable, but you’re still manually pointing-and-clicking all those jobs. Updates are a hassle (even with scriptler). So you go the job-dsl route as described in previous posts, you script your jobs, but now you still have to create the seed jobs for all those projects. Or you have a gigantic single seed job that listens to all those repos (a totally legit approach). Either way, you don’t want to do much pointing and clicking to create seed jobs.

We solve that via a Registration pattern.

Registering job repositories

We haven’t yet open sourced this code (we will), so I’ll provide all the meaningful bits in this post for now.

The idea is simple: a single configuration file that teams can add their Jobs repository to via pull request. Once merged, a “Mother Seed Job” runs and then creates a seed job for their jobs. It’s a seed job for seed jobs.

Even after reading the problem statement above, you might still be thinking: you really need that? Like, can’t you just manually configured some seed jobs and move on? I thought the same thing. And in your org, maybe you don’t! It has turned out to really benefit us, though. More on that doubt, and trust, at the end of this post because it’s a good story.

Configuration file

Let’s start with the configuration file we use. It’s kept in an internal git repo, and additions are made via pull request. It looks something like this:

We have multiple hosts (the above are fake), and we split those into traditional dev/test/prod/etc environments. Some repos will only ever have automations that live in one host/environment combination, and some with multiple, so we account for those possibilities in this configuration.

The idea is simple: you put your repo with job-dsl jobs in here, based on what Jenkinses you want your jobs to be created in.

“all” is a special case that means what you’d expect: create this repo’s jobs in all the Jenkinses for this host.

Mother seed job

We then have the “Mother Seed Job” configured in all our Jenkinses. Here’s where it gets trippy: the mother seed job is itself a straightforward Freestyle job that runs a job-dsl script.The mother job pulls from 2 git repos: the jenkins-automation repo I talked about last time, and an internal git repo with configuration like above (and, currently, this seed job script below, with some other stuff). All the interesting bits are in that job-dsl script.

I am reproducing it here, in full:

This then creates seed jobs for each repo configured in the config file above, in the appropriate Jenkinses.

What’s the global variables stuff?

If you actually read that closely, you also noticed some global variables business going on. In short, we also use this seed job to generate a “globals.properties” file that we put on each Jenkins, so that we don’t have to manually manage global vars in Jenkins. That’s just another config file, which looks something like this:

 

How does a Jenkins identify itself?

We have these config files that have a host/environment matrix, and a mother seed job that creates seed jobs based on configuration. So how do each of the Jenkinses know which jobs are theirs?

Two global variables that we manage outside of jenkins-as-code (via configuration management or manual global var config in Jenkins):

  • JAC_HOST
  • JAC_ENVIRONMENT

Those 2 global variables exist in all of our jenkinses, and they are unique.

For example, for our Jenkinses in AWS, their JAC_HOST value will be “aws”. Then, because we only have one Jenkins master per environment, the JAC_ENVIRONMENT will be “dev”, “test”, “stage”, or “prod”.

The JAC_HOST and JAC_ENVIRONMENT values correlate exactly to the keys (aws, heroku, azure… dev, test, prod) in those config files.

There’s nothing special about these host or environment names; all that matters is that those two vars configured in Jenkins align with their corresponding names in the config files.

(disclaimer: I’m using aws, heroku, and azure as examples. I cannot publicly speak to our infrastructure)

Short story about trust

Irina M is ultimately responsible for leading design and implementation of the solution I’ve been writing about in this series. At one point, when work was winding down, she said she was working on a “mother seed job” (her words) to make it easier for teams to get their job repos in our Jenkinses. I told her then, more than once, that I didn’t really understand the problem.

She patiently explained it, but I still didn’t grok it as an actual-actual problem. But here’s the thing: as a team lead, people join your team and they’re smarter than you and you have to trust them. Sure, you can be skeptical, but you have to be confident enough to let go of control. For a lot of people, trust, and letting go, is not easy. It’s not for me.

This is one in a countless number of reminders of how often that trust pays off. Had I said “No, it’s not worth it, please spend your time on other things” we would not have had this solution, Irina would’ve rightfully been disengaged, and on and on.

The enlightened among you are thinking, “Duh, dumbass”. If you’re not thinking that, I implore you: put your ego down and trust your team.

Next up: my favorite development helper

As our team was working on adopting this jenkins-as-code solution, I confess that local development was kind of a pain in the ass. Because seed jobs pull from source control and then process job dsls, we’d have to build the dsl scripts, push to git, run the seed job, and run the built jobs, repeating as necessary till the job was working as desired. The workflow was juuust slow enough to be annoying.

We solve that by adapting a command line utility, a tiny nugget of gold, unassumingly linked in the job-dsl-plugin wiki. And now, local development of job-dsl scripts is, for me, a pleasure. More on that in the next post.

Jenkins-as-code: creating reusable builders

In the first post in this series, I covered the problems we were having with job creation and maintenance and a very high level look at our solution. In the second post, I covered the job-dsl-plugin which underpins our solution.

In this post, I’ll dig into the Builders we’ve added on top of job-dsl-plugin, which do the following for us

  • add sensible defaults for all jobs
  • make it easy to do the right thing for potentially complicated jobs or configurations
  • provide utilities for common decision-making and code blocks
  • establish a Builders pattern for use internally even if a job type isn’t all that reusable across our projects

Why am I writing about these?

While I think it’d be great if others were to take advantage of our jenkins-automation repo, that’s not my motivation for writing about our customizations. Rather, I want to use them as an example of what I consider to be sensible extensions of on top of job-dsl-plugin that we’ve found well worth the investment, for the reasons listed above.

Reusable builders

job-dsl-plugin is Groovy-based, and the domain specific language (DSL) used to create jobs via text is Groovy. Thus it’s natural to follow additional groovy idioms when building on top of job-dsl-plugin. Builders is one of those idioms.

Essentially, these builders are just wrappers around job-dsl. At their most basic, they provide sensible defaults.

Builders for sensible defaults

For example, we want all of our jobs to have:

  • Build discard configured (aka “log rotation”)
  • Emails for failure and fixed
  • Broken build claiming
  • Colorized log output

To achieve that in the Jenkins UI, for all of our jobs, would require much discipline, oversight, or both. So in the spirit of making it easy to do the right thing, that’s part of our BaseJobBuilder. In one of our jobs.groovy files — which, remember from last time, gets turned Jenkins jobs via a seed job — we’d have something like this:

Here, BaseJobBuilder will create start creating DSL for a new job, add some goodies to it, and then return that job-dsl job object, which we can then use to further customize with plain-old-job-dsl.

Builders for complicated jobs / configurations

Another use case for Builders is to make it easy to do the right thing for potentially complicated jobs, especially if we want to provide the one right way to do something.

For example, we use a security tool called bdd-security for running penetration tests against our software, and we integrate that into our Jenkins pipelines. The Jenkins job for that can seem a little gnarly at first, and so we wrap it up in a BddSecurityJobBuilder, and then use it like so:

The “guts” of all our bdd-security jobs will then be exactly as we want them; should updates be required, we simply update the builder, and all jobs built from that builder get those updates within minutes. Obviously, a builder like this should still provide for us all the sensible defaults described above.

Common decision-making and code blocks

As we surveyed our Jenkins jobs, we observed some configuration patterns we realized we could simplify. First, as I mentioned in the first post, some jobs run in some Jenkinses, some run in all, some run in one. We needed a way to make that decision-making consistent and simple, without littering out Jenkins-as-code repositories with similar-but-different conditionals that mostly do the same thing. Thus, we created an Environment utility, which depends on a few global variables we ensure exist in all our Jenkinses. It looks like this in practice:

We also use these utilities — a better word would probably be “blocks” — for adding custom DSL blocks, especially for gnarly configs that would require a configure block since a nice DSL API doesn’t exist.

Those utils are here.

Builders pattern for use internally

Finally, the 4th motivation for Builders was to establish a pattern for use internally in our Jenkins-as-code repositories, particularly for jobs / job types that didn’t rise to the level of reusability that it’d be useful in our public git repo, or which would’ve been unduly burdened by prematurely genericizing (or, when we simply can’t justify the time to invest in genericizing)

For example, one of our projects has a common pattern of “Task” style jobs. These aren’t build pipelines or related to deployments; rather, they’re just… for running stuff. They almost all use a python virtual env, often run on some type of schedule, and, well, other stuff.

Here’s a stripped down version, and this is kept in an internal (non-github.com) repo:

And then jobs in that repo can use it just like any other Builder:

It may come to pass that we realize patterns in these custom internal builders, and at that time, put in the time to make them generic and pull them out into our public repo.

Getting started with the custom builders

A starter repository for easily getting started with these customizations. We use this repo as the starting point for all our internal Jenkins-as-code repositories.

Next up: orchestrating project job repositories

I’ve mentioned “internal Jenkins-as-code repositories” several times throughout this post. Rather than have all our Jenkins jobs in a single monolithic repository, we believe that projects / project teams should be able to keep their jobs in repo locations that make the most sense to them. That does impose several burdens, however:

  1. finding those repositories
  2. configuring their seed job in all the Jenkinses in which those project’s jobs should run

We solve that via a Registration pattern, which I cover in the next post.