Jenkins-as-code: registering jobs for automatic seed job creation

In the previous post in this series, I covered custom builders we’ve added on top of job-dsl-plugin. In this post, I’ll cover an innovative solution that’s made working with all these Jenkins-as-code repositories much easier. It centers around a seed job for seed jobs.

What are seed jobs again?

The job-dsl-plugin tutorial covers them thoroughly. In short, a seed job processes your job-dsl scripts and thus creates the Jenkins jobs from those scripts using the “Process Job DSLs” build step. You generally configure your seed jobs to listen for changes to the source code repo where you keep your jobs, such that your jobs are automatically rebuilt whenever changes occur.

The problem

I’ve mentioned “internal Jenkins-as-code repositories” several times throughout past posts in this series. Rather than have all our Jenkins jobs in a single monolithic repository, we believe that projects / project teams should be able to keep their jobs in repo locations that make the most sense to them, maybe even sitting right inside their main project repo (think: .travis.yml or Jenkinsfile, but on steroids). That does impose several burdens, however:

1. finding those Jenkins job repositories
2. manually configuring their seed job in all the Jenkinses in which those project’s jobs should run

In an organization of any size, discovery is a problem, and we anticipate dozens (hundreds?) of Jenkins job repositories scattered across the organization. And the whole point of jenkins-as-code is to get out of the pointy-clicky job creation business, so it violates the principle of the initiative to require manual seed job configuration.

Just for now, imagine that you have 20 projects. They all have their own jobs. You’ve tried like hell to de-snowflake them so that they’re practically automatable or templatable, but you’re still manually pointing-and-clicking all those jobs. Updates are a hassle (even with scriptler). So you go the job-dsl route as described in previous posts, you script your jobs, but now you still have to create the seed jobs for all those projects. Or you have a gigantic single seed job that listens to all those repos (a totally legit approach). Either way, you don’t want to do much pointing and clicking to create seed jobs.

We solve that via a Registration pattern.

Registering job repositories

We haven’t yet open sourced this code (we will), so I’ll provide all the meaningful bits in this post for now.

The idea is simple: a single configuration file that teams can add their Jobs repository to via pull request. Once merged, a “Mother Seed Job” runs and then creates a seed job for their jobs. It’s a seed job for seed jobs.

Even after reading the problem statement above, you might still be thinking: you really need that? Like, can’t you just manually configured some seed jobs and move on? I thought the same thing. And in your org, maybe you don’t! It has turned out to really benefit us, though. More on that doubt, and trust, at the end of this post because it’s a good story.

Configuration file

Let’s start with the configuration file we use. It’s kept in an internal git repo, and additions are made via pull request. It looks something like this:

We have multiple hosts (the above are fake), and we split those into traditional dev/test/prod/etc environments. Some repos will only ever have automations that live in one host/environment combination, and some with multiple, so we account for those possibilities in this configuration.

The idea is simple: you put your repo with job-dsl jobs in here, based on what Jenkinses you want your jobs to be created in.

“all” is a special case that means what you’d expect: create this repo’s jobs in all the Jenkinses for this host.

Mother seed job

We then have the “Mother Seed Job” configured in all our Jenkinses. Here’s where it gets trippy: the mother seed job is itself a straightforward Freestyle job that runs a job-dsl script.The mother job pulls from 2 git repos: the jenkins-automation repo I talked about last time, and an internal git repo with configuration like above (and, currently, this seed job script below, with some other stuff). All the interesting bits are in that job-dsl script.

I am reproducing it here, in full:

This then creates seed jobs for each repo configured in the config file above, in the appropriate Jenkinses.

What’s the global variables stuff?

If you actually read that closely, you also noticed some global variables business going on. In short, we also use this seed job to generate a “globals.properties” file that we put on each Jenkins, so that we don’t have to manually manage global vars in Jenkins. That’s just another config file, which looks something like this:

 

How does a Jenkins identify itself?

We have these config files that have a host/environment matrix, and a mother seed job that creates seed jobs based on configuration. So how do each of the Jenkinses know which jobs are theirs?

Two global variables that we manage outside of jenkins-as-code (via configuration management or manual global var config in Jenkins):

• JAC_HOST
• JAC_ENVIRONMENT

Those 2 global variables exist in all of our jenkinses, and they are unique.

For example, for our Jenkinses in AWS, their JAC_HOST value will be “aws”. Then, because we only have one Jenkins master per environment, the JAC_ENVIRONMENT will be “dev”, “test”, “stage”, or “prod”.

The JAC_HOST and JAC_ENVIRONMENT values correlate exactly to the keys (aws, heroku, azure… dev, test, prod) in those config files.

There’s nothing special about these host or environment names; all that matters is that those two vars configured in Jenkins align with their corresponding names in the config files.

(disclaimer: I’m using aws, heroku, and azure as examples. I cannot publicly speak to our infrastructure)

Short story about trust

Irina M is ultimately responsible for leading design and implementation of the solution I’ve been writing about in this series. At one point, when work was winding down, she said she was working on a “mother seed job” (her words) to make it easier for teams to get their job repos in our Jenkinses. I told her then, more than once, that I didn’t really understand the problem.

She patiently explained it, but I still didn’t grok it as an actual-actual problem. But here’s the thing: as a team lead, people join your team and they’re smarter than you and you have to trust them. Sure, you can be skeptical, but you have to be confident enough to let go of control. For a lot of people, trust, and letting go, is not easy. It’s not for me.

This is one in a countless number of reminders of how often that trust pays off. Had I said “No, it’s not worth it, please spend your time on other things” we would not have had this solution, Irina would’ve rightfully been disengaged, and on and on.

The enlightened among you are thinking, “Duh, dumbass”. If you’re not thinking that, I implore you: put your ego down and trust your team.

Next up: my favorite development helper

As our team was working on adopting this jenkins-as-code solution, I confess that local development was kind of a pain in the ass. Because seed jobs pull from source control and then process job dsls, we’d have to build the dsl scripts, push to git, run the seed job, and run the built jobs, repeating as necessary till the job was working as desired. The workflow was juuust slow enough to be annoying.

We solve that by adapting a command line utility, a tiny nugget of gold, unassumingly linked in the job-dsl-plugin wiki. And now, local development of job-dsl scripts is, for me, a pleasure. More on that in the next post.