In the first post in this series, I covered the problems we were having with job creation and maintenance and a very high level look at our solution. In the second post, I covered the job-dsl-plugin which underpins our solution.
In this post, I’ll dig into the Builders we’ve added on top of job-dsl-plugin, which do the following for us
- add sensible defaults for all jobs
- make it easy to do the right thing for potentially complicated jobs or configurations
- provide utilities for common decision-making and code blocks
- establish a Builders pattern for use internally even if a job type isn’t all that reusable across our projects
Why am I writing about these?
While I think it’d be great if others were to take advantage of our jenkins-automation repo, that’s not my motivation for writing about our customizations. Rather, I want to use them as an example of what I consider to be sensible extensions of on top of job-dsl-plugin that we’ve found well worth the investment, for the reasons listed above.
Reusable builders
job-dsl-plugin is Groovy-based, and the domain specific language (DSL) used to create jobs via text is Groovy. Thus it’s natural to follow additional groovy idioms when building on top of job-dsl-plugin. Builders is one of those idioms.
Essentially, these builders are just wrappers around job-dsl. At their most basic, they provide sensible defaults.
Builders for sensible defaults
For example, we want all of our jobs to have:
- Build discard configured (aka “log rotation”)
- Emails for failure and fixed
- Broken build claiming
- Colorized log output
To achieve that in the Jenkins UI, for all of our jobs, would require much discipline, oversight, or both. So in the spirit of making it easy to do the right thing, that’s part of our BaseJobBuilder. In one of our jobs.groovy files — which, remember from last time, gets turned Jenkins jobs via a seed job — we’d have something like this:
Here, BaseJobBuilder will create start creating DSL for a new job, add some goodies to it, and then return that job-dsl job object, which we can then use to further customize with plain-old-job-dsl.
Builders for complicated jobs / configurations
Another use case for Builders is to make it easy to do the right thing for potentially complicated jobs, especially if we want to provide the one right way to do something.
For example, we use a security tool called bdd-security for running penetration tests against our software, and we integrate that into our Jenkins pipelines. The Jenkins job for that can seem a little gnarly at first, and so we wrap it up in a BddSecurityJobBuilder, and then use it like so:
The “guts” of all our bdd-security jobs will then be exactly as we want them; should updates be required, we simply update the builder, and all jobs built from that builder get those updates within minutes. Obviously, a builder like this should still provide for us all the sensible defaults described above.
Common decision-making and code blocks
As we surveyed our Jenkins jobs, we observed some configuration patterns we realized we could simplify. First, as I mentioned in the first post, some jobs run in some Jenkinses, some run in all, some run in one. We needed a way to make that decision-making consistent and simple, without littering out Jenkins-as-code repositories with similar-but-different conditionals that mostly do the same thing. Thus, we created an Environment utility, which depends on a few global variables we ensure exist in all our Jenkinses. It looks like this in practice:
We also use these utilities — a better word would probably be “blocks” — for adding custom DSL blocks, especially for gnarly configs that would require a configure block since a nice DSL API doesn’t exist.
Those utils are here.
Builders pattern for use internally
Finally, the 4th motivation for Builders was to establish a pattern for use internally in our Jenkins-as-code repositories, particularly for jobs / job types that didn’t rise to the level of reusability that it’d be useful in our public git repo, or which would’ve been unduly burdened by prematurely genericizing (or, when we simply can’t justify the time to invest in genericizing)
For example, one of our projects has a common pattern of “Task” style jobs. These aren’t build pipelines or related to deployments; rather, they’re just… for running stuff. They almost all use a python virtual env, often run on some type of schedule, and, well, other stuff.
Here’s a stripped down version, and this is kept in an internal (non-github.com) repo:
And then jobs in that repo can use it just like any other Builder:
It may come to pass that we realize patterns in these custom internal builders, and at that time, put in the time to make them generic and pull them out into our public repo.
Getting started with the custom builders
A starter repository for easily getting started with these customizations. We use this repo as the starting point for all our internal Jenkins-as-code repositories.
Next up: orchestrating project job repositories
I’ve mentioned “internal Jenkins-as-code repositories” several times throughout this post. Rather than have all our Jenkins jobs in a single monolithic repository, we believe that projects / project teams should be able to keep their jobs in repo locations that make the most sense to them. That does impose several burdens, however:
- finding those repositories
- configuring their seed job in all the Jenkinses in which those project’s jobs should run
We solve that via a Registration pattern, which I cover in the next post.