Going Serverless: using Twilio and AWS Lambda to make phone calls with an AWS IoT button



Full credit for this idea and post goes to my colleague Andy, who presented this approach at a recent lunch-and-learn at work. A few months after his presentation, I decided to try replicating what he did. This blog post represents what I learned from that effort. Thanks Andy!

The Goal

For the 3 phones in our house, when one is misplaced, press an Amazon IoT button and have it dial the phone. A single click should call one phone number, double click another, and long click the third number. Then, the phone would ring, and I would listen for it, and I would find the phone, and that’d be swell.

This was most assuredly not an effort to create something terribly useful. My actual goal was just to learn more about Lambda and IoT, with some practical (ish) real-world utility.

In this post, I will not go into excruciating detail on every single step. I’ll get into the weeds where I had particular trouble or learned something useful. This is not an Introduction to Lambda post. It also assumes some familiarity with AWS and the AWS console.

The Tech

AWS IoT Button

First, an Amazon IoT button. The button supports single, double, and long click, and so ultimately I want my function to respond differently depending on the type of click. These things are ridiculously priced at $20 and get you about 2000 clicks before the battery dies.

Second, AWS Lambda. I used Python for the code (shown below). I also used nficano/python-lambda to deploy the code, although I definitely recommend learning how to use lambda without any such deploy library first.

Third, a Twilio account and phone number. You get $15 credit for a trial account, but I decided just to buy a number ($1/month). Total cost is $1 / month for the number + 1.3 cents per call. Once you create an account, you’ll get a phone number and “sid” and “token” values to use in the code.

Fourth, twimlets. In short, using a twimlet URL within the lambda function was the easiest way I could find to have the phone call say something (in a terrible robo-voice) when you pick up the phone. More on this later.

Tying it all together: Click Button –> Triggers lambda function –> uses Twilio API to dial a phone number –> that API call uses a twimlet URL that says something to the person who answers the phone call

Let’s dig in. I’ll be using Python for this. Andy has a version that uses JavaScript / NodeJS.

Using the Twilio API

Before jumping in to the Lambda bits, let’s look at the meat of the code, which is using the Twilio API to make a phone call. You can do that with a single command.

pip install the Twilio library and run that in a python shell (set the variables, obviously). In a few seconds, you should get a phone call with a greeting and then some hold music.

The code, draft 1

I mentioned above that I’m using the nficano/python-lambda library to make developing lambda functions a bit simpler. I also mentioned that when you’re just getting started with Lambda — doing your Hello World version — I strongly recommend learning how to do it with the plain old Lambda console before bringing in any tooling.

But once you get that under your belt, a library such as python-lambda really helps.

All of this code lives at https://github.com/marcesher/twilio-lambda, and I’ll walk through it here, modifying it as I go.

First, I created a new local directory, created a virtualenv, and created requirements.txt file with these contents:

After activating the virtualenv, I installed those libraries into it:

pip install -r requirements.txt

Second, I ran

lambda init

which generated several files, including skeletons for service.py and some config files.

Third, I knew I’d be configuring environment variables for twilio sid and token, the “from” phone number, and the various “to” phone numbers.

I keep the environment variables in a “.env” file:

lambda init also created an “event.json” file and added some dummy variables. That file will eventually become important for local testing when we want to simulate double and long button clicks on the button. But I’m going to hold off on that for now.

I then wiped out the stuff in the generated “service.py” and replaced it with this:

Then, using the python-lambda library, and ensuring that the service.py has the code above, I ran it like so:

Invoke the function locally

If all goes well, then the number you’ve configured to get called should get called by your Twilio number. If you’re using a trial account, you’ll first hear a kindly gentleman’s voice telling you you’re using a trial account, and then when you press any key on your phone, you’ll hear some hold music. Later in this post, we’ll change that hold music to something more useful.

All this lambda-invoke thing is just a nice way of running your python function and having it simulate the event object using your event.json file. It’s quite similar to how you’d use the “Test” button in the AWS Lambda console, where you also can create a JSON event object for testing.

Changing the phone call

First, that deceptively simple, single Twilio API call was, for me, infuriating to get working as I wanted. The URL it uses in the sample code is to some hold music, and I wanted to change it to actually say something, which according to the docs means pointing to a URL that returns “Twiml” — Twilio XML — which tells twilio how to behave. Here’s a sample:

So I figured I’d just host a static XML file on github or S3. But when I tried those URLs in the client call, it failed, with an error indicating that the URL didn’t accept a POST request (GH and S3 only support GET). I got to the point where I thought I might actually have to set up a web server just to serve the damn XML file. But before I did that, I took one more look at the sample code that Twilio provides and that’s when I noticed that Twimlets thing:

So then I went and checked out Twimlets.com, and lo and behold, there are all manner of  handy helpers. I read the one for “Simple Message”, and then used its “Twimlet Generator” UI to create a properly encoded URL that I could then use in my service. Here’s a sample that just says “Hello from twilio” when you pick up the phone

Just with that simple message twimlet, you can configure it to say any number of different things, or play any number of mp3s. It’s slick.

Getting it working in Lambda

The AWS Lambda function

With the function working locally — dialing a number with twilio and saying something when I picked up — it was time to start configuring the Lambda function. This is going to be a multi-step affair:

  1. create the function (and IAM role if it doesn’t exist)
  2. configure the environment variables
  3. deploy the code
  4. testing in the Lambda console

Wiring up the button, and modifying the service to respond to different types of clicks, will come last.

I used the AWS console to create a new Lambda function named “hello_twilio”. I configured it thusly in the console:

Lambda function configuration

I want to talk about the “Role” stuff. When I first started with Lambda, I got hung up a bit there. Ultimately, it’s just adding an IAM role with appropriate permissions. Here’s the role I created, and which you see in that screenshot above, called “lambda_execution_role”. At a minimum, your role will require CloudWatch and Lambda execution privileges. I threw in a few more for good measure.

Configuring environment variables

On the “Code” tab of the Lambda function, I added environment variables for the twilio sid and token, and from and to numbers. They’re named identically to the variables shown in the .env file, above.

Deploying the function

Because this Lambda function has dependencies — in this case, the Twilio library — you’ll need to bundle your function up as a zip file and upload it to Lambda. The AWS documentation explains how to do this manually. But if you’re using the nficano/python-lambda library, you can use it to do that for you.

First, you’ll want to be sure that the config.yml file created via lambda init has the right values. In my case, those are the values that appear above in the console. Note that “service.handler” means “the handler function in the service.py file”. This will now overwrite anything you previously configured, so make sure this file has the correct values:

Then, it’s simply a matter of:

lambda deploy

Testing in the Lambda console

With the code working locally, I wanted to then test it in the Lambda console. After ensuring that environment variables were set, I clicked the “Test” button and then inspected the CloudWatch log output below. Any invocations of your Lambda function — from the Test button, or from the real IoT button, will stuff the output into CloudWatch. This will become really important in a minute, when I discuss testing the function with the button and trying to change behavior based on the type of button click.

Wiring up the IoT button

The button comes with a tiny manual that walks you through how to activate the button, secure it, and so forth. My experience was that it was quite straightforward and only took a few minutes. Once the button is configured, you’ll be able to add it as a trigger to the Lambda function.

You wire up the IoT button in the Lambda console “Triggers” tab. Click “Add Trigger”, then “AWS IoT”. This will walk you through a wizard where it’ll create some files that you then have to add to your Button.

Once the button is all wired up, you’re ready to press!

You can then view the output in CloudWatch. Additionally, assuming that the code successfully invokes the Twilio API, you can also view the Twilio logs from your Twilio dashboard. This ended up being really helpful for me when I was trying to get that URL working as described above.

 Changing behavior based on click type

Once I had the function successfully invoked with the button, I wanted to add one last thing: change the behavior of the function so that it would use a different number for a double and a long click.

This was way harder to figure out than I expected, even though the resultant answer is dead simple.

I simply could not figure out from reading documentation how to have the lambda function respond differently to double and long clicks. I knew they were supported, but no manner of googling “AWS Lambda IoT Double Click” or other such things led me to relevant docs. I went on an “IoT Rules” goose chase for a while, to no avail. Then, on a whim, I decided to just add a print statement of the event object to see if anything was in there.

After adding the print statement, re-deploying, clicking the button, and looking at the logs in CloudWatch, I noticed that there was a “clickType” key in the event object with a value of “SINGLE”. Naturally, I tried double-clicking, then long clicking, and saw in the logs the values of “DOUBLE” and “LONG”. So it ended up being really simple to respond to different click types, but it took a fair bit of time and just dumb luck to figure out how to do it.

And although I admittedly haven’t searched much since getting it working, I still haven’t found the documentation that spells out the clickType being added to the event object by the button.

The final code ended up looking like this:

Supporting clickType for local invocation and Lambda console testing

Once I figured out that different button click was simply a matter of creating a “clickType” key in the event object, it was fairly straightforward to mimic that locally and within the Lambda console.

For local development: that lambda init way up above had created an “event.json” file. I opened that up and replace the contents, like so:

Then, lambda invoke will automatically inject that as an entry in the event object.

For testing in the Lambda console, you need to configure the Test Event object. This is under the “Actions” dropdown menu.

Then, when you click the “Test” button in the console, it’ll use that as the event object.

Wrapping up

Again, huge props to Andy for inspiring this little Lambda / Twilio journey. It was a lot of fun.

Finally, I know this post hand-waved over a bunch of stuff (“to wire up the button… wire up the button” 🙂 ). But if you have questions about any of the stuff I’ve left out, please do ask.


This started out as mostly just a silly-ish way to learn more about Lambda and IoT. But it certainly impressed the kids, and they get a kick out of pressing the button to annoy us. It’s like Leave it to Beaver: clean, wholesome fun the whole family can enjoy 😉


Jenkins-as-code: comparing job-dsl and Pipelines

In the previous post in this series, I covered my favorite development-time helper: running job scripts from the command line. In this post, I’ll cover the differences between job-dsl and Pipelines, and how I currently see the two living together in the Jenkins ecosystem.

job-dsl refresher

If you’re coming into this post directly, without reading the preceding articles in the series, I strongly encourage you to start at the start and then come back. For the rest of you, a quick refresher:

job-dsl is a way of creating Jenkins jobs with code instead of the GUI. Here’s a very simple example:

When processed via a Jenkins seed job, this code will turn into the familiar Jenkins jobs you know and love.

What are Pipelines?

Jenkins Pipelines are a huge topic, way more than I am going to cover in a single blog post. So I’m going to give the 30,000 foot view, leave a whole bunch of stuff out, and I hope whet your appetite for learning more. For the impatient, skip these next few paragraphs and head straight for the Jenkins Pipeline tutorial.

At its simplest, a Pipeline is very job-dsl-ish: it lets you describe operations in code. But you have to shift your mindset quite a bit from the Freestyle jobs you know well. When configuring a Freestyle job, you have the vast array of Jenkins plugins at your fingertips in the GUI (and job-dsl) — SCM management, Build Triggers, Build Steps, Post-build actions.

Top-level view of a Freestyle job configuration
Top-level view of a Freestyle job configuration

But with Pipelines, it’s different. You get Build Triggers and your Pipeline definition. But what about the other stuff, you ask? This is where the mindshift comes in. Those things are no longer configured at the job level, but at the Pipeline level. And plugins are not automatically supported in Pipelines, so currently you get a subset of all available Jenkins functionality in Pipelines.

Top-level view of a Pipeline job configuration

Thus in practice, that means things like git repos, build steps, email, test recording/reporting, publishers, etc are all done — in text — in the Pipeline definition.

Here’s an example that ships with Jenkins:

Example Pipeline script
Example Pipeline script

This is kinda sorta like…

Probably confusing, I know. Let’s try to think of it this way: If you’ve read Jez Humble and David Farley’s Continuous Delivery, or have otherwise implemented build/deploy pipelines in Jenkins for years, you already have a solid conceptual sense of pipelines. It’s just that in Jenkins world until rather recently, you probably did this in one of two ways:

  1. Upstream / downstream jobs (possibly in combination with the Delivery Pipeline plugin); or
  2. Via the Build Flow plugin, with independent jobs being orchestrated via a simple text DSL

Either way, you probably had independent Freestyle jobs tied together somehow to make a pipeline.

Well, Jenkins Pipelines still certainly enable you to do that — and I’ll talk specifically about option #2 momentarily — but the big change here is that Pipelines enable you to do all that orchestration in a single job.

Whereas before you might have separate BuildJob, TestJob, and DeployJob tied together in one of the manners above, with Pipelines, you can do all that in a single job, using the concept of Stages to separate the discrete steps of the pipeline.

Cool! What else do I get with this?

Even with the simplest of Pipelines, you get:

  • Durability, to survive Jenkins restarts
  • Pausing for user input
  • Parallelism built in
  • Pipeline snippet generator to help you build pipelines
  • Nice visualization of your Pipelines right in the UI

But wait, there’s more

You also get Travis CI style CI via a Jenkinsfile, and Multibranch pipelines which enable pipeline configuration for different branches of a repo. From the Jenkins Pipeline tutorial: “In a multibranch pipeline configuration, Jenkins automatically discovers, manages, and executes jobs for multiple source repositories and branches.”

In addition, Jenkins Blue Ocean is shaping up to have beautiful visualizations for Pipeline jobs

Using Pipelines right now, today

Let’s say you still really like loosely coupled jobs that can run independently or together as part of a Pipeline (caveat: I have encouraged this approach for years, and it’s why I’ve long used Build Flow plugin over Build/Delivery Pipeline plugin). Right now, today, you can replace your Build Flow jobs with Pipelines.

In fact, you should do this. From the Build Flow wiki page:

A simple Pipeline script to build other jobs looks like this:

Overall, pretty similar to BuildFlow. And don’t worry, you get all the parallelism, retry, etc that you’re used to.

I am tremendously grateful for the people who’ve built and maintained the Build Flow plugin over the years. For me, it’s been a cornerstone of continuous delivery, enabling independent, reusable, loosely coupled jobs. Build Flow developers: Thank you!

But, it’s time to move on, Pipeline will replace Build Flow.

Do Pipelines replace job-dsl?

Now, to the final question: should Pipelines replace job-dsl?

I believe that’s the wrong question.

job-dsl will be complementary to Pipelines. Even if I were to stop using Freestyle jobs entirely, and build nothing but Pipelines, I’d still use job-dsl to create those jobs. In fact, if you go back to my initial post where I described the problems we were trying to solve when we adopted job-dsl, none of them are solved by Pipelines. In that respect, Pipeline is just another type of job.

A friggin’ awesome type of job, no doubt. I am incredibly excited about Pipelines and look forward to using them more. And here’s how I’ll be building those jobs, as job-dsl has full support for Pipelines:

So what is the right question?

If asking whether Pipelines replace job-dsl is the wrong question, what’s the right question?

I believe it’s:

  1. When should Pipeline replace Freestyle jobs?
  2. When should Pipeline — via Jenkinsfile — replace creating jobs directly in Jenkins (via GUI or job-dsl)?

I’m going to mostly cop out of answering those questions right now, as, for me, the answers are still evolving as I work more with Pipelines.

My initial gut reactions are:

  1. replace Build Flows, as mentioned above
  2. replace Freestyle jobs when there’s no value in running that set of jobs independently
  3. replace Freestyle jobs when you’d benefit from what Multibranch provides
  4. replace Jenkins-built jobs with Jenkinsfile when you have a TravisCI-style workflow that you want to use in Jenkins instead and you’ve seriously considered the safety and security implications for your organization (my thoughts are in the very earliest stages here)

Next up: encouraging adoption of Jenkins-as-code among teams

In the final planned post in this Jenkins-as-code series, I will address how we encouraged adoption of this approach amongst our development teams. I’ll cover where we succeeded, where we stumbled, and the work I think we still have to do.

Remaining more present by controlling information overload

A while back, I was made well aware of my, shall we say, un-presence. I’d be physically present with my family, but distracted the point of absence. Usually, it was due to a device stealing my time from them. The phone in my pocket was becoming a time-and-attention thief.

A Facebook message. A Twitter reply. An email.

I am still not great at this — my kids would probably say I’m terrible — but I’ve come a long way and thought I’d share some of my tactics for helping reduce social media and email overload.

The point of this is to show how I use friction and delay to help keep myself from being pulled out of the present moment when it comes to notifications that would otherwise constantly distract me.

In short:

  1. Using web apps, not native apps
  2. Turning off most email notifications
  3. Rolling up the email notifications that remain

Using web apps, not native apps

Heresy: when I can, I use web apps instead of native mobile apps. This is first and foremost a matter of principle: I believe in the browser as a platform and the ability to produce rich, accessible applications in standard technologies such as HTML, CSS, and JavaScript. So, in support of this principle, I will opt to use a web application over a native device application unless the native application offers compelling features that I want.

And here’s a feature I don’t want: device notifications. If I get a Facebook comment or twitter mention, I don’t care to know about it right away.

By using web apps, I am forced to go to there rather than having that information pushed to me. In other words, I use that little bit of friction to my advantage.

Disabling notifications

For nearly all social media — Twitter, Facebook, etc — I disable most email notifications. Facebook PMs and likes and comments… nope. Twitter follows… nope. I force myself to go that information; I don’t want it pushing to me.

The combination of using web apps, and disabling notifications, means that I am often hours, if not days, behind on interactions on these platforms, by choice.

Rolling up the email notifications that remain

I still get a fair amount of non-personal (i.e. person-to-person) email — newsletters, LinkedIn, GitHub, meetups, etc.

To stop those from distracting me as soon as I receive them, I use a free service called Unroll.me to manage that email. This service watches my inbox, and when it detects something that looks automated, it asks whether I want to unsubscribe, roll it up into a daily email, or keep it in the inbox.

After almost a year of using this service, I’ve unsubscribed from a lot of things I don’t care about, and rolled up many disparate communications into a daily email. I cannot recommend this service highly enough. (Caveat: it has access to all your email.)


How we interact with the digital world is a highly personal affair. The choices I’ve described above — to ultimately favor the present moment vs interacting with people digitally at all hours of day and night except when I explicitly choose to do so — clearly come with tradeoffs. I am slower to respond to people. I am probably not servicing certain relationships as well as I should. When I do interact on those services, it’s slightly less convenient.

But the real-time web — powerful and important as it may be — can also be a bug and not a feature, especially when it comes to interactions that, perhaps, aren’t urgent. In these cases, I give myself permission not to participate in real time.

The way I think about it is that these choices help me to favor the people I am with, right here, right now. And I cannot do that effectively if there’s a constant trickle of requests for attention coming from across the ether.




Jenkins-as-code: creating jobs from the command line during development

In the previous post in this series, I covered how to make a seed job for seed jobs via registration. In this post, I’ll cover my favorite development-time helper: running job scripts from the command line.

The problem

As our team was working on adopting this jenkins-as-code solution, I confess that local development was suboptimal. Because seed jobs pull from source control and then process job dsls, we’d have to build the dsl scripts, push to git, run the seed job, and run the built jobs, repeating as necessary till the job was working as desired. The workflow was slow enough to be annoying. Because, let’s face it, how often do you really get your automations right on the first try?

The solution

The solution ended up serendipitously coming from something Irina had built for the registration job… a way to “recover” that job on a Jenkins should something unfortunate happen to it or to easily put that job onto a new Jenkins without manually configuring it. The idea was to be able to do the following from the command line, given that a dsl script called recover_mother_job.groovy existed and whose purpose was to rebuild the mother-seed-job:

./gradlew rest -Dpattern=jobs/recover_mother_job.groovy -DbaseUrl=http://our-jenkins/ -Dusername=foo -Dpassword=bar

She found it in this bit of gold from the job-dsl folks.

Broader applicability

Eventually, it hit us that if we could use it for the mother seed job, why not for any job?

The workflow would look like this:

  1. You’ve got a target Jenkins server, either local or remote
  2. You’re editing your dsl script locally
  3. You want to quickly build that on the target Jenkins server, without pushing to SCM yet
  4. You run a single command in your terminal, then go check out the job in your target Jenkins
  5. Once you’re satisfied, you push to SCM

So we copied-and-slightly-modified Matt Sheehan’s rest runner into our jenkins-automation project, and then added the rest task into our starter project, which we use for the basis of all our jenkins job repositories internally.

Try it out

Here’s what I want you to do right now:

  1. Identify an existing local or remote Jenkins to run this against
  2. Clone https://github.com/cfpb/jenkins-as-code-starter-project.git locally
  3. cd jenkins-as-code-starter-project/
  4. create a new file in the “jobs” directory called “simple.groovy” and give it these contents:
job('hello-world-dsl') {
   steps {
     shell("echo 'Hello World!'")

Now, from that same directory, run:

./gradlew rest -DbaseUrl=http://localhost:8080 -Dpattern=jobs/simple.groovy

If you need to pass credentials, also pass

-Dusername=<username> -Dpassword=<password>

You should see output  like this:

:compileJava UP-TO-DATE
:processResources UP-TO-DATE

processing file: jenkins-as-code-starter-project/jobs/simple.groovy
Processing provided DSL script
hello-world-dsl - created

Go to your jenkins, and behold your new glorious ‘hello-world-dsl’ job.

Now, keep playing with that job in your edit. Or add more jobs to that file (or other files). Run that command over and over, and see your job updated in Jenkins. This is about as close to an “F5 Refresh” Jenkins job development workflow as I’ve found.


A few caveats:

  1. Don’t get in the bad habit of only running jobs from the terminal. Commit them to SCM and let seed jobs do the real work
  2. If you’re running against a Jenkins with default CSRF crumb protection, this won’t work
  3. The latest version of job-dsl-plugin has an “auto-generated dsl” feature to replace the need for manually adding new DSL APIs for plugins; this will flat-out break this method of job creation. For that reason, we have not adopted into our code any of the auto-generated bits and are sticking, for now, with either APIs that exist or with configure blocks, because that’s how important this tool has become (to me, anyway)

Next up: what about Workflow/Pipeline and Jenkinsfile?

The Workflow plugin, aka Pipelines, is standard as of Jenkins 2.0. It, also, is a groovy DSL approach to configuring automations.

In the next post, I’ll cover the differences between job-dsl and Pipelines, and how I currently see the two living together in the Jenkins ecosystem

Jenkins-as-code: registering jobs for automatic seed job creation

In the previous post in this series, I covered custom builders we’ve added on top of job-dsl-plugin. In this post, I’ll cover an innovative solution that’s made working with all these Jenkins-as-code repositories much easier. It centers around a seed job for seed jobs.

What are seed jobs again?

The job-dsl-plugin tutorial covers them thoroughly. In short, a seed job processes your job-dsl scripts and thus creates the Jenkins jobs from those scripts using the “Process Job DSLs” build step. You generally configure your seed jobs to listen for changes to the source code repo where you keep your jobs, such that your jobs are automatically rebuilt whenever changes occur.

The problem

I’ve mentioned “internal Jenkins-as-code repositories” several times throughout past posts in this series. Rather than have all our Jenkins jobs in a single monolithic repository, we believe that projects / project teams should be able to keep their jobs in repo locations that make the most sense to them, maybe even sitting right inside their main project repo (think: .travis.yml or Jenkinsfile, but on steroids). That does impose several burdens, however:

  1. finding those Jenkins job repositories
  2. manually configuring their seed job in all the Jenkinses in which those project’s jobs should run

In an organization of any size, discovery is a problem, and we anticipate dozens (hundreds?) of Jenkins job repositories scattered across the organization. And the whole point of jenkins-as-code is to get out of the pointy-clicky job creation business, so it violates the principle of the initiative to require manual seed job configuration.

Just for now, imagine that you have 20 projects. They all have their own jobs. You’ve tried like hell to de-snowflake them so that they’re practically automatable or templatable, but you’re still manually pointing-and-clicking all those jobs. Updates are a hassle (even with scriptler). So you go the job-dsl route as described in previous posts, you script your jobs, but now you still have to create the seed jobs for all those projects. Or you have a gigantic single seed job that listens to all those repos (a totally legit approach). Either way, you don’t want to do much pointing and clicking to create seed jobs.

We solve that via a Registration pattern.

Registering job repositories

We haven’t yet open sourced this code (we will), so I’ll provide all the meaningful bits in this post for now.

The idea is simple: a single configuration file that teams can add their Jobs repository to via pull request. Once merged, a “Mother Seed Job” runs and then creates a seed job for their jobs. It’s a seed job for seed jobs.

Even after reading the problem statement above, you might still be thinking: you really need that? Like, can’t you just manually configured some seed jobs and move on? I thought the same thing. And in your org, maybe you don’t! It has turned out to really benefit us, though. More on that doubt, and trust, at the end of this post because it’s a good story.

Configuration file

Let’s start with the configuration file we use. It’s kept in an internal git repo, and additions are made via pull request. It looks something like this:

We have multiple hosts (the above are fake), and we split those into traditional dev/test/prod/etc environments. Some repos will only ever have automations that live in one host/environment combination, and some with multiple, so we account for those possibilities in this configuration.

The idea is simple: you put your repo with job-dsl jobs in here, based on what Jenkinses you want your jobs to be created in.

“all” is a special case that means what you’d expect: create this repo’s jobs in all the Jenkinses for this host.

Mother seed job

We then have the “Mother Seed Job” configured in all our Jenkinses. Here’s where it gets trippy: the mother seed job is itself a straightforward Freestyle job that runs a job-dsl script.The mother job pulls from 2 git repos: the jenkins-automation repo I talked about last time, and an internal git repo with configuration like above (and, currently, this seed job script below, with some other stuff). All the interesting bits are in that job-dsl script.

I am reproducing it here, in full:

This then creates seed jobs for each repo configured in the config file above, in the appropriate Jenkinses.

What’s the global variables stuff?

If you actually read that closely, you also noticed some global variables business going on. In short, we also use this seed job to generate a “globals.properties” file that we put on each Jenkins, so that we don’t have to manually manage global vars in Jenkins. That’s just another config file, which looks something like this:


How does a Jenkins identify itself?

We have these config files that have a host/environment matrix, and a mother seed job that creates seed jobs based on configuration. So how do each of the Jenkinses know which jobs are theirs?

Two global variables that we manage outside of jenkins-as-code (via configuration management or manual global var config in Jenkins):


Those 2 global variables exist in all of our jenkinses, and they are unique.

For example, for our Jenkinses in AWS, their JAC_HOST value will be “aws”. Then, because we only have one Jenkins master per environment, the JAC_ENVIRONMENT will be “dev”, “test”, “stage”, or “prod”.

The JAC_HOST and JAC_ENVIRONMENT values correlate exactly to the keys (aws, heroku, azure… dev, test, prod) in those config files.

There’s nothing special about these host or environment names; all that matters is that those two vars configured in Jenkins align with their corresponding names in the config files.

(disclaimer: I’m using aws, heroku, and azure as examples. I cannot publicly speak to our infrastructure)

Short story about trust

Irina M is ultimately responsible for leading design and implementation of the solution I’ve been writing about in this series. At one point, when work was winding down, she said she was working on a “mother seed job” (her words) to make it easier for teams to get their job repos in our Jenkinses. I told her then, more than once, that I didn’t really understand the problem.

She patiently explained it, but I still didn’t grok it as an actual-actual problem. But here’s the thing: as a team lead, people join your team and they’re smarter than you and you have to trust them. Sure, you can be skeptical, but you have to be confident enough to let go of control. For a lot of people, trust, and letting go, is not easy. It’s not for me.

This is one in a countless number of reminders of how often that trust pays off. Had I said “No, it’s not worth it, please spend your time on other things” we would not have had this solution, Irina would’ve rightfully been disengaged, and on and on.

The enlightened among you are thinking, “Duh, dumbass”. If you’re not thinking that, I implore you: put your ego down and trust your team.

Next up: my favorite development helper

As our team was working on adopting this jenkins-as-code solution, I confess that local development was kind of a pain in the ass. Because seed jobs pull from source control and then process job dsls, we’d have to build the dsl scripts, push to git, run the seed job, and run the built jobs, repeating as necessary till the job was working as desired. The workflow was juuust slow enough to be annoying.

We solve that by adapting a command line utility, a tiny nugget of gold, unassumingly linked in the job-dsl-plugin wiki. And now, local development of job-dsl scripts is, for me, a pleasure. More on that in the next post.