Jenkins-as-Code: Creating Jenkins jobs with text, not clicks

By Marc Esher

This is the first in a series of posts on how we upped our Jenkins game by treating Jenkins jobs as code, rather than pointing-and-clicking to create jobs.

In this series, I’ll cover:

the problems we had as our Jenkins use scaled throughout the organization
the target conditions we wished to achieve
how we addressed those problems using the job-dsl-plugin along with some sugar on top
what the development workflow looks like
what a realistic set of jobs looks like for a sample project
the sugar we built on top of job-dsl-plugin
how we encouraged adoption of this approach across teams
how this can be used complementary to the new pipeline jobs in Jenkins 2.x

Credit

Before I even get started, I want to be very clear that I had very little to do with any of this. On our team, these people did all the hard work, notably David G and Dan D for initial experiments; and especially Irina M for ultimately executing on the vision, to whom I am eternally grateful. And none of this would be possible without the heroes behind the job-dsl-plugin.

The problem

At work, we have:

Multiple Jenkins servers, in multiple separate hosting environments; none of these can communicate with one another
Most Jenkins jobs run in 1 of those hosting environments, not both. But some run in both.
Hundreds of Jenkins jobs across all these environments
Of the jobs that run in only 1 environment, many of them run on multiple Jenkinses in that environment, with slight differences (eg, in Dev all projects deploy on SCM change; on prod, most projects manually deploy and prompt for a tag to be deployed)
No control over the hosting/environment situation
Dozens of developers, working on dozens of projects
Significant growth in number of developers, demand for automation, and consequently number of Jenkins jobs
A very small group of folks who know Jenkins well

We also have:

A fantastic team of people
An organizational commitment, with leadership support, to solving the problems described above

For the jobs that were duplicated — with slight differences per environment — we found ourselves doing a significant amount of redundant pointing-and-clicking in different Jenkinses. In addition, we were creating a lot of snowflake jobs that did similar things differently, because of silos, skill gaps, absence of consistency / standards, etc. We were witnessing job configuration drift both between environments, and also between teams.

In practice, it looked like this:

“Why does this deploy job do [Thing A] in dev, but [Thing A+] in prod?”

“Why does this app deploy [this way], but this other app which is structurally the same deploy [that way]?”

“Who wants to build this [some job useful everywhere] we need in all of our Jenkinses?”

“What’s our policy for discarding old builds? Because these jobs retain for 30 days, these for 50 builds, and most just retain forever.”

“Why do these jobs use Extended email, and these use plain email?”

“I really, really wish every job would have failure claiming turned on by default. Why the hell is that an option, anyways?”

And on and on. In other words, we accumulated a lot of organizational deployment technical debt, and we were not happy.

The solution: job-dsl-plugin

I’ll spare you the history and cut to where we landed. We realized we couldn’t solve the multi-environment problem… that is our infrastructure reality. And our automations team isn’t big enough — nor would we want to — police hundreds of Jenkins jobs across multiple environments and turn into the consistency enforcement team. We wanted to continue to empower all developers to use Jenkins, and we wanted to satisfy our own needs for increased consistency. We wanted to make it easy to do the right thing. After a several month discovery phase to investigate solutions to the problems above, we ended up adopting an approach to creating Jenkins jobs that centered around the job-dsl-plugin.

This enabled us to:

use text to create Jenkins jobs
store those jobs in source control
easily code necessary differences per environment
more easily see and eradicate unnecessary differences in jobs across environments
easily create these jobs in multiple jenkinses, with a bit of config
“make it easy to do the right thing”… providing the consistent defaults we wanted, for free
simplify the small handful of jobs where we wanted “the one and only one way to do this thing”
foster knowledge sharing and discovery for job configuration
perform peer review of Jenkins job configuration

In short, we treat our Jenkins jobs like configuration-as-code

Real-world usage

Some really high-level info just to make what comes below grokkable until I get to the nitty-gritty details:

Jobs are configured in text, using Groovy. No, you do not need to become a Groovy expert, retool, learn a whole new language, etc. The syntax is very basic, and the API viewer makes it trivial to copy/paste snippets for job configuration
These jobs, contained in one or more .groovy file, are kept in source control
One or more “seed jobs” are manually configured to pull those .groovy files from source control and then “process” them, turning the Groovy text into Jenkins jobs (truth: we even automate the creation of seed jobs; more in a future post)
Nearly all of that happens via the job-dsl-plugin; the only exception is creating the seed jobs.

I also want to mention now — and I’ll repeat this a lot — that the job-dsl API viewer is pure friggin gold: https://jenkinsci.github.io/job-dsl-plugin/

Apologies in advance for starting a few steps ahead and leaving out a lot of hand-wavey stuff for now. I want to begin with the end in mind. I’ll fill in all the gaps later, I promise. As you’re reading along wondering “What’s this Builder stuff? How do these actually turn into Jenkins jobs?”, trust me, I’ll fill it all in.

I’ll start with the dead simplest job you can create with job-dsl. This is, say, step 0. Then I’ll do what everyone hates and rocket ahead to step 10.

When the seed job runs and pulls that config from SCM, it’ll create a job named “example-job-from-job-dsl”, pulling from a github repo, triggered by a push, with a gradle step, and archiving artifacts.

Now, truth be told, at work we don’t use straight up “job” but instead have Builders that wrap job, and add some useful defaults for us that we want on all our jobs.

Here’s a fairly representative sample of what a simple Jenkins job looks like for us, in code. Ignore the “BaseJobBuilder” business for now, as it’s just some sugar on top of job-dsl-plugin that adds some sane (for us) defaults:

When the seed job for this job runs, it results in a Jenkins job named “operations-jira-restart”, configured to pull from a git repo, with a shell step of “fab restart_jira”. The “BaseJobBuilder” bits for us add some other things to the job that we want to exist in all of our jobs (log rotation, failure claiming, and so on)

Here’s another, using a different “Builder“, with a bit more config added. We use this “SiteMonitorJobBuilder” builder

This dsl script will result in a Jenkins job named “jenkins-outbound-connectivity-check”, which we have configured in every single one of our Jenkinses. It runs hourly, runs http requests against configured URLs, and pulls from an external and internal GitHub repo, to confirm that the Jenkins instance can talk to all the things it should talk to.

I included this example because it demonstrates how easy it is for us now to solve one of the problems above, namely, how to change a job easily that runs in multiple Jenkinses. If we want to change how this connectivity check runs — or, heck, even delete it entirely — we just change it in code and push to SCM. The seed job responsible for this will run, and update the job in all our Jenkinses.

Next up: Getting started with job-dsl

Now that I’ve covered the problems we needed to solve, and a very high level look at our solution, I’ll go more in depth in the next post, covering the job-dsl-plugin and how to use it.

Relevant links

job-dsl-plugin
job-dsl api
our builders on top of job-dsl
our starter repo for storing jenkins jobs as code

It’s my chicken!

By Marc Esher

This is a quick story about parenting, and about positive self-talk.

Several months ago, our youngest daughter joined a local soccer rec league. It’s her first experience playing soccer outside of the occasional game in gym class. We’re not a rabid soccer family; my wife never played, I played a year in high school, and our oldest daughter played for a few seasons when she was little, much to her displeasure.

Now, being her first time and all, our youngest had pretty much no clue what to do, or what was going on in general. There’s a ball; there’s a goal. there’s lots of other girls; some of them are standing around, some of them are chasing the ball like crazy people. Some dude has a whistle. A bunch of parents are on the sidelines; most are quiet and staring at their gizmos (it’s practice, after all), and some are yelling weird things.

After the first practice, she was hooked and wanted to practice at home every night. “Dad,” she’d say after dinner, “let’s go kick.” So we’d go out in the yard and practice. That’s when I yelled(ish) something weird:

“It’s your chicken! Don’t let me steal your chicken!”

WTF, her eyes said

“Yell it with me. It’s MY chicken! Say it! Say, ‘It’s my chicken!'”

More WTF.

We do not own chickens.

I explained to her that, hey, there’s this movie, Rocky, and in the movie Rocky, who is a boxer, has to get lightning fast on his feet otherwise he’s going to get his butt kicked, so he chases a chicken around because chickens are fast and hard to catch apparently. And he fails at it, and then finally he catches the chicken. So, imagine that ball is the chicken. It’s yours, no one else’s. You don’t let anyone on that other team take your chicken away from you.

Imagine you’re 8 and hearing this.

I tried to do the same thing with my oldest daughter, back when she first started, except I told her to pretend the ball was her guinea pig. In hindsight, not smart. What kid wants to kick their own guinea pig?

So this chicken, it becomes… a thing. With my wife, me, both daughters. It cracks us up. Eldest is yelling at youngest to get her chicken. We’re at games, yelling from the sidelines, “it’s your chicken!”

We don’t really get invited to parties. Not sure why.

The season ended a few weeks ago, but we still practice nearly every night. And the other night, we were duking it out for the ball, and she yells “Get away from my chicken! It’s mine!”

I asked her, had she ever yelled that while she was playing.

“No,” she said. “But I think it.”

No, but I think it.

As parents — and coaches, and teachers — we’re in our children’s heads all the time. We don’t know when or how, but we’re there. The off-hand remarks that we don’t even know they heard; the things we say over and over and over and are damn sure they never hear; the things we say to them when we’re proud, happy, encouraging, loving… and angry, disappointed, tired, mentally somewhere else.

And children have remarkable memories. Yes, they’re amazingly resilient; but they hold on to things, and you never know what’s going to stick.

You can stack the cards in your children’s favor. Encourage positive self talk. When you hear them saying “I can’t do this” or “I’m not good” or “I’m stupid” or “I’m a failure,” don’t let them get away with it. Tell them “Yes you can, and I want you to say that to yourself right now. Let’s say it out loud together.” Teach them the habit of positive self talk.

Don’t just say “Sweetie, you shouldn’t say that” and brush it off. Stop and replace the negative self talk with a habit of positive self talk. You do that over and over again, relentlessly, and that habit will stick.

Yes, it’ll seem silly. Especially to them. That’s OK. Keep doing it.

Sure, as parents, our first charge is to keep our children alive. But life’s about thriving, not just surviving. They will not grow into strong, confident, resilient, kind, loving adults if they’re not first kind to themselves. Teach them how to do that, starting with the language they use to talk to themselves.

2015 Year in review; What’s coming in 2016

By Marc Esher

Before diving head-first in to 2016, I want to look back a year and look forward a bit less than that. It’s said, “The days go by so slowly, and the years go by so fast,” and I see evidence of that when reflecting over a timeline of at least as long as a year

This was my 2015:

Changes at work

Since about mid-2013, at my dayjob I’ve been the nominal, non-supervisory team lead for what was once called our “Release Management” team and is now called “Software Delivery” team. It’s grown considerably since its inception, both in terms of people and breadth of responsibility.

When I took the lead role in 2013, it was a team of 2 and grew to about 5 by the end of the year. At the start of 2015, the team numbered 11 people.

The biggest change for me over the past, say, 1.5 years, has been accepting that successfully leading a team of this size cannot mean “Individual Contributor ++”. I no longer measure my performance by my own productivity, but instead on whether my team members are growing, making meaningful progress on important, fulfilling work in service of organizational and individual goals. It’s been a seismic shift for me, wholly uncomfortable. All growth is.

A correspondent change has been more attention to influence outside my immediate team. It’s difficult without any formal authority, but it’s not impossible. Building relationships with others outside your immediate vertical is critical to enabling your and other teams to doing their best work. Probably not surprising that this has been the greatest source of frustration for me this year; but gains in this area are sometimes the most meaningful. I’ll continue to work to improve here in 2016.

I’m incredibly proud of what we’ve accomplished and excited about what’s coming up.

Books and other learning

The two most meaningful professional books for me this year were Lean Enterprise and Tribal Leadership. I credit Lean Enterprise with helping me rethink how we do projects at work, and it led directly to a significant early-stage pivot on one project (more on that later) that is one of my proudest work-related contributions of 2015. I credit Tribal Leadership with helping me rethink how relationships work on successful teams.

I strongly recommend both.

In addition, I listened to all episodes published up to the end of 2015 of Revolutions Podcast. These episodes covered the English, American, and French revolutions. What a thrill!

Subsequent to the French Revolution, I re-read Charles Dickens’ A Tale of Two Cities, which I encountered first in high school. Fantastic.

Related: this is the 2nd book in 2 years I’ve re-read with a span of 15+ years since the first reading, the other being Siddhartha. I’m discovering that one of the joys of aging is revisiting past treasures but with more experience and, one hopes, wisdom. It’s a blessing to encounter great books when young enough to be wide-eyed about the world and old enough to appreciate them; and a double-blessing to revisit them.

Conferences

For the 3rd year in a row, I failed miserably to spend all of my dayjob’s generous training allowance. Still, I managed two conferences: VelocityConf 2015 in Santa Clara, and DevOpsDays DC.

This is my second VelocityConf, and the content was as expected spectacular. However, I’m starting to question the value proposition for an East Coaster like me to take two 13-hour travel days for a two-day conference. If I attend in the future, I’ll stick with New York.

I took notes on sessions I attended.

DevOpsDays DC was phenomenal. Considering the cost ($100), I can hardly imagine a more valuable 2-day conference. I was kind of dubious about the open spaces, but I was sold after the first session. I met great people and had great conversations about topics I care deeply about. I’m a fan.

Presentations

I started the year with a lightning talk at work on “How Organizations Learn”, with a focus on learning from failure. This presentation is one in a series of years-long, persistent nudges in my organization toward a more deliberate approach to dealing with failure, and I’m proud to see some results. We’ve been doing production incident post-mortems, to some small degree, for a while, but not so much on the larger project level. I’m seeing gains there. My mantra on organizational learning is: “If the lessons aren’t shared, then the organization hasn’t really learned anything”. Another way to phrase it is: “‘I learned’ does not equal ‘we learned'”.

After taking 2 years off from conference presentations, I got back on the horse this year and presented at DevOpsDays DC (Video | Slides/Notes). This presentation was the story of applying some of the lessons from Lean Enterprise to a project at work, mentioned above. I think it’s a good story and I invite you to watch.

Live entertainment

Enough about work, for now.

In 2015, our family took in a fair bit of live entertainment.

We saw a Cinderella ballet and Chinese National Acrobats. For the former, my daughters were freaked out by the tight leotards (kids these days); the latter was a stunning testament to the talent of humans. Jaw-dropping. My 12yo daughter and I saw a Frank Sinatra tribute performance, now that she’s declared herself Sinatra’s biggest fan.

My wife and I saw David Sedaris, who was funny.

We also saw Justin Townes Earle and Bela Fleck & Abigail Washburn. The latter was my musical highlight for the year. Though Bela Fleck is one of the most accomplished banjo players ever, I was blown away by Abigail Washburn. She’s a consummate performer, both in terms of banjo ability and stage presence.

Finally, I crossed seeing Primus off the bucket list. They did about an hour of their classic material and an hour of their new Chocolate Factory stuff. Totally awesome and creepy.

Travel

2015-10-23 06.20.59 — Sunrise from Mt. Haleakala, Maui

This year, my wife and I celebrated 15 years of marriage (yay us), and we honored it with an October trip to Maui for 10 days.

It was, in short, the trip of a lifetime. Aside from simply relaxing and growing closer, we did a bunch of stuff, too: snorkeling, jumping off Black Rock, zip-lining, sunrise on Mt Haleakala and bike-riding from Haleakala to Paia, the Road to Hana, ridiculous and fun dinner theater Luau emceed by a guy I affectionately referred to as Luau Don King.

Cycling

I started and ended the year with cold bike rides, and mostly rode year-found, which was new for me. I put somewhere between 2 and 3k miles on the bike this year, far below what I had hoped but far more than previous years. I did 1 century and a bunch of metric centuries, including 2 self-supported ones which was my big cycling goal for the year.

That said, I was inexplicably beset by thigh cramps on long rides this year, with the severest resulting in total leg lock during one of the metrics. A former Army medic, bless his soul, stopped and helped.

I attribute most of these problems to improper ride nutrition, suboptimal bike fit (now corrected), and some degree of absence of conditioning. In addition, this leads to:

Meditation

I started meditating more regularly in November of 2014, and one of my 2015 goals was to make meditation stick. I did so-so with this until the summer, when the cycling leg cramps led me to modify my meditation routine and practice.

This probably violates all manner of strict meditation rules, but I started using my 20 minutes of meditation for stretching, and since then I’ve both more consistently meditated and haven’t cramped since. Win-win.

I started using Headspace, which I like, with a focus on relationships and kindness. I’d like to think it’s helped me be more empathetic in difficult situations.

So, yeah, I try to spend 20 minutes stretching and thinking about being kind.

Perhaps a day will come when I can sit and meditate for meditation’s sake. Right now, though, the motivator is the feel and physical benefit of stretching.

Plateaus and un-met goals

I fell short in several important-to-me areas for 2015.

First, I hoped to drop about 20 pounds but instead ended up gaining weight.

Second, I didn’t improve my performance at all on the bike, even with the additional miles. This is a total rookie move… it was all riding, but no training.

Third, I made zero progress on the banjo this year. Again, I played, but I didn’t improve.

I have all manner of reasons for this, but when I started to become concerned later in 2015 about my absence of improvement, I focused on one keystone change I could make in my life that would, I hoped, help:

Alcohol-free

In November, 2015, I decided to take an extended break from alcohol. The whys, hows, and consequences are enough for a series of posts. Ultimately, though, it came down to a desire to be more present in my own life.

Spend enough time in tech culture, and you’ll readily observe how alcohol permeates it. I hesitated to even mention it here because, strangely, abstaining can be seen as some weird stigma. But life’s too short for worrying about that, so if I can serve as a model by acknowledging that alcohol wasn’t doing any favors, and I decided to take action, so be it.

2016

Breaking past “Expert Beginner”

When I was young, I spent hours most days riding my bike.

In Will Smith’s words, they were hours and hours and hours of beating on my craft. Learning, executing, and perfecting new tricks is the closest I’ve ever been to pure joy in this life, excepting the birth of my children.

Over the past few years, though, I notice that when it comes to learning, I get stuck at Expert Beginner stage. Or, whether it’s real or imagined, it feels that way. I get far enough with something to be competent at it, and then plateau. I see this manifested directly in how I spend my time, and while I think it’d be worth spending a lot more words in another post on this topic, I’ll sum it up here:

For me, a result of moving from a maker’s schedule to a manager’s schedule has been a near absence of time for getting into flow. My work calendar is this swiss-cheesy thing, with days being broken up between meetings, 1:1s, and small odds and ends, leaving no long, uninterrupted blocks of time for diving deep on any one thing. I’ve referred to it as a “fast food schedule”, because it can be filling but not always nutritious.

I see the most insidious effects of this in my decision-making on how to spend the 30-60 minute chunks of empty space that do come up. Rather than start or continue something difficult and fulfilling, I’ll often choose something that I know I can complete in that time, because it has become emotionally and damn-near-physically painful for me to try to start something and then have it drag out over days and weeks because the schedule does not accommodate significant near-term progress.

Now, apply that to the banjo: given a 30 minute chunk of time at home, should I start learning this new ditty that’s played so fast and whose notes I can barely pick out, knowing full well that to really get it will mean hours spent listening to the song at half-tempo, and then practicing and practicing and practicing until maybe a few weeks or a month from now — in 30 minute increments — it’s going to sound somewhat like it should? Or should I just practice something I already know because at least that feels a bit satisfying?

Spread that kind of decision-making over the course of a year, and it’s easy to see how you can do a thing for a long time without growing at it. It’s an unsatisfying way to live.

I see this pattern too much in my life, and I will break it. I refuse to live a life of the same year of experience, over and over. It starts today.

Discovering what’s next

Not much in life is certain, but here’s something: In 2016, my job is guaranteed to change (this is good!). The change may be small (same role, but official) or significant (new role? new org?). Regardless of outcome, discovering what’s next is exhilarating.

Remember the kindness meditation I talked about way back when? Well… this is one reason I started with that theme. I’ve been bracing myself for this change for quite some time, and come what may, I’ll face it with optimism, grace, and gratitude.

2016 Themes

Rediscover the joy of deep learning and deliberate improvement in all areas of life
Continue building relationships and deriving joy from helping others do kick-ass work
Continue working on being the best dad and husband I can be, which are the greatest privileges of my life

A Learning Approach to processing emails after returning from vacation

By Marc Esher

Anyone who works in an email-reliant organization knows the pain of dealing with emails after a vacation of more than a few days. Hours, perhaps even days, spent dealing with hundreds or thousands of emails that have piled up.

Herein, I’ll describe an approach to processing emails with the goals of:

Catching up on important decisions
Learning about your team
Quickly moving on to more important work

Most email is time-sensitive and will be irrelevant by the time you read it after vacation. Spending hours or days trying to catch up is a fool’s errand, prompted primarily by Fear of Missing Out (FOMO).

My audience here is anyone who is generally relied upon for decision-making, especially team leads and those responsible for other people’s work.

How to process email after vacation

All automated emails (GitHub, Jira, server alerts, most likely anything you have a rule that moves into a separate folder): mark as read. Don’t even click into those folders

Now that you’ve removed what was perhaps >50% of the noise, let’s move on to the important part, wherein you become a human filter for a few minutes.

Next, quickly scan through the rest with the goal of creating 2 buckets:

Important decisions that were made and which you need to be aware of
Decisions not made because they await your input

Just about everything else is junk.

Here’s why: if they’re truly important, they’ll come around again in one form or another, soon. You’ll get another email, or a phone call, or someone will bring it up in your daily standup as a blocker. This is critical: you need to approach this step knowing that really important stuff will announce itself again; otherwise, it’s not important.

I will grant one exception here, and that is if you think there’s an opportunity to learn by detecting patterns for the time you were out. For example, maybe you think you can learn something by scanning your “alerts” folder (if you have such a thing) and find stuff to ask your team about during standup: “I noticed a lot more alerts than normal this week. What happened?”

Important decisions made

This one’s easy. Scan email looking for phrases such as “we decided”, “I decided”, “the decision was made”, “we’re moving forward with”, etc. In addition, before you left for vacation, there were possibly a handful of topics that were approaching decision making, and you’ll notice those from the subject of the email and know to pay attention.

Some decisions will come from above; others will come from your own team. The latter interest me most.

When you find one, decide on the spot:

Do you heartily disagree and think you have any ability to influence an already-made decision? If not, move on
For decisions your team made, reinforce healthy behavior by acknowledging it to the group. Reply immediately, praise, and move on.
I hesitate to even write this, but it’s probably not realistic to omit: If you think this email will take time for you to chew on later, mark it as unread, but know that every time you do this, you’re burdening your future self. Limit this aggressively.

Decisions not made

This is the most interesting and probably most important because it’s a rare chance to learn how your team acts in your absence. This is the part where you focus on learning.

Create your bucket of emails where your team did not make a decision because you were gone. Some of these deferrals will have been the right call, but others are an indication that your team was perhaps not confident, informed, empowered, authorized, or able to make a decision that they otherwise should have been able to make on their own.

You want to dig in here. If you’re going to spend hours or days on post-vacation email, spend it on learning and strengthening your team.

For each of these un-made decisions, figure out what the blocker was on the decision.

Did your team fear consequences if they made the wrong call?
Did your team not have all the information they needed and thus deferred the decision?
Did your team not have physical/virtual access to something (network connectivity, etc) that prevented them from making a call?
Did your team lack authority?

The point here is to get at the nature(s) of absence of decision making. You’re trying to determine ways in which they are too dependent on you, ways in which you’re painfully necessary (not a good thing), and especially ways in which your team has learned helplessness.

It may take a long time to resolve these. But it only takes minutes or a few hours of reading post-vacation email to start the important work of identifying those problems. Then, make a concerted effort to resolve them.

Over time, watch as your team becomes more empowered and less reliant on you. In 6 months, take another vacation, and follow this process again. If you’ve done your job well, you should see an increase in decisions made without you, and a decrease in decisions relying on your input.

Postscript

If you’re thinking “shouldn’t I be doing this all the time anyway?”, the answer is “Of course.” But during vacations, you should be completely inaccessible, and you can really observe your team in action when you’re not present. If you are not completely inaccessible during vacation, that’s a different problem that you need to fix.

On 18F: past and future

By Marc Esher

Disclosures: at the time of this writing 1) I work for the U.S. federal government. 2) several former colleagues currently work at 18F 3) my perspectives are decidedly civil and not defense

18F is a digital services delivery team within the U.S. federal government General Services Administration (GSA), recently celebrating its 1 year anniversary.

Since inception, it’s created or strongly promoted several public applications and initiatives, including Hub, Midas, Analytics, HTTPS initiative, and much more. 18F publicizes its project statuses on a dashboard. I know first hand how hard it can be to get things done in federal government IT, and 18F’s accomplishments in such a short amount of time are quite impressive.

These applications and initiatives are important, and they have been built transparently and in the open. The manner in which this work gets done is also encouraging: agile, perhaps even lean. I don’t see year+-long projects comprising large teams and costing millions of dollars. They appear to be greenfield, relatively small applications that will likely not require considerable resources to maintain over time.

Importantly, we know about all of this because 18Fers are vocal about their work and successes, rightly so. They blog frequently and tweet, a lot. They market and promote their work, and they do so consistently. Considering the government’s general risk aversion, message obsession, and concern about “optics,” 18F’s prolificness in writing about their work is, again, impressive.

I do not know first-hand what it’s like to work at 18F. But it sure does seem like a good place to spend part of a career in technology.

That, right there, is in my estimation the most important thing that 18F has done: 18F has made working in government seem like an attractive option for talented people.

In this regard, they are not the first. The Consumer Financial Protection Bureau has for several years drawn incredibly talented designers and developers through its design and technology fellowship program. Likewise, Presidential Innovation Fellows (out of which 18F was born) and HHS Fellows. Even those more public successes are standing on the shoulder of many giants, the hundreds of thousands of dedicated, thoughtful federal IT workers who for decades have been heads down, gettin’ it done. But right now, 18F are far and away the best marketers.

If 18F were doing all this great work, but not promoting it and using it as a recruiting vehicle to bring more people into the federal government, the 18F organization would not be as important. Their impact on the world is not primarily the dozens of projects they’ve already built and touted publicly. Their effect will be on the talented, dedicated, persistent, optimistic humans they bring into the federal government to work on much harder problems… women and men who otherwise would be working in the private sector.

I suspect most folks coming to work at 18F will be working on applications for other federal agencies. Look at the Dashboard and see the “Partner” label beside the projects. And know this: working in federal government is hard damn work. It’ll be on those partner projects where mettle is tested, and impact is greatest.

It’s in this role in Federal IT — as a project delivery team for other agencies — that I’d like to see even more transparency from 18F, because that’s where its impact and challenge will lie.

Projects such as USCIS, FOIA modernization, and so forth… how are they doing? 18F’s GitHub contains 100+ repositories, but the Dashboard shows only a dozen or so projects. For those projects, what’s it like to work on them? How are these project teams succeeding despite the difficulties of working for federal clients? What innovations are they bringing to bear on these problems, and how are they spreading these innovations across government to enable even more federal IT workers to be successful? What challenges are they facing and how are they overcoming them? How are they exploiting their unique role to multiply effectiveness and joy for other federal workers?

That’s a lot to put on their fledgling organization, I admit. I also know they have recruited amazing people, and they are up to the challenge.

I want to read more blog posts about the really hard projects they’re delivering for other federal agencies. I want to hear about their challenges, their failures, their frustrations, the shitshows — all the gnarly stuff. I want to hear about mistakes, what they’re learning, where they still need help. I want to hear more about the human side of software delivery and how the great people at 18F are getting things done.

Over the past half decade, the winds have slowly been shifting about the idea of the federal government being a great place to build amazing, meaningful, high quality software, particularly in the public-facing space. The Consumer Financial Protection Bureau, with its focus on user experience and design aesthetic, has helped tremendously, and you see similar initiatives and conversations in other federal agencies. Agencies are talking about UX, design, DevOps, agile, lean.

These are necessary, but not sufficient.

Talented people are motivated by challenge. I want to hear more about 18F’s challenges, and I speculate that being as transparent and vocal about those challenges is the next step toward recruiting even more great people into the federal government who otherwise would have gone to work in Silicon Valley or New York unicorn companies.

Happy 1st birthday, 18F. You’ve done well. You’re doing good work. You have amazing people whom I deeply admire. I am excited for your future, and keen to see how the work you’re doing affects the lives of other federal IT workers, both current and future. Your most important work, and most difficult, is yet to come.