Category Archives: Teams

Continuous Delivery, not Continuous Deployment

Engineering teams like Etsy’s have popularized the idea of continuous deployment: infrastructure that automatically rolls-out newly minted code to production in a safe and gradual manner. In Lean Startup and Web Operations, Eric Ries explains the rationale behind continuous deployment: making a safe habit out of shipping product. I loved the motive, but it was clear that the practice as described required heavy operations infrastructure:

  • A continuous deployment server for automatic deploys after successful continuous integration runs
  • Live monitoring to discern if a code change negatively affected business metrics
  • Automatic rollback if business metrics are negatively affected
  • Incremental rollouts to production servers so as not to deploy to all servers at once
  • Code architecture that allows for both old and new code to run in production simultaneously while code rollouts are in-progress
  • Feature switches

While leading a team at LivingSocial, I set out to achieve the goal of safe code shipping as a habit but without the complicated and time-costly infrastructure. We were successful by incorporating good software engineering and deployment practices–all of which were generally good for us and didn’t require as much dedicated tooling or time. Later we discovered others outside the company were starting to do the same under the label “continuous delivery.” We have been even more successful with continuous delivery at LearnZillion, where I am today.

Unfortunately, the cost of continuous deployment infrastructure can discourage engineering teams from investing time in their development and deployment process because they don’t realize the lower-cost alternative, continuous delivery, is also a viable option. I want to share how we do continuous delivery at LearnZillion, so that others can achieve similar results without the overhead of extra infrastructure.

0. Assumptions

I am going to assume the year is 2015, or even 2010 or 2006, and that you have a deployment script or tool like Capistrano to automate the basic deployment steps for your application. As well, I’m going to assume your team or organization wants to do continuous delivery. If neither of these are in-place, start with them.

1. Individual responsibility

Although we work as a team, individuals are responsible for carrying work forward to completion and doing that well. Staff are responsible for taking features from initial definition to production shipment. Along the way, they collaborate with and incorporate input from the broader team and company. (See Multipliers and Drive for reasons to give employees meaningful responsibility in the workplace.)

With these responsibilities come expectations:

Do not break the site. Do not break features. Do not break the test suite. Do not commit code you did not write (this is a smell of a bad development database on your machine, failed merge, etc.). Run the tests regularly–especially before merging into the master branch. If the master branch code changes in-between your test suite run and your re-attempt at commit, run the tests again after cleanup, as appropriate.

Unfortunately, I have found that in many organizations, lack of trust is the default. A tech lead or manager is responsible for scrutinizing code from all team members, merging, deploying, and ensuring the application won’t break. This may make sense for new team members until they understand and are comfortable with the team conventions and have demonstrated that they are capable engineers. Otherwise, it should not be the norm.

2. Smallest overlap of responsibilities

We often pair a product designer (design, UX, HTML/CSS) with a full-stack engineer (SQL, Rails, Ruby, JavaScript, HTML/CSS) to work on a feature. However, we avoid assigning multiple engineers the same feature. We try to keep engineers working on “orthogonal capabilities.” (See “The Three Musketeers” and The Mythical Man Month for the rationale behind this approach.)

3. The master branch is sacred

We deploy to production from our master branch. Developers can depend on master as a reliable foundation to fork, merge, and rebase from. Features are developed, reviewed, and QA-ed in separate branches. If you have test failures, it’s most likely your code. Feature branches are only merged into master immediately before deployment. It is the responsibility of the feature owner to make sure the branch is reasonably current with master before it is merged itself. There are loads of articles on “simple git workflow,” which you can find online, like this one. git and GitHub make this paradigm easy to follow.

4. Follow “The Twelve-Factor App” methodology

I will let the methodology speak for itself. See part X in particular. The biggest continuous delivery benefit is no surprises during deployment.

At LivingSocial, my team ensured the application development environment behaved like production, except where Rails intentionally separates the two. Truth be told, we didn’t have a reliable staging environment at our disposal, so we went straight from development to production. Believe it or not, because of our practices, this still worked quite well.

At LearnZillion, we take this further by using similar SaltStack configurations for production, staging, and a Vagrant-powered development environment. In development, the Ruby process and gems for the app are still installed on the host operating system but everything else runs inside VirtualBox. It has the side benefit of speeding-up the on-boarding process for new engineers.

5. A test suite

At both LivingSocial and LearnZillion, we used Ruby on Rails, which strongly encourages use of a unit testing framework. Engineers make certain to have a passing test suite before merging a branch into master, must have a passing test suite on master post merge, and a failure on the master branch takes top priority–second only to a live site outage.

At LearnZillion we took this farther by integrating CircleCI with GitHub to minimize the execution burden on engineers.

6. An automated QA test suite

At LearnZillion, we have a QA team. They naturally have the potential to be a bottleneck for getting features out. Since quality is their main objective, you want them to be gatekeepers. What you don’t want is for their review and gatekeeping processes to be cumbersome or inefficient. The most powerful lever you can maneuver within your QA team for continuous delivery is to automate their testing. Our team has an extensive QA test suite, which QA engineers can run against any branch, at any time, on a staging server. Automated tests are usually written soon after deployment to production, but sometimes are completed before then. Manual QA of emerging features still takes place, of course.

7. Look at your dashboards

It doesn’t take much effort to have a short list of links to Google Analytics, Mixpanel, or your error reporting service like Bugsnag or Honeybadger. An engineer can inspect them after deploy to see if something broke. Engineers and product designers should be doing this anyway to see how users are responding to changes or new features.

Bonus 1: Manual QA in a different time zone

When an engineer’s code has passed peer review and the automated QA test suite, it is sent along to QA for manual inspection. Test results are back by the next business morning because some of our QA team members are located in India. They test our work while we sleep.

Bonus 2: Continuous QA

At LearnZillion, we’ve integrated a GitHub pull request web hook that deploys a branch to a staging server and runs the QA test suite against it. This means that a branch has been regression tested before it gets to the QA team and usually before it gets to peer review. If you want to read more about our automated QA process, see Kevin Bell’s article about us over at CircleCI.

In Summary

With the good engineering and deployment practices of continuous delivery, you can achieve the same benefit of continuous deployment: safe, consistent delivery of product as a habit. You don’t have to build-out a dedicated infrastructure, and you can build a better engineering team and environment in the process.

Looking for your next gig?

If this sort of engineering environment is appealing to you, and you are interested in being a Senior Software Engineer or Senior Product Designer at LearnZillion, please apply. We would love to meet you.

[Thanks to my team for reviewing this post and recommending improvements to it.]

The 10x Engineer and Delegated Responsibility

Whenever I do an introductory phone call with an engineering candidate, I make sure to explain my management style and how my approach directs our team’s process. Our process is agile, but it is decidedly not a formal Agile methodology. It’s not Agile Scrum; it’s not Extreme Programming; it’s not Kanban. Instead, it’s delegated responsibility in a culture of continuous deployment. I delegate the responsibility of something important to an employee–usually in the form of a significant feature–and let them take it from concept through implementation to deployment.

One of our co-founders serves as our product manager, and we have an experience design team that translates spoken words into diagrams and pictures. However, I make it very clear to my team that any text or visual content they receive are merely representations of product vision. We need them to guide us from here to there. The people on the front-lines–the ones doing the actual building of code and product–are the ones most equipped with information. They face the real constraints of the problem domain and existing code base; they have the best insights into how we can be most economical with their time; and they have the capacity to see all the options before us. I’m there to help them sift through that information, when needed, and to be that supportive coach, but my goal is for them to be carrying us forward. I manage, but I aim to lead, not micro manage.

Delegated responsibility is a very common and efficient practice in the business world. However, the practice has largely been abandoned in the software industry by practices and processes that shift responsibility onto a team of replaceable cogs. The team is expected to churn through a backlog of dozens of insignificantly small bits of larger features, which often lack foresight into the constraints that will be discovered and the interdepencies between smaller bits that result in developer deadlock. On top of this, a generalized backlog of small pieces creates room for misinterpretation by omitting full context around features or results in excessive communication overhead (see The Mythical Man Month).

We are most definitely inspired by Agile. We build a minimum viable product iteratively. We build-measure-learn, pair program when needed, collaborate, peer review each step of the way, and let our QA engineer find our leaky parts. However, my team members are individually responsible for their work and ship whenever they have something ready to show the world.

Some candidates would much rather be working on a team with equally-shared responsibility, collective code ownership, and continuous pair programming. I realize some people need this model, which is why I always discuss it with potential hires. However, others thrive with delegated responsibility. They take ownership, require little to no management or direction, make the right decisions, take pride in what they have built with their own two hands, and are extremely productive. Not surprisingly, others understand their code. It integrates well with the code base. They avoid the dangers that formal methodologies try to curtail. Often they are, or are becoming, that 10x developer. They are liberated, thrilled, and at their best working in this environment. It’s a joy to provide it to them.

If this sort of environment sounds exciting to you, please check out our careers page at LearnZillion.

Pivotal Tracker Dashboard

UPDATE: The script I wrote is no longer needed for the latest release of Pivotal Tracker, as pinned panes persist between browser refreshes. The script below is for the last release of classic Pivotal Tracker on June 20, 2013.

I like Pivotal Tracker. It’s a step-up from the cumbersome ticket tracking systems I’ve used in the past. As a manager though, it’s too cumbersome to see how my individual team members are doing and get an overall picture of how the team is doing at the same time. I’m only given a single backlog, which intermingles everyone’s tickets. I can see all tickets for a single engineer by using search, and I can pin each search results panel to get what I want. But if I reload the page, I lose all my efforts to build a usable dashboard. This is what I’m aiming for but without the hassle: a view of our current sprint (column one), our backlog (column two), our icebox (column three), and each engineer’s backlog (the remaining columns).

Yeah, the screenshot is a bit small here, but I’m not allowed to show you what we’re working on. With a little TamperMonkey grease, you too can have a comprehensive, and persistent dashboard. (GreaseMonkey if you’re still using Firefox.) Here is the script to pull it off. All you have to do is customize the project number and list of engineer name abbreviations.

// ==UserScript==
// @name       Pivot Tracker Dashboard
// @namespace
// @version    1.0
// @description  Show Pivotal Tracker panels for each engineer
// @match
// @copyright  2012, Ian Lotinsky
// ==/UserScript==
function main() {
  setTimeout(function() {
    var devs = 'mms, ay, hkb, bh, js, jw, np'.split(', ');
    for (i = 0; i < devs.length; i++) {
      $('.search .std').attr('value', 'mywork:' + devs[i]);
  }, 2000);

// Source:
var script = document.createElement('script');
script.appendChild(document.createTextNode('('+ main +')();'));
(document.body || document.head || document.documentElement).appendChild(script);

Now, this is a bit of hack, so not everything is roses. You still have to move tickets around in the standard backlog and icebox; you can’t move tickets around inside the developer backlogs since they’re just pinned, search result panels; and you need to refresh your browser from time-to-time to refresh the developer backlogs. It’s not as smooth as a first-class feature, but it gets the job done.

Team Debt

I’m currently having a blast leading the technical team behind the LivingSocial Takeout & Delivery web site. One of the challenges of a growing team is maintaining appropriate amounts of communication. You want everyone to know everything that’s important, but not everything. Otherwise, you end up being a case study in The Mythical Man Month.

Although our team did not follow this plan when it was ramping-up, hindsight reveals the need for a team debt management strategy as it grows. After mulling over it for awhile, I’m fairly sure that if I lead a new team in the future, we will follow this path:

First engineer to join the team

  1. Sets-up the source code repository
  2. Writes a starter project README
  3. Provisions the application and team notification email addresses
  4. Wires-up application notification email(s)

Second engineer

  1. Sets-up the continuous integration (CI) server
  2. Provisions the CI notification email address(es)
  3. Wires-up CI notification emails

Third engineer

  1. Sets-up the team’s Campfire
  2. Wires-up commit and deployment notifications (Campfire and/or email)

Fourth engineer

  1. Sets-up a scrubbed production database dump that engineers can use for local development

What tech team debt tools do you typically employ, and when do you employ them?