The 10x Engineer and Delegated Responsibility

Whenever I do an introductory phone call with an engineering candidate, I make sure to explain my management style and how my approach directs our team’s process. Our process is agile, but it is decidedly not a formal Agile methodology. It’s not Agile Scrum; it’s not Extreme Programming; it’s not Kanban. Instead, it’s delegated responsibility in a culture of continuous deployment. I delegate the responsibility of something important to an employee–usually in the form of a significant feature–and let them take it from concept through implementation to deployment.

One of our co-founders serves as our product manager, and we have an experience design team that translates spoken words into diagrams and pictures. However, I make it very clear to my team that any text or visual content they receive are merely representations of product vision. We need them to guide us from here to there. The people on the front-lines–the ones doing the actual building of code and product–are the ones most equipped with information. They face the real constraints of the problem domain and existing code base; they have the best insights into how we can be most economical with their time; and they have the capacity to see all the options before us. I’m there to help them sift through that information, when needed, and to be that supportive coach, but my goal is for them to be carrying us forward. I manage, but I aim to lead, not micro manage.

Delegated responsibility is a very common and efficient practice in the business world. However, the practice has largely been abandoned in the software industry by practices and processes that shift responsibility onto a team of replaceable cogs. The team is expected to churn through a backlog of dozens of insignificantly small bits of larger features, which often lack foresight into the constraints that will be discovered and the interdepencies between smaller bits that result in developer deadlock. On top of this, a generalized backlog of small pieces creates room for misinterpretation by omitting full context around features or results in excessive communication overhead (see The Mythical Man Month).

We are most definitely inspired by Agile. We build a minimum viable product iteratively. We build-measure-learn, pair program when needed, collaborate, peer review each step of the way, and let our QA engineer find our leaky parts. However, my team members are individually responsible for their work and ship whenever they have something ready to show the world.

Some candidates would much rather be working on a team with equally-shared responsibility, collective code ownership, and continuous pair programming. I realize some people need this model, which is why I always discuss it with potential hires. However, others thrive with delegated responsibility. They take ownership, require little to no management or direction, make the right decisions, take pride in what they have built with their own two hands, and are extremely productive. Not surprisingly, others understand their code. It integrates well with the code base. They avoid the dangers that formal methodologies try to curtail. Often they are, or are becoming, that 10x developer. They are liberated, thrilled, and at their best working in this environment. It’s a joy to provide it to them.

If this sort of environment sounds exciting to you, please check out our careers page at LearnZillion.

Advertisements

Document the Why

Like many coders, I am a proponent of writing self-documenting code. The more I have worked with intentional code that omits unnecessary or misleading comments, the more efficient I have been as a software engineer. I read clear code and I understand what is going on.

However, regardless of whether I have spelunked self-documenting code or code with a girth of extraneous comments, both styles often omit why something is being done when it’s not obvious. Maybe an API you’re interacting with has a bug, so you have to do things in a roundabout or undocumented way. Maybe there is a non-obvious edge case that your main conditional branch code covers that another programmer would expect to be solved in a more conventional manner. Maybe you made a calculated business decision for a particular user experience when other sites behave differently.

Make it obvious why a non-obvious approach was taken. Save your fellow engineers and your future self from re-exploring the explored, re-arguing the argued, and re-deciding the decided.

Google Analytics Crash Course Notes

Thinking that you will adequately learn Google Analytics by clicking around the product, even over years, is a foolish concept. You will only understand a subset of its features and how they work together. You need to do your homework.

I cannot improve upon Google Analytics’ (GA) own crash course, titled Google Analytics IQ Lessons. It covers just about all the material in the paid Analytics courses (101, 201, and 301) at just the right level–not too high, and not too deep.

Here are my notes of the key gotcha’s and items to configure for your GA Web Properties. As well, I’ve linked to other helpful learning resources. As is my standard practice, this is mostly for my reference down-the-road, so it’s not comprehensive. However, I figure others can benefit from them as well.

Gotcha’s

  • Incognito mode and other browser privacy sessions count as new Visitors, Visits, and Page Views, as if the user had cleared his cookies. Not a huge surprise to most, I’m sure. (Although other trackers can still track you.)
  • Visits are separated by exits from the site or a 30-minute cookie timeout while on the site. Advertising Campaign attribution expires after a 6-month cookie timeout. Both are customizable.
  • Time on Exit Pages is not tracked because time is calculated between page loads on the same site. This also means that Bounce Page time is not tracked either. This has serious implications for some genres of sites, like blogs where a bit of traffic goes into and out of a single article. Know how to track Exit Page times and Bounce Pages, if you need to.
  • A Visitor can only trigger a Goal conversion once during a Visit, but can trigger an E-commerce Goal multiple times in a Visit.
  • Filters are applied between raw data capture and the Account’s Profile where the data is ultimately stored. Even if you change a Filter that sits in-between the raw feed and Profile, you cannot recover historical data. Try accomplishing the same filtering with Advanced Segments instead, which don’t run the risk of losing data. At the least, you should use Advanced Segments or other features to test concepts before creating a real Filter for them.
  • Domains and subdomains can break tracking in many glorious ways–especially E-commerce Goal tracking.

Basic Checklist

  • Always have a raw Profile that has no Filters, Advanced Segments, etc.
  • Have a Profile that excludes internal IP address so you’re not tracking yourself and your staff as they click around your site.
  • Have a Profile that exclusively tracks internal IP addresses for debugging Google Analytics code on your site.
  • Use the Google Analytics Debugger Chrome Extension for your own debugging and analysis of competitors’ tracking.
  • Enable Auto-Tagging between Google AdWords and Analytics if you are using both products.
  • If you create an AdWords Profile, set up two Filters to focus-in on AdWords traffic (Campaign Source: google, Campaign Medium: cpc).
  • Set up E-commerce tracking.
  • Set up Goal tracking.
  • Set up Internal Site Search tracking. (It’s much easier than you think.)
  • Utilize _addIgnoredOrganic to attribute Organic Search Visits for your web site’s address (i.e. someone searching for “example.com”) to a Direct Visit instead.
  • Set up appropriate Custom Variables to track additional information about Visitors, Visits, and Page Views.

Hopefully, all these will help us do a better job optimizing our customer average lifetime value (LTV).

Pivotal Tracker Dashboard

UPDATE: The script I wrote is no longer needed for the latest release of Pivotal Tracker, as pinned panes persist between browser refreshes. The script below is for the last release of classic Pivotal Tracker on June 20, 2013.

I like Pivotal Tracker. It’s a step-up from the cumbersome ticket tracking systems I’ve used in the past. As a manager though, it’s too cumbersome to see how my individual team members are doing and get an overall picture of how the team is doing at the same time. I’m only given a single backlog, which intermingles everyone’s tickets. I can see all tickets for a single engineer by using search, and I can pin each search results panel to get what I want. But if I reload the page, I lose all my efforts to build a usable dashboard. This is what I’m aiming for but without the hassle: a view of our current sprint (column one), our backlog (column two), our icebox (column three), and each engineer’s backlog (the remaining columns).

Yeah, the screenshot is a bit small here, but I’m not allowed to show you what we’re working on. With a little TamperMonkey grease, you too can have a comprehensive, and persistent dashboard. (GreaseMonkey if you’re still using Firefox.) Here is the script to pull it off. All you have to do is customize the project number and list of engineer name abbreviations.

// ==UserScript==
// @name       Pivot Tracker Dashboard
// @namespace  https://www.pivotaltracker.com/s/projects/
// @version    1.0
// @description  Show Pivotal Tracker panels for each engineer
// @match      https://www.pivotaltracker.com/s/projects/453961
// @copyright  2012, Ian Lotinsky
// ==/UserScript==
function main() {
  setTimeout(function() {
    var devs = 'mms, ay, hkb, bh, js, jw, np'.split(', ');
    for (i = 0; i < devs.length; i++) {
      $('.search .std').attr('value', 'mywork:' + devs[i]);
      $('#search_button').click();
      $('.pin_search').last().click();
    }
    $('searchString').clear();
  }, 2000);
}

// Source: http://stackoverflow.com/questions/2303147/injecting-js-functions-into-the-page-from-a-greasemonkey-script-on-chrome
var script = document.createElement('script');
script.appendChild(document.createTextNode('('+ main +')();'));
(document.body || document.head || document.documentElement).appendChild(script);

Now, this is a bit of hack, so not everything is roses. You still have to move tickets around in the standard backlog and icebox; you can’t move tickets around inside the developer backlogs since they’re just pinned, search result panels; and you need to refresh your browser from time-to-time to refresh the developer backlogs. It’s not as smooth as a first-class feature, but it gets the job done.

Time Zones, Ruby on Rails, and MySQL


Ad: Need an Application Security Audit or Training for your Engineering Team?

If you are looking for a firm to do an application security audit for your web or mobile application or to train your software engineers on writing secure code, I highly recommend nVisium Security. (Tell them Ian Lotinsky sent you.)


The Blog Post

A few months ago, I had the privilege of taking a Rails 3 application national. One of the requirements for doing so was ensuring that it supported each of the four time zones in the contiguous United States of America. Just switching the application from EST to UTC and using JavaScript to render times to local time would have been ideal. However, there were several API, native application, and legacy code constraints that prevented us from using that trick. We needed the entire request/response cycle to be in the local time zone.

While doing my research, I was unable to find a single source of reference material that thoroughly explained how a standard Ruby on Rails stack dealt with time zones and how an engineer could leverage it. This post is meant to serve those who find themselves in a similar position.

MySQL

The Basics

Before we dive into Ruby on Rails, we need to start at the foundation: how MySQL stores and retrieves dates and times.

  1. DATE, TIME, and other date and time column types are time zone agnostic.1 Even worse, we can store invalid dates and times in them. Think of them as specially-formatted string types.
  2. The MySQL system variable time_zone is used internally by functions like NOW() to tell MySQL what time zone it’s in so it knows what hour to output. The variable defaults to the value SYSTEM, but it can be set to any specific time zone. When its value is SYSTEM, any functions that rely on it fall back to system_time_zone for that directive instead.
  3. system_time_zone is the time zone of the database’s operating system. If our ops team knows what it’s doing, all our servers are set to UTC. However, we can’t assume that’s true or will always be true. Besides, each developer on our team could have his or her laptop set to a different local time zone.

Let’s Standardize

Rails itself doesn’t require any sort of time zone setting in MySQL because it just looks at the data like it’s a string. However, we want to stay sane when comparing data or inspecting the results of query execution in development versus production. It would be good to standardize.

To get every system on the same page, we go to each machine (laptops and servers) and set the MySQL time zone in /etc/my.cnf2 to UTC:

  [mysqld]
  default-time-zone = '+00:00'

This requires the MySQL server to be restarted in each of those environments. We can double check our work after restarting by running the following query:

  show variables like "%time_zone"

Note that system_time_zone will be the zone of the operating system, but time_zone is going to be +00:00 regardless. If we run SELECT NOW(); on any machine with this configuration, we’ll get the same, consistent time in UTC—of course, offset by any delay we introduce when switching between each MySQL console.

Implications and Further Reading

The only downside to all this is that we have to start thinking in UTC when developing. Well, time to level-up, Mr. Engineer.

If you want more information on MySQL’s handling of dates, times, and zones, let me recommend the MySQL documentation as well as Sheeri Cabral’s video Time Zones in MySQL.

Ruby (sprinkled with ActiveSupport)

Ruby itself does not have time zone support. Everything is in system time. Rails strengthens Ruby’s time related classes with time zones (ActiveSupport::TimeWithZone). The result of this combo is:

  1. Each Ruby thread keeps track of the time zone in Time.zone. Just like MySQL, Ruby defaults to the system time zone.
  2. We can change the time zone of the current Ruby thread via Time.zone = 'Eastern Time (US & Canada)' thanks to ActiveSupport.
  3. We can shift a Time instance to another time zone via #in_time_zone('Pacific Time (US & Canada)'). We omit the parameter if we want to convert a Time instance to the time zone of the current Ruby thread (#in_time_zone).
  4. Time.now will always return the current time in the system time zone regardless of the time zone setting (bleh). So, instead of using Time.now, we have to use Time.current. (Time.current itself actually calls Time.now.in_time_zone.)
  5. Time.parse shares the same fate. Instead, we use Time.parse('2012-1-1').in_time_zone('Pacific Time (US & Canada)').

Rails

Rails 3 gets with the program and acknowledges that any real app is going to want to run in UTC.

ActiveRecord

  1. ActiveRecord has a time zone setting. Any dates or times that we assign to a model instance get auto-converted to that zone when being stored to the database, and back again when read from it. Smart.
  2. The standard ActiveRecord config defaults to :local, which is the system time zone. However the ActiveRecord railtie sets it to :utc. A tad confusing, but the result in our Rails app is UTC out of the box.

ActionPack

  1. application.rb has a config.time_zone setting. This is the time zone of any request/response thread and does not affect what’s stored through ActiveRecord. The default is UTC, but we can set it to something else if we want.

What to Do

All examples I’ve seen online recommend setting the request/response thread’s time zone in a before_filter via something like Time.zone = current_user.time_zone. The problem with this approach is that we need to make sure every request goes through that filter and that we recover from any exceptions that happen during that request/response cycle to set the time zone back to UTC appropriately. We could accomplish this by using an around_filter with an ensure so that subsequent requests start from a good time zone baseline of UTC3:

  around_filter :reset_timezone_to_utc
  before_filter :set_timezone_to_user

  def reset_timezone_to_utc
    yield
  ensure
    Time.zone = Time.zone_default
  end

  def set_timezone_to_user
    Time.zone = current_user.time_zone
  end

Although this looks good on a blog, making all requests go through the same time zone logic in a larger Rails app is unrealistic—believe me. Not every action in your application is going to be user-time-zone-centric. Instead, I recommend running everything in UTC and converting to the appropriate time zone each place you need to display a date or time in a view or mailer. For example, in one place we might have:

  some_order.created_at.in_time_zone(some_order.purchaser.time_zone)

and in a different view of the same record somewhere else in our application, we may want:

  some_order.created_at.in_time_zone(some_order.seller.time_zone)

Queries

If any of our queries, whether MySQL (or Sphinx), need to utilize time zones and their offsets, we’re going to need a table of time zones in our database. MySQL has a way to load TZInfo into special tables. However, there are a few problems with this approach:

  1. It’s an unusual MySQL feature that we would have to wrap our brain’s around and translate between MySQL and Ruby on Rails.
  2. It requires up-to-date TZInfo to be regularly loaded into our database server.
  3. It requires calculations of offsets for Daylight Savings at query execution time.

There is a more Railsy way of accomplishing the same goal: build our own time zone table, and fill it with time zone info from ActiveSupport::TimeZone.all.

class AddTimezones < ActiveRecord::Migration
  def self.up
    create_table :timezones do |t|
      t.string :name, :null => false
      t.string :offset, :null => false
      t.timestamps
    end

    add_index :timezones, :name, :unique => true
  end

  def self.down
    drop_table :timezones
  end
end

We add a time zone column to each model that needs it.

class AddTimezoneToSellers < ActiveRecord::Migration
  def self.up
    add_column :sellers, :timezone, :string, :null => false
  end

  def self.down
    remove_column :sellers, :timezone
  end
end

We associate records by time zone name and not ID because:

  1. We’re going to be regularly updating the time zone table by time zone name anyway.
  2. When using Rails’ #in_time_zone method, we just need to pass the time zone name in. There’s no point to joining on another table just to get a string back.

And, finally, we’re going to keep the time zone table up-to-date each morning.

class Timezone < ActiveRecord::Base
  # This model is really just for filling/updating the timezones table for MySQL and Sphinx
  # I tried removing the ID column and making `name` the primary key, but ran into an obscure Rails `schema.rb` generation bug
  def self.reload_tzinfo_into_database
    ActiveSupport::TimeZone.all.map do |tz|
      tz_record = Timezone.find_or_initialize_by_name(:name => tz.name)
      tz_record.offset = Time.now.in_time_zone(tz.name).formatted_offset
      tz_record.save!
    end
  end
end

With the following rake task:

namespace :timezones do
  desc 'Creates and/or updates timezones records with time zones and their current UTC formatted offsets'
  task :reload => :environment do
    ActiveSupport::TimeZone.all.map do |tz|
      tz_record = Timezone.find_or_initialize_by_name(:name => tz.name)
      tz_record.offset = Time.now.in_time_zone(tz.name).formatted_offset
      tz_record.save!
    end
  end
end

And then we set up the rake task as a daily cron job on our server.

The End

Setting up a Ruby on Rails application and MySQL (and Sphinx) to support multiple times zones is quite the journey. I’ve tried my best to take good notes, but if you spot something funny, feel free to leave a comment. Just don’t be a troll. Thanks.

Footnotes

1Except TIMESTAMP, which is stored as UTC and shifted when queried. TIMESTAMP is a MySQL-proprietary column type.
2my.cnf may be somewhere else if you have a different installation of MySQL than I do.
3Hat tip to Nathan Herald for the around filter idea.

Gemfiles and GitHub Enterprise

It took me about 30 minutes to figure out how to point a Gemfile to a GitHub Enterprise repository, in a GitHub organization, in a branch. Here’s the solution I whipped-up:

gem 'mygem', :git => "http://username:password@ourgithub.ourdomain.com/organization/mygem.git", :branch => "bugfix/mybranch"

However, do not check this in to your repo. Your username/password is in plain text.

If anyone has a better suggestion, please post it in the comments. Thanks.

Team Debt

I’m currently having a blast leading the technical team behind the LivingSocial Takeout & Delivery web site. One of the challenges of a growing team is maintaining appropriate amounts of communication. You want everyone to know everything that’s important, but not everything. Otherwise, you end up being a case study in The Mythical Man Month.

Although our team did not follow this plan when it was ramping-up, hindsight reveals the need for a team debt management strategy as it grows. After mulling over it for awhile, I’m fairly sure that if I lead a new team in the future, we will follow this path:

First engineer to join the team

  1. Sets-up the source code repository
  2. Writes a starter project README
  3. Provisions the application and team notification email addresses
  4. Wires-up application notification email(s)

Second engineer

  1. Sets-up the continuous integration (CI) server
  2. Provisions the CI notification email address(es)
  3. Wires-up CI notification emails

Third engineer

  1. Sets-up the team’s Campfire
  2. Wires-up commit and deployment notifications (Campfire and/or email)

Fourth engineer

  1. Sets-up a scrubbed production database dump that engineers can use for local development

What tech team debt tools do you typically employ, and when do you employ them?