Category Archives: Ruby on Rails

Time Zones, Ruby on Rails, and MySQL


Ad: Need an Application Security Audit or Training for your Engineering Team?

If you are looking for a firm to do an application security audit for your web or mobile application or to train your software engineers on writing secure code, I highly recommend nVisium Security. (Tell them Ian Lotinsky sent you.)


The Blog Post

A few months ago, I had the privilege of taking a Rails 3 application national. One of the requirements for doing so was ensuring that it supported each of the four time zones in the contiguous United States of America. Just switching the application from EST to UTC and using JavaScript to render times to local time would have been ideal. However, there were several API, native application, and legacy code constraints that prevented us from using that trick. We needed the entire request/response cycle to be in the local time zone.

While doing my research, I was unable to find a single source of reference material that thoroughly explained how a standard Ruby on Rails stack dealt with time zones and how an engineer could leverage it. This post is meant to serve those who find themselves in a similar position.

MySQL

The Basics

Before we dive into Ruby on Rails, we need to start at the foundation: how MySQL stores and retrieves dates and times.

  1. DATE, TIME, and other date and time column types are time zone agnostic.1 Even worse, we can store invalid dates and times in them. Think of them as specially-formatted string types.
  2. The MySQL system variable time_zone is used internally by functions like NOW() to tell MySQL what time zone it’s in so it knows what hour to output. The variable defaults to the value SYSTEM, but it can be set to any specific time zone. When its value is SYSTEM, any functions that rely on it fall back to system_time_zone for that directive instead.
  3. system_time_zone is the time zone of the database’s operating system. If our ops team knows what it’s doing, all our servers are set to UTC. However, we can’t assume that’s true or will always be true. Besides, each developer on our team could have his or her laptop set to a different local time zone.

Let’s Standardize

Rails itself doesn’t require any sort of time zone setting in MySQL because it just looks at the data like it’s a string. However, we want to stay sane when comparing data or inspecting the results of query execution in development versus production. It would be good to standardize.

To get every system on the same page, we go to each machine (laptops and servers) and set the MySQL time zone in /etc/my.cnf2 to UTC:

  [mysqld]
  default-time-zone = '+00:00'

This requires the MySQL server to be restarted in each of those environments. We can double check our work after restarting by running the following query:

  show variables like "%time_zone"

Note that system_time_zone will be the zone of the operating system, but time_zone is going to be +00:00 regardless. If we run SELECT NOW(); on any machine with this configuration, we’ll get the same, consistent time in UTC—of course, offset by any delay we introduce when switching between each MySQL console.

Implications and Further Reading

The only downside to all this is that we have to start thinking in UTC when developing. Well, time to level-up, Mr. Engineer.

If you want more information on MySQL’s handling of dates, times, and zones, let me recommend the MySQL documentation as well as Sheeri Cabral’s video Time Zones in MySQL.

Ruby (sprinkled with ActiveSupport)

Ruby itself does not have time zone support. Everything is in system time. Rails strengthens Ruby’s time related classes with time zones (ActiveSupport::TimeWithZone). The result of this combo is:

  1. Each Ruby thread keeps track of the time zone in Time.zone. Just like MySQL, Ruby defaults to the system time zone.
  2. We can change the time zone of the current Ruby thread via Time.zone = 'Eastern Time (US & Canada)' thanks to ActiveSupport.
  3. We can shift a Time instance to another time zone via #in_time_zone('Pacific Time (US & Canada)'). We omit the parameter if we want to convert a Time instance to the time zone of the current Ruby thread (#in_time_zone).
  4. Time.now will always return the current time in the system time zone regardless of the time zone setting (bleh). So, instead of using Time.now, we have to use Time.current. (Time.current itself actually calls Time.now.in_time_zone.)
  5. Time.parse shares the same fate. Instead, we use Time.parse('2012-1-1').in_time_zone('Pacific Time (US & Canada)').

Rails

Rails 3 gets with the program and acknowledges that any real app is going to want to run in UTC.

ActiveRecord

  1. ActiveRecord has a time zone setting. Any dates or times that we assign to a model instance get auto-converted to that zone when being stored to the database, and back again when read from it. Smart.
  2. The standard ActiveRecord config defaults to :local, which is the system time zone. However the ActiveRecord railtie sets it to :utc. A tad confusing, but the result in our Rails app is UTC out of the box.

ActionPack

  1. application.rb has a config.time_zone setting. This is the time zone of any request/response thread and does not affect what’s stored through ActiveRecord. The default is UTC, but we can set it to something else if we want.

What to Do

All examples I’ve seen online recommend setting the request/response thread’s time zone in a before_filter via something like Time.zone = current_user.time_zone. The problem with this approach is that we need to make sure every request goes through that filter and that we recover from any exceptions that happen during that request/response cycle to set the time zone back to UTC appropriately. We could accomplish this by using an around_filter with an ensure so that subsequent requests start from a good time zone baseline of UTC3:

  around_filter :reset_timezone_to_utc
  before_filter :set_timezone_to_user

  def reset_timezone_to_utc
    yield
  ensure
    Time.zone = Time.zone_default
  end

  def set_timezone_to_user
    Time.zone = current_user.time_zone
  end

Although this looks good on a blog, making all requests go through the same time zone logic in a larger Rails app is unrealistic—believe me. Not every action in your application is going to be user-time-zone-centric. Instead, I recommend running everything in UTC and converting to the appropriate time zone each place you need to display a date or time in a view or mailer. For example, in one place we might have:

  some_order.created_at.in_time_zone(some_order.purchaser.time_zone)

and in a different view of the same record somewhere else in our application, we may want:

  some_order.created_at.in_time_zone(some_order.seller.time_zone)

Queries

If any of our queries, whether MySQL (or Sphinx), need to utilize time zones and their offsets, we’re going to need a table of time zones in our database. MySQL has a way to load TZInfo into special tables. However, there are a few problems with this approach:

  1. It’s an unusual MySQL feature that we would have to wrap our brain’s around and translate between MySQL and Ruby on Rails.
  2. It requires up-to-date TZInfo to be regularly loaded into our database server.
  3. It requires calculations of offsets for Daylight Savings at query execution time.

There is a more Railsy way of accomplishing the same goal: build our own time zone table, and fill it with time zone info from ActiveSupport::TimeZone.all.

class AddTimezones < ActiveRecord::Migration
  def self.up
    create_table :timezones do |t|
      t.string :name, :null => false
      t.string :offset, :null => false
      t.timestamps
    end

    add_index :timezones, :name, :unique => true
  end

  def self.down
    drop_table :timezones
  end
end

We add a time zone column to each model that needs it.

class AddTimezoneToSellers < ActiveRecord::Migration
  def self.up
    add_column :sellers, :timezone, :string, :null => false
  end

  def self.down
    remove_column :sellers, :timezone
  end
end

We associate records by time zone name and not ID because:

  1. We’re going to be regularly updating the time zone table by time zone name anyway.
  2. When using Rails’ #in_time_zone method, we just need to pass the time zone name in. There’s no point to joining on another table just to get a string back.

And, finally, we’re going to keep the time zone table up-to-date each morning.

class Timezone < ActiveRecord::Base
  # This model is really just for filling/updating the timezones table for MySQL and Sphinx
  # I tried removing the ID column and making `name` the primary key, but ran into an obscure Rails `schema.rb` generation bug
  def self.reload_tzinfo_into_database
    ActiveSupport::TimeZone.all.map do |tz|
      tz_record = Timezone.find_or_initialize_by_name(:name => tz.name)
      tz_record.offset = Time.now.in_time_zone(tz.name).formatted_offset
      tz_record.save!
    end
  end
end

With the following rake task:

namespace :timezones do
  desc 'Creates and/or updates timezones records with time zones and their current UTC formatted offsets'
  task :reload => :environment do
    ActiveSupport::TimeZone.all.map do |tz|
      tz_record = Timezone.find_or_initialize_by_name(:name => tz.name)
      tz_record.offset = Time.now.in_time_zone(tz.name).formatted_offset
      tz_record.save!
    end
  end
end

And then we set up the rake task as a daily cron job on our server.

The End

Setting up a Ruby on Rails application and MySQL (and Sphinx) to support multiple times zones is quite the journey. I’ve tried my best to take good notes, but if you spot something funny, feel free to leave a comment. Just don’t be a troll. Thanks.

Footnotes

1Except TIMESTAMP, which is stored as UTC and shifted when queried. TIMESTAMP is a MySQL-proprietary column type.
2my.cnf may be somewhere else if you have a different installation of MySQL than I do.
3Hat tip to Nathan Herald for the around filter idea.

Advertisements

strip_attributes, Rails 3, shoulda 2.11 hack

I have never understood why Rails doesn’t strip attributes by default. I know at least one person who tried committing it to core, only to have it rejected. I always end up installing the strip_attributes plugin.

I’m ramping-up a new Rails 3 project, with Shoulda 2.11. I installed strip_attributes. It works, but the strip_attributes Shoulda macros don’t work anymore. I could take the route of upgrading the plugin, but then I would “have to” also refactor the Shoulda macros since macros have been ditched for matchers. Then I’d have to re-write the tests. At that point, I might as well make it a gem and add that feature I’ve always wanted. But, I’m just not ready for that sort of commitment right now (#gtd_maybe_someday).

So, here’s a short line to add to the end of strip_attributes/init.rb to get the old strip_attributes Shoulda macros working again:

require File.expand_path(File.join(File.dirname(__FILE__), 'shoulda_macros', 'macros.rb')) if Rails.env.test?

SSL in Ruby on Rails


Ad: Need an Application Security Audit or Training for your Engineering Team?

If you are looking for a firm to do an application security audit for your web or mobile application or to train your software engineers on writing secure code, I highly recommend nVisium Security. (Tell them Ian Lotinsky sent you.)


Update

The world has shifted since I wrote this solution and blog post. The popularization of Firesheep and concerted efforts in understanding and optimizing SSL performance has led to the wise trend of enforcing HTTPS everywhere. There are much better (and simpler) ways to secure Rails applications. The one aspect of my solution that I still recommend incorporating is logging non-GET or HEAD requests so you can identify any forms that are misbehaving. Reader beware…


I was supposed to simply verify that everything in a Rails 2 application was properly using SSL to serve pages and receive requests over HTTPS. The web server had already been configured with the appropriate SSL certs, and developers had been using different constructs in Rails to force certain actions to be protected. There was no reason to believe things were broken or spewing secrets; I just needed to dot the I’s and cross the T’s. Specifically, I needed to verify that:

  1. Pages that needed the familiar SSL security lock icon showed it in the browser.
  2. That any such pages’ forms actually POSTed over HTTPS.
  3. That the margin for developer coding error was slim.
  4. And that we did all these without hurting developers’ brains. This meant ensuring things were DRY and as simple to use as possible.

Being my first experience validating usage of SSL in a Rails application, I set out to read-up on the topic. To my surprise, I discovered that there are multiple ways of using SSL in Rails. To make matters worse, the application I was verifying used all three of them equally, none with conviction, and this mix gave us a false sense of assurance that everything was correct and safe, when in fact it wasn’t. The core of how we utilized SSL in Rails needed to be re-written.

The Options

The popular ways of requiring SSL in Rails are:

  1. Passing in :protocol => 'https' in calls to *_url helpers when generating links and forms that should be GET or POSTed over HTTPS.
  2. Specifying :requirements => { :protocol => 'https' } in specific routes in routes.rb to accomplish the same thing, but in a central place.
  3. Using DHH’s ssl_requirement plugin.

(I suggest reading the pages at the links above before continuing. SSL in Rails isn’t cake and understanding each will help you follow the rest of my article.) Using option 1 exclusively will quickly fill your views and helpers with extraneous code and your ERB output with absolute URLs, which will pack more bytes in your pages than relative ones. You will also have to make sure you’re specifying HTTPS in each URL helper call for that URL, which is not DRY. However, that’s not the worst of it: it will do nothing on the controller-side to ensure communication between browser and server actually took place over HTTPS.

    <% form_tag(:method => 'post', create_user_url(:protocol => 'https')) do %>

    def create
      # I'll take HTTPS. Heck, I'll take HTTP too!
    end

It’s way too easy to create an HTTP URL to the same action without even thinking.

    <% form_tag(:method => 'post', create_user_url) do %>

Using option 2 will take care of centralized HTTPS requirements for helpers, but it will make your routes messy because you have to require SSL for an entire resource or break out actions on resources that require SSL into their own route lines.

    map.resources :users, :only => [:new, :create], :requirements => { :protocol => 'https' }
    map.resources :users, :only => [:show]

This gets particularly messy when you have nested resources. This also does nothing on the controller-side to ensure SSL actually took place. It only helps in generating SSL paths and URLs like option 1. Option 3 is the best because it not only ensures SSL actually took place on the controller-side, it also allows you to more-DRYly define SSL requirements in one-place: the controller because of redirects.

    class UsersController < ApplicationController
      ssl_required :new, :create
      ...
    end

A before_filter ensures the proper protocol is used for an action. If the right protocol is used, the action executes as expected. If the wrong protocol is used, Rails tells the browser to try the request again over the proper protocol. This makes your life a lot easier and better by requiring less code and producing thinner ERB output. How? First, you can completely ignore specifying HTTPS in helpers to URLs for pages that need to have the security lock icon. When fetched with HTTP, the server will tell the browser to request it again, but this time over HTTPS. Second, all of your URLs can be generated by *_path helpers instead of *_url helpers. If the origin page is HTTP, the relative URL will be HTTP. If the origin page is HTTPS, the relative URL will be HTTPS. In either case, ssl_requirement will ensure the right protocol is used.

    def UsersController < ApplicationController
      ssl_required :new, :create
      ...
    end

    def new
      # I require HTTPS. If you try to GET to me over HTTP, ssl_requirement will redirect you to the HTTPS URL of this request.
    end

    <% form_tag(:method => 'post', create_user_url) do %><%# This will produce a relative path, which will resolve to HTTPS, because :new was fetched or redirected to HTTPS. %>

    def create
      # I require HTTPS. If you hack your browser and try to POST to me over HTTP, ssl_requirement will redirect you to POST again to the HTTPS URL.
    end

So, specifying both the origin and the target pages in your calls to ssl_required, and ssl_requirement will take care of you.

My Mods

Now, as you can see, ssl_requirement works the same for form requests too. However, do you really want a form POSTed over HTTP to redirect to HTTPS? Most browsers will warn a user that they are about to make a request from a secure page to an insecure URL. And yet you’ll never know this is happened unless you are watching your Rails logs like a hawk. Even if you did, do you really want your application to let that happen? It makes for a very bad user experience. On top of that, even though the browser will try again over SSL, any sensitive data will have already been transmitted insecurely over HTTP. By that point, who cares if the data is resent securely? The secret has already been exposed! You never want your application to let a user do this. To ensure your code never has a bug that allows for this to happen, you can tweak ssl_requirement by overriding ensure_proper_protocol such that it raises an exception if anyone performs on action over HTTP that requires HTTPS. (I’m assuming you’re watching your application exceptions. You should be.) If someone does try to request an HTTPS action over HTTP, and exception is thrown, and now you know you missed something.

    class ApplicationController < ActionController::Base
      ...
      protected
        def ensure_proper_protocol
          if !ssl_allowed? && ssl_required? && !request.ssl? && !(request.get? || request.head?)
            raise 'SSL required!' # either we have a bug somewhere or someone is playing with us.
          else
            super
          end
        end
    end

You don’t throw an exception for GETs or HEADs because you rely on ssl_requirement to redirect them to HTTPS so you can DRYly define SSL requirements in controllers only. If exceptions gets out-of-control in your production environment because someone is playing with you or has a misbehaving browser, you can always switch to logging the issue and redirect the user to a 404 page or your home page. Just watch your logs! Now, you might be even lazier, like me, and not have SSL setup on your laptop for development. To get around setting it up, you can also override ssl_required? for the development and local test environments so that ssl_requirement doesn’t function. Although, you probably want it in integration tests, because integration tests can simulate SSL. To use it there, you can use an ENV variable and temporarily toggle it in setup and teardown so that ssl_requirement functions.

    environment.rb
      ENV['SSL_PROTOCOL'] = 'https'

    development.rb
      ENV['SSL_PROTOCOL'] = 'http'

    test.rb
      ENV['SSL_PROTOCOL'] = 'http'

    class ApplicationController < ActionController::Base
      ...
      private
        def ssl_required?
          return false if ENV['SSL_PROTOCOL'] == 'http'
          super
        end
    end

    an_integration_test_that_wants_to_test_the_new_and_improved_ensure_proper_protocol.rb
      def setup
        ENV['SSL_PROTOCOL'] = 'https'
      end

      def teardown
        ENV['SSL_PROTOCOL'] = 'http'
      end

      def test_that_exception_is_raised
        post '/users', {:user => {:name => 'Natalia Rose'}}
        assert_response :error, 'SSL required!'
      end

Caveats

Query Strings

Okay, I’m assuming here that you don’t want to put secure data in a GET or HEAD request query string. You can’t use the DRY technique of omitting HTTPS in your helper calls and only putting them in your controllers if you need to encrypt query strings. The reason I make this assumption is because it’s not the best idea. The string will show up in all sorts of server logs unencrypted without some extra configuration. Even if you do configure them, it will still show up in users’ browser histories too. The better option is to use POSTs to send params instead of GETs.

Explicit HTTP to HTTPS POSTS

In a few instances, you might need to POST from an insecure page to a secure one. (This should be rare.) If you do, you can use option 1 (use a *_url helper and specify the :protocol => 'https'.)

Give me the Code!

Okay, so here’s everything in one place:

  environment.rb
    ENV['SSL_PROTOCOL'] = 'https'

  development.rb
    ENV['SSL_PROTOCOL'] = 'http'

  test.rb
    ENV['SSL_PROTOCOL'] = 'http'

  application_controller.rb
    protected
      def ensure_proper_protocol
        if !ssl_allowed? && ssl_required? && !request.ssl? && !(request.get? || request.head?)
          raise 'SSL required!' # either we have a bug somewhere or someone is playing with us.
        else
          super
        end
      end

    private
      def ssl_required?
        return false if ENV['SSL_PROTOCOL'] == 'http'
        super
      end

  users_controller.rb
    ssl_required :new, :create, :edit, :update

  a_form_on_an_insecure_page_that_prefills_users_new_which_is_a_secure_page.html.erb
    <% form_tag(:method => 'get', new_user_url(:protocol => ENV['SSL_PROTOCOL'])) do %>

  an_integration_test_that_wants_to_test_the_new_and_improved_ensure_proper_protocol.rb
    def setup
      ENV['SSL_PROTOCOL'] = 'https'
    end

    def teardown
      ENV['SSL_PROTOCOL'] = 'http'
    end

    def test_that_exception_is_raised
      post '/users', {:user => {:name => 'Natalia Rose'}}
      assert_response :error, 'SSL required!'
    end

  a_integration_test_that_uses_shoulda.rb
    context 'some request' do
      setup do
        ENV['SSL_PROTOCOL'] = 'https'

        get '/users'
      end

      should_require_ssl

      teardown do
        ENV['SSL_PROTOCOL'] = 'http'
      end
    end

  shoulda_extensions.rb
    # Functional Testing
    def should_require_ssl_for(*actions)
      should 'require ssl for' do
        assert_equal actions, controller.class.read_inheritable_attribute(:ssl_required_actions)
      end
    end

    def should_not_require_ssl
      should 'not require ssl' do
        assert_equal nil, controller.class.read_inheritable_attribute(:ssl_required_actions)
      end
    end

    def should_allow_ssl_for(*actions)
      should 'allow ssl for' do
        actions = [:controller] + actions unless actions.include?(:controller)
        assert_equal actions, controller.class.read_inheritable_attribute(:ssl_allowed_actions)
      end
    end

    def should_not_allow_ssl
      should 'not allow ssl' do
        assert_equal [:controller], controller.class.read_inheritable_attribute(:ssl_allowed_actions)
      end
    end

    # Integration Testing
    def should_require_ssl
      should 'require SSL' do
        assert https?
      end
    end

As I mentioned, SSL in Rails isn’t cake, so there is a very, very good chance I’ve got something wrong in this article, or even in my solution. If you spot something, please take a minute to voice your concern, question, or finding. We can all benefit from this. Thanks!

Rails path and url helpers

Sometimes I forget the simple differences between Rail’s helpers. This mini post is so I don’t forget.

*_path
Generates relative URLs: /users
Used in views by link_to, form_for, etc. (per DHH)
The browser maps relative URLs to absolute URLs based on the current page’s protocol and host (/users on the page http://domain.com/new translates to http://domain.com/users)

*_url
Generates absolute URLs: http://domain.com/users
Used in controllers by redirect_to (per DHH) because RFC 2616 states that redirects must be absolute URLS. This is true, however, modern browsers can handle relative URL redirects now too

assert_raise in Rails Integration Tests

Today I wanted to assert that a POST from an integration test resulted in an exception being raised under certain conditions. I had done this bunch of times in unit and controller tests, so I just ported that methodology to my integration test. Here is a contrived example of how my original test read:

  def test_that_exception_is_raised
    exception = assert_raise(RuntimeError) {
      post '/comments', {:comment => {:message => 'I hate you!'}}
    }
    assert_equal 'Say something nicer please. Thank you.', exception.message
  end

The problem was that the assertions were failing because <RuntimeError> exception expected but none was thrown. But I clearly saw in my test.log that the exception was thrown:

  RuntimeError (Say something nicer please. Thank you.):
    app/controllers/comments_controller.rb:789:in `create'
    test/integration/comments_test.rb:456:in `__bind_1278520981_664592'
    /ruby/lib/ruby/8.8/test/unit/assertions.rb:123:in `assert_raise'

I thought about about the differences between functional and integration tests in Rails, and then realized that although the exception was thrown, that’s not what is returned to the browser. Instead, the browser is sent an error page. Then it became glaringly-obvious, how to write my assertion:

  def test_that_exception_is_raised
    post '/comments', {:comment => {:message => 'I hate you!'}}
    assert_response :error, 'Say something nicer please. Thank you.'
  end

I still feel like a newbie sometimes.

Ruby include and extend

My employer, Razoo, treated its developers to RailsConf 2010 this year. Very late the second night of the conference (the first night I was there–we skipped the tutorials), I saw Yehuda Katz give an impromptu Birds of a Feather talk about upcoming changes in Rails. It was good, but the thing I remember most clearly was his explanation of include and extend in Ruby. He let code do the explaining:

# a module with a method
module Says
  def hello
    puts 'hello'
  end
end

# a class we want to have the method (not an instance; the class)
class Person
end

# (our end goal)
class Person
  def self.hello
    puts 'hello'
  end
end

# there are two ways to achieve this
# using extend
class Person
  extend Says
end

# or using include
class Person
  class << self
    include Says
  end
end

So, in short, include just includes the module’s methods as instance methods and extend includes the module’s methods as class methods. I always do better with examples than reading technical documentation!

Our Code Is in 15% of Rails Applications (Roughly)

New Relic just posted some basic statistics on the Rails applications they monitor. They include lists of the top Ruby gems and top Rails plugins installed for each application they monitor. Patrick and I were delighted to discover that our ActiveMerchant code contribution is installed with 15% of Rails applications. We wrote the code for SandwichBoard and Patrick continues to improve upon it for LivingSocial Daily Deals.

Whether or not those 15% are actually using our code for Authorize.Net ARB or CIM credit card charging is unclear; but, it’s cool to know your code is installed with that many applications.