Hampton Catlin - Ruby on Rails Podcast

Posted over 7 years back at Ruby on Rails Podcast

The author of HAML rants on markup, Macs, diversity, and tech writing.
Also mentioned

Capistrano Off the Beaten Path

Posted over 7 years back at Revolution On Rails

Introduction

If you use Capistrano, most likely you use it to deploy rails applications by running it from the project directory. The plugem management tool piggy-backs on Capistrano to execute some recipes without using the current path because recipes operate on gems. Recipes do not, however, completely ignore the current directory, but instead use it for optional customization. Customization is not limited to a deployment recipe from the current directory but can be environment-specific across multiple sites if they each have a special extension gem installed.

Background

We use a bunch of ruby scripts at RHG for deployment and delivery of our numerous applications and shared component. When we started preparing some of that functionality for a public release, we needed to provide a way to customize the scripts. Warren Konkel suggested to migrate them to Capistrano since it was the most familiar tool for rails developers and it has a recipes hierarchy that can be used for customization. So, we wrote the plugem command tool that feeds its parameters to Capistrano via Capistrano::CLI.new(plugem_converted_arguments).execute!
after loading its own recipes.

The Code

    1 module Capistrano
2 class Configuration
3
4 alias :standard_cap_load :load
5
6 def load(*args, &block)
7
8 standard_cap_load(*args, &block)
9
10 if args == ["standard"]
11
12 load_plugem_deploy_recipes(File.dirname(__FILE__) + '/../..')
13
14 begin
15 require 'plugems_deploy_ext'
16 load_plugem_deploy_recipes(PLUGEMS_DEPLOY_EXT_DIR) # Overriding from extensions
17 rescue Exception
18 # No extension is loaded
19 end
20
21 end
22
23 end
24
25 def load_plugem_deploy_recipes(dir)
26 Dir[File.join(dir, 'recipes', '*')].each { |f| standard_cap_load(f) }
27 end
28
29 end
30 end

Ignoring lines 14-19 for now, all it does is inject the plugem deployment gem recipes to Capistrano. They are now available for execution by Capistrano. So when I call plugem update my_app it is being translated to cap plugem_update -s plugem_name=my_app(we use the plugem_ namespace for tasks and variables to avoid clashing with standard recipes). Since loading of plugem recipes is purposely not handled via the -f flag of capistrano, the standard deployment recipes like config/deploy.rb are still being loaded.

Customization

The package is extending Capistrano but we wanted it to be customized too. For example, you might want to define your own list of gem servers to download gems from. Capistrano allows to define recipes per project, user, or host. The plugems_deploy takes it a step further and provides a way to customize it per deployment environment, which might contain many hosts. It does this by expecting an optional plugems_deploy_ext gem to be installed on a system (lines 14-19). If it finds the extension gem, it loads recipes from there overriding the default ones.


Conclusion

Capistrano is not only a great deployment tool, it can be used as a base of highly-customizable general purpose tools.

Capistrano Off the Beaten Path

Posted over 7 years back at Revolution On Rails

Introduction

If you use Capistrano, most likely you use it to deploy rails applications by running it from the project directory. The plugem management tool piggy-backs on Capistrano to execute some recipes without using the current path because recipes operate on gems. Recipes do not, however, completely ignore the current directory, but instead use it for optional customization. Customization is not limited to a deployment recipe from the current directory but can be environment-specific across multiple sites if they each have a special extension gem installed.

Background

We use a bunch of ruby scripts at RHG for deployment and delivery of our numerous applications and shared component. When we started preparing some of that functionality for a public release, we needed to provide a way to customize the scripts. Warren Konkel suggested to migrate them to Capistrano since it was the most familiar tool for rails developers and it has a recipes hierarchy that can be used for customization. So, we wrote the plugem command tool that feeds its parameters to Capistrano via Capistrano::CLI.new(plugem_converted_arguments).execute!
after loading its own recipes.

The Code

    1 module Capistrano
2 class Configuration
3
4 alias :standard_cap_load :load
5
6 def load(*args, &block)
7
8 standard_cap_load(*args, &block)
9
10 if args == ["standard"]
11
12 load_plugem_deploy_recipes(File.dirname(__FILE__) + '/../..')
13
14 begin
15 require 'plugems_deploy_ext'
16 load_plugem_deploy_recipes(PLUGEMS_DEPLOY_EXT_DIR) # Overriding from extensions
17 rescue Exception
18 # No extension is loaded
19 end
20
21 end
22
23 end
24
25 def load_plugem_deploy_recipes(dir)
26 Dir[File.join(dir, 'recipes', '*')].each { |f| standard_cap_load(f) }
27 end
28
29 end
30 end

Ignoring lines 14-19 for now, all it does is inject the plugem deployment gem recipes to Capistrano. They are now available for execution by Capistrano. So when I call plugem update my_app it is being translated to cap plugem_update -s plugem_name=my_app(we use the plugem_ namespace for tasks and variables to avoid clashing with standard recipes). Since loading of plugem recipes is purposely not handled via the -f flag of capistrano, the standard deployment recipes like config/deploy.rb are still being loaded.

Customization

The package is extending Capistrano but we wanted it to be customized too. For example, you might want to define your own list of gem servers to download gems from. Capistrano allows to define recipes per project, user, or host. The plugems_deploy takes it a step further and provides a way to customize it per deployment environment, which might contain many hosts. It does this by expecting an optional plugems_deploy_ext gem to be installed on a system (lines 14-19). If it finds the extension gem, it loads recipes from there overriding the default ones.


Conclusion

Capistrano is not only a great deployment tool, it can be used as a base of highly-customizable general purpose tools.

Episode 31: Formatting Time

Posted over 7 years back at Railscasts

Learn how to use the trusty strftime method to format a time, and see how Rails allows you to save this format for later use.

Wayfaring Stranger

Posted over 7 years back at PJ Hyett

With all of this talk of the future, I’d be remiss not to mention the other website that I’m apart of that’s been chugging along for almost two years now:

Wayfaring! With the team working at CNET last year we haven’t been able to give it much love, but we did manage to upgrade the Google Maps API to version 2 and even built a version for Land Rover.

Some quick stats:

  • 29k users
  • 35k maps
  • 101k waypoints

It has a very loyal following and its design is still fresh, have a look.

Plugin I Cannot Live Without

Posted over 7 years back at Revolution On Rails

The Enhanced Rails Migrations plugin was written to end the constant battle we had with clashing names in db migrations within our large development team. We tried everything: special commit policies, rake tasks, even claiming the next migration number in subversion. Nothing worked and CI server was sending 'broken build due to conflicting migration number' messages almost daily. Since the plugin was introduced to all our rails applications around six months ago, I have not heard of a single case of conflicting migrations. Seemingly, the goal was well accomplished.

What I found over time, is that the plugin is not only useful for large projects. Any rails development effort with more than one programmer involved benefits from using it. If you ever had to renumber your new migration after doing svn up you know what I am talking about. It makes sense to install this plugin as the very first one in your project since an amount of migrations at the beginning tends to grow much faster then later in the game.

The plugin works for rails versions 1.1.6 up to the latest edge. When you start your next project with multiple developers, use it and you should be able to forget that you ever had problems with clashing migrations.

Plugin I Cannot Live Without

Posted over 7 years back at Revolution On Rails

The Enhanced Rails Migrations plugin was written to end the constant battle we had with clashing names in db migrations within our large development team. We tried everything: special commit policies, rake tasks, even claiming the next migration number in subversion. Nothing worked and CI server was sending 'broken build due to conflicting migration number' messages almost daily. Since the plugin was introduced to all our rails applications around six months ago, I have not heard of a single case of conflicting migrations. Seemingly, the goal was well accomplished.

What I found over time, is that the plugin is not only useful for large projects. Any rails development effort with more than one programmer involved benefits from using it. If you ever had to renumber your new migration after doing svn up you know what I am talking about. It makes sense to install this plugin as the very first one in your project since an amount of migrations at the beginning tends to grow much faster then later in the game.

The plugin works for rails versions 1.1.6 up to the latest edge. When you start your next project with multiple developers, use it and you should be able to forget that you ever had problems with clashing migrations.

Episode 30: Pretty Page Title

Posted over 7 years back at Railscasts

If you are like me, you avoid creating page titles because it is kind of a pain. But in this episode I will show you a clean way to add titles to your pages.

Moving models to a different database

Posted over 7 years back at Revolution On Rails

There many reasons to use multiple databases (DBs) and when this is done, there is often a case when a model needs to be moved from one DB to another. The impetus could be that part of the data is referential and this is being reflected by moving it to a read-only DB. Another possibility is that we want to protect some data with an additional layer of security, so we extract it to a secure DB. In all cases, the challenge is to migrate the existing data. When the amount of data is considerably large, there is no choice but to do it via SQL data loaders or similar techniques. On the other hand, if it is acceptable to leverage rails db migrations and you prefer to do any data manipulation through them, there are some challenges to face.

Often, you need to have an access to old and new models during data migration. One solution is to move or copy an existing model to a separate namespace and put the new model instead at the old namespace. Let's look at a couple of examples:


Extracting referential data

I have a model Fruit in our main DB which gets its data from an external source, so we only access it read-only. We want to enforce it by moving the data to a DB which we access with a read-only account. First, I create a referential_db entry in database.yml:

dbs:
database: main

referential_db:
database: referential

Then, I copy the original model, Fruit, to a dedicated namespace, so the model becomes RetiredModels::Fruit. I add establish_connection to the original namespace model:
# create_table :fruits do |t|
# t.column :name, :string
# end
class Fruit < ActiveRecord::Base
establish_connection configurations[RAILS_ENV]['referential_db']
end

Everything is set for migration. Since it is a referential data, the migration needs to preserve data integrity so the models belonging to Fruit can still reference it by an old id:
def self.up
RetiredModels::Fruit.find(:all).each do |old_record|
Fruit.new(old_record.attributes) { |new_record| new_record.id = old_record.id }.save!
end
end

After the successful migration run, all data is replicated to a new DB. The retired model can be removed during next deployment, and the original table dropped.

There is one caveat for development and test modes. If you don't want to bother with multiple databases in those modes, you need to take care of having no table name clashing. So, the new model would have to use different table names via set_table_name.


Securing sensitive data

One of the models belonging to Fruit is SecretFruit. It contains a secret name for every fruit out there. Our legal department asked the development team to protect that data in case our DB is stolen. We decided to migrate the existing SecretFruit data to a protected DB and keep sensitive data encrypted with help from Acts As Secure. First, I create a secure_db entry in database.yml:
dbs:
database: main

secure_db:
database: secure
host: protected_host

Then, I copy the original model, SecretFruit, to a dedicated namespace, so the model becomes RetiredModels::SecretFruit. I modify the model in the original namespace to reflect the new requirements:
# create_table :secret_fruits do |t|
# t.column :name, :binary
# t.column :fruit_id, :integer
# end
class SecretFruit < ActiveRecord::Base
establish_connection configurations[RAILS_ENV]['secure_db']
acts_as_secure :crypto_provider => MasterKeyProvider
belongs_to :fruit
end

Since data encryption is done on-the-fly and there are no data integrity requirements, the migration is straightforward:
def self.up
RetiredModels::SecretFruit.find(:all).each { |old| SecretFruit.create!(old.attributes) }
end

I can now safely delete RetiredModels::SecretFruit and associated data.

Moving models to a different database

Posted over 7 years back at Revolution On Rails

There many reasons to use multiple databases (DBs) and when this is done, there is often a case when a model needs to be moved from one DB to another. The impetus could be that part of the data is referential and this is being reflected by moving it to a read-only DB. Another possibility is that we want to protect some data with an additional layer of security, so we extract it to a secure DB. In all cases, the challenge is to migrate the existing data. When the amount of data is considerably large, there is no choice but to do it via SQL data loaders or similar techniques. On the other hand, if it is acceptable to leverage rails db migrations and you prefer to do any data manipulation through them, there are some challenges to face.

Often, you need to have an access to old and new models during data migration. One solution is to move or copy an existing model to a separate namespace and put the new model instead at the old namespace. Let's look at a couple of examples:


Extracting referential data

I have a model Fruit in our main DB which gets its data from an external source, so we only access it read-only. We want to enforce it by moving the data to a DB which we access with a read-only account. First, I create a referential_db entry in database.yml:

dbs:
database: main

referential_db:
database: referential

Then, I copy the original model, Fruit, to a dedicated namespace, so the model becomes RetiredModels::Fruit. I add establish_connection to the original namespace model:
# create_table :fruits do |t|
# t.column :name, :string
# end
class Fruit < ActiveRecord::Base
establish_connection configurations[RAILS_ENV]['referential_db']
end

Everything is set for migration. Since it is a referential data, the migration needs to preserve data integrity so the models belonging to Fruit can still reference it by an old id:
def self.up
RetiredModels::Fruit.find(:all).each do |old_record|
Fruit.new(old_record.attributes) { |new_record| new_record.id = old_record.id }.save!
end
end

After the successful migration run, all data is replicated to a new DB. The retired model can be removed during next deployment, and the original table dropped.

There is one caveat for development and test modes. If you don't want to bother with multiple databases in those modes, you need to take care of having no table name clashing. So, the new model would have to use different table names via set_table_name.


Securing sensitive data

One of the models belonging to Fruit is SecretFruit. It contains a secret name for every fruit out there. Our legal department asked the development team to protect that data in case our DB is stolen. We decided to migrate the existing SecretFruit data to a protected DB and keep sensitive data encrypted with help from Acts As Secure. First, I create a secure_db entry in database.yml:
dbs:
database: main

secure_db:
database: secure
host: protected_host

Then, I copy the original model, SecretFruit, to a dedicated namespace, so the model becomes RetiredModels::SecretFruit. I modify the model in the original namespace to reflect the new requirements:
# create_table :secret_fruits do |t|
# t.column :name, :binary
# t.column :fruit_id, :integer
# end
class SecretFruit < ActiveRecord::Base
establish_connection configurations[RAILS_ENV]['secure_db']
acts_as_secure :crypto_provider => MasterKeyProvider
belongs_to :fruit
end

Since data encryption is done on-the-fly and there are no data integrity requirements, the migration is straightforward:
def self.up
RetiredModels::SecretFruit.find(:all).each { |old| SecretFruit.create!(old.attributes) }
end

I can now safely delete RetiredModels::SecretFruit and associated data.

Episode 29: group_by Month

Posted over 7 years back at Railscasts

Learn how to use the very useful group_by method to group an array by anything you want! In this episode I group an array of tasks by month then sort it properly.

Enhanced Rails Plugin to Validate (X)HTML and CSS

Posted over 7 years back at

It feels like forever ago that I was last working with ruby on rails. But I am back hacking away at a few rails and I slowly getting more familiarity with ruby itself rather than just using it as part of the rails platform.

Something I have wanted to do for a long time is automagically validate content such as (X)HTML, CSS, atom feeds etc that is generated by the web application. A while ago I put together the assert-valid-asset plugin that allowed you to assert (in functional tests) that the content generated is valid. However you still had to explicitly call the assert.

So recently I enhanced the plugin so that it can automatically validate generated content when configured to do so. To configure auto validation you need to set a class variable in Test::Unit::TestCase via code such as;

    class Test::Unit::TestCase
      self.auto_validate = true
    end

Then anytime content is generated in tests (such as via get and post methods) it will check the mime type of the content. If the content has a mime type of ‘text/html’ or ‘text/xhtml’ it will pass it to the ‘assert_valid_markup’ method. If the content is ‘text/css’ then it will be validated by the ‘assert_valid_css’ method.

Of course you may have tests that generate invalid (X)HTML or CSS (to work with specific unnamed browsers) and you may want to exclude these tests from the automatic content validation. This can be done by adding the test symbol to the exclude list or alternatively by adding the desired test symbols to an include list. Both of the following examples have identical behavior;

Example 1:
    class FooControllerTest < Test::Unit::TestCase
      ...
      self.auto_validate_excludes = [:test_foo,:test_bar]

      def test_foo; ... ; end
      def test_bar; ... ; end
      def test_baz; ... ; end
    end
Example 2:
    class FooControllerTest < Test::Unit::TestCase
      ...
      self.auto_validate_includes = [:test_baz]

      def test_foo; ... ; end
      def test_bar; ... ; end
      def test_baz; ... ; end
    end

You can grab it from subversion at;

svn checkout http://assert-valid-asset.googlecode.com/svn/trunk/

The original article describing the plugin is here .

Deconstructing date_select

Posted over 7 years back at zerosum dirt(nap) - Home

date_select and it’s friend datetime_select are handy little helpers for rendering date selection widgets in your form, but what they hand back to your controller is a little bit unintuitive. If you’re just going to go ahead and use update_attributes on your model, then ActiveRecord magically takes care of converting these to an appropriate Date or Time object for you. However, if you’re not doing that, you’ll find yourself staring at keys in your hash named odd things like “attribute(1i)”.

Here’s a handy little helper method, in case you ever need to manually reconstruct a Date object from the params:

# Reconstruct a date object from date_select helper form params
def build_date_from_params(field_name, params)
  Date.new(params["#{field_name.to_s}(1i)"].to_i, 
           params["#{field_name.to_s}(2i)"].to_i, 
           params["#{field_name.to_s}(3i)"].to_i)
end

date = build_date_from_params(:published_at, params[:article])

Put this in your ApplicationController so inherited controller classes can access it, or (better yet) put it in a suite of helper methods to mix in to application.rb. Not exactly brain surgery here, but I’m a little surprised that there isn’t something already baked into the helpers to do this.

DRYing Models via Acts As

Posted over 7 years back at Revolution On Rails

ActsAs is an idiom familiar to every Rails developer, which makes it a good candidate for a shared functionality between models. Using it as early in the game as possible allows one to work on its functionality without a need to touch the code in multiple models. Let's look at a couple of examples.

Acts As Unique

I have some models that I want to have uniqueness across my application. I use some UUID mechanism (initially, a db call) to set a field (:token) after creation. Since I have multiple models, I decide to extract it the code for uniqueness setting to acts_as_unique. After refactoring, my model Fruit looks like:

# create_table :fruits do |t|
# t.column :name, :string
# t.column :token, :string
# end
class Fruit < ActiveRecord::Base
acts_as_unique
end

My acts_as_unique might look like:
module ActiveRecord; module Acts; end; end
module ActiveRecord::Acts::ActsAsUnique

def self.included(base)
base.extend(ClassMethods)
end

module ClassMethods
def acts_as_unique(field = :token)
validates_uniqueness_of field
before_validation_on_create do |o|
o.send("#{ field }=", connection.select_one('SELECT UUID() AS UUID', "#{name} UUID generated")['UUID'])
end
end
end
end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsAsUnique)

Let's try it:
>> f = Fruit.create(:name => 'apple')
>> p f.token
"0a4d7c46-4df0-102a-a4b9-59b995bffdb7"

Now I can work on acts_as_unique to replace the DB call with a UUID gem or some other implementation without affecting the rest of the code.


Acts As Trackable

I have some models for which I want to keep track of when instances are created or updated. I have a polymorphic Event model for storage of such events. Since there are multiple models I want to track, I extract the functionality to acts_as_trackable. After refactoring, my models look like:
# create_table :fruits do |t|
# t.column :name, :string
# end
class Fruit < ActiveRecord::Base
acts_as_trackable
end

# create_table :events do |t|
# t.column "action", :string
# t.column "created_at", :datetime, :null => false
# t.column "trackable_type", :string
# t.column "trackable_id", :integer
# end
class Event < ActiveRecord::Base
belongs_to :trackable, :polymorphic => true
end

module ActiveRecord; module Acts; end; end 
module ActiveRecord::Acts::ActsTrackable

def self.included(base)
base.extend(ClassMethods)
end

module ClassMethods
def acts_as_trackable
has_many :events, :as => :trackable, :dependent => :destroy
after_update { |o| o.events.create(:action => 'updated') }
after_create { |o| o.events.create(:action => 'created') }
end
end

end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsTrackable)

Let's see what we got:
>> f = Fruit.create(:name => 'apple')
>> p f.events.collect(&:action)
["created"]
>> f.name = 'passionfruit'
>> f.save!
>> p f.events.collect(&:action)
["created", "updated"]

The Event model is likely to evolve but it would be easier to support it since the only place where I need to reflect the changes is acts_as_trackable. The goal is achieved.

DRYing Models via Acts As

Posted over 7 years back at Revolution On Rails

ActsAs is an idiom familiar to every Rails developer, which makes it a good candidate for a shared functionality between models. Using it as early in the game as possible allows one to work on its functionality without a need to touch the code in multiple models. Let's look at a couple of examples.

Acts As Unique

I have some models that I want to have uniqueness across my application. I use some UUID mechanism (initially, a db call) to set a field (:token) after creation. Since I have multiple models, I decide to extract it the code for uniqueness setting to acts_as_unique. After refactoring, my model Fruit looks like:

# create_table :fruits do |t|
# t.column :name, :string
# t.column :token, :string
# end
class Fruit < ActiveRecord::Base
acts_as_unique
end

My acts_as_unique might look like:
module ActiveRecord; module Acts; end; end
module ActiveRecord::Acts::ActsAsUnique

def self.included(base)
base.extend(ClassMethods)
end

module ClassMethods
def acts_as_unique(field = :token)
validates_uniqueness_of field
before_validation_on_create do |o|
o.send("#{ field }=", connection.select_one('SELECT UUID() AS UUID', "#{name} UUID generated")['UUID'])
end
end
end
end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsAsUnique)

Let's try it:
>> f = Fruit.create(:name => 'apple')
>> p f.token
"0a4d7c46-4df0-102a-a4b9-59b995bffdb7"

Now I can work on acts_as_unique to replace the DB call with a UUID gem or some other implementation without affecting the rest of the code.


Acts As Trackable

I have some models for which I want to keep track of when instances are created or updated. I have a polymorphic Event model for storage of such events. Since there are multiple models I want to track, I extract the functionality to acts_as_trackable. After refactoring, my models look like:
# create_table :fruits do |t|
# t.column :name, :string
# end
class Fruit < ActiveRecord::Base
acts_as_trackable
end

# create_table :events do |t|
# t.column "action", :string
# t.column "created_at", :datetime, :null => false
# t.column "trackable_type", :string
# t.column "trackable_id", :integer
# end
class Event < ActiveRecord::Base
belongs_to :trackable, :polymorphic => true
end

module ActiveRecord; module Acts; end; end 
module ActiveRecord::Acts::ActsTrackable

def self.included(base)
base.extend(ClassMethods)
end

module ClassMethods
def acts_as_trackable
has_many :events, :as => :trackable, :dependent => :destroy
after_update { |o| o.events.create(:action => 'updated') }
after_create { |o| o.events.create(:action => 'created') }
end
end

end

ActiveRecord::Base.send(:include, ActiveRecord::Acts::ActsTrackable)

Let's see what we got:
>> f = Fruit.create(:name => 'apple')
>> p f.events.collect(&:action)
["created"]
>> f.name = 'passionfruit'
>> f.save!
>> p f.events.collect(&:action)
["created", "updated"]

The Event model is likely to evolve but it would be easier to support it since the only place where I need to reflect the changes is acts_as_trackable. The goal is achieved.