Dangers of Cargo-culting

Posted about 7 years back at The Rails Way - all

“Cargo culting”, when used in a computer-programming context, refers to the practice of using techniques (or even entire blocks of code) seen elsewhere without wholly understanding how they work. (The term “cargo cult”, if you are unfamiliar with it, has its own fascinating etymology, which is covered nicely at wikipedia.) Cargo culting is a dangerous phenomenon, watering down the state of the art and encouraging cookie-cutter code shoved blindly into black boxes.

Consider the following snippet of code, taken from a project that was submitted to us some time ago. (Alas, I cannot find the original submitter—I apologize for that!)

1
2
3
def account_code?
  !! @account_code.nil?
end

To me, this looks cargo-culted, since it is seems that the programmer did not understand what the ”!!” idiom was all about. They probably saw it used somewhere and “cargo culted” it, using it without knowledge, assuming that it was, for some reason, “necessary”.

Now, the way ”!!” works is this: take the value behind the ”!!”, negate it, and negate it again. It’s just double-negation: !(!(@account_code.nil?)). The ultimate effect is to take some value, and convert it into an honest-to-goodness “true” or “false”. (In my ever-so-humble opinion, the ”!!” idiom is an abomination: it’s far too clever for its own good. First of all, you rarely ever need a real boolean value, and for those times you do, it is better to be explicit in the conversion, by using a ternary operator or full-blown if statement, for instance.)

In other words, the double-negation of nil? results in absolutely no difference from the use of nil? by itself, since nil? will return a true/false value. This, in turn, means the effect in the original is actually not what was intended for the account_code? predicate. It should have returned “true” if the account code existed (was “non-nil”), not “false”. Thus, the method should have actually been written thus:

1
2
3
def account_code?
  ! @account_code.nil?
end

In this case, cargo-culting resulted in the code being buggy. This is not an uncommon outcome of using techniques or code without understanding their purpose. If you ever find yourself copying something into your own code, with a justfying “I-don’t-know-what-it-does, but-it-appears-to-work”, stop immediately. Do some research. Figure it out. Learn what it means.

Further, note that unless you actually need a true boolean value from that, you can shorten the implementation of the account_code? predicate even further:

1
2
3
def account_code?
  @account_code
end

This works because Ruby treats nil and false as false, and everything else as true.

If there is one thing that Koz and I want you, our readers, to come away from this site with, it is an understanding of why you should do things one way and not another. Ultimately, it makes the difference between being a mediocre programmer, and becoming a great programmer.

Positive Reasons for Typo

Posted about 7 years back at Wood for the Trees

I highly respect quick responses to a post. In Simeon’s post, he laments my lack of a positive reasoning for choosing Typo, so I’d like to quickly outline why I originally selected Typo and why I was drawn back to it.

Typo leverages the modern technologies that are out there and, true to a Rails application, has a love of shiny things without overdoing it. Nevertheless, it is fast (70 req/sec is my entire readership in a second, which makes me feel very warm and cuddly). Despite the speed, it’s not stripped down at all. What it comes with built-in (every social networking/bookmarking sidebar imaginable, multiple feed formats, multiple database support) is easily supplemented by its easy-to-use plugin architecture. It now uses the Rails plugin architecture, which makes it even easier. Adding themes is a doddle. Even editing the core code gives you the result you want, more often than not.

Basically, Typo is written to offer complex features using the simplest code. There are areas you could quibble with me on that score, and I may agree, but generally Typo is a sleek and sexy blogging engine with lots of features and yet which still has limitless potential. Just look at their Trac. The active development suggests a project which is going somewhere and getting there fast. I stopped using Typo after 4.0 and came back at 4.1. Already I see a lot of changes under the hood that would suggest more than just a minor version change. Even with all these changes, reverting back to Typo took me no less than 10 minutes. That says a lot to me.

Sure, WordPress and other blogging engines may have plugins or built-in features which rival those in Typo, but the sheer speed and flexibility of development behind Typo is peerless. I like that. It feels edgy.

I’m no old man and I don’t need Zimmerframe 2.2.1. Give me the bloody nose brawls, the midday groggy writer’s block, the street-vomiting hard nights, the raving ecstasy of Typo any day. She’s the kind of girl you can take out for a walk in the park or on a bruiser’s night and return home proud and glowingly post coital, because she’s always good fun, day or night.

Typo’s no dumb blonde.

Positive Reasons for Typo

Posted about 7 years back at Wood for the Trees

I highly respect quick responses to a post. In Simeon’s post, he laments my lack of a positive reasoning for choosing Typo, so I’d like to quickly outline why I originally selected Typo and why I was drawn back to it.

Typo leverages the modern technologies that are out there and, true to a Rails application, has a love of shiny things without overdoing it. Nevertheless, it is fast (70 req/sec is my entire readership in a second, which makes me feel very warm and cuddly). Despite the speed, it’s not stripped down at all. What it comes with built-in (every social networking/bookmarking sidebar imaginable, multiple feed formats, multiple database support) is easily supplemented by its easy-to-use plugin architecture. It now uses the Rails plugin architecture, which makes it even easier. Adding themes is a doddle. Even editing the core code gives you the result you want, more often than not.

Basically, Typo is written to offer complex features using the simplest code. There are areas you could quibble with me on that score, and I may agree, but generally Typo is a sleek and sexy blogging engine with lots of features and yet which still has limitless potential. Just look at their Trac. The active development suggests a project which is going somewhere and getting there fast. I stopped using Typo after 4.0 and came back at 4.1. Already I see a lot of changes under the hood that would suggest more than just a minor version change. Even with all these changes, reverting back to Typo took me no less than 10 minutes. That says a lot to me.

Sure, WordPress and other blogging engines may have plugins or built-in features which rival those in Typo, but the sheer speed and flexibility of development behind Typo is peerless. I like that. It feels edgy.

I’m no old man and I don’t need Zimmerframe 2.2.1. Give me the bloody nose brawls, the midday groggy writer’s block, the street-vomiting hard nights, the raving ecstasy of Typo any day. She’s the kind of girl you can take out for a walk in the park or on a bruiser’s night and return home proud and glowingly post coital, because she’s always good fun, day or night.

Typo’s no dumb blonde.

Apache2 Redirect To Feedburner

Posted about 7 years back at Mike Mondragon

This is how I’m doing a 302 redirect to Feedburner from my Apache2 virtual host settings for this blog. Its only for the main articles syndication in RSS and Atom . Everyone is redirected to Feedburner except for their web crawling bot. I’m doing this so people can see Feedburner’s shiny widget ( ) in the sidebar showing that only 1 or 2 people subscribe to my blog.

I put the following in between the RewriteEngine On statement and the static maintenance statement RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f covered in this post describing an Apache vhosts setup for a Rails app

# 302 temporary, 301 permanent
RewriteCond %{HTTP_USER_AGENT}  !^FeedBurner/.*
RewriteRule ^/xml/rss20/feed.xml$ http://feeds.feedburner.com/mondragon/PQVR [R=302,L]
RewriteRule ^/xml/atom/feed.xml$ http://feeds.feedburner.com/mondragon/PQVR [R=302,L]

Apache2 Redirect To Feedburner

Posted about 7 years back at Mike Mondragon

This is how I’m doing a 302 redirect to Feedburner from my Apache2 virtual host settings for this blog. Its only for the main articles syndication in RSS and Atom . Everyone is redirected to Feedburner except for their web crawling bot. I’m doing this so people can see Feedburner’s shiny widget ( ) in the sidebar showing that only 1 or 2 people subscribe to my blog.

I put the following in between the RewriteEngine On statement and the static maintenance statement RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f covered in this post describing an Apache vhosts setup for a Rails app

# 302 temporary, 301 permanent
RewriteCond %{HTTP_USER_AGENT}  !^FeedBurner/.*
RewriteRule ^/xml/rss20/feed.xml$ http://feeds.feedburner.com/mondragon/PQVR [R=302,L]
RewriteRule ^/xml/atom/feed.xml$ http://feeds.feedburner.com/mondragon/PQVR [R=302,L]

Using Rails with Flex to manage long running tasks

Posted about 7 years back at work.rowanhick.com

UPDATE Jul 2008 - The API for BackgroundRB has changed and docs improved significantly since this blog post. Read the latest API calls over here http://backgroundrb.rubyforge.org/rails/. Don't reference the Rails code below if using latest BackgroundRB As soon as you start doing anything with photos sooner or later someone says "It would be really nice to upload these in a zip file", which then leads into a whole rabbit warren of issues, one of which you will inevitably come across is how to deal with a long running task on the server, in terms of a) getting it to actually complete and b) telling the user what's going on. There are various hacks for doing these, like running a shell script and writing out tmp files all over the place. Just plain ugly. Enter Backgroundrb, probably one of the greatest little inventions for Rails. It lets you kick off a background process, outside of the main rails app process, with the option to interact with the rails models or not. The beauty of this plugin is a little thing called MiddleMan, as the name implies it lets you interact with your background processes (Workers) from your Rails app. So, this gives us the ability, from your Flex app to invoke a Worker on the server side, then monitor that Worker until it's finished, and/or kill off the worker if need be (for slacking on the job?). What do we need. 1. First off install BackgroundRB from here. 2. Next we need to create our worker in your Rails app dir 'script/generate worker ResizerWorker . Here's a simple chunk of worker code. class ResizerWorker < BackgrounDRb::Rails def do_work(args) @total_images = 50 @current_image = 1 50.times do @current_image += 1 sleep(1) end end def get_progress @current_image < 50 ? status = 'In progress' : status = 'Finished' info = { :status => status, :current_image => @current_image, :total_images => @total_images} return info end end As you can see we have a do_work method which is generated by default, this is called whenever the worker is instantiated so it kicks off our work to be done. Then I added a get_progress method to return a hash of the worker's current status. 3. Great so now we need to kick it off, there's some options to do this automatically at timed intervals, but I'm not going to go into that here, all we're doing is getting our rails app, by a web request to start/stop and monitor the progress (the web request is then called from Flex) . So we create a watcher controller, with actions 'start_task', 'status' and 'stop_task' as below which is all pretty self explanatory: class WatcherController < ApplicationController def start_task session[:worker_key] = MiddleMan.new_worker(:class => :resizer_worker) info = {:message => 'job_started', :key => session[:worker_key]} render :xml => info.to_xml(:dasherize => false) end def status @worker = MiddleMan.get_worker(session[:worker_key]) if @worker info = @worker.get_progress if info[:status] == 'Finished' MiddleMan.kill_worker(session[:worker_key]) end else info = { :status => 'Finished' } end render :xml => info.to_xml(:dasherize => false) end def stop_task @worker = MiddleMan.kill_worker(session[:worker_key]) info = { :status => 'Task Stopped' } render :xml => info.to_xml(:dasherize => false) end end Gotcha! I found out the method names in the Documentation, on MiddleMan, were wrong compared to the version I had installed. I needed to use .new_worker, .get_worker and .kill_worker. 4. Fantastic, so go into console, do "rake backgroundrb:start" in your rails directory, then fire up a rails server. You should now be able to hit http://localhost:3000/watcher/start_task, http://localhost:3000/watcher/status, http://localhost:3000/watcher/stop_task and see the appropriate results. 5. Now you can fire HTTPService requests on the Flex side and consume the results as required. For example : (for the sake of berevity I haven't put in any timer function to recheck the status, just a manual click event) import mx.rpc.events.ResultEvent; import mx.controls.Alert; private function onSvcWatchTaskResult(event:ResultEvent):void { trace(event.result.valueOf()); var info:XML = XML(event.result); pbProgress.setProgress( info.current_image, info.total_images ); } private function onSvcStartTaskResult(event:ResultEvent):void { Alert.show('Task Started'); } And there you go. A few seemingly simple lines of code has knocked off one of the age old problems of dealing with long running tasks. I heart Rails (and Flex). (As always with code examples, don't copy and paste, rewrite it yourself. You'll understand it better and won't copy in any errors I've made whilst cutting it down for blogging)

Seriously Sick of WordPress -- Back to Typos

Posted about 7 years back at Wood for the Trees

I should have listened to myself ages ago when I discovered Ruby and said to myself: “I hated PHP before there was an alternative. I hate PHP even more now that there is one.” So why did I convert from Typo to WordPress? I really, really do not understand why there are more posts about converting from Typo to Wordpress than vice versa.

My dear peers, pelt me with rancid tomatoes. I don’t know what I was thinking when converting to a PHP monstrosity. WordPress is such a bloated and badly coded blog engine (and that is much due to the way PHP itself encourages you to code). Every time I try to dig into its code, I become nauseous and frustrated, despite years of PHP experience.

What brought me to this realisation is an innocent attempt to upgrade… the quality of a software distribution and its development practices always reveal themselves when you upgrade, because it is in the care and attention to backwards compatibility which show developers have really considered what they are doing, not just chasing the new shiny thing.

So in trying to upgrade from WordPress 2.0 to 2.2, I found my entire day wasted chasing down errors, discovering none of my plugins will work, finding my theme broken in numerous places, and just plain frustrated by the idiotic process of removing, moving, copying, overwriting and double-checking files and directories. Upgrading WordPress is the most painful form of manual labour I’ve experienced in a while.

So, without further ado, my list of complaints before saying Adieu! to WordPress:

  • Biggest of all: upgrading is a serious pain in the arse, even with my upgrade.rb script (notice the irony)
  • Upgrading breaks almost every plugin you use
  • Plugin compatibility is atrocious. There may be a lot of them in their plugin repository, but look at the fine print and you’ll see they range from pre-1.x to 2.2 compatibility, and that doesn’t mean 2.2 works with 1.5 or vice versa. You need the right version or, more likely than not, it won’t even be recognised. So, really, each version has more like 30 plugins available, not 3000.
  • With every upgrade, you are welcoming security bugs (the worst kind) and I just don’t like the idea that upgrading to fix one is just introducing a new one. The reason for this is that WordPress is not a framework; its core functionality is part of its higher functionality, and with every upgrade they are mucking with both. I’d suggest to the WordPress development team that they extract the underlying HTTP management and the overlying CMS functionality… but then, why not use something like Django, CakePHP or Rails?
  • Their plugin architecture is horrendously 90’s
  • The admin interface is hideous and unintuitive—where do I set pings and trackbacks again? Why do plugins enable new menus in weird places? I can never remember where anything is because nothing is organised properly
  • Creating and editing posts is so slow I find myself making a 10-cup cafetiere every time.
  • Does WordPress send pings? I never see my blog appear after a post. I can only assume it doesn’t, even when I tell it to.
  • The WordPress helper functions are megolithic and cryptic. I can’t cuss them more, really. And they don’t even cover every situation. I’d rather have thousands of little ones to learn that three which do vastly different things, none of which do what I want. That means editing themes is a serious pain.
  • Lastly, I just despise PHP and giving it support by using it.

But this isn’t just about me whinging. I have positive reasons for moving back to Typo:

  • It’s written in Rails, which lets me do whatever I want with it, in a pleasurable way. I know Rails well enough to muck around. All I need to do is learn how Typo works, and it isn’t that complicated—nothing written in Rails, that I’ve encountered, is too complicated to grok in a day or two.
  • Typo 4.2 is going to introduce multiple blogs within a single installation, which I really would like to have. So I’m in for the long haul.
  • The latest Typo (4.1) uses Rails 1.2, which was one of my concerns with 4.0.
  • The rumour of an admin interface redesign is rather thrilling, but then, I still really like Typo’s admin interface as it stands. I’m very curious what they’re going to do.
  • Typo now uses Rails plugins to do everything. I intend to write a FeedBurner plugin that lets you do all sorts of things, like FeedFlare, Feedburner redirects and other nifty FeedBurner stuff. Now that it is easy to write a plugin, I’m keen to test it out and prove its worth.

So, back to Typo 4.x. Let’s hope its developers sorted out some of the issues that put me off it and caused me to so whimsically (and stupidly) change to WordPress.

Note: Over the next week, my dear readers, you may encounter a few problems. These are only temporary as I muck around with Typo 4.1 and juice it for all its worth.

Oh, and thank you for waiting the few months while I was deeply involved with a major project. I’m back permanently now, on freelance. It’s good to be free and back on the Rails.

Seriously Sick of WordPress -- Back to Typos

Posted about 7 years back at Wood for the Trees

I should have listened to myself ages ago when I discovered Ruby and said to myself: “I hated PHP before there was an alternative. I hate PHP even more now that there is one.” So why did I convert from Typo to WordPress? I really, really do not understand why there are more posts about converting from Typo to Wordpress than vice versa.

My dear peers, pelt me with rancid tomatoes. I don’t know what I was thinking when converting to a PHP monstrosity. WordPress is such a bloated and badly coded blog engine (and that is much due to the way PHP itself encourages you to code). Every time I try to dig into its code, I become nauseous and frustrated, despite years of PHP experience.

What brought me to this realisation is an innocent attempt to upgrade… the quality of a software distribution and its development practices always reveal themselves when you upgrade, because it is in the care and attention to backwards compatibility which show developers have really considered what they are doing, not just chasing the new shiny thing.

So in trying to upgrade from WordPress 2.0 to 2.2, I found my entire day wasted chasing down errors, discovering none of my plugins will work, finding my theme broken in numerous places, and just plain frustrated by the idiotic process of removing, moving, copying, overwriting and double-checking files and directories. Upgrading WordPress is the most painful form of manual labour I’ve experienced in a while.

So, without further ado, my list of complaints before saying Adieu! to WordPress:

  • Biggest of all: upgrading is a serious pain in the arse, even with my upgrade.rb script (notice the irony)
  • Upgrading breaks almost every plugin you use
  • Plugin compatibility is atrocious. There may be a lot of them in their plugin repository, but look at the fine print and you’ll see they range from pre-1.x to 2.2 compatibility, and that doesn’t mean 2.2 works with 1.5 or vice versa. You need the right version or, more likely than not, it won’t even be recognised. So, really, each version has more like 30 plugins available, not 3000.
  • With every upgrade, you are welcoming security bugs (the worst kind) and I just don’t like the idea that upgrading to fix one is just introducing a new one. The reason for this is that WordPress is not a framework; its core functionality is part of its higher functionality, and with every upgrade they are mucking with both. I’d suggest to the WordPress development team that they extract the underlying HTTP management and the overlying CMS functionality… but then, why not use something like Django, CakePHP or Rails?
  • Their plugin architecture is horrendously 90’s
  • The admin interface is hideous and unintuitive—where do I set pings and trackbacks again? Why do plugins enable new menus in weird places? I can never remember where anything is because nothing is organised properly
  • Creating and editing posts is so slow I find myself making a 10-cup cafetiere every time.
  • Does WordPress send pings? I never see my blog appear after a post. I can only assume it doesn’t, even when I tell it to.
  • The WordPress helper functions are megolithic and cryptic. I can’t cuss them more, really. And they don’t even cover every situation. I’d rather have thousands of little ones to learn that three which do vastly different things, none of which do what I want. That means editing themes is a serious pain.
  • Lastly, I just despise PHP and giving it support by using it.

But this isn’t just about me whinging. I have positive reasons for moving back to Typo:

  • It’s written in Rails, which lets me do whatever I want with it, in a pleasurable way. I know Rails well enough to muck around. All I need to do is learn how Typo works, and it isn’t that complicated—nothing written in Rails, that I’ve encountered, is too complicated to grok in a day or two.
  • Typo 4.2 is going to introduce multiple blogs within a single installation, which I really would like to have. So I’m in for the long haul.
  • The latest Typo (4.1) uses Rails 1.2, which was one of my concerns with 4.0.
  • The rumour of an admin interface redesign is rather thrilling, but then, I still really like Typo’s admin interface as it stands. I’m very curious what they’re going to do.
  • Typo now uses Rails plugins to do everything. I intend to write a FeedBurner plugin that lets you do all sorts of things, like FeedFlare, Feedburner redirects and other nifty FeedBurner stuff. Now that it is easy to write a plugin, I’m keen to test it out and prove its worth.

So, back to Typo 4.x. Let’s hope its developers sorted out some of the issues that put me off it and caused me to so whimsically (and stupidly) change to WordPress.

Note: Over the next week, my dear readers, you may encounter a few problems. These are only temporary as I muck around with Typo 4.1 and juice it for all its worth.

Oh, and thank you for waiting the few months while I was deeply involved with a major project. I’m back permanently now, on freelance. It’s good to be free and back on the Rails.

ActiveRecord and mysql: show my databases

Posted about 7 years back at Saaien Tist

Working on a ruby API for the Ensembl databases, I bumped into the issue of having to connect to a database without knowing its name.

The ensembl database server hosts databases for each species. Every two months or so, there's a new release which means a new database for every single species. To see what databases are there, you can login to the ensembl server with mysql:


mysql -h ensembldb.ensembl.org -u anonymous

(for more information, see here).

The command "show databases;" on the mysql prompt lists a total of 1035 databases at the moment, a short selection looks like this:

bos_taurus_core_41_2d
bos_taurus_core_42_2e
bos_taurus_core_43_3
bos_taurus_core_44_3a
bos_taurus_core_45_3b
bos_taurus_est_36_2
bos_taurus_est_37_2a
homo_sapiens_core_36_35i
homo_sapiens_core_37_35j
homo_sapiens_core_38_36
homo_sapiens_core_39_36a
homo_sapiens_core_40_36b
homo_sapiens_core_41_36c
homo_sapiens_core_42_36d
homo_sapiens_core_43_36e
homo_sapiens_core_44_36f
homo_sapiens_core_45_36g
homo_sapiens_core_expression_est_34_35g
homo_sapiens_core_expression_est_45_36g
homo_sapiens_core_expression_gnf_34_35g
homo_sapiens_core_expression_gnf_45_36g

To connect to the homo_sapiens_core_45_36g database, type "use homo_sapiens_core_45_36g;" at the mysql prompt. However, as all 'core' databases have the same database schema, the API applies to all of these species, and just has to connect to different databases. But how do you go about doing that? What you could do, is provide the full database name in the establish_connection statement. But having to memorize these full names, or having to open mysql connections prior to writing your scripts is, to say the least, far from optimal. But how do you query a database system without connecting to a particular database?

Basically, you make a connection to the host without specifying a database, and send the raw sql query "show databases;" over that connection. The code below does just that.


ENSEMBL_RELEASE = 45
DB_ADAPTER = 'mysql'
DB_HOST = 'ensembldb.ensembl.org'
DB_USERNAME = 'anonymous'
DB_PASSWORD = ''

class DummyDBConnection < ActiveRecord::Base
self.abstract_class = true

establish_connection(
:adapter => DB_ADAPTER,
:host => DB_HOST,
:database => '',
:username => DB_USERNAME,
:password => DB_PASSWORD
)
end

class CoreDBConnection < ActiveRecord::Base
self.abstract_class = true

def self.connect(species)
db_name = DummyDBConnection.connection.select_values('show databases').select{|v| v =~ /#{species}_core_#{ENSEMBL_RELEASE.to_s}/}[0]


if db_name.nil?
warn "WARNING: No connection to database established. Check that the species is in snake_case (was: #{species})."
else
establish_connection(
:adapter => DB_ADAPTER,
:host => DB_HOST,
:database => db_name,
:username => DB_USERNAME,
:password => DB_PASSWORD
)
end
end
end



And then just have your classes (e.g. CoordSystem, SeqRegion, Gene) inherit from CoreDBConnection instead of ActiveRecord::Base.

To make the actual connection, start your script with:

CoreDBConnection.connect('bos_taurus')


I'm currently at Ensembl for a week ("Geek for a Week") to work on the full-blown ruby API, and am planning to give an introduction on how to use it in one of the later posts.

Episode 64: Custom Helper Modules

Posted about 7 years back at Railscasts

Rails designates one helper module per controller, but that shouldn't stop you from making custom helper modules to help structure the code. Learn how in this episode.

Rails Models in a Namespace

Posted about 7 years back at Mike Mondragon

If you are starting to get a cluttered model space in your Rails application you should consider placing your models in a namespace. As an example I’m going to go through a Rails application I’m calling Recipes. If my models were starting to have the namespace implied in the class names such as AppleFruit in app/models/apple_fruit.rb then that’s starting to smell like rotten apples. A better namespace would be Fruit::Apple in app/models/fruit/apple.rb

This is what we’ll be modeling. Fruits of Apples and Oranges via single table inheritance. And Vegetables of Potatoes and Carrots through single table inheritance.

We’ll have Ingredients that belong to Fruit, Vegetables, and Recipes. Ingredients are a limited kind of join model going from the recipe through to the kind of ingredient (i.e. fruit or vegetable). Ingredients are a true join model from the fruit or vegetables back to their associated recipes. The Ingredient is polymorphic because Fruits and Vegetables are different kinds of objects.

Finally Recipes are another single table inheritance model but by convention they will only have ingredients, they won’t be associated to the kinds of ingredients through the polymorphic Ingredient class. To access the specific kinds of ingredients from the recipe’s perspective you must access the collection of ingredients and then program the desired behavior to access the kinds of ingredients in your application logic.

Here’s a graph of the models we are designing (click for bigger picture):

The graph was made with Railroad which uses Graphiz to generate the graphs.

All of the source for this example as available at the following Subversion code repository

svn checkout http://svn.mondragon.cc/svn/recipes/trunk/ recipes

http://svn.mondragon.cc/svn/recipes/trunk/

Setup

For simplicity we’ll be using a sqlite3 database for this application, now that we are eating fruits and vegetables we don’t need to be any fatter with an external database server floating around. This example is done in Rails 1.2.3

Before we go on let me give you a quote that Eric Hodel has been putting in the footer of his emails:

Poor workers blame their tools. Good workers build better tools. The
best workers get their tools to do the work for them. -- Syndicate Wars

I’ve been learning many things from Eric and I try to emulate what he does as a developer. Two things he always does is practice test driven development and uses a tool he wrote called autotest to make TDD easier to accomplish. autotest does supports Rails out of the box. Now on with our recipes…

This is the config/database.yml we’ll be using:

market: &market
  adapter: sqlite3

development:
  database: "db/development.db" 
  <<: *market

production:
  database: "db/production.db" 
  <<: *market

test:
#  database: ":memory:" 
  database: "db/test.db" 
  <<: *market

And we’ll start off by putting our sessions in the database and then running the migration to ensure we have our database settings correct.

rake db:sessions:create
rake db:migrate
rake test

Now in a separate console cd into the root of your application and start autotest

autotest

it will run in your directory and whenever you save a test or code file the corresponding unit tests will be fired for those files.

Fruits and Vegetables models

Fruits

Now lets make a Base of our Apples and Orange model with single table inheritance, after the fixture is generated we need to fix where its placed as in the example. Note we are declaring a string attribute named ‘type’ to the model generator. The string is really a column and having a column named ‘type’ is a Rails idiom signaling single table inheritance.

ruby script/generate model Fruit::Base type:string
mv test/fixtures/fruit/fruit_bases.yml test/fixtures/
rmdir test/fixtures/fruit/

In your unit test test/unit/fruit/base_test.rb you need to clue the test into knowing which table/fixture is to be used in the namespace. Coincidently note that your tables and fixtures will still look somewhat flat even though your model classes have depth. After you save the test autotest should be complaining the about an error with the SQL since you haven’t yet migrated your schema. Lets also change the default truth test the generator runs so that autotest is testing something of value for better test driven development

require File.dirname(__FILE__) + '/../../test_helper'

class Fruit::BaseTest < Test::Unit::TestCase
  fixtures :fruit_bases
  set_fixture_class :fruit_bases => Fruit::Base

  # if our notion of a valid new fruit changes then we'll catch it that here
  def test_should_be_valid
    f = Fruit::Base.new
    assert f.valid?
  end

end

In your base fruit model you also have to mark which table to assign the class in its namespace

class Fruit::Base < ActiveRecord::Base
  set_table_name :fruit_bases
end

Now migrate your schemas and then re-save your app/models/fruit/base.rb and you should see autotest is happy, it doesn’t have any errors or failures.

rake db:migrate

See, autotest is happy, it doesn’t have any errors or failures

/usr/local/bin/ruby -I.:lib:test -rtest/unit -e "%w[test/unit/fruit/base_test.rb].each { |f| require f }" | unit_diff -u
Loaded suite -e
Started
.
Finished in 0.48792 seconds.

1 tests, 1 assertions, 0 failures, 0 errors

Now lets generate our Apples and Oranges, but the generator is going to create test/fixtures/fruit/fruit_apples.yml and test/fixtures/fruit/fruit_oranges.yml and we won’t need those fixtures because we are using single table inheritance and we’ll only have one fixture for all of the fruits: test/fixtures/fruit_bases.yml Migrations for Fruit::Orange and Fruit::Apple are also generated. We don’t need those because we are doing single table inheritance from the Fruit::Base Migrate the schema while you are at it.

ruby script/generate model Fruit::Apple
rm db/migrate/*_create_fruit_apple.rb
rm test/fixtures/fruit/fruit_apples.yml
ruby script/generate model Fruit::Orange
rm db/migrate/*_create_fruit_apple.rb
rm test/fixtures/fruit/fruit_oranges.yml
rmdir test/fixtures/fruit/
rake db:migrate

For simplicity of this example we want to have a Apple and a Orange in our fruits fixture test/fixtures/fruit_bases.yml

one:
  id: 1
  type: Apple
two:
  id: 2
  type: Orange

This is what their models and tests should look like, as you go through saving your changes lining up the files watch what autotest is telling you. Do not react to autotest until after you have set up apple.rb, orange.rb, apple_test.rb, and orange_test.rb files. See how our inheritance is denoted in the models Fruit::Apple < Fruit::Base and Fruit::Orange < Fruit::Base

app/models/fruit/apple.rb

class Fruit::Apple < Fruit::Base
end

test/unit/fruit/apple_test.rb

require File.dirname(__FILE__) + '/../../test_helper'

class Fruit::AppleTest < Test::Unit::TestCase
  fixtures :fruit_bases
  set_fixture_class :fruit_bases => Fruit::Base

  # loading from the fixture there is only one Apple
  def test_there_should_only_be_one_apple_in_the_fixture
    assert_equal 1, Fruit::Apple.find(:all).length
  end

end

Orange will follow the same pattern as Apple.

Once completed have a sanity check with the rake test:unit task

rake test:unit

Vegetables

Do everything we just did for Fruits, but this time for Vegetables. We want to end up with Vegetable::Base, Vegetable::Carrot, and Vegetable::Potato . Don’t forget to trigger single table inheritance when you generate the base and trim out the non single table inheritance from the Carrots and Potatoes fixtures.

Ingredient model

Now we’ll make the Ingredient model. It will use a polymorphic association so that it can refer to fruits and vegetables. From a Fruit::Base perspective the ingredient model will be a join model to recipes (we’ll update our Fruit::Base code shortly to accomplish this) From recipe’s perspective (which we will generate shortly) the ingredient model can not be used as a join model to fruits and vegetables because the polymorphic side the of ingredients can not be used in this manner.

Generate the model with ‘kind’ being the name used in the polymorphic idiom (kind_id, kind_type) for heterogeneous ingredients and recipe_id used to join a kind of ingredient (fruit, vegetable, etc.) back to the recipe that uses it.

ruby script/generate model Ingredient::Base kind_id:integer kind_type:string recipe_id:integer
mv test/fixtures/ingredient/ingredient_bases.yml test/fixtures/ingredient_bases.yml
rmdir test/fixtures/ingredient/

app/models/ingredient/base.rb

##
# A polymorphic model to associate different kinds of
# specific ingredients with a recipe.  The joining nature
# of the ingredient is one way from its kind to the recipe.
# The recipe cannot go through the ingredient to its kind
# due to a limitation in the polymorphic model.

class Ingredient::Base < ActiveRecord::Base
  set_table_name :ingredient_bases
  belongs_to :kind, :polymorphic => true
  belongs_to :recipe, :class_name => "Recipe::Base", :foreign_key => "recipe_id"
end

There is not anything of significance about the polymorphic declaration of the Ingredient model. However since the Recipe is itself in a namespace we need to help ActiveRecord with the recipe association declaring the class_name of the recipe and the foreign key to it.

Don’t forget to write a unit test for you Ingredient model.

We also need to update the basic fruit and vegetable base models.

Here is the updated app/models/fruit/base.rb

##
# A fruit base class that uses single table inheritance.
# Specific kinds of fruits should inherit from this class.
# A fruit has ingredients as join a model through which
# recipes that include the fruit can be found.

class Fruit::Base < ActiveRecord::Base
  set_table_name :fruit_bases
  has_many :ingredient, :class_name => 'Ingredient::Base',
           :foreign_key => :kind_id, :conditions => "kind_type LIKE 'Fruit::%'"
  has_many :recipes, :through => :ingredient, :uniq => true
end

Notice that the fruit base has many ingredients. But because ingredients are polymorphic (has an kind_id column and a kind_type column) the fruit base needs to declare the foreign key that the ingredient uses to refer to it and what the kind_type column will look like when an ingredient is pointing to fruit. Once that is established then the ingredient model can be used as a join model which we go through to get to the recipe that includes this kind of ingredient.

Update your vegetable base model accordingly.

Recipe model

Now lets generate the recipe model. Its using single table inheritance and we’ll give each recipe a title so this is what our generation looks like. Don’t forget to flatten the fixtures again.

ruby script/generate model Recipe::Base type:string title:string
mv test/fixtures/recipe/recipe_bases.yml test/fixtures/recipe_bases.yml
rmdir test/fixtures/recipe/

app/models/recipe/base.rb

class Recipe::Base < ActiveRecord::Base
  set_table_name :recipe_bases
  has_many :ingredients, :class_name => 'Ingredient::Base', :foreign_key => :recipe_id

  # If we could go through ingredients to their kinds this is how we would make
  # the association.  However polymorphic models cannot be used as a join model
  # when the join is towards the heterogeneous type referenced by the model
  # has_many :kinds, :through => :ingredients
end

Again, we need to clue AR in to which ingredient model we are associating with and what the foreign key used. Don’t for get to write your tests.

Runtime

Integration test

Be sure to check out the source code of the example. It has an integration test that runs the model through its paces using predefined fixtures. This is the test

test/integration/recipes_test.rb

require File.dirname(__FILE__) + '/../test_helper'

##
# Tests the recipes system with the simple yaml fixtures

class RecipesTest < ActionController::IntegrationTest

  # load up all the fixtures
  fixtures :fruit_bases
  set_fixture_class :fruit_bases => Fruit::Base
  fixtures :vegetable_bases
  set_fixture_class :vegetable_bases => Vegetable::Base
  fixtures :ingredient_bases
  set_fixture_class :ingredient_bases => Ingredient::Base
  fixtures :recipe_bases
  set_fixture_class :recipe_bases => Recipe::Base

  def test_fruit_salad_recipe_should_have_apples_and_oranges
    r = Recipe::Base.find(:first, :conditions => {:title => "Fruit Salad"})
    assert r
    r.ingredients.collect do |i|
      assert(i.kind.class == Fruit::Apple || i.kind.class == Fruit::Orange)
    end
  end

  def test_apple_pie_recipe_should_only_have_apples
    r = Recipe::Base.find(:first, :conditions => {:title => "Apple Pie"})
    assert r
    r.ingredients.collect do |i|
      assert_equal Fruit::Apple, i.kind.class
    end
  end

  def test_apple_should_be_in_fruit_salad_and_apple_pie
    a = Fruit::Apple.find(:first)
    # there are 3 recipes but check that the through across the polymorphic
    # ingredients is limited to fruit
    assert_equal 2, a.recipes.length
    a.recipes.each do |r|
      assert(r.title == "Apple Pie" || r.title == "Fruit Salad")
    end

  end

end

You can explicitly run only the integration test with rake thus:

rake test:integration

Or run a specific function with the integration test such as:

ruby test/integration/recipes_test.rb -n test_apple_pie_recipe_should_only_have_apples

Rails console

In the Rails console the following code also shows some behavior that can be exercised with our Recipes, Ingredients, Fruits and Vegetables:

ruby script/console

Run this code in the console

# create an apple and orange ingredient
a = Fruit::Apple.create!
o = Fruit::Orange.create!
apple = Ingredient::Base.create! :kind => a
orange = Ingredient::Base.create! :kind => o

# notice that the recipe hasn't been assigned
# for this ingredient  "recipe_id"=>nil
apple.attributes

r = Recipe::Base.create! :title => "Fruit Salad"
r.ingredients << apple
r.ingredients << orange

# now the apple ingredient is associated with the
# recipe "recipe_id"=>1
apple.attributes

# look at the ingredients in this recipe, we have to go
# through the ingredient to inspect their kinds because
# we can not go through the join model from its polymorphic side
r.ingredients.collect{|i| i.kind}
r.ingredients.collect{|i| i.kind.type}

# make another recipe using the apple object
# (not the first apple ingredient) so the apple
# object can tell us which recipes it belongs to
r = Recipe::Base.create! :title => "Apple Pie"
apple = Ingredient::Base.create! :kind => a
r.ingredients << apple

# and we can see that the apple instance knows which recipes
# it is included with now
a.recipes
a.recipes.collect{|r| r.title}

a.ingredient.collect{|i| i.recipe}
a.ingredient.collect{|i| i.recipe.title}

# note STI finders are smart by its class, base
# returns all fruit, orange only returns oranges
Fruit::Base.find(:all)
Fruit::Orange.find(:all)

Wrap-up

All of the source for this example as available at the following Subversion code repository

svn checkout http://svn.mondragon.cc/svn/recipes/trunk/ recipes

http://svn.mondragon.cc/svn/recipes/trunk/

Here is my lib/tasks/diagrams.rake to generate Railroad’s graphs with these Rake tasks:

rake doc:diagram:controllers   # generate controllers diagram
rake doc:diagram:models        # generate models diagram
rake doc:diagrams              # generate object graphs of models and controllers
namespace :doc do
  namespace :diagram do
    desc "generate models diagram"
    task :models do
      sh "railroad -i -l -a -m -M | dot -Tsvg | sed 's/font-size:14.00/font-size:11.00/g' > doc/models.svg"
    end

    desc "generate controllers diagram"
    task :controllers do
      sh "railroad -i -l -C | neato -Tsvg | sed 's/font-size:14.00/font-size:11.00/g' > doc/controllers.svg"
    end
  end

  desc "generate object graphs of models and controllers"
  task :diagrams => %w(diagram:models diagram:controllers)
end

Rails Models in a Namespace

Posted about 7 years back at Mike Mondragon

If you are starting to get a cluttered model space in your Rails application
you should consider placing your models in a namespace. As an example I’m
going to go through a Rails application I’m calling Recipes. If my
models were starting to have the namespace implied in the class names such as
AppleFruit in app/models/apple_fruit.rb then that’s starting to smell like
rotten apples. A better namespace would be Fruit::Apple in app/models/fruit/apple.rb

This is what we’ll be modeling. Fruits of Apples and Oranges via single table
inheritance. And Vegetables of Potatoes and Carrots through single table
inheritance.

We’ll have Ingredients that belong to Fruit, Vegetables, and Recipes.
Ingredients are a limited kind of join model going from the recipe through
to the kind of ingredient (i.e. fruit or vegetable). Ingredients are a true
join model from the fruit or vegetables back to their associated recipes.
The Ingredient is polymorphic because Fruits and Vegetables are different kinds of
objects.

Finally Recipes are another single table inheritance model but by
convention they will only have ingredients, they won’t be associated
to the kinds of ingredients through the polymorphic Ingredient class.
To access the specific kinds of ingredients from the recipe’s perspective
you must access the collection of ingredients and then program the desired
behavior to access the kinds of ingredients in your application logic.

Here’s a graph of the models we are designing (click for bigger picture):

The graph was made with Railroad which uses
Graphiz to generate the graphs.

All of the source for this example as available at the following Subversion
code repository

svn checkout http://svn.mondragon.cc/svn/recipes/trunk/ recipes

http://svn.mondragon.cc/svn/recipes/trunk/

Project Management and Best Practices in Retrospect-iva

Posted about 7 years back at Wood for the Trees

I don’t hear a lot about project management, even though there’s a lot about how to manage a project. Testing, deployment and source code management get the most attention, and project management seems to get the least. Maybe that’s because it hasn’t been done properly yet and all the solutions out there only address pieces of the overall problem.

So I’m going to try to clarify, for myself mostly, the kind of project management that is needed and why it is so important in development.

Getting It Right

When I say management, I mean a combination of something like Lighthouse and Basecamp, with a serious overhaul of perpsective. An integral part of good development is developing ideas hand-in-hand with the code. Management is all about keeping this communication as agile as the coding process, sticking to priorities, and addressing the right things at the right time.

Is Basecamp sufficient for managing a project? No. DHH even says it is not meant for managing Rails projects; it’s for marketers and managers. It’s only a piece of the puzzle, because it provides no way to easily track code.

Is Trac sufficient for managing a project? No. It is too much like a big todo list and a bug tracker combined. It is very developer-centric—even when the developer is also the designer and manager, there’s no way to make known the other roles. Trac too is only a piece, because it provides no way to easily communicate ideas. A wiki doesn’t cut it.

Are help desks sufficient? No. They are too customer/support-centric. They have no way to easily communicate the ideas of developers and designers.

What about Lighthouse or Unfuddle? For a hosted solution, Lighthouse and Unfuddle combine Trac and Basecamp. That’s going in a decent direction. Anything which integrates different parts of the development process is addressing the need for management. But it’s not enough, because it has no integration with the customer.

What about [insert megolithic answer to everything]? No. It’s too complicated, has too many options and forms, too much information on each page. Something as complicated as Google Analytics, for example, is pushing the boundary of what is acceptable. Complicated applications get in the way of communication and understanding, even if they integrate everything. Simplicity first.

What all of these solutions lack is a focus on the different kinds of users for a project, ways of easily communicating their needs and ways of addressing those needs. Even when the developer, designer, marketer and manager are rolled into the same person, it is important to separate the roles, make them clear and integrate each one’s concerns at the right points.

It’s all about the development process.

The Development Process

I see that there are 9 stages in the development process:

  1. Management: find out the next need to address
  2. Specification/Testing: specify how the need is addressed
  3. Coding: code until the specification passes
  4. Continuous Integration: combine the efforts of multiple coders
  5. Refactoring: clean up the code
  6. Graphic Design (if needed): make the new feature appealing
  7. Deployment: release the latest revision
  8. Marketing: advertise the latest feature
  9. Customer Feedback: find out what is going well and what isn’t

If you see this process as organic, the importance of management becomes much clearer. In fact, I think management is the most important stage, more important than the code itself, for a number of reasons:

  • Management is the first step; without management, testing/coding is arbitrary
  • Management brings everyone into the development process
  • Management gives everyone an overview so they can see the wood for the trees
  • Management encourages communication between everyone
  • Management naturally focuses on the most important aspects
  • Management reinforces and rewards good development
  • Management operates organically, reflecting needs and their importance
  • Management begins the specification/coding cycle
  • Management draws from and feeds into all other stages of development

Disconnecting from these aspects of development is a serious mistake because it denies the organic element of development. Everything needs to converge at some point and management is the most natural way of acknowledging and converging all of a project’s members, roles, ideas, problems and concerns.

Poor management will try to force the development process into a linear pattern. It will approach things as ‘things to do’, ‘features to have’, ‘milestones to reach’, ‘deadlines to meet’, ‘code to test’. Everything will have its place, need to be addressed by a particular person… in short, it’ll look like a Trac installation. The code will feel strained, regimented and will generally be a rather boring thing to deal with. The developer is being strait-jacketed.

On the other hand, a lack of management will result in the process becoming chaotic. The developer will code whatever takes his fancy. Occasionally e-mails or posts containing feedback will find their way into the code, but mostly the code will diverge from the customer’s interests. The organic element has gone mad in this case because the developer is too isolated.

But good management acknowledges the organic aspect of development and lets the code flow. It translates ideas into specifications just as test-first translates specifications into code. Good project management will create and maintain strong channels of communication between developer, designer, marketer, manager and customer. The real needs for the project will appear of their own accord as different ideas converge in one place.

But that sounds much easier than it actually is. There isn’t yet an application out there which integrates all of those roles together, but some are closer than others. I think Retrospectiva could be the one which gets there first.

What is needed for good management

The three major aspects of project management are development, collaboration and integration. There needs to be a way to develop, track that development, and focus it. There needs to be collaboration and communication surrounding that development. There needs to be a process of integration between management and the other stages of development.

Ideally, a project management system will have the following aspects:

  • Stories: isolated stories to be resolved (bug, feature, question, idea)
  • Dynamic properties: status, milestone, persona, feature, assigned user, assigned group
  • Reinforcement: aspects of the application NOT to change (positive feedback, robust code)
  • Personae: ability to define personae, like power users, buyers, sellers, novices, etc.
  • Milestones: rough organisation of stories and deadlines
  • Messages: site-wide (like Basecamp) and for each milestone
  • Roles: developer, guest, manager, designer, customer, marketer, administrator
  • Groups: optional story development by groups
  • Interfaces: different interfaces for each role and/or group
  • Cross-referencing: referencing between stories, messages, milestones, source
  • Testing integration: update stories with progress on tests (e.g. Tesly)
  • Coverage Integration: stories for area of test coverage, whether 100% covered or not
  • SCM integration: update stories through commit logs
  • Continuous Build Integration: create stories for failed builds
  • Error Notification Integration: create stories for application errors
  • Customer Integration: create stories for customer feedback (positive & negative)

Most importantly, the interface(s) needs to be extremely clean. Lighthouse goes a long way in making a highly readable, even pleasurable interface. Most of the aspects I list above can exist on their own, meaning the application itself will have many facets, but each very easily understood. Cross-referencing is probably the most important of them all, since it will bring together the various aspects.

The shift in perspective I suggest for project management is to not focus on managing people (like Basecamp) or code (like Trac) or users, but ideas. Those ideas will never disappear from view, unlike tickets on Trac or todo lists on Basecamp. As stories grow and connect with new stories (like associating tickets), everyone will see the evolution and development of features, in which way the project is going, and be able to react better to the movement of the project. In a way, project management also begins to document the project, but more importantly, it shows in black and white how ideas become code and how they evolve. At the centre of the project should be a cloud of ideas which each role can see differently.

Just to give a little hypothetical situation: at the beginning of your project you had a simple user authentication system. Over time, users talked about adding Open ID. Management wanted an authorisation system and an admin interface. Designers wanted a cute widget that pops down with AJAX. Developers wanted to extract it into a plugin. All of these ideas would be associated and appear together in a good management system, showing the time each one was added and completed, the role which initially suggested it, and the group or user responsible for implementing it. All the bugs, notes, support questions, requests and feature stories will clump together and naturally point towards what is needed next, if anything.

This form of project management could very well revolutionise the way development is perceived. Or is it a load of bullshit?

Project Management and Best Practices in Retrospect-iva

Posted about 7 years back at Wood for the Trees

I don’t hear a lot about project management, even though there’s a lot about how to manage a project. Testing, deployment and source code management get the most attention, and project management seems to get the least. Maybe that’s because it hasn’t been done properly yet and all the solutions out there only address pieces of the overall problem.

So I’m going to try to clarify, for myself mostly, the kind of project management that is needed and why it is so important in development.

Getting It Right

When I say management, I mean a combination of something like Lighthouse and Basecamp, with a serious overhaul of perpsective. An integral part of good development is developing ideas hand-in-hand with the code. Management is all about keeping this communication as agile as the coding process, sticking to priorities, and addressing the right things at the right time.

Is Basecamp sufficient for managing a project? No. DHH even says it is not meant for managing Rails projects; it’s for marketers and managers. It’s only a piece of the puzzle, because it provides no way to easily track code.

Is Trac sufficient for managing a project? No. It is too much like a big todo list and a bug tracker combined. It is very developer-centric—even when the developer is also the designer and manager, there’s no way to make known the other roles. Trac too is only a piece, because it provides no way to easily communicate ideas. A wiki doesn’t cut it.

Are help desks sufficient? No. They are too customer/support-centric. They have no way to easily communicate the ideas of developers and designers.

What about Lighthouse or Unfuddle? For a hosted solution, Lighthouse and Unfuddle combine Trac and Basecamp. That’s going in a decent direction. Anything which integrates different parts of the development process is addressing the need for management. But it’s not enough, because it has no integration with the customer.

What about [insert megolithic answer to everything]? No. It’s too complicated, has too many options and forms, too much information on each page. Something as complicated as Google Analytics, for example, is pushing the boundary of what is acceptable. Complicated applications get in the way of communication and understanding, even if they integrate everything. Simplicity first.

What all of these solutions lack is a focus on the different kinds of users for a project, ways of easily communicating their needs and ways of addressing those needs. Even when the developer, designer, marketer and manager are rolled into the same person, it is important to separate the roles, make them clear and integrate each one’s concerns at the right points.

It’s all about the development process.

The Development Process

I see that there are 9 stages in the development process:

  1. Management: find out the next need to address
  2. Specification/Testing: specify how the need is addressed
  3. Coding: code until the specification passes
  4. Continuous Integration: combine the efforts of multiple coders
  5. Refactoring: clean up the code
  6. Graphic Design (if needed): make the new feature appealing
  7. Deployment: release the latest revision
  8. Marketing: advertise the latest feature
  9. Customer Feedback: find out what is going well and what isn’t

If you see this process as organic, the importance of management becomes much clearer. In fact, I think management is the most important stage, more important than the code itself, for a number of reasons:

  • Management is the first step; without management, testing/coding is arbitrary
  • Management brings everyone into the development process
  • Management gives everyone an overview so they can see the wood for the trees
  • Management encourages communication between everyone
  • Management naturally focuses on the most important aspects
  • Management reinforces and rewards good development
  • Management operates organically, reflecting needs and their importance
  • Management begins the specification/coding cycle
  • Management draws from and feeds into all other stages of development

Disconnecting from these aspects of development is a serious mistake because it denies the organic element of development. Everything needs to converge at some point and management is the most natural way of acknowledging and converging all of a project’s members, roles, ideas, problems and concerns.

Poor management will try to force the development process into a linear pattern. It will approach things as ‘things to do’, ‘features to have’, ‘milestones to reach’, ‘deadlines to meet’, ‘code to test’. Everything will have its place, need to be addressed by a particular person… in short, it’ll look like a Trac installation. The code will feel strained, regimented and will generally be a rather boring thing to deal with. The developer is being strait-jacketed.

On the other hand, a lack of management will result in the process becoming chaotic. The developer will code whatever takes his fancy. Occasionally e-mails or posts containing feedback will find their way into the code, but mostly the code will diverge from the customer’s interests. The organic element has gone mad in this case because the developer is too isolated.

But good management acknowledges the organic aspect of development and lets the code flow. It translates ideas into specifications just as test-first translates specifications into code. Good project management will create and maintain strong channels of communication between developer, designer, marketer, manager and customer. The real needs for the project will appear of their own accord as different ideas converge in one place.

But that sounds much easier than it actually is. There isn’t yet an application out there which integrates all of those roles together, but some are closer than others. I think Retrospectiva could be the one which gets there first.

What is needed for good management

The three major aspects of project management are development, collaboration and integration. There needs to be a way to develop, track that development, and focus it. There needs to be collaboration and communication surrounding that development. There needs to be a process of integration between management and the other stages of development.

Ideally, a project management system will have the following aspects:

  • Stories: isolated stories to be resolved (bug, feature, question, idea)
  • Dynamic properties: status, milestone, persona, feature, assigned user, assigned group
  • Reinforcement: aspects of the application NOT to change (positive feedback, robust code)
  • Personae: ability to define personae, like power users, buyers, sellers, novices, etc.
  • Milestones: rough organisation of stories and deadlines
  • Messages: site-wide (like Basecamp) and for each milestone
  • Roles: developer, guest, manager, designer, customer, marketer, administrator
  • Groups: optional story development by groups
  • Interfaces: different interfaces for each role and/or group
  • Cross-referencing: referencing between stories, messages, milestones, source
  • Testing integration: update stories with progress on tests (e.g. Tesly)
  • Coverage Integration: stories for area of test coverage, whether 100% covered or not
  • SCM integration: update stories through commit logs
  • Continuous Build Integration: create stories for failed builds
  • Error Notification Integration: create stories for application errors
  • Customer Integration: create stories for customer feedback (positive & negative)

Most importantly, the interface(s) needs to be extremely clean. Lighthouse goes a long way in making a highly readable, even pleasurable interface. Most of the aspects I list above can exist on their own, meaning the application itself will have many facets, but each very easily understood. Cross-referencing is probably the most important of them all, since it will bring together the various aspects.

The shift in perspective I suggest for project management is to not focus on managing people (like Basecamp) or code (like Trac) or users, but ideas. Those ideas will never disappear from view, unlike tickets on Trac or todo lists on Basecamp. As stories grow and connect with new stories (like associating tickets), everyone will see the evolution and development of features, in which way the project is going, and be able to react better to the movement of the project. In a way, project management also begins to document the project, but more importantly, it shows in black and white how ideas become code and how they evolve. At the centre of the project should be a cloud of ideas which each role can see differently.

Just to give a little hypothetical situation: at the beginning of your project you had a simple user authentication system. Over time, users talked about adding Open ID. Management wanted an authorisation system and an admin interface. Designers wanted a cute widget that pops down with AJAX. Developers wanted to extract it into a plugin. All of these ideas would be associated and appear together in a good management system, showing the time each one was added and completed, the role which initially suggested it, and the group or user responsible for implementing it. All the bugs, notes, support questions, requests and feature stories will clump together and naturally point towards what is needed next, if anything.

This form of project management could very well revolutionise the way development is perceived. Or is it a load of bullshit?

Capistrano 2.0, upgrading & fitting into a size 0 dress

Posted about 7 years back at Wood for the Trees

The improvements to Capistrano are much welcomed. My deployment recipe is now half the length it used to be and it is much easier to follow what is happening for my many types of deployment. I love the new features added, mostly dealing with manipulating scopes and enhancing the user’s ability to extend the core framework.

Review of new features

namespaces: Like Rake, you can namespace your tasks and group them together more sensibly. This feature alone is worth upgrading for just to make your scripts more sensible and easier to read.

events: Like Rails, you can now perform tasks before or after other ones rather than using the hacky ‘before_something’ and ‘after_something’. Much cleaner and much faster too.

strategies: In addition to checkout, you can now deploy via export and copy and use different strategies for deployment, such as using export for your copy_strategy rather than zips and tarballs.

scoping: All sorts of scoping has been introduced in Capistrano 2.0, from namespacing to single execution of “run” and “sudo”, allowing you to define specific roles or hosts in which your commands run.

help: Capistrano 2.0 now has a more verbose way of explaining tasks with cap -e task_name. You’ll realise how useful this is when you use it for the built-ins as well as your own.

All in all, Capistrano is pretty simple, but it is the way it is written that makes it appear so much simpler than it really is. Capistrano 2.0 takes that to a new level, not groundbreaking perhaps, but definitely a lot cleaner than its previous releases.

Upgrading from 1.4.1

There is no need to change config/deploy.rb out of the box. Capistrano 2.0 is nicely backwards compatible, unlike other things out there, and, at least for me, nothing broke because of the upgrade.

You can look at Capistrano’s instructions for upgrading, if you want to know what is being done, but for the impatient, here are the steps you have to follow before we can start drying up your deploy script.

1. Install the new version of capistrano:
sudo gem install capistrano
2. cd project_root & run capify
~# cd projroot
projroot# capify .
3. Upgrade previous deployments to use the new revision tracking system
projroot# cap -f upgrade -f Capfile upgrade:revisions

4. Rinse and repeat for each of your deployment targets

Getting your deploy.rb into its new size 0 dress

You may now have the very understandable urge to slim down your deployment recipes. With the introduction of Capistrano 2.0, I found my deploy.rb reduced to less than half the size. Below, I cover the areas which you should focus on to get that deploy script into its new size 0 dress.

Anatomy of my deploy.rb

  1. requires: capistrano-ext, mongrel_cluster, etc.
  2. global, stage and custom variables
  3. event chains
  4. rewriting built-ins: web:disable and web:enable
  5. extra tasks: fixing permissions, copying mongrel confs, etc.
  6. custom deploy tasks: long, normal, quick
  7. maintenance tasks: backup, restore

Variables

More than before, variables are the lynch-pin of slimming everything down. The first thing you should do is look over every task rewrite or custom task and see how it can be turned into a simple set :var, true/false/whatever. Capistrano 2.0 will make it very easy to do this.

With Capistrano 2.0, you should use the set command religiously, both for built-in and custom tasks.

I personally set the following at the top of my recipe.

  • Global variables: stages, deploy_via
  • Application specific: application, repository, user, scm_username
  • Deployment specific: deploy_to, rails_env
  • Custom variables: serving_via, suexec, suexec_user, suexec_group, disable_template

Deployment Strategy

I would personally suggest using xport for your deploy_via strategy unless you have a reason for using heckout or copy.

Using Namespaces

Namespaces make it dead simple to group common tasks, like different restart methodologies. I use a serving_via variable which translates into the reload:whatever task to run for restarting the application. For example:

namespace :reload do
  desc "Default reloading procedure"
  task :default do
    mongrels
  end
  desc "Reload an FCGI application"
  task :fcgi, :roles => :app do
    sudo "#{current_path}/script/process/reaper -a graceful -d #{current_path}/public/dispatch.fcgi"
  end
  desc "Reload an LSAPI application"
  task :lsapi, :roles => :app do
    sudo "/usr/local/litespeed/bin/lswsctrl restart"
  end
  desc "Give the mongrels a bath"
  task :mongrels, :roles => :app do
    restart_mongrel_cluster
  end
end

Note: I warn against using restart as a namespace because it clashes with the built-in task and, in certain instances, results in infinite recursion.

Maintenance Splash

The biggest change in Capistrano you may need to worry about is the removal of delete and render. Don’t despair, though, because creating a maintenance splash is still easy. This is my rewrite:

desc "Generate a maintenance.html to disable requests to the application."
deploy.web.task :disable, :roles => :web do
  remote_path = "#{shared_path}/system/maintenance.html"
  on_rollback { run "rm #{remote_path}" }
  template = File.read(disable_template)
  deadline, reason = ENV["UNTIL"], ENV["REASON"]
  maintenance = ERB.new(template).result(binding)
  put maintenance, "#{remote_path}", :mode => 0644
end

desc "Re-enable the web server by deleting any maintenance file."
deploy.web.task :enable, :roles => :web do
  run "rm #{shared_path}/system/maintenance.html"
end

Using events

Like the before and after filters in Rails, you can now cleanly chain together tasks. I’m a sucker for one-line solutions and these are really so simple that it makes my heart bleed:

before "deploy:restart", "fix:permissions"
before "deploy:migrate", "db:backup"
after "deploy:symlink", "deploy:cleanup"
after "deploy:update_code", "deploy:web:disable"
after "deploy:restart", "deploy:web:enable"

capistrano-ext & multistage

I highly recommend the use of multistage. It comes with the capistrano-ext gem (which has been upgraded to Capistrano 2.0, of course).

Basically, it separates the concerns of different deployments. If, like me, you like having a few other versions of your application out there, like a staging area, a testing area for bleeding edge features, and, of course, the production site, separating these in Capistrano before 2.0 was very irritating. Multistage sorts that out very nicely.

By default, you must specify the stage you wish to deploy. This behaviour can be overridden by setting the default_stage variable, but I like being explicit. This is what using stages looks like:

# cap production deploy

If you don’t provide ‘production’, it’ll complain and abort.

Using multistage is dead easy. Put this at the top of your deploy.rb:

  require 'capistrano/ext/multistage'
  set :stages, %w(staging production testing)

Run the task for generating your stage deploy files:

projroot# cap multistage:prepare

This will create a recipe file for each stage in a new config/deploy directory (exactly like Rails environments). Now, in each stage recipe, add all of your stage-specific tasks and variables. For example:

set :rails_env, "stage"
set :application, "staging.example.com"
set :deploy_to, "/var/www/#{application}"

Now switching between different deployments is a breeze. Just make a new recipe file for it with the necessary variables and you’re set.