PORO validators in Rails

The other day, I wanted to extract some validation logic from an Active Record model into its own class.

Initially I tried Rails’ validates_with and ActiveModel::Validator.

It went something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

class MyModel < ActiveRecord::Base
  validates_with Validator
end

class MyModel
  class Validator < ActiveModel::Validator
    def validate(record)
      @record = record
      validate_not_bad
    end

    private

    def validate_not_bad
      record.errors.add_to_base("Bad!") if bad?
    end

    def bad?
      properties.include?(:evil) || properties.include?(:nasty)
    end

    def properties
      @properties ||= record.some_expensive_lookup
    end

    def record
      @record
    end
  end
end

I wanted private helper methods, and I didn’t want to pass the record around as method arguments to each, so I treated the validator as a regular object, though Rails only offered me a validate method, not an initializer.

But it soon became apparent that it indeed wasn’t the regular object I hoped for. In my tests, the memoized properties from one validation run would still be around when validating a second time.

The validator was not initialized once per validation run, as one might expect, but only once when the class loads.

So I rewrote it as a plain old Ruby object (“PORO”) with a minimum of boilerplate and glue, and as far as I can tell, it works better, with less magic and less surprises:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

class MyModel < ActiveRecord::Base
  validate do |record|
    Validator.new(record).validate
  end
end

class MyModel
  class Validator
    def initialize(record)
      @record = record
    end

    def validate
      validate_not_bad
    end

    private

    # The exact same private methods.
  end
end

After a brief look at the Rails Validator code, I suspect the class is defensible as a base for Rails’ built-in validations, but it doesn’t seem worthwhile to build your own validators around it, to me. But please let me know if I’m missing something.

Richard Paterson will kill you

Richard Paterson is Whyte & Mackay’s “master blender”, which means he doubtless knows a lot about whisky, and as a result he’s clearly asked to appear on TV to guide people around the world of whisky and how best to appreciate it.

Let him guide you now:

But what is a fun bit of banter once can quickly become sinister when you hear it again. And again. And again.

“I’ll kill you.”

“I’ll kill you.”

“I’ll kill you.”

… so don’t say you haven’t been warned.

Building Internal Apps

My first job out of college was with JPMorgan Chase. I worked on the Production Assurance team which was responsible for incident and change management. One thing that quickly became apparent while working there is that JPMorgan Chase depends on a lot of software systems. Interestingly, many of those systems were hacked together and poorly supported.

Here’s how that situation commonly happened: somebody in a business unit identifies a way to become more efficient at what they do using software. Having an app built internally would be too involved and costly, so the person learns how to work with Lotus Notes or Access to build the app themselves. Other people want to use the software as well, and before long a large number of people depend on a poorly supported application to do their jobs. Of course, because there isn’t any official IT support for the application, it’s running on a desktop under somebody’s cubicle and isn’t very reliable. Repeat times a few hundred.

I contributed to the proliferation of poorly supported software myself while I was there by building some tools to help the Production Assurance team monitor services and build incident reports.

Factors

Most companies would rather spend time solving their customers’ pain than their own internal pain. The tendency to favor building customer-facing features over internal tools is typical, but most companies do too little of the latter.

Key factors that deter from building internal apps:

Cost. If only considering the financial cost, it’s often cheaper to pay somebody to do a task manually than it is to build (and support) software to automate the work. However, for software companies, the opportunity cost will likely be even greater than the financial cost.

And a few reasons that favor internal apps:

Scale. Doing the work manually may be cheaper at current scale, but not if the business grows 10x. This could be taken as a reason to wait to build, although once an organization becomes reliant on a process or alternative solution, it can be difficult to change (example at many companies: Salesforce).

Quality of Life. Most people would rather not spend time doing repetitive manual tasks. Even if you can hire somebody to do the work, will it be fulfilling? Good people want good tools.

Next Best Alternative. As described based on my experience at Chase, if you don’t build good internal tools, your company will likely end up relying on Excel macros or Access applications. That’s good for some tasks, but not great for something that is a core competency of the business.

Error Rates. Software can have bugs, but it’s more likely for people to make mistakes when doing tasks manually than it is when those tasks are automated. Bugs in software can be fixed, but manually errors can only be solved with involving more people to do QA.

Customer Support. Having good internal tools will likely enable a company to provide better support. Faster response times are valuable, especially when on the phone with a customer.

Perfect Fit. If you build an internal tool, it will do exactly what you need it to do, it in the way that’s best for your business. Using another product often forces you to shape your processes around the tool, rather than the other way around.

37signals

I admire 37signals, so I often try to think how they would approach problems based on what they’ve written publicly. A quick search revealed that they’ve built numerous internal apps: Iterations, In/Out, Extra Extra, Smiley, 37, Dash, and likely more.

RDoc XSS vulnerability CVE-2013-0256 releases 3.9.5, 3.12.1, 4.0.0.rc.2

RDoc versions 3.9.5, 3.12.1 and 4.0.0.rc.2 have been released!

• home: github.com/rdoc/rdoc
• rdoc: docs.seattlerb.org/rdoc
• bugs: github.com/rdoc/rdoc/issues

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the rdoc and ri tools for generating and displaying documentation from the command-line.

Vulnerability Description

RDoc documentation generated by rdoc 2.3.0 through rdoc 3.12 and prereleases up to rdoc 4.0.0.preview2.1 are vulnerable to an XSS exploit. This exploit may lead to cookie disclosure to third parties.

The exploit exists in darkfish.js which is copied from the RDoc install location to the generated documentation.

RDoc is a static documentation generation tool. Patching the library itself is insufficient to correct this exploit. Those hosting rdoc documentation will need to apply the following patch. If applied while ignoring whitespace, this patch will correct all affected versions:

diff --git darkfish.js darkfish.js
index 4be722f..f26fd45 100644
--- darkfish.js
+++ darkfish.js
@@ -109,13 +109,15 @@ function hookSearch() {
 function highlightTarget( anchor ) {
   console.debug( "Highlighting target '%s'.", anchor );

-  $("a[name=" + anchor + "]").each( function() {
-    if ( !$(this).parent().parent().hasClass('target-section') ) {
-      console.debug( "Wrapping the target-section" );
-      $('div.method-detail').unwrap( 'div.target-section' );
-      $(this).parent().wrap( '<div class="target-section"></div>' );
-    } else {
-      console.debug( "Already wrapped." );
+  $("a[name]").each( function() {
+    if ( $(this).attr("name") == anchor ) {
+      if ( !$(this).parent().parent().hasClass('target-section') ) {
+        console.debug( "Wrapping the target-section" );
+        $('div.method-detail').unwrap( 'div.target-section' );
+        $(this).parent().wrap( '<div class="target-section"></div>' );
+      } else {
+        console.debug( "Already wrapped." );
+      }
     }
   });
 };

RDoc 3.9.5, 3.12.1 and RDoc 4.0.0.rc.2 and newer are not vulnerable to this exploit.

This exploit was discovered by Evgeny Ermakov corwmh@gmail.com.

This vulnerability has been assigned the CVE identifier CVE-2013-0256.

RDoc 3.9.5

RDoc 3.9.5 was released to match the RDoc in ruby 1.9.3-p385.

• Bug fixes
  • Fixed an XSS exploit in darkfish.js. This could lead to cookie disclosure to third parties. See CVE-2013-0256.rdoc for full details including a patch you can apply to generated RDoc documentation.

RDoc 3.12.1

RDoc 3.12.1 was updated as the latest stable release as RDoc 4 is not yet ready.

• Bug fixes
  • Fixed an XSS exploit in darkfish.js. This could lead to cookie disclosure to third parties. See CVE-2013-0256.rdoc for full details including a patch you can apply to generated RDoc documentation.
  • Ensured that rd parser files are generated before checking the manifest.

RDoc 4.0.0.rc.2

• Minor enhancements

  • Added current heading and page-top links to HTML headings.

• Bug fixes

  • Fixed an XSS exploit in darkfish.js. This could lead to cookie disclosure to third parties. See CVE-2013-0256.rdoc for full details including a patch you can apply to generated RDoc documentation.
  • Fixed parsing of multibyte files with incomplete characters at byte 1024. Ruby bug #6393 by nobu, patch by Nobuyoshi Nakada and Yui NARUSE.
  • Fixed rdoc -E. Ruby Bug #6392 and (modified) patch by Nobuyoshi Nakada
  • Added link handling to Markdown output. Bug #160 by burningTyger.
  • Fixed HEREDOC output for the limited case of a heredoc followed by a line end. When a HEREDOC is not followed by a line end RDoc is not currently smart enough to restore the source correctly. Bug #162 by Zachary Scott.
  • Fixed parsing of executables with shebang and encoding comments. Bug #161 by Marcus Stollsteimer
  • RDoc now ignores methods defined on constants instead of creating a fake module. Bug #163 by Zachary Scott.
  • Fixed ChangeLog parsing for FFI gem. Bug #165 by Zachary Scott.
  • RDoc now links #=== methods. Bug #164 by Zachary Scott.
  • Allow [] following argument names for TomDoc. Bug #167 by Ellis Berner.
  • Fixed the RDoc servlet for home and site directories. Bug #170 by Thomas Leitner.
  • Fixed references to methods in the RDoc servlet. Bug #171 by Thomas Leitner.
  • Fixed debug message when generating the darkfish root page. Pull Request #174 by Thomas Leitner.
  • Fixed deletion of attribute ri data when a class was loaded then saved. Issue #171 by Thomas Leitner.

Mama Otter and 3 Cubs

Last fall Nicole and I had the pleasure of filming and photographing these wild river otters in Yellowstone. Mama (on the left) mostly sleeps while the cubs roughhouse on the bank and in the water, then she preens them before putting them all back to bed.

Handle incoming email with Griddler

For all the likes, shares, tweets, pokes, follows, and friends, there’s a fundamental core to the internet that, no matter how hard some might hope, will never go away—email. Rails has built-in support for outgoing mail with ActionMailer, but nothing on the omakase menu resembles handling incoming mail. To help with that, today we’re happy to release Griddler, hot off the… ahem… presses.

Griddler is a Rails engine that provides an endpoint for the SendGrid Parse API that hands off a preprocessed email object to a class implemented by you. We’re happy to look at pull requests that interface with other email services.

Setup

To get Griddler integrated with your app, you may start by adding Griddler to your Gemfile, and bundle away:

gem 'griddler'

When adding it to your app, Griddler automatically adds an endpoint to your routes table resembling the following:

match '/email_processor' => 'griddler/emails#create', via: :post

But you may copy, paste, and modify that anywhere else in your routes for the purposes of your application.

Once Sendgrid posts to your endpoint Griddler will take care of packaging up the important bits of that data and providing a nice Griddler::Email object for you. The contract we expect you to go in on with Griddler at this point is that you will implement a class called EmailProcessor, containing a class method called process, which we will be passing that packaged up instance of Griddler::Email into.

For example, in lib/email_processor.rb:

class EmailProcessor
  def self.process(email)
    # all of your application-specific code here - creating models,
    # processing reports, etc
  end
end

In implementing that interface you will have a method to handle the data coming in from SendGrid in the form of the email object, which contains the following attributes:

• to
• from
• subject
• body
• raw_body

Of those, to, from, and subject fall on the obvious side as to their purpose. What isn’t entirely obvious (but very cool) is that Griddler helps you handle the email body by cleaning up replies and providing the important parts of an email before -- Reply ABOVE THIS LINE -- in the body attribute. Note that the reply delimeter is adjustable in the configuration. We keep raw_body around, as contains everything before Griddler scrubs it into body so that you may use the contents for other purposes.

There is much more information in the Griddler README explaining the details, configuration, testing, and other bits.

If you like it, let us know what you think! As always, you can find the code on GitHub. We look forward to hearing all of the ways you use Griddler!

Episode 34: Very little comes to those who wait

Episode 34: Very little comes to those who wait:

In this week’s podcast episode, Ben Orenstein is joined by Steve Snyder, Entrepreneur in Residence at the law firm, Gesmer Updegrove LLP. Ben and Steve discuss Steve’s history, his unique position at the law firm, mistakes to avoid, and advice and guidance to entrepreneurs just starting out.

• Episode Notes and Links
• Subscribe via iTunes
• Subscribe via RSS
• Direct Download

2.0.0.pre8: final prerelease

I’m proud to announce the release of Hobo 2.0.0.pre8. We’re planning on releasing this as 2.0.0 final in a week or so if no significant issues are found, so please test this release against your applications.

This release has been tested against Rails 3.2.11 on Ruby 1.9.3-p374, Ruby 1.8.7-p378 and JRuby 1.7.2.

Changes

The vast majority of updates to Hobo for 2.0.0.pre8 were to the documentation.

• The default theme has been changed to Bootstrap

• MarkdownString will now use Kramdown, RDiscount or Maruku in preference to Bluecloth if these are available.

Minor Changes

• <sortable-input-many> and <sortable-collection> have been moved to the hobo_jquery_ui gem

• fix for recognize_page_path when used for non-GET requests

• fix for AJAX response when no part is specified

• <view> has gained a force attribute

• ILIKE is now used for automatic scopes on Postgres

• hobo_login now respects the return code of its block

• the user class now works without a lifecycle

• fixes for non-empty relative url root

2.0.0.pre8: final prerelease

I’m proud to announce the release of Hobo 2.0.0.pre8. We’re planning on releasing this as 2.0.0 final in a week or so if no significant issues are found, so please test this release against your applications.

This release has been tested against Rails 3.2.11 on Ruby 1.9.3-p374, Ruby 1.8.7-p378 and JRuby 1.7.2.

Changes

The vast majority of updates to Hobo for 2.0.0.pre8 were to the documentation.

• The default theme has been changed to Bootstrap

• MarkdownString will now use Kramdown, RDiscount or Maruku in preference to Bluecloth if these are available.

Minor Changes

• <sortable-input-many> and <sortable-collection> have been moved to the hobo_jquery_ui gem

• fix for recognize_page_path when used for non-GET requests

• fix for AJAX response when no part is specified

• <view> has gained a force attribute

• ILIKE is now used for automatic scopes on Postgres

• hobo_login now respects the return code of its block

• the user class now works without a lifecycle

• fixes for non-empty relative url root

Wrestling with Bundler

I’ve been working on upgrading an app from Rails 2.3 to 3.0. As one of the first steps, I’m converting the app to use Bundler. Currently, all of the gems are in the vendor/gems directory, except for gems which need to be compiled — those are installed as system gems. I ran into some unexpected challenges with the upgrade. Thankfully, a friend had some advice.

wrestling with Bundler. and losing.

— Dan Manges (@dan_manges) February 3, 2013

@dan_manges it’s ok… Bundler is in a heavier weight class.

— Mike Busch (@mikelikesbikes) February 3, 2013

@dan_manges just don’t let it shoot out your legs. Beat it with speed and agility.

— Mike Busch (@mikelikesbikes) February 3, 2013

Approach

To introduce Bundler as incrementally as possible, I wanted to try to have Bundler load the system gems and the gems from the vendor/gems directory without making any other changes. I didn’t want to have to install Bundler as a system gem, or change Capistrano scripts to run any additional commands as part of the deployment process. I was also hoping that the production runtime environment would remain as much the same as possible. I ran into two issues, both related to groups.

For an example, I’m going to use this Gemfile.

source :rubygems

gem "rails", "2.3.16"

group :development do
  gem "faker", "1.1.2"
  gem "thin", "1.5.0"
end

I was able to follow Bundler's instructions for Rails 2.3 successfully. If you remove the error handling the instructions, you end up with:

require "rubygems"
require "bundler"
Bundler.setup

I wanted to make sure that development gems weren’t loaded in production, so I added a group to the setup call to only load the default gems and the gems for the current environment.

Bundler.setup(:default, Rails.env)

Load Path

The first problem that I noticed is that my development gems were available on the load path in a non-development irb console. Since the Faker gem is only in the development group, it shouldn’t be available.

:001 > $LOAD_PATH.grep(/faker/)
 => ["~/.rvm/gems/ruby-1.8@blog-bundler/gems/faker-1.1.2/lib"] 

It took me a while to figure out why this was happening. I learned that Bundler adds -r bundler/setup to the RUBYOPT environment variable. Rails starts its console by making an exec call to irb. The require of bundler/setup is equivalent to calling Bundler.setup without any argument, so Bundler adds all gems to the load path. It doesn’t actually require the gems, but I’d still prefer to not have development gems on the load path in production.

require "rubygems"
require "bundler"
Bundler.setup(:default)
p $LOAD_PATH.grep(/faker/)
#=> []
exec "irb"
:001 > $LOAD_PATH.grep(/faker/)
 => ["~/.rvm/gems/ruby-1.8@blog-bundler/gems/faker-1.1.2/lib"] 

Bundler::GemNotFound

The next problem that I ran into was that Bundler was raising an exception stating that gems needed to be installed in production even if they were only declared in the development group. Using the Gemfile from above and uninstalling thin, I’d get an error when calling Bundler.setup(:default).

# gems/bundler-1.2.0/lib/bundler/spec_set.rb:90:in `materialize':
# Could not find thin-1.5.0 in any of the sources (Bundler::GemNotFound)

Intuitively, I wouldn’t think that Bundler.setup(:default) (before the exec irb) would require development gems to be installed since they’re not actually used.

Bundle Install

After digging through the Bundler source code for a while, I figured out the problem. A side effect of running bundle install is that Bundler writes a .bundle/config file. If you run bundle install --without development, then the file looks like:

--- 
BUNDLE_WITHOUT: development

Setting BUNDLE_WITHOUT makes the irb exec and the Bundler.setup call work as expected. Development gems no longer get added to the load path inside irb, and bundler no longer requires development gems to be installed even though they’re not loaded. You can test this by adding this line to the script above:

ENV["BUNDLE_WITHOUT"] = "development"

I think this issue happens because Bundler expects bundle install to be run to install gems. Because I already had system gems installed and all other gems unpacked into the vendor/gems directory, I didn’t think that I needed to run it. More importantly, I specifcally didn’t want to run it in production yet because I wanted to roll this release out without modifying the Capistrano scripts. It’s unintuitive that Bundler.setup(:some_group) will require gems in other groups to be installed unless BUNDLE_WITHOUT is also set. Or, if Bundler does expect install to run before any calls to setup, then Bundler could raise an exception to communicate that. I’d like to submit a patch to make this more clear, although changes may cause issues with backwards compatibility. I’ll either work around this issue for now, or go ahead and update deployment scripts to run bundle install when deploying, though it should be unnecessary.

Rails 3.0.20 and 2.3.16

In this episode, Jason and Peter talk about the latest security updates to Rails, Threads in Ruby, and the usual round up of interesting blog posts.

What happens when MiniTest runs, or, what I think about testing using classes

I think I can see the end of my Ruby Testing Quest in sight.

As one part of the final leg of this journey, I want to take a not-too-deep dive into how the some principle testing frameworks actually work, so that I can better clarify in my own mind what distinguishes them, and perhaps, if we are lucky, draw out some attributes that may help me. Somehow.

We’re going to start with MiniTest. We’ll also look at RSpec and Kintama, but not right now. This is already crazy-long.

A simple MiniTest example

Let’s say you have the following test case:

require "minitest/autorun"

class SomethingTest < MiniTest::Unit::TestCase
  def setup
    @something = Object.new
  end

  def test_something
    refute_nil @something
  end

  def teardown
    @something = nil
  end
end

This test is obviously extremely dull and pointless, but it contains just enough to exercise the major parts of MiniTest that I care about.

The two hallmark attributes here are:

• creating an explicit subclass of a framework class (SomethingTest < MiniTest::Unit::TestCase)
• defining test behaviour within explicit methods (def setup, def test_something and def teardown).

Running the test

The simplest way of running this test would be to save it in some file (something_test.rb) and run it from the command-line.

$ ruby something_test.rb
Run options: --seed 24486

# Running tests:

.

Finished tests in 0.000866s, 1154.7344 tests/s, 1154.7344 assertions/s.

1 tests, 1 assertions, 0 failures, 0 errors, 0 skips

So – what’s actually happening here?

Autorun

The first line in the file (require "minitest/autorun"), when evaluated, loads the MiniTest library and then calls MiniTest::Unit.autorun, which installs an at_exit hook – a block of code that will be run when this Ruby interpreter process starts to exit.

Our command in the shell (ruby something_test.rb) tells Ruby to load the contents of something_test.rb, which after loading MiniTest simply defines a class with some methods, and nothing else, so after the definition of SomethingTest is finished Ruby starts to exit, and the at_exit code is invoked.

Within this block, a few things happen, but only a small part is particularly relevant to us at the moment: the method MiniTest::Unit.new.run is run, with the contents of ARGV from the command line (in this case an empty Array, so we’ll ignore them as we continue).

MiniTest::Unit, a.a. the “runner”

The call to MiniTest::Unit.new.run simply calls MiniTest::Unit.runner._run, passing the command-line arguments through. runner is a class method on MiniTest::Unit, which returns an instance of MiniTest::Unit by default, although it can be configured to return anything else by setting MiniTest::Unit.runner = <something else>.

So, an instance of MiniTest::Unit was created in the unit test, which then calls run on another newly-created instance of it. It’s mildly confusing, but I believe the purpose is to allow you to completely customise how the tests run by being able to use any object with a _run method. From here on, we’ll assume that the default runner (an instance of MiniTest::Unit) was used.

The default _run method parses the ARGV into arguments (which we’ll ignore right now since in our example they are empty) and then loops through the plugins (another modifiable property of MiniTest::Unit class), which is really just an array of strings which correspond to methods on the MiniTest::Unit “runner” instance. By default, this is all methods which match run_*, and unless you’ve loaded extensions to MiniTest, it is just run_tests:

$ MiniTest::Unit.plugins
# => ["run_tests"]

The run_tests method calls the _run_anything method with the argument :tests. Within _run_anything, the argument is used to select the set of “suites” by kind (“test” suites or “bench” suites, but basically the classes that contain your actual tests).

The actual set of “suites” is returned by calling TestCase.test_suites in this instance. So what does it return? Let’s take a diversion to see what’s going on there.

The test suites, a.k.a TestCase subclasses, a.k.a. your actual tests

Take another look at the content of our test file:

class SomethingTest < MiniTest::Unit::TestCase
  def setup
    @something = Object.new
  end

  def test_something
    refute_nil @something
  end

  def teardown
    @something = nil
  end
end

When we subclassed MiniTest::Unit::TestCase as SomethingTest, the inherited hook on the superclass is called by Ruby with SomethingTest as an argument.

This stashes a reference to the class SomethingTest in an class variable¹. The TestCase.test_suites method that we were looking at above returns all those subclasses, sorted by name:

MiniTest::Unit::TestCase.test_suites
# => [SomethingTest]

Running a “suite”²

Back in the _run_anything method, those suites are passed to the _run_suites method, which maps them into their results by passing each to the _run_suite method.

The _run_suite method is responsible for selecting those tests within a suite (returned by the test_methods method on your TestCase subclass) which match any filters (i.e. -n /test_something/).

SomethingTest.test_methods
# => ["test_something"]

The default filter is /./, which will match everything that test_methods returns. For each matching method name, it instantiates a new instance of your suite class, with the method name as an argument to the intialiser, i.e. SomethingTest.new("test_something").

The run method is then called on that instance, with the runner (the instance of MiniTest::Unit that was returned by MiniTest::Unit.runner) as an argument. If you wanted to do the same in the console, it basically amounts to this:

runner = MiniTest::Unit.new
suite = SomethingTest.new("test_something")
suite.run(runner)
# => "."

An actual test running

We’re now at the point where the code from your test is significantly involved. Within the run method, the following methods are called³:

• setup – this is the method defined in your TestCase subclass. In our example, this results in the instance variable @something being set:

    def setup
      @something = Object.new
    end
  

• run_test, with the test name that passed to the initializer as an argument. This method is simply an alias for __send__, so the effect is that the method corresponding to your test name is invoked. In our case, the body of test_something runs:

    def test_something
      refute_nil @something
    end
  

• runner.record – this passes information about the name of the test, how long it took and how many assertions were called back to the runner instance

If we reach this point in the method, it means that the test method returned without raising any exceptions, and so the test is recorded as a pass.

However, if an exception was raised – either by the test, or by a failing assertion – then the test is marked as a failure, and the exception is passed as an argument in a corresponding call to runner.record.

• teardown – This method is run via an ensure block, so that it will be invoked whether or not an exception occured. In our example, the @something instance variable is set to nil:

    def teardown
      @something = nil
    end
  

Various other things happen, but this is the essential core of how MiniTest works: an instance of your TestCase subclass is created, and then the setup, test and teardown methods are invoked on it.

After the test has run

The run method returns a “result”, which is normally a character like . or F or E. This ultimately gets spat out to whatever is going to be doing the output (normally STDOUT). We saw this output above when we manually instantiated SomethingTest and then called run on it.

Actually, the puke method is called for anything other than a pass, which writes a more detailed string into a @report instance variable, and then returns the first character of that string (Skipped ... → S, Failed ... → F and so on).

Back up into MiniTest

Once run method finishes, the result is printed out, and the number of assertions stored on the instance is collected. The test method names that we were iterating over – the result of SomethingTest.test_methods above – are sequentially mapped into this number of assertions, and the final returned value of the _run_suite method is a two element array, the first being the number of tests and the second being the total number of assertions, for each test that ran. In our example, this would be [1,1] – one test and one assertion in total:

runner = MiniTest::Unit.new
runner._run_suite(SomethingTest, :test)
# => [1, 1]

Back up in the _run_suites method, each TestCase is being mapped via into this pair of numbers:

runner._run_suites([SomethingTest], :test)
# => [[1, 1]]

Back up one level further in the _run_anything method, those numbers are summed to return the total number of tests and the total number of assertions, across the whole run of test suites. Finally, these numbers are printed out, and then any failures that were gathered by the calls to runner.record when each test was running.

When the _run method itself finally finishes, taking us back into the at_exit block we started in, it returns the number of errors plus failures that were counted. This value doesn’t seem to be used, and disappears into the quantum foam of energy and information to which we all, ultimately, return.

Running tests within the console

We’ve actually seen already how we could start to poke around with tests without running them all. We can run a single test relatively easily, and determine whether or not it passed:

runner = MiniTest::Unit.new
suite = SomethingTest.new("test_something")
suite.run(runner)
# => "."

Unfortunately, there’s no simple way to run a group of tests (a “suite” or a “testcase” or what have you) aside from using the runner to specify a filter based on names. In other words, there’s no behaviour inherent within the TestCase class that lets you examine the result of the tests it contains. The information about which test failed, and why, leaves the instance when runner.report is called, and it’s only the runner that “knows” (in a very, very weak sense) about the state of more than just the test that is running now.

Instead the TestCase subclass is really just a collection of test_ methods along with the underlying behaviour to execute them (the run method that we examined above, and all of the supporting methods it invokes).

A test’s environment

One of the aspects of test frameworks that interests me most is what provides the environment for each test. What I mean by environment here are things like

• the implicit value of self
• how instance variables declared outside of the test relate to the code within the test
• how methods defined outside the test relate to the code within the test

When a TestCase subclass is instantiated, that instance provides the environment for the test to execute. MiniTest, like test-unit before it, is using the familiar conceptual relationship between classes, objects and methods in Ruby to implicitly communicate that instance variables created or modified in one method, like setup, will be available within our tests, just like normal Ruby code.

This is, I believe, the main reason behind some of the preference towards MiniTest or test-unit style frameworks – they use “less magic”, they are “closer to the metal” – because they use the same conceptual relationships between methods, variables and self as we use when doing all other programming.

This may be so familiar as to seem obvious; methods can of course call methods within the same class, and instance variables set in one method (e.g. setup) can of course be accessed by other methods (e.g. test_something) within the same class. Therefore implementing test suites like this is surely only natural!

Yes, indeed. But doing so is not without consequence.

For example, it’s not typical behaviour to create a new instance of a class just to invoke a single method on it, but that happens for every test_ method. I hope you’ll agree that that seems far less natural. But this has to happen so that each test runs within a clean environment, without any of the changes the previous test might have made to the instance variables they both use, and without any trace of the instance variables previous tests may have created.

If your test framework has those hallmark attributes I mentioned above – a class definition to contain tests, and tests defined as methods – then creating a new instance of that class to run each individual test is an inevitable consequence, unless you want to do some incredible gymnastics behind the scenes.

Examining test environments

Before I climb the ivory tower at the end of this post, let’s have one final code interlude, using these test objects we are creating in the console.

I’ve often imaginged that it would be very useful if, when a test fails, you got a dump of all of the values of every instance value in that test. I don’t know about you, but I am very bored of peppering tests with puts statements, or trying to use logs to decipher what happened, when I know that if I could just see the instance variables then I could tell what was failing, and why.

How about this:

require "minitest/unit"

class AnotherTest < MiniTest::Unit::TestCase
  def setup
    @value = 123
  end

  def test_something
    assert_nil @value
  end
end

class MiniTest::Unit::TestCase
  def environment(hide_framework_variables = true)
    variables = instance_variables
    variables.reject! { |v| v.to_s =~ /^@(_|passed)/ } if hide_framework_variables
    variables.inject({}) do |h, v|
      h[v] = instance_variable_get(v)
      h
    end
  end
end

runner = MiniTest::Unit.new
test = AnotherTest.new("test_something")
test.run(runner)
# => "F"

We can see that the test failed, but now we can also look at the instance variables within that test:

test.environment
# => {:@value=>123}

In this test it’s pretty trivial, but maybe you can imagine that being useful when you have a ton of ActiveRecord objects flying around? Particularly if you also patch whatever is outputting your test results to print the contents of environment for all failing tests.

If you’re curious, you can also take a look at the other instance variables that MiniTest has created behind the scenes, mostly prefixed with _ to indicate an informal ‘privacy’:

test.environment(false)
# => {:@__name__=>"test_something", :@__io__=>nil, :@passed=>false, :@value=>123, :@_assertions=>1}

Perhaps this might be worth developing into something useful? Maybe. It’s very much related to the other ideas that I’ve had about Rerunning tests in Ruby.

Using classes for test cases?

So, here we are at the foot of my ivory tower.

There’s nothing wrong with implementing test suites like MiniTest does, but it’s interesting to understand the consequences, both in terms of the impact to the test implementer and the design choices that it forces on the framework implementer. This is obviously particularly try if you’re trying to understand the different ways that one could compose test suites.

Using classes and methods is one way, but it’s not the only way to produce blocks of code (indeed, blocks are another) to be run in some specific way.

If we choose Ruby’s existing class system as the mechanism for collecting test behaviour together, we are bound by the rules and limitations of that class system when trying to do anything slightly more out of the ordinary, like dynamically composing abstract behaviour specifications.

Of all languages I’ve used, Ruby is by far the most forgiving regarding this; you can get an amazing amount of mileage out of subclassing, and including modules, and using “class methods” to modify the definition of classes at run-time.

It’s really a credit to Ruby that, even within the niche of the ecosystem that testing libraries represent, and even within that, the libraries that build on MiniTest or test-unit, so much richness exists. Things like shoulda or coulda or contest could not possibly exist without this flexibility.

But that doesn’t mean that there aren’t occasions where you hit a problem using things like inclusion or inheritance. This has been on my mind for a while.

Hmm

It’s my intuition that these test suites that we’re writing… well, they shouldn’t be classes. They don’t describe things that you can instantiate sensibly and that then have behaviour. They certainly don’t send messages to one another, like “proper objects” do. Classes are just convenient containers for these loosely-related essentially-procedural test bodies.

I believe that this intuition is what lies behind my interest in other test frameworks. From it springs all the ideas about composing or describing the systems under test in more dynamic or more natural ways.

In the next article, we’ll look at how RSpec works under the hood, and finally how Kintama does. Without having done the comparison yet, my guess is that they are very similar, but even within the alternate approach of block-defined tests there are many different paths you can take…

Sort lines alphabetically in vim

Imagine you’re working in vim. You come across this code:

gem 'clearance', '1.0.0.rc4'
gem 'neat'
gem 'stripe'
gem 'pg'
gem 'thin'
gem 'rails', '3.2.11'
gem 'bourbon'
gem 'simple_form'
gem 'strong_parameters'

You want to sort the list alphabetically. You select the lines visually:

Shift + V

You invoke the sort function:

:sort

You rejoice:

gem 'bourbon'
gem 'clearance', '1.0.0.rc4'
gem 'jquery-rails'
gem 'neat'
gem 'pg'
gem 'rails', '3.2.11'
gem 'responders'
gem 'simple_form'
gem 'stripe'
gem 'strong_parameters'
gem 'thin'

You dig deeper:

:help sort