Fox Pups At Play

After several late snowstorms, it feels like spring has finally arrived. And that means young critters are coming out of their dens for the first time! Nicole and I have been studying a red fox family: mom, dad, and their five new pups. When it's time to come out and play, it's non-stop running, jumping, and wrestling as they practice the skills they'll need for life on their own. A scavenged tennis ball rolled down the hill makes for a fun game of catch the "prey".

Big Snapper

Came across an unexpected traveler in the field headed for the greener waters of a new pond.

Up close, he resembles of Jabba the Hut.

Emacs Lisp: Toggle Between a Clojure String and Keyword

When I was doing a fair bit of Ruby I often used the TextMate's shortcut (Ctrl+:) to convert a Ruby String to a Symbol or a Ruby Symbol to a String. It's something I've periodically missed while doing Clojure, and yesterday I found myself in the middle of a refactoring that was going to force the conversion of 5+ Clojure Keywords to Strings.

The following emacs lisp is my solution for toggling between Clojure Strings and Keywords. The standard disclaimers apply - it works on my machine, and I've never claimed to know emacs lisp well.

<script src="https://gist.github.com/jaycfields/5502257.js"></script> A quick video of the behavior:

<iframe width="560" height="315" src="http://www.youtube.com/embed/UQU7xasn3eA" frameborder="0" allowfullscreen=""></iframe>

strong_parameters are good and you should feel good

Besides moving attribute whitelisting to the controller rather than the model, Rails 4’s move to Strong Parameters over attr_accessible provides great documentation about the data with which records are being created.

Here is an example of a controller many of us have written, using strong_parameters:

class CommentsController < ApplicationController
  respond_to :html

  def create
    @comment = Comment.create!(comment_params)
    respond_with @comment
  end

  private

  def comment_params
    params.
      require(:comment).
      permit(:body).
      merge(user: current_user, commentable: commentable)
  end

  def commentable
    # find and return a commentable record
  end
end

Notice how the comment_params method tells you at a glance what object’s parameters this controller/action cares about (comment), the specific data being used (body), and the extra information being added. After glancing at the method, you hardly have to concern yourself with the rest of the class: everything just makes sense.

strong_parameters will be standard in Rails 4.0, but they can be used now in Rails 3.*.

Written by Caleb Thompson

P.S. You can include ActiveModel::ForbiddenAttributesProtection on a model-by-model basis, but given the level of awesomeness provided I wouldn’t recommend it.

Deploying WebP via Accept Content Negotiation

Deploying new image formats on the web is a challenge, but not an unsolvable one, and one that would pay high dividends in terms of performance. In fact, there are many ways to tackle the problem: client-side logic, server negotiation, and hybrid strategies. There is a time and place for each one - they are not exclusive.

Having said that, I'm partial to server negotiation as it requires the least amount of work to move the web forward in a significant way: a single flip of a switch by a CDN or a large hosting provider, instantly enabling significant byte savings for all of their users. Computers are great at performing routine negotiation work, humans are not: we get bored, we get lazy, we routinely forget to optimize our images.

WebP deployment checklist

As a hands on example, let's take a look at what we would need to enable Accept negotiation for deploying WebP:

1. User agents which support WebP should advertise it in the Accept header
2. Servers and caches should use the Accept header to serve appropriate assets
3. Origin servers must specify Vary: Accept in generated responses

Most importantly, notice what's missing: there is no mention of having to modify the markup or logic of millions of individual web applications. Having said that, the above steps still require some work, let's dig a bit deeper.

Configuring Accept negotiation in Nginx

The great news is that Chrome Canary is now advertising image/webp in its Accept header for all image requests - with that in place, we can check off the first item on our checklist above. Now, we need to configure our Nginx server to automatically serve the right file. To start, let's assume we have a pre-generated a WebP asset on disk:

location / {
  # check Accept header for webp, check if .webp is on disk
  if ($http_accept ~* "webp") { set $webp T; }
  if (-f $request_filename.webp) { set $webp "${webp}T"; }

  # if WebP is supported, and WebP is on disk, serve it!
  if ($webp = TT) {
    add_header Vary Accept;
    rewrite (.*) $1.webp break;
  }
  # ...
}

First, we check if the Accept header is advertising WebP. Then we check if there is a corresponding file with a .webp extension on disk. If both conditions match, we serve the WebP asset and add Vary: Accept - done.

Now, let's consider the case where Nginx acts as a proxy cache. Here we need to teach Nginx to cache and serve multiple variants of the same image (WebP and non-WebP versions) based on the value of the Accept header:

server {
  location / {
    # lookup inbound requests with exta WebP flag if advertised in Accept
    if ($http_accept ~* "webp") { set $webp T; }
    proxy_cache_key $scheme$proxy_host$request_uri$webp;

    # proxy requests to remote servers
    proxy_pass http://backend;
    proxy_cache my-cache;
  }
}

Not much different from the previous example! Instead of checking the local disk for the asset, we pass the request through to some set of backends, and append the WebP bit to our cache key - that's all there is to it.

Accept fragmentation and Key

Wait, why didn't we just append the $http_accept variable to our proxy cache key above? Unfortunately, the Accept header varies from browser to browser, which means that using the raw header would unnecessarily fragment the cache. Hence, we scan for WebP support in the header and add an additional bit to the cache key.

However, wouldn't it be nice if we had a standard mechanism to define a custom cache key? Well, good news, there is work underway on the new Key response header, which will act as a smart replacement to Vary:

The 'Key' header field for HTTP responses allows an origin server to describe the cache key for a negotiated response: a short algorithm that can be used upon later requests to determine if the same response is reusable.

IE is special, as always!

Unfortunately, Vary support in IE, even in the latest versions, remains very poor: IE does not cache outbound headers, which means it cannot perform the appropriate matching algorithm, and is forced to issue a revalidation request on any asset with Vary. As a result, to avoid the extra requests, we can create a special case for IE:

  if ($http_user_agent ~* "(?i)(MSIE)") {
    # drop Vary for IE users, mark resource as private
    proxy_hide_header Vary;
    add_header Cache-Control private;
  }

With the above configuration in place, IE users won't see Vary, and resource is marked as private: it can still be cached on the client, but it won't be cached by any other intermediate proxy.

The "action plan", continued...

Small steps, but we're heading in the right direction. We've addressed the issue on the client, we know how to configure our servers, now we need to push for better support by proxies and CDN's - ideally, by deploying support for Key! The good news is, this requires work, but it is not rocket science. As a case in point, CDNConnect recently enabled transparent Accept+WebP negotiation for all of their users.

Finally, let's take a look at Facebook's recent experiment with WebP, as summarized by ArsTechnica:

It turns out that Facebook users routinely do things like copy image URLs and save images locally... As a result, users were left with unusable URLs and files; files they couldn't open locally, and URLs that didn't work in Internet Explorer, Firefox, or Safari.

URL sharing is not an issue if Accept negotiation is configured correctly - see instructions above. Next, once a file is saved locally, you need a local WebP codec or viewer. The good news is, Chrome just landed a patch to register itself as a default WebP handler across all platforms - if you have Chrome, you can view local WebP files, and there is a growing list of editors, viewers, and plugins. Also, there are ongoing discussions about enabling the browser to save a "safe copy" of the asset to mitigate the problem entirely. One step at a time, we're getting there!

Embedding Canvas and SVG charts in emails

In our app Freckle Time Tracking we’re sending out weekly reports to users by email, reporting to them what they achieved last week.

Our emails not only include a report, listing all entries they logged, but also our “Mini-Pulse”, a graphical representation of how much they worked for each project.

Here’s how a typical email we send out looks like:

Now, generating a table and styling it for the HTML email is (relatively) easy, and beyond the scope of this article, but let’s have look at the charts in the email. We already generate the Mini-Pulse graph in our web app—we use an ancient version of Raphael.js to generate SVG, and this gets the job done nicely (SVG works on practically every modern browser, and Raphael falls back to VML on older Internet Explorer versions). Of course, you could use Canvas or any other HTML supported by WebKit just as well.

This is all great in your web browser, but not for emails. First off, obviously, JavaScript
is disabled in HTML emails, so we can’t use that to generate the SVG on the fly; moreover
SVG only works in a handful of email clients.

The only image formats that reliably work in HTML emails are GIF, PNG and JPEG, which means we have to dynamically generate such an image containing the charts and refer to it from the email.

There’s two possible ways to do this:

1. Reimplement the logic and rendering with a tool specifically made for generating chart images
2. Reuse the existing JavaScript/SVG code in a headless web browser and make “screenshots”

We chose to reuse the code we have, so we can easily adapt and extend both the web app and the HTML emails in the future (plus no need to learn yet another tool!).

A great way to create screenshots of the graphs is to use PhantomJS, which is a headless WebKit with an API that has support for taking screenshots.

We also have the following requirements for our report emails:

• Don’t generate the Mini-Pulse if the email is never opened, to conserve server resources
• Cache the generated image once the email was opened once
• Securely serve the image and use encrypted URLs with embedded authentication (the user the email was sent to may no longer have permission to access Freckle at the time the email is opened)
• Charts should be retinafied (your HTML emails are retinafied, are they?)
• File size of image should be small so it loads fast on mobile email clients

To fulfill these requirements, here’s what happens when a user opens an email that has a chart embedded:

1. HTML email is shown
2. Email client or browser accesses URL in the form of https://app.letsfreckle.com/m/xxxx/yyy.gif
3. If there’s a cached version of the image serve it and go to step 10, otherwise continue to step 4
4. Rails app decrypts account ID, user ID, chart type and date range from the given encrypted URL*
5. Rails app calls internal PhantomJS web service with a URL to call to generate the chart
6. PhantomJS web service calls the Freckle Rails app internally
7. Rails app serves Raphael.js, and our chart generation code and the data needed for it. (We use a special, stripped-down layout that only serves the chart and doubles resolution on everything to simulate rendering on a high-density screen (a feature that Raphael.js doesn’t yet directly support).
8. PhantomJS renders the page
9. PhantomJS returns a GIF to Rails (the call from step 4)
10. Rails returns the GIF and caches it into a file (Rails page caching)
11. Email client or web browser renders the GIF

All this sounds pretty complex, but it’s actually implemented in just about a hundred lines of code.

There’s a few tricky things you have to deal with when installing Phantom.js on a Linux server, such as adding fonts that may not be part of your default Linux server setup, but it’s pretty easy to get going. For Ubuntu 10.04, you can check out this gist with instructions on getting decent rendering quality.

*To accommodate passing parameters along from an HTML email to our Rails app, I’ve released URLcrypt, an open-source, MIT-licensed Ruby library for ‘elegant’ encrypted URLs. Alternatively, you could also use a table the holds tokens, but I find encrypting the account/user id information a more scalable solution.

Year Five

The average lifespan for a software engineering job is 4 years. Okay, I've never actually seen proof (or contradiction), but that's the general feeling in the groups I associate with. Perhaps that's selection bias - my employer has generally changed on year 3 or 4. Perhaps this is the exception and not the rule, in that case feel free to simply read this as an experience report. However, I do think it's somewhat common for developers to leave around year 3 or 4. This entry contains speculation on why they leave, and offers one idea on what employers can do to break that cycle.

My 4 year employment cycle generally looks like this

• Year One: "I'm in over my head. My semi-bluff was in-fact a bluff. They're going to fire me any day."
• Year Two: "It's nice to feel like a productive team member"
• Year Three: "This is fun, and I'm not bad at it. It's satisfying to pass on knowledge to teammates."
• Year Four: "This feels repetitive, that grass over there sure looks greener"

• A company you don't work at always seems to have infinite possibilities; however, after a few years with an employer, it's extremely clear what your options are. More importantly, it's very clear what limitations will likely always be there.
• A company you don't work at contains no code you're responsible for. Conversely, any company you've been with for 4 years probably has plenty of code you're not proud of. If you're responsible for that code, it's a constant reminder of your previous limitations. If you're not responsible for it, your co-workers aren't likely to let you forget about it anytime soon.
• There's always someone willing to pay you more than you're worth. After several years with a company it's likely that they're going to pay you what you're worth, but not what some other company thinks you're worth. I'm surprised that more companies don't pay (the employees they want to keep) what their "flawless market value" would be. In other words, what would you pay them if they interviewed, you determined what they knew, you determined what value they would bring, and you were completely ignorant of their flaws? That's what your competition is likely doing. That's what you're fighting against if you want to keep them around.
• A new job often offers a new challenge. Once you feel like you've given that challenge your best shot, what remains? If you did a great job, it's likely that you'll have plenty of other options. However, if you've done a good job, you may be stuck in a spot where there aren't as many open doors and challenges to choose from - not nearly as many as a position at another company will appear to offer.