Technology

Track the change you want to see in the world

September 26, 2011
By Pieter de Zwart

Most tech blogs focus on solving difficult problems, or learning lessons from the unexpected, like outages. This post, however, is about foresight. Software engineers often don’t have time for foresight—they’re always building the next thing in the Product pipeline that will make more money/time/happiness, leaving little time to pursue forward-thinking endeavors that at first glance might not contribute to the bottom line. Here at the Rubicon Project, however, we’re allowed to roam free every so often to see what crazy hare-brained scheme we can come up with.

One of these schemes wound up paying dividends thousands of times over, and is now a deeply ingrained, mission-critical portion of our entire technological stack.

Last year Brad Rodriguez, one of our engineers, started building a messaging system as part of our web application stack. There was no immediate business need; our engineering justification was tenuous at best. But our gut feeling was that a messaging system would be a cool way to solve a lot of problems down the road. After some basic research and testing we settled on RabbitMQ, an Erlang-based implementation of the Advanced Message Queue Protocol (AMQP) boasting lots of community support and active development. We highly recommend you check them out.

With any AMQP environment, decisions have to be made in the beginning that determine how the system will be used thereafter. We opted for a topic exchange, allowing any queue to bind to any portion of a message, so that anyone could choose to “listen in” to anything they might find interesting. We settled on a routing key structure that looked like this:

[version].[environment].[system].[message type].[type].[id].[action].[status]

An example of which is:

v1.dev.revv.event.campaign.1234.updated.success

All users must comply with the first four elements (version, environment, system, and message type) to avoid conflicts and allow for easy upgrades, if they’re even needed—it’s been a year and we’ve found this structure to still be extremely effective, and we believe it will be around for quite a while longer.

Another major decision was crafting the message body. In order to keep it as small and information-dense as possible, we built a JSON-encoded object that mimics a lot of information found in the routing key but also includes details about what was changed, and by whom.

{
    "message_version": 1,
    "routing_key": "v1.dev.revv.event.campaign.1234.updated.success",
    "item_type": "campaign",
    "item_id": "1234",
    "operation": "updated",
    "status": "success",
    "sent": "2011-04-08 15:32:07",
    "body": {
        "user_id": "123",
        "description": {
            "modifications": {
                "campaign_name": {
                    "new_value": "Super Campaign",
                    "old_value": "Super Duper Campiagn"
                },
                "updated": {
                    "new_value": "2011-04-08 15:32:07",
                    "old_value": "2011-02-23 15:50:48",
                }
            }
        }
    }
}

When we first implemented our AMQP based messaging system, we weren’t really sure what we were going to use it for. There were a lot of ideas: distributed instant messaging, a background job manager, asynchronous messaging between two different systems, and so on. The lightbulb moment came during our Quarterly Hack Day event, when one of our beloved users requested for a way to track ad campaign changes to help clients with troubleshooting.

The engineer fielding the request had an epiphany: “Why don’t I just hook into the messaging system and look for anything that creates or updates a campaign object?” And in the space of a few hours, we’d put together a new queue that logged campaign creations and updates into a new table. Sure, we could’ve just overridden the Campaign model save event and dispatched an additional database query (or two) to add the columns. But not only is publishing a single message faster, taking only 0.002 seconds to complete, it also lets anyone listen in and do as they see fit. The messaging system also decouples our front end from mission-critical operations (i.e. saving new campaign information) and non-critical operations (i.e. tracking changes). Perfect for systems like ours that need to scale—and scale quickly.


Tags: , ,