This is a short post about SQS fanout queues, how to start using them using the AWS SDK for Ruby, and how they can help you get staging and test data for free for all your SQS-dependent apps.

Overview

In 2012, SQS got a new feature that linked SQS and SNS, allowing for something called “fanout”. You can now send a message to an SNS topic, and any SQS queues subscribed to that topic will receive that message.

I had a hard time figuring out how to do this via the AWS SDK docs, so hopefully this post will help someone else trying to do the same thing.

How to do it

Let’s cut right to the chase.

topic = AWS::SNS.new.topics.create('foo_topic')
queue = AWS::SQS.new.queues.create('foo_queue')
subscription = topic.subscribe(queue)

If you don’t want the default JSON document format (described here), you can get your raw message by setting a flag:

subscription.raw_message_delivery = true

Now you can send messages to your queue via SNS!

topic.publish('hello world')

So what?

What is the use of fanout? Test queues.

Let’s say you have a process that pulls from SQS. This process can tough to do integrations testing on, because you have to either create test input by hand or copy real stuff from production. Both options are pretty inconvenient.

Enter, test queues! Start sending SQS messages via SNS, and at any time you can choose to also subscribe a test queue to the SNS topic to get real messages in your testing environment without affecting production.

If you have a staging environment set up, you can even run your process in parallel with production. This could be useful for company-critical processes, especially for those with high throughput. Spin up a staging box with your new feature branch and let it run for a day, reading from a staging queue subscribed to your SNS topic, and keeping an eye on the results.

Here’s a quick example of what this could look like in code:

# $env = Rails.env

class ProcessThatSendsToSNS
  def run
    topic = AWS::SNS.new.topics.named('foo_topic')
    topic.publish('hello world')
  end
end

class ProcessWeWantToIntegrationsTest
  def run
    # let's say this queue is already created and subscribed to "foo_topic"
    queue = AWS::SQS.new.queues.named("#{$env}_foo_queue")
    message = queue.receive_message
    do_crazy_stuff(message)
  end
end

When you want to test, just set your $env to staging or development or whatever is appropriate, and boom! You’re playing with real data in a safe environment.

What’s the catch?

So you might be wondering, why send to SQS at all? Let’s send all of our messages via SNS!

Ladies, one at a time, please

Unfortunately, unlike SQS, there is no batch send for SNS (yet), only one-at-a-time publish. Depending on the throughput of your application, this may become a performance bottleneck.

One trick you could try using is packing a single message with multiple items, such as via a JSON collection. In 2013, Amazon announced that a single message can now have up to 256KB. That’s quite a lot! So instead of sending payloads like {"foo":"bar"}, you can accumulate a big batch of them like [{"foo":"bar"},{"baz":"qux"}], taking care to stay under the 256KB maximum.

Another trick is to thread your API calls to reduce time waiting on IO.

$$$!!! Wait, nope, that’s fine.

Let’s say you’re sending to production, staging, and development queues at all times using the test queues pattern described above. That means there are three times more SQS messages being sent, which means our SQS send costs triple, right?

Turns out, no, your costs don’t triple. Sending to SQS via SNS is no extra charge, so you can rest easy.

Wrap it up

Fanouts are a pretty neat way to get staging environments set up for free for your SQS-munching apps. There are some negative performance implications in publishing to SNS versus sending to SQS. But the value of having real test data handy whenever you need it potentially outweights any bit of extra work to get SNS fanouts set up.