Kapost Engineering

Recent Posts


Recent Comments


Archives


Categories


Meta


Decoupling Ruby Applications with Amazon SNS & SQS

Matt HugginsMatt Huggins

As the Kapost development team has grown, we found that we’ve reached a threshold where it has become necessary to break our application apart into smaller, more manageable pieces.  This includes customer-facing applications that match components of our product vision, APIs that have specific tasks or domains, and internal services responding to application events.  Building smaller applications and APIs are topics that are relatively easy insofar as the amount of help and documentation you can find online.  However, handling application events within new internal services is something that took a bit more architecting for us before settling on an approach.

There were four primary considerations in what we wanted in an eventing system:

  1. Decoupling: There should be a clear separation of concern between our applications.  Whatever app is publishing an event shouldn’t care what other apps are subscribed.  There should never be a need to update the code relating to publishing an event such that the application needs to specify subscribers.  We simply want to fire-and-forget.
  2. Fan-out: One event could have any number of subscribers.  There should not be a one-to-one mapping between a published event and a subscriber.
  3. Speed: We want to be able to publish an event without waiting for it to be published.  Related to the first point, we also don’t want to have to publish more than once (i.e.: one publish vs. one publish per subscriber).
  4. Reliability: Systems require downtime, and as we build more of them, the timing of migrations and updates for any given system will vary.  We don’t want any published messages to be missed by any particular system due to unavailability.

After some research and consideration, we decided that using Amazon SNS in conjunction with Amazon SQS would allow for all these conditions to be met.  Amazon SQS is “a fast, reliable, scalable, fully managed message queuing service” that can continue to enqueue messages regardless of internal system availability.  Amazon SNS is “a fast, flexible, fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients” that integrates with SQS out of the box.

We Have the Technology!

With our research complete, we set out to build circuitry, a simple Ruby gem that encapsulates publishing to SNS and subscribing to SQS.

With circuitry, any object can published; it is simply serialized via its to_json method prior to delivery.  This means that helpful serialization tools such as active_model_serializers can be used to wrap an object, only publishing data that is appropriate for your use case.  Conversely, subscribers will receive the JSON payload along with the topic name that was given when publishing.

An example of how this might be used to track changes to a document is as follows:

# publisher
class Document < ActiveRecord::Base
  before_update :publish_changes

  def publish_changes
    serializer = DocumentUpdateSerializer.new(self)
    Circuitry.publish('document-update', serializer, async: true)
  end
end

# subscriber
Circuitry.subscribe('https://sqs.region.amazonaws.com/account-id/queue-name') do |message, topic|
  HANDLERS[topic].handle(message)
end

HANDLERS = {
  'document-update' => Handlers::DocumentUpdate
}.freeze

module Handlers
  module DocumentUpdate
    def handle(message)
      DocumentChangeDeserializer.from_json(message).save
    end
  end
end

It should also be pointed out that publishing accepted an async option.  This means that we’re able to publish in the background while our process continues working normally, preventing any additional network latency during user requests.

AWS Configuration

We still need to configure SNS and SQS to work properly before this code will work.  Fortunately, AWS gives us the necessary tools to do this.

The first step is to create our SQS queues.  A queue should be application-specific, and in order to meet our reliability criteria, we should set up a failure queue as well.  AWS requires that we set up our failure queue first.

Create SQS Failure Queue

Next, we have to create our normal event queue.  This should be set up with the “Redrive Policy” enabled, which is what allows us to take advantage of the failure queue from the previous step.

Create SQS Event Queue

If done correctly, you should now see both queues in your SQS management console.

SQS Management Console

Before we can check the SQS setup off our list, we need to set permissions on each of these queues.  Select either queue from your list, and click the “Add Permission” button as seen above.  Ensure that the “SendMessage” permission is enabled.  Although I enabled this for “Everybody” in the demo below, a more secure solution would be to only permit it for the AWS account number(s) that are associated with your SNS topics.  Be sure to repeat this step for both the event and failure queues.

Add SQS Permission

After our application queues are set up, we need to create any SNS topics that we plan to publish to.  Although we’ll only create one topic for this example, the number and names of topics actually created should reflect the events for your apps will be publishing.  The topic name can be anything, as long as it matches the topic name used in your code.

Create SNS Topic

Once your topic is created, the last step is to create a subscription between it and the SQS event queue we created earlier.  Ensure that the Protocol is set to “Amazon SQS”, while the Endpoint matches the ARN generated for your SQS event queue.

Create SNS Subscription

Results

We’ve just begun using circuitry, so our mileage remains to be seen.  With that said, we’re happy with the architecture conceptually and are excited to continue working with it internally.  Given that we’ve recently taken Flux head-on for our JavaScript development, it should come as no surprise that we’re fans of the dispatching paradigm.

The circuitry gem is completely open-source, so feel free to share and contribute.  We’re excited to see its continued development!

Hello, I'm a senior software engineer at Kapost. While I enjoy both front- and back-end development, I am especially passionate about the Ruby language. I'm excited to help Kapost grow by solving architectural problems as we continue to scale our product.

Comments 3
  • David Hersey
    Posted on

    David Hersey David Hersey

    Reply Author

    Hi Matt, thanks for this post! I would be very interested to hear what your performance and scalabilty results are using this approach — I am considering using SNS to push messages to mobile apps and it would be nice to know what the performance looks like.


    • Matt Huggins
      Posted on

      Matt Huggins Matt Huggins

      Reply Author

      Our engineering team (~20) has been slowly implementing this across our apps. I’d estimate it’s being used in at least 4-5 of our apps for fan-out messaging now. Everyone who uses it says what a great experience it is, and there have been no performance issues at all. Our CTO even said how nice it was to restart the server on a new app a few times without worrying about losing circuitry messages.

      The only thing we have yet to address is some kind of versioning scheme. We’ve needed to update messages structures a couple times, and there’s not a smooth way to do this presently without releasing dependent apps in sync, more or less. The alternative would be to publish all possible versions of message formats for a topic, which doesn’t seem like a great solution either, so it’ll take some consideration before we have a best practice or coded solution for this.