Content Management Systems (CMSs), the backbone of blogs and product announcements, are extremely important. What happens when you want to build one into a publishing pipeline and it doesn't feature a mechanism to do so?
Most people would probably write something in PHP to do the trick, but this isn't a long-term solution, as maintaining a fork of an open source project can leave you without security updates. Staying secure is especially important in the FinTech space, so how can the gap in the publishing chain be filled while minimising risk? The answer: go serverless.
Serverless is a term that has come to signify zero setup, maintenance and management of services running production code, all hosted by a cloud provider (in our case AWS). AWS were first to market with AWS Lambda, a way of running arbitrary code in JavaScript, Python or Java, invoked by other Amazon services or by events driven by other infrastructure.
An example of this is Amazon's first use case for Lambda: uploading an image to AWS Simple Storage Solution (S3). This creates an event, including information such as the file name, file size, the path to the file, and when it was uploaded. This information is then sent to a Lambda function and your code handles the event, which in this scenario, could be resizing the image to a 16:9 format, or compressing the image.
Event driven compute architecture such as this is fast becoming the way to solve gaps in infrastructure and applications.
The Missing Ingredient
At FundApps we chose Directus, an API-driven CMS (written in PHP of course) to publish some notifications. Directus was perfect for our needs, but we had one issue: identifying when a post had been published. As our posts are only going to be visible to customers, we can't put the posts on the public Internet. We wanted our customers to actually read our posts, though. Our solution was to tie together the CMS with our existing notification service to automatically notify our customers when a new post gets published, with a link to the post.
However, sending a notification turned out to be more than a simple webhook setup. Directus does not support sending publishing events to a URL via webhooks. Directus does support database hooks though, when a post has been written or updated.
We had one remaining problem. We could create a hook for when a post was created or updated, but not when it went from a draft to an active post. Thankfully this information did come through with the hooks, but it meant we had to track some state to only notify our users once, when the post was set to active. To solve these issues we decided to create a pipeline in AWS that would ensure delivery, while not having any servers to maintain. For this pipeline, we decided to use all the TLAs at our disposal: DynamoDB, SNS, Lambda and SQS.
AWS All The Things
DynamoDB is a NoSQL database where we will be storing the state of the post, whether or not it has been posted or not. If we didn't hold the state of the post, we would not know if the post was brand new or just update to an existing post. This is because the Directus event doesn't indicate an existing post update, just that something has been posted. By using DynamoDB, we can ensure that notifications are not sent on every update and know all the content that has been posted. DynamoDB is completely managed by AWS, so our only interaction is via the AWS SDK.
This interaction is done inside our Lambda function, which is where we check DynamoDB for posts, then send on a newly formatted message. The Lambda function is triggered by SNS, AWS's Simple Notification Service. This service is a way to send messages to a topic, with different functions, such as Email and Lambda, subscribing to that topic. Once a message has been received, the Lambda is passed the message body, which is the postID and metadata about the post published in Directus. The code evaluates the message and will act if the message meets the criteria.
While most of what we have seen so far has been all new, the changes to the Directus code are still needed, but this is a small extension. Our hook code essentially extracts the data we want from the database event and then creates a formatted message. This message is then encoded into JSON and sent to SNS.
So far, our chain is:
CMS (Directus) -> SNS -> Lambda <-> DynamoDB
The Lambda will call DynamoDB with the postID, looking for a reference. If there isn't one, it will write to the database to indicate a new post is present. Finally, we can inform our notifications service of a new event for all our users, which is where SQS comes in. SQS is AWS's Simple Queuing Service, a way to send messages with an application pulling them off the queue to process.
The notifications service we run already uses SQS, so we just need to format the message for the notifications service and send it. This is also done in the Lambda function.
Performance
Despite the fact there are several parts to this process, the time to glass from submission to notification is less than a second. Fast performance wasn't something that we had in mind when we were designing the system, just reliability. What the performance does show however, is that despite the complexity, the serverless approach didn't introduce delays in the publishing chain.
Final Picture
While there are some missing features from our CMS of choice, with less than 30 lines of PHP and 100 lines of Golang in our Lambda, we filled in the gaps to complete our publishing pipeline. It may seem complex, but in reality, the amount of work to put the AWS pieces together was minimal. The majority of the work was making the PHP code work.
Solutions like this one require no human interaction to keep them going; everything is managed by AWS and our most likely issues will come when we upgrade the CMS. In years gone by, this would have required a huge amount of work to achieve the same result. Setting up MongoDB, RabbitMQ and writing code to handle the incoming and outgoing messages would have seemed far too convoluted.
Serverless Computing removes these barriers and lets you focus on getting the code right and solution you want into production.