Today Cloud adoption is growing rapidly and so is the management of your IaaS, and while the reasons of why you should aim for Automation in the Cloud are obvious, there are a few things you would like to consider when building your Architecture and I would like to dedicate this post to Serverless and to Serverless Architecture.

This topic is general available for most major Cloud providers like AWS, Google, Microsoft, IBM, Oracle. However, to be more specific in my article, I will pick AWS Cloud and AWS Lambda and will make reference to AWS related Cloud Services where needed.

Warm up

Audience

Serverless.

Scenario

Within Disaster Recovery Category: recovering from a Failure Event Notification.

Difficulty

Easy.

Post Type

Easy Reading

🎧 Playlist 🎧

1. Nick Cave & The Bad Seeds – Red Right Hand ⭐️
2. HAIM – Summer Girl ⭐️⭐️
3. The Water Boys – This is the Sea
4. Mark Knopfler – Every Heart In The Room
5. Tom Misch, De La Soul – It Runs Through Me
6. Bruce Springsteen – Secret Garden
7. The Teskey Brothers – Crying Shame ⭐️

The Genesis

Several years ago, when I was starting to use Cloud Services in IaaS & PaaS, one “easy” way for me to manage the very initial setup of built instances, was to write some scripts that were calling the relevant APIs.With more and more logic inside over the time, scripting with CLI seemed the easiest at the beginning, then, later, with python and later, with cloud python libs, like for example boto3 for AWS, and today with options for more and more programming languages as per the available Cloud SDKs.

I’ve used to run the desired setup, whenever I wanted to (manually or scheduled, well, “scheduled” more likely).

I’ve used to save (and run) those scripts on (and from) my local machine. Quick note here: depending on where my local machine network was, that could require some adjustments, like for example asking for special outbound permissions, or using special network proxies etc. However, as the Cloud Services and the scenarios complexity started to grow, in a very natural and ordinary way, the scripts evolved as well, and the “local machine” turned into a small footprint VM that I could spin it initially from my “local machine”. That VM, even was on a very small footprint, started to ask for maintenance, so it was moved into the Cloud as an instance of some “kind” (it was the smallest EC2 instance at starters). Not good enough though, so it was converted into a nicely “pre-baked” container (a Docker Container) that was by far the easiest to maintain (Container Repo) and the fastest to spin “anywhere” (pretty much anywhere), deployable to any instance (supporting docker container, or even to a Container Managed Service). Hmm… while this final setup wasn’t that bad as it may sound, something was still missing …

We wanted more: Serverless !

Sample Scenarios

Starting from the most basics ones to more complex:

• starting/stopping some (EC2) instances
• notifications (email and/or SMS) triggered by specific events (i.e. Billing Alarm SMS etc)
• Common AWS Lambda Use Cases
• AWS Customer Case Studies
• most notable today in Enterprise world being:
  • Data Lakes
  • Disaster Recovery

The Goal

Regardless of the complexity of your scenario, but in particular if your scenarios are more complex, you should be interested in a good Serverless Architecture, aiming for a unified experience in developing, building, testing, deploying, monitoring, and securing your Serverless Applications.

The Approach

Depending on who you are:

• your size (large enterprise, medium, small or very small companies)
• your Business constraints (choosing strategic Cloud providers or partnerships with more than one Cloud vendors)
• … others …

just as I was mentioning in my Automation in the Cloud article: “achieving IaaC is not easy, particularly if you are planning to be “Cloud Agnostic” and have a Multi Cloud approach as you should really do, especially if you are an enterprise …” (for example, try thinking Terraform or other similar), the same is valid for Serverless and Serverless Architecture , and today more than in the past, you can think Serverless Framework (or other similar). But before jumping straight into using other 3d party frameworks, be an expert on what your selected Cloud Vendor offers; AWS definitely offers a large palette of services that you can combine to build your own solutions, so it worth to analyze them and see if they will suit your needs.

NOTE: by“Cloud Agnostic” above I mean the approach in itself, by using a single Interface for achieving IaaC (you still need of course to know each cloud provider features).

My Sample Scenario

A recent scenario I ran into, that I could classify it in the Disaster Recovery Category, was:

On a special AWS Event Notification (a failure notification), I had to initiate a workflow of logic steps, trying to recover from that failure. If after running the workflow logic, the result would have lead to the same initial failure, the workflow logic would have been repeated only for a limited number of times. That doesn’t sound too complex and it isn’t. To make things very simple I’m inserting a clean simplified diagram of such a workflow logic.

However, the workflow time duration could have vary from a minimum of several minutes (more than 15 minutes) to many more (even hours).

I’ve implemented the solution using another Serverless Service from AWS named: AWS Step Functions. The workflow in its details is irrelevant, but do note that for many of the workflow steps, a unique Lambda function was associated and was executed when the State Machine reaches that step. All those Lambdas would have been executed within a maximum of 10 second.

So how to achieve the Serverless Architecture in this case, remember: “aiming for a unified experience in developing, building, deploying, monitoring, and securing your Serverless Applications” !

Here is how I’ve achieved an AWS tailored Solution

• I’ve sketch the AWS Step Functions workflow logic (as illustrated above briefly)
• I’ve developed the needed AWS Lambda functions; in many of the workflow steps, an AWS Lambda function was used to execute various commands, like for example, checking the status of a specific AWS service, or triggering a start, suspend or stop action for some other AWS Services etc.

• I’ve manually created the 1st AWS Step Functions to test the workflow including the various AWS Lambda functions in a SandBox stage
• I’ve moved into a DEV stage by automating with IaaC the creation of the AWS Step Functions, i.e. building an AWS Cloud Formation stack (could be multiple stacks as needed)
• I’ve added to the IaaC the Lambda functions (other CF stacks) for developing, building, deploying and monitoring all of the AWS Lambda function through CodePipeline, CodeBuild, CodeDeploy, CodeCommit.

The Result

I’ve achieved Cloud Formation Stack(s) representing the IaaC for the Step Function workflow logic, but also the IaaC for CI CD for the AWS AWS Lambda functions. So I’ve achieved a unified experience in developing, building, testing, deploying, monitoring, and securing the Serverless Applications.

Yes, monitoring and security is included (comes with AWS) and can be customized more, adding to the IaaC CF template as per my needs.

Do notice, that I had full control in the IAM when it comes to Roles and Permissions that need to be built in the solution developed.

Closure

You could still ask: “- How did you build the AWS CloudFormation stack(s) ?“

At least a few answers here:

1. You could theoretically build those templates manually – but even for the not that complex scenario I presented here, this could be a real nightmare, I am sure there are some people so familiar with AWS Cloud Formation syntax that could write straight into the yaml (json) template, however, I am not one of those.
2. Building your own framework around your preferred programming language would be the ideal solution. This may cost you some time spent in the beginning, but it will pay off on the long run. This would be my preferred solution because I would like to be in full control of all I am doing in IaaC, including the Security (Permissions and Roles) , and also have a programmatic logic approach in building the IaaC as template in AWS Cloud Formation.
3. More than 1 years ago AWS launched such a framework that continues today as an open source project, mostly coordinated by AWS experts, named AWS CDK . While in my opinion it didn’t reach yet a strong maturity, it looks very promising, and maybe just before diving into building your own, you may like to consider investigating this one. And being open source, well, if you find some bugs or missing features, you can always go and fix it yourself and share it back with the community.
4. Why wouldn’t you consider building all through Serverless Framework for example ? Or other frameworks ? Maybe you would like another kind of CI CD pipelines for your DEV or another type of Dashboards where to monitor your Serverless Apps ?

There is no one solution that fits all eventually, and I hope the info I’ve shared with you in this post will help you in selecting your best Solutions and Architectures for your own Serverless Applications challenges.

Happy Sailing !