AWS Lambda! But On Steroids

Do you ever feel that your workload is restricted by AWS Lambda limitations? Even though lambda is great at many things, like event-driven computing and speedy executions, there are some situations in which you might need more processing power or longer execution times. It’s possible that your tasks require more CPU cores or longer runs times than Lambda’s 15-minute maximum. It makes sense to want something as simple as a Lambda but with more power—a Lambda on steroids, if you will.

Even though AWS provides EC2 for more demanding workloads and control over computing resources, Lambda’s effortless scalability and event-driven nature can make managing EC2 instances challenging. Here’s where the concept of a more capable serverless solution, one that combines the simplicity of Lambda with the scalability and expanded functionalities of a more conventional compute service, comes into play.

Imagine a serverless platform that offers more processing power and longer execution times, but keeps all of Lambda’s benefits, such as auto-scaling and pay-per-use billing. A platform like this would support a wider range of applications, from jobs requiring more computational power and longer processing windows for machine learning inference to complex data processing tasks.

Solutions that combine the ease of use of Lambda with the power of EC2 are becoming more and more necessary as serverless architectures develop. Businesses and developers alike are looking for scalable, adaptable serverless solutions that can easily manage a variety of workloads. Future serverless computing approaches could be completely changed by a Lambda on steroids, regardless of the task—real-time analytics, batch processing, or high-performance computing, for example.

Well Ecs might just be the solution you need. Ecs works very similar to Ec2 except this is a task based solution where you

  1. Create Docker images
  2. Create Containers out of the docker images
  3. Define a basic layout of the sort of tasks you might be doing
  4. Use anything you like to initiate a task all the while replacing the original parameters that you defined your tasks with

Yes, it’s that simple

Docker Image

Aws lets you use custom images to run your tasks on. Which means you can have your scripts or binaries or applications in the image that you want. Once built from Dockerfile, you will tag and upload this image into your AWS accounts ECR. Pretty simple to do; you can follow the official docs here.

You can also use public images available in docker hub; all you need is the tag. For this example we will use the python:3.8 image available publicly in docker hub.

Creating ECS Clusters

Once you have a docker image in your hands, head over to your AWS account and go into ECS. Then find clusters and then hit Create cluster. Follow along.

Using fargate as the cluster template

We will use the first option to use the cluster as Fargate (Serverless compute for containers) Hit Next step.

Give it a name and make sure to enable Cloudwatch insights cause they are always helpful. We are not creating a VPC because this account already has one, you can create one if you still need it. Hit Create.

Creating Task Definition

Once complete, we will create a Task definition, this will lay down the ground rules for our tasks. Find the Task Definition option from the sidebar and hit Create new Task Definition.

We will use FARGATE here too to manage our clusters and tasks. Click Next step.

Give your task a name. You can select the amount of default resources you want your container to have; for example, we will use 2 gigs of ram, and a single vCpu but you can always change these for each task you run.

Task Execution Role

You can create a custom task execution role if you want; this role will be used to pull images from ecs, and manage ecs and cloudwatch logs for your task. The console will create one by default, for us that’s okay so we are going with ecsTaskExecutioinRole.

Task Role

The task role is crucial for services using sdks like boto3 or aws-sdk. As this is the role that aws ecs will use to inject permissions into your container for your task to be able to use aws apis. For us the default ecsTaskExecutioinRole role will do just fine.

Container Definition

Click Add container. The major two settings we will use are these ones.

But you can do loads of things with it but for us these two will suffice for a demo. Hit Add and then hit Create. Everything else will be default.

Now that you’ve created a task definition, You can head over to the task definition page and select on your task definition and hit Run Task.

Creating Tasks

Let’s make this image do something. Create a task with the following steps.

For this account, we already have a vpc and subnets, we are just selecting them and the console automatically assigns a security group for us. You can always create your own. Our launch type will be Fargate. As you can see it automatically picked up the cluster we made in step one otherwise you would have to choose a cluster. Since we are using public images, Auto-assign public ip must be ENABLED.

Container settings for each task

In the Advanced Options, we can override any task definition settings we created, for example, we are overriding the command for our task definition. Note that you must provide the commands in comma separated manner and then hit Run Task. It will take you to your cluster’s tasks option and you will see a task being Provisioned. Click on it and then hit Logs. Wait for the logs to appear and voila;

The Logs are conveniently placed along with the execution.

Now, ecs is not meant to be a lambda substitution, this can be used for tasks that require custom applications and scripts or heavy resources or when you are just not comfortable with the 15 minutes timeout.

Caveats

Not many;

  1. It is slower than a lambda to start up; from my experience it takes around a minute for the container to spin up but it’s faster than ec2
  2. It’s a little pricier than a lambda but cheaper than ec2

That’s all folks. Hope this tutorial was useful for all reading. Take care.

Picture of Sakib Alam

Sakib Alam

Software Development Engineer

Hire Exceptional Developers Quickly

Share this blog on

Hire Your Software Development Team

Let us help you pull out your hassle recruiting potential software engineers and get the ultimate result exceeding your needs.

Contact Us Directly

Address:

Plot # 272, Lane # 3 (Eastern Road) DOHS Baridhara, Dhaka 1206

Talk to Us
Scroll to Top