Trigger task on ECS Fargate based on SQS Queue

[karkir0003]

For my application, I'm trying to use ECS Fargate to run a computing task. I want ECS Fargate to be able to service a queue of requests and autoscale as needed. In other words, the backend of my app should ideally write to a queue in SQS with the data needed to run the task and ECS Fargate should poll the SQS queue and run the tasks as needed and return the output. How would I construct an architecture like this using Python Boto 3? Can someone provide a working code example?

[RyanFitzSimmonsAK]

Hi @karkir0003, thanks for reaching out. Your questions are a bit broad in scope relative to what this forum is intended for. I'd encourage you to take a look at the AWS documentation and the Boto3 developer guide. These resources should contain all the information you need about the services you mentioned, as well as details about any other services you might need. I would also specifically mention CDK (Cloud Development Kit) as a good solution to your use case. Here is the CDK Workshop to help you get started. Hope that helps!

For more specific information, here are some relevant links to the services you mentioned. I also included some information about AWS Lambda, since it is a straightforward way to process SQS messages and trigger an ECS Fargate task. If that solution doesn't work for your use case, you'd have to add code to process the queue yourself in ECS.

• Create queue: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.create_queue
• Send message: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.send_message
• Getting queue attributes (including the number of messages): https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs.html#SQS.Client.get_queue_attributes
• Configuring a queue as an event source in Lambda: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#events-sqs-eventsource

Can you clarify whether using Lambda to trigger the tasks works for your use case? Thanks!

[karkir0003]

So I'm flexible on this. I was initially trying to have backend write message to sqs queue and have fargate directly pick up messages from the queue, process message and mark the message as deleted so that 2 ec2 instances for example dont run the same task/message from sqs

[RyanFitzSimmonsAK]

In that case, I'd recommend you read this page of the documentation which discusses the SQS visibility timeout. There's also a few nuances that are discussed in the first few paragraphs of the Lambda documentation I linked previously. With that in mind, both solutions (Lambda or writing it yourself in ECS) should be fine.

[karkir0003]

Got it. Thanks so much for the detailed response