SQS function being called twice?

[eroltoker]

I'm trying to use a function to process messages from SQS. I believe I've followed the demo exactly... here is the file that the function is calling:

`<?php
require_once 'vendor/autoload.php';

// This is a PHP file example.
// Replace it with your application.

$logger = new \Bref\Logger\StderrLogger();
$rand = rand(0,100);
$logger->warning("SQS Handler entered {$rand}");

use Bref\Context\Context;
use Bref\Event\Sqs\SqsEvent;
use Bref\Event\Sqs\SqsHandler;

class Handler extends SqsHandler
{
public function handleSqs(SqsEvent $event, Context $context): void
{
foreach ($event->getRecords() as $record) {
// We can retrieve the message body of each record via ->getBody()
$body = $record->getBody();

    }
}

}

return new Handler();
`
If you look at the logs below, it looks like the code outside the handler is getting called twice. When I log messages from inside the SQS function, it looks like it's only happening once, so that's good. However, I just want to make sure I'm not doing something wrong before starting to build more functionality on top of this.

[eroltoker]

somewhat related... what is the best practice for deleting the message from the queue once it's processed?

Am I supposed to get the messageId >> create an SQS client >> delete message

OR

Is there some built-in best practice in bref that i should be following

[eroltoker]

because i don't know how to 'not delete' the SQS message, I've fallen back on requeue'ing the message in a separate SQS queue with a longer holding time to allow the first request to complete. If there is a better solution you suggest I would appreciate it. Thanks!

[afu-dev]

Hello,

If I understand correctly, you want to set a VisibilityTimeout in the SQS configuration. If you use Serverless Framework to deploy your queue, you can add the property to your CloudFormation declaration (ref: serverless/serverless#6565 (comment)).

It's a good practice to set the VisibilityTimeout to 6x the timeout of your lambda function. Or at least the maximum duration of it. Ref: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#events-sqs-queueconfig.

If you want to fail a duplicated message to retry it later (with cache), you will need a deduplication id in your message. Be aware that SQS already handles a deduplication ID, but this is not what you want: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html.

In your case, you want a deduplication ID in your message. When you start to handle it, save it somewhere (database, efs, ...), and mark it as "processing". If you already have the ID marked as processing, fail the message to let it return to the SQS. It will be re-processed a few minutes later (with a VisibilityTimeout correctly set).

I hope this can help.

[eroltoker]

thank you @afu-dev , this is exactly what i have set my code up to do. What I don't know is how to "fail the message to let it return to SQS".

Normally the way to do this with a worker hitting SQS (I believe) is to simply do nothing, because the way to mark a message as completed is to explicitly delete it from SQS after the message has been successfully handled.

Here it is the opposite; it looks like bref is automatically handling removing the SQS message from the queue. I tried throwing generic exceptions in my handler but couldn't stop the message from being removed from the queue. If there is a simple code snippet you can point me to on how to do this 'failure', it would be so much appreciated. Thank you!

[eroltoker]

The approach I mentioned above seems to have failed in a strange way. I specified two separate queues in the serverless.yml file (one for the original SQS queue and one to handle the messages that I wanted to delay). To recap, the flow is:

• message placed in queue1
• message processed via sqshandler
• if message should be delayed, it is placed in queue2, which has a delay built in.

serverless.yaml config:

    events:
        -   sqs:
                arn: arn:aws:sqs:us-east-1:752544013807:real-time-job-queue
                batchSize: 1
        -   sqs:
                arn: arn:aws:sqs:us-east-1:752544013807:duplicate-job-queue
                batchSize: 1

What happens here is the code seems to execute as expected except I don't see any logs for the SqsHandler in cloudwatch😭. Deleting the second queue makes it work. Is there a limit of 1 queue per function?

For now, I don't have a solution to handle these messages @afu-dev , if you can point me to docs/code sample of how to fail a message being handled by the SqsHandler sample in the docs, I would really appreciate it!

[afu-dev]

To elaborate on the retry system:

You can use the maxReceiveCount property to allow a message to be retried: https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html.
Note that AWS Lambda also has a system to retry messages: https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html.

Another ref: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

I don't have time to create a PoC, but you should be able to do what you describe with only one queue.

[mnapoli]

To answer your first question, this is normal.

The lambda function is preparing for the next message, so all the initialization code runs again to be ready for the next invocation.