Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ioredis doesn't emit error for ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established #1927

Open
Steffen911 opened this issue Oct 29, 2024 · 1 comment
Open

ioredis doesn't emit error for ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established #1927

Steffen911 opened this issue Oct 29, 2024 · 1 comment

Comments

@Steffen911
Copy link

What happened

We use ioredis within bullmq and observe errors doing message processing when the load on the instance is high. Our redis is configured as follows:

import Redis, { RedisOptions } from "ioredis";
import { env } from "../../env";
import { logger } from "../logger";

export const createNewRedisInstance = () => {
  const instance = new Redis(env.REDIS_CONNECTION_STRING, {
        maxRetriesPerRequest: null,
        enableAutoPipelining: env.REDIS_ENABLE_AUTO_PIPELINING === "true",
        retryStrategy: (times: number) => {
          // Retries forever. Waits at least 1s and at most 20s between retries.
          logger.warn(`Connection to redis lost. Retry attempt: ${times}`);
          return Math.max(Math.min(Math.exp(times), 20000), 1000);
        },
        reconnectOnError: (err) => {
          // Reconnects on READONLY errors and auto-retries the command.
          logger.warn(`Redis connection error: ${err.message}`);
          return err.message.includes("READONLY") ? 2 : false;
        },
      });

  instance?.on("error", (error) => {
    logger.error("Redis error", error);
  });

  return instance;
};

As we see in the logs below, eventually we see the following error:

"2024-10-29T07:18:09.787Z",,"""worker""","}"
"2024-10-29T07:18:09.787Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.787Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.787Z",,"""worker"""," host: 'master.prod-eu-redis-cluster.<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.787Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.787Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.787Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.787Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.787Z",,"""worker""","2024-10-29T07:18:09.786Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","Queue worker cloud-usage-metering-queue failed: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","}"
"2024-10-29T07:18:09.778Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.778Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.778Z",,"""worker"""," host: 'master.prod-eu-redis-cluster..<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.778Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.778Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.778Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.778Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.778Z",,"""worker""","2024-10-29T07:18:09.776Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"

What I expect to happen

Given my code above, I would expect a log message Redis error for the on("error") handler and a Redis connection error warning from the reconnectOnError method. The only logs I observe from our application are the Connection to redis lost. Retry attempt logs, though.

How would I need to adjust the code to correctly log the ECONNRESET errors and then process them in my reconnectOnError?

Versions

[email protected]
[email protected]

Logs

Date,Host,Service,Message
"2024-10-29T07:18:10.809Z","""unknown""","""worker""","2024-10-29T07:18:10.808Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: reconnecting -> connecting"
"2024-10-29T07:18:10.806Z","""unknown""","""worker""","2024-10-29T07:18:10.806Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: reconnecting -> connecting"
"2024-10-29T07:18:10.806Z","""unknown""","""worker""","2024-10-29T07:18:10.806Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:10.806Z","""unknown""","""worker""","2024-10-29T07:18:10.805Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.800Z","""unknown""","""worker""","2024-10-29T07:18:10.800Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:10.799Z","""unknown""","""worker""","2024-10-29T07:18:10.798Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.799Z","""unknown""","""worker""","2024-10-29T07:18:10.798Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.798Z","""unknown""","""worker""","2024-10-29T07:18:10.797Z ioredis:redis write command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:10.796Z","""unknown""","""worker""","2024-10-29T07:18:10.795Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('c95b6be40a2863dd03813001fa069bcd90a2c6c7,8,bull:trace-upsert:wait,bull:trace-upsert:paused,bull:trace-upsert:meta,bull:trace-upsert:id,bull:trace-upsert:completed,bull:trace-upsert:active,bull:trace-u ... <REDACTED full-length=""660"">')"
"2024-10-29T07:18:10.796Z","""unknown""","""worker""","2024-10-29T07:18:10.795Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('c95b6be40a2863dd03813001fa069bcd90a2c6c7,8,bull:trace-upsert:wait,bull:trace-upsert:paused,bull:trace-upsert:meta,bull:trace-upsert:id,bull:trace-upsert:completed,bull:trace-upsert:active,bull:trace-u ... <REDACTED full-length=""660"">')"
"2024-10-29T07:18:10.795Z","""unknown""","""worker""","2024-10-29T07:18:10.794Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.783Z","""unknown""","""worker""","2024-10-29T07:18:10.783Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:10.771Z","""unknown""","""worker""","2024-10-29T07:18:10.770Z ioredis:redis write command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> client([ 'setname', 'bull:dHJhY2UtdXBzZXJ0' ])"
"2024-10-29T07:18:10.770Z","""unknown""","""worker""","2024-10-29T07:18:10.769Z ioredis:redis write command[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> client([ 'setname', 'bull:aW5nZXN0aW9uLXF1ZXVl' ])"
"2024-10-29T07:18:10.769Z",,"""worker""","2024-10-29T07:18:10.768Z ioredis:connection set the connection name [bull:aW5nZXN0aW9uLXF1ZXVl]"
"2024-10-29T07:18:10.769Z","""unknown""","""worker""","2024-10-29T07:18:10.768Z ioredis:redis status[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: connect -> ready"
"2024-10-29T07:18:10.769Z","""unknown""","""worker""","2024-10-29T07:18:10.769Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.768Z",,"""worker""","2024-10-29T07:18:10.767Z ioredis:connection send 1 commands in offline queue"
"2024-10-29T07:18:10.768Z",,"""worker""","2024-10-29T07:18:10.767Z ioredis:connection set the connection name [bull:dHJhY2UtdXBzZXJ0]"
"2024-10-29T07:18:10.768Z","""unknown""","""worker""","2024-10-29T07:18:10.767Z ioredis:redis status[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: connect -> ready"
"2024-10-29T07:18:10.768Z","""unknown""","""worker""","2024-10-29T07:18:10.768Z ioredis:redis write command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:10.751Z",,"""worker""","2024-10-29T07:18:10.751Z ioredis:connection send 1 commands in offline queue"
"2024-10-29T07:18:10.751Z",,"""worker""","2024-10-29T07:18:10.750Z ioredis:connection resend 0 unfulfilled commands"
"2024-10-29T07:18:10.751Z",,"""worker""","2024-10-29T07:18:10.750Z ioredis:connection set the connection name [bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=]"
"2024-10-29T07:18:10.751Z","""unknown""","""worker""","2024-10-29T07:18:10.751Z ioredis:redis write command[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: 0 -> bzpopmin([ 'bull:evaluation-execution-queue:marker', '2.514' ])"
"2024-10-29T07:18:10.751Z","""unknown""","""worker""","2024-10-29T07:18:10.750Z ioredis:redis status[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: connect -> ready"
"2024-10-29T07:18:10.745Z","""unknown""","""worker""","2024-10-29T07:18:10.744Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:10.745Z","""unknown""","""worker""","2024-10-29T07:18:10.744Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.744Z","""unknown""","""worker""","2024-10-29T07:18:10.743Z ioredis:redis write command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> info([])"
"2024-10-29T07:18:10.743Z","""unknown""","""worker""","2024-10-29T07:18:10.743Z ioredis:redis write command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> auth([ 'default', '<pw>' ])"
"2024-10-29T07:18:10.743Z","""unknown""","""worker""","2024-10-29T07:18:10.743Z ioredis:redis status[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: connecting -> connect"
"2024-10-29T07:18:10.734Z","""unknown""","""worker""","2024-10-29T07:18:10.733Z ioredis:redis write command[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> info([])"
"2024-10-29T07:18:10.734Z","""unknown""","""worker""","2024-10-29T07:18:10.733Z ioredis:redis write command[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> auth([ 'default', '<pw>' ])"
"2024-10-29T07:18:10.733Z","""unknown""","""worker""","2024-10-29T07:18:10.733Z ioredis:redis status[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: connecting -> connect"
"2024-10-29T07:18:10.730Z","""unknown""","""worker""","2024-10-29T07:18:10.729Z ioredis:redis write command[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: 0 -> auth([ 'default', '<pw>' ])"
"2024-10-29T07:18:10.730Z","""unknown""","""worker""","2024-10-29T07:18:10.729Z ioredis:redis status[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: connecting -> connect"
"2024-10-29T07:18:10.730Z","""unknown""","""worker""","2024-10-29T07:18:10.730Z ioredis:redis write command[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: 0 -> info([])"
"2024-10-29T07:18:10.726Z","""unknown""","""worker""","2024-10-29T07:18:10.726Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""619"">')"
"2024-10-29T07:18:10.715Z","""unknown""","""worker""","2024-10-29T07:18:10.714Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.715Z","""unknown""","""worker""","2024-10-29T07:18:10.715Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.714Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.711Z","""unknown""","""worker""","2024-10-29T07:18:10.710Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:trace-upsert:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.695Z","""unknown""","""worker""","2024-10-29T07:18:10.694Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""665"">')"
"2024-10-29T07:18:10.695Z","""unknown""","""worker""","2024-10-29T07:18:10.695Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:10.694Z","""unknown""","""worker""","2024-10-29T07:18:10.694Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:10.690Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.681Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.600Z","""unknown""","""worker""","2024-10-29T07:18:10.600Z ioredis:redis status[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: reconnecting -> connecting"
"2024-10-29T07:18:10.599Z","""unknown""","""worker""","2024-10-29T07:18:10.598Z ioredis:redis status[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: reconnecting -> connecting"
"2024-10-29T07:18:10.597Z","""unknown""","""worker""","2024-10-29T07:18:10.596Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.597Z","""unknown""","""worker""","2024-10-29T07:18:10.596Z ioredis:redis status[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: reconnecting -> connecting"
"2024-10-29T07:18:10.587Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.285Z","""unknown""","""worker""","2024-10-29T07:18:10.284Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:10.284Z","""unknown""","""worker""","2024-10-29T07:18:10.284Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""665"">')"
"2024-10-29T07:18:10.007Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:10.003Z","""unknown""","""worker""","2024-10-29T07:18:10.003Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('c95b6be40a2863dd03813001fa069bcd90a2c6c7,8,bull:trace-upsert:wait,bull:trace-upsert:paused,bull:trace-upsert:meta,bull:trace-upsert:id,bull:trace-upsert:completed,bull:trace-upsert:active,bull:trace-u ... <REDACTED full-length=""660"">')"
"2024-10-29T07:18:09.993Z","""unknown""","""worker""","2024-10-29T07:18:09.992Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.992Z","""unknown""","""worker""","2024-10-29T07:18:09.991Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:trace-upsert:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.992Z","""unknown""","""worker""","2024-10-29T07:18:09.992Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:09.833Z","""unknown""","""worker""","2024-10-29T07:18:09.833Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""619"">')"
"2024-10-29T07:18:09.833Z","""unknown""","""worker""","2024-10-29T07:18:09.832Z ioredis:redis queue command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:09.817Z","""unknown""","""worker""","2024-10-29T07:18:09.816Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:09.804Z",,"""worker""","2024-10-29T07:18:09.803Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.804Z","""unknown""","""worker""","2024-10-29T07:18:09.803Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: close -> reconnecting"
"2024-10-29T07:18:09.803Z",,"""worker""","Connection to redis lost. Retry attempt: 2"
"2024-10-29T07:18:09.803Z","""unknown""","""worker""","2024-10-29T07:18:09.802Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: close -> reconnecting"
"2024-10-29T07:18:09.802Z",,"""worker""","2024-10-29T07:18:09.802Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.802Z",,"""worker""","Connection to redis lost. Retry attempt: 2"
"2024-10-29T07:18:09.802Z","""unknown""","""worker""","2024-10-29T07:18:09.801Z ioredis:redis status[master.prod-eu-redis-cluster.<id>..euw1.cache.amazonaws.com:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: connecting -> close"
"2024-10-29T07:18:09.801Z","""unknown""","""worker""","2024-10-29T07:18:09.801Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:legacy-ingestion-queue:wait,bull:legacy-ingestion-queue:active,bull:legacy-ingestion-queue:prioritized,bull:legacy-ingestion-queue:events,bull:legacy-i ... <REDACTED full-length=""770"">')"
"2024-10-29T07:18:09.801Z","""unknown""","""worker""","2024-10-29T07:18:09.800Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('909ce352e2491c9082e6150ad29edff258ef9833,14,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""665"">')"
"2024-10-29T07:18:09.801Z","""unknown""","""worker""","2024-10-29T07:18:09.801Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:09.800Z","""unknown""","""worker""","2024-10-29T07:18:09.800Z ioredis:redis write command[10.30.27.201:6379]: 0 -> ping([])"
"2024-10-29T07:18:09.800Z","""unknown""","""worker""","2024-10-29T07:18:09.799Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:trace-upsert:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.800Z","""unknown""","""worker""","2024-10-29T07:18:09.800Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:09.798Z","""unknown""","""worker""","2024-10-29T07:18:09.797Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.787Z",,"""worker""","Queue worker batch-export-queue failed: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.787Z",,"""worker""","}"
"2024-10-29T07:18:09.787Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.787Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.787Z",,"""worker"""," host: 'master.prod-eu-redis-cluster.<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.787Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.787Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.787Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.787Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.787Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.787Z",,"""worker""","2024-10-29T07:18:09.786Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","Queue worker cloud-usage-metering-queue failed: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.778Z",,"""worker""","}"
"2024-10-29T07:18:09.778Z",,"""worker"""," localAddress: undefined"
"2024-10-29T07:18:09.778Z",,"""worker"""," port: 6379,"
"2024-10-29T07:18:09.778Z",,"""worker"""," host: 'master.prod-eu-redis-cluster..<id>.euw1.cache.amazonaws.com',"
"2024-10-29T07:18:09.778Z",,"""worker"""," path: undefined,"
"2024-10-29T07:18:09.778Z",,"""worker"""," code: 'ECONNRESET',"
"2024-10-29T07:18:09.778Z",,"""worker"""," at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {"
"2024-10-29T07:18:09.778Z",,"""worker"""," at endReadableNT (node:internal/streams/readable:1696:12)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.emit (node:events:531:35)"
"2024-10-29T07:18:09.778Z",,"""worker"""," at TLSSocket.onConnectEnd (node:_tls_wrap:1730:19)"
"2024-10-29T07:18:09.778Z",,"""worker""","2024-10-29T07:18:09.776Z ioredis:connection error: Error: Client network socket disconnected before secure TLS connection was established"
"2024-10-29T07:18:09.709Z","""unknown""","""worker""","2024-10-29T07:18:09.709Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:trace-upsert:wait,bull:trace-upsert:active,bull:trace-upsert:prioritized,bull:trace-upsert:events,bull:trace-upsert:stalled,bull:trace-upsert:limiter,b ... <REDACTED full-length=""435"">')"
"2024-10-29T07:18:09.709Z","""unknown""","""worker""","2024-10-29T07:18:09.708Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:ingestion-queue:wait,bull:ingestion-queue:active,bull:ingestion-queue:prioritized,bull:ingestion-queue:events,bull:ingestion-queue:stalled,bull:ingesti ... <REDACTED full-length=""472"">')"
"2024-10-29T07:18:09.708Z","""unknown""","""worker""","2024-10-29T07:18:09.708Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:09.693Z",,"""worker""","Processing ingestion event"
"2024-10-29T07:18:09.676Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:09.528Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:09.528Z",,"""worker""","2024-10-29T07:18:09.528Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.528Z",,"""worker""","2024-10-29T07:18:09.527Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.528Z","""unknown""","""worker""","2024-10-29T07:18:09.527Z ioredis:redis status[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: close -> reconnecting"
"2024-10-29T07:18:09.528Z","""unknown""","""worker""","2024-10-29T07:18:09.528Z ioredis:redis status[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: close -> reconnecting"
"2024-10-29T07:18:09.528Z","""unknown""","""worker""","2024-10-29T07:18:09.528Z ioredis:redis status[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: ready -> close"
"2024-10-29T07:18:09.527Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:09.527Z",,"""worker""","2024-10-29T07:18:09.526Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:09.527Z","""unknown""","""worker""","2024-10-29T07:18:09.526Z ioredis:redis status[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: ready -> close"
"2024-10-29T07:18:09.527Z","""unknown""","""worker""","2024-10-29T07:18:09.527Z ioredis:redis status[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: ready -> close"
"2024-10-29T07:18:09.527Z","""unknown""","""worker""","2024-10-29T07:18:09.526Z ioredis:redis status[10.30.27.201:6379 (bull:ZXZhbHVhdGlvbi1leGVjdXRpb24tcXVldWU=)]: close -> reconnecting"
"2024-10-29T07:18:09.526Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:09.524Z","""unknown""","""worker""","2024-10-29T07:18:09.524Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('a2b97b03b7d21d69b152446117520ec099d22a84,9,bull:cloud-usage-metering-queue:stalled,bull:cloud-usage-metering-queue:wait,bull:cloud-usage-metering-queue:active,bull:cloud-usage-metering-queue:failed,bu ... <REDACTED full-length=""452"">')"
"2024-10-29T07:18:09.487Z","""unknown""","""worker""","2024-10-29T07:18:09.487Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ '99b6eb3d7a4777fb2271fd6cdf3ac5a290a4aac4', '2', 'bull:ingestion-queue:30470371:lock', 'bull:ingestion-queue:stalled', '7b8d9abf-b30f-463b-9ce3-3dfd7637d2ad:37478', '30000', '30470371' ])"
"2024-10-29T07:18:09.484Z","""unknown""","""worker""","2024-10-29T07:18:09.483Z ioredis:redis status[10.30.27.201:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: reconnecting -> connecting"
"2024-10-29T07:18:09.483Z","""unknown""","""worker""","2024-10-29T07:18:09.482Z ioredis:redis status[10.30.27.201:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: reconnecting -> connecting"
"2024-10-29T07:18:00.918Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:00.916Z","""unknown""","""worker""","2024-10-29T07:18:00.915Z ioredis:redis write command[10.30.27.201:6379 (bull:aW5nZXN0aW9uLXF1ZXVl)]: 0 -> bzpopmin([ 'bull:ingestion-queue:marker', '5' ])"
"2024-10-29T07:18:00.911Z",,"""worker""","Processing legacy ingestion"
"2024-10-29T07:18:00.897Z","""unknown""","""worker""","2024-10-29T07:18:00.896Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:00.897Z","""unknown""","""worker""","2024-10-29T07:18:00.896Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha([ 'a8034c631460c6803c87966ce881636246de2398', '1', 'bull:legacy-ingestion-queue:', 'wait', 'paused', 'delayed', 'prioritized', 'waiting-children' ])"
"2024-10-29T07:18:00.897Z",,"""worker""","Processing ingestion event"
"2024-10-29T07:18:00.896Z","""unknown""","""worker""","2024-10-29T07:18:00.895Z ioredis:redis queue command[10.30.27.201:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: 0 -> bzpopmin([ 'bull:cloud-usage-metering-queue:marker', '5' ])"
"2024-10-29T07:18:00.880Z","""unknown""","""worker""","2024-10-29T07:18:00.879Z ioredis:redis status[10.30.27.201:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: close -> reconnecting"
"2024-10-29T07:18:00.880Z",,"""worker""","2024-10-29T07:18:00.879Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:00.879Z","""unknown""","""worker""","2024-10-29T07:18:00.879Z ioredis:redis status[10.30.27.201:6379 (bull:YmF0Y2gtZXhwb3J0LXF1ZXVl)]: ready -> close"
"2024-10-29T07:18:00.879Z","""unknown""","""worker""","2024-10-29T07:18:00.879Z ioredis:redis status[10.30.27.201:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: close -> reconnecting"
"2024-10-29T07:18:00.879Z","""unknown""","""worker""","2024-10-29T07:18:00.878Z ioredis:redis status[10.30.27.201:6379 (bull:Y2xvdWQtdXNhZ2UtbWV0ZXJpbmctcXVldWU=)]: ready -> close"
"2024-10-29T07:18:00.879Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:00.879Z",,"""worker""","2024-10-29T07:18:00.879Z ioredis:connection reconnect in 1000ms"
"2024-10-29T07:18:00.879Z",,"""worker""","Connection to redis lost. Retry attempt: 1"
"2024-10-29T07:18:00.877Z","""unknown""","""worker""","2024-10-29T07:18:00.876Z ioredis:redis write command[10.30.27.201:6379 (bull:dHJhY2UtdXBzZXJ0)]: 0 -> bzpopmin([ 'bull:trace-upsert:marker', '5' ])"
"2024-10-29T07:18:00.877Z","""unknown""","""worker""","2024-10-29T07:18:00.876Z ioredis:redis write command[10.30.27.201:6379]: 0 -> evalsha('accb36ea979fc4cd20f4a56e3c4ad6df0ec0b14a,11,bull:cloud-usage-metering-queue:wait,bull:cloud-usage-metering-queue:active,bull:cloud-usage-metering-queue:prioritized,bull:cloud-usage-metering-queue:even ... <REDACTED full-length=""602"">')"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Steffen911