How to debug performance?

[angel-rs]

First of all, thanks for such simple & great library :)

I have a queue that is critical for it to perform tasks as fast as possible.
Jobs take some time to go from "waiting" to "active" queue

Here are some data points from latest jobs:

(3 Different runs for the same job)

Job run #1
Added at 11:06:07
Process started at 11:06:15
Finished at 11:06:18
Took 8s to start processing (bull)
Took 2s to process job (user code)

Job run #2
Added at 10:21:22
Process started at 10:21:42
Finished at 10:21:45
Took 20s to start processing (bull)
Took 3s to process job (user code)

Job run #3
Added at 10:13:23
Process started at 10:13:36
Finished at 10:13:48
Took 13s to start processing (bull)
Took 12s to process job (user code)

Bull version: 4.8.3 (Recently upgraded from v3.x, thinking version could be the problem)
Redis version: 6.2.6 (Cloud instance of redislabs free plan)

Code implementation is fairly simple so wondering if it's a config/hardware/network-roundtrips issue?
Did some local benchmarking & couldn't replicate the issue, so thought in asking here:
How could I go about debugging performance root issue?

Just for reference what I'm doing:

import Queue from "bull";

const commonOptions = {
  maxRetriesPerRequest: null,
  enableReadyCheck: false,
};

const REDIS_CONNECTION = process.env.REDISCLOUD_URL;

const redis = {
  client: new Redis(REDIS_CONNECTION, commonOptions),
  subscriber: new Redis(REDIS_CONNECTION, commonOptions),
  fallback: new Redis(REDIS_CONNECTION, commonOptions),
};

const options = {
  createClient(type) {
    switch (type) {
      case "client":
        return redis.client;
      case "subscriber":
        return redis.subscriber;
      default:
        return redis.fallback;
    }
  },
};

const queue = new Queue("TEST_QUEUE", options);
queue.process(async () => {
  // my job logic
})

// somewhere else in the project codebase
queue.add(params)

[manast]

Are the above 3 cherry-picked jobs on a larger flow of jobs or just the 3 jobs there are?

[angel-rs]

They are the same job, just three different runs.

[manast]

You can use Redis command MONITOR to see what is going on, several seconds for processing a dummy job is wrong, but I have no idea based on the information that you are providing.

[angel-rs]

Managed to solve the issue & jobs now stay in the waiting queue for <1s 🎊

Not 100% what the issue was. I think it had something to do with the redis connections. Reading through the tests, updated my createClient logic to be similar to this. i.e:

const redis = {
  client: null,
  subscriber: null,
};

export const bullOpts = {
  createClient(type, redisOpts) {
    switch (type) {
      case "client":
        if (!redis.client) {
          redis.client = createRedisConnection(redisOpts);
        }
        return redis.client;
      case "subscriber":
        if (!redis.subscriber) {
          redis.subscriber = createRedisConnection(redisOpts);
        }
        return redis.subscriber;
      case "bclient":
        return createRedisConnection(redisOpts);
      default:
        throw new Error("Unexpected connection type: ", type);
    }
  },
};

Main difference is it will always create a new connection when the client type is bclient (Perhaps the root issue?)

[manast]

Yes, you need to always create a new blocking client. Performance could be thousands of jobs per second if you increase the concurrency.