chore(bug): fixed a bug where processing step was over counting #28
Conversation
Sources/Jobs/JobQueue.swift
Outdated
| let jobName = id.name | ||
| let id = try await self.queue.push(buffer, options: options) | ||
| Meter(label: "swift.jobs.meter", dimensions: [("status", "queued"), ("name", jobName)]).increment() |
There was a problem hiding this comment.
I assume this was causing issues when a push fails?
There was a problem hiding this comment.
The meter label is hard coded here, but not in JobQueueHandler
There was a problem hiding this comment.
I am moving the labels into an internal structure named JobMetricsHelper so they can be accessible in the swift jobs library.
Sources/Jobs/JobQueueHandler.swift
Outdated
| @@ -27,6 +27,7 @@ final class JobQueueHandler<Queue: JobQueueDriver>: Sendable { | |||
| self.logger = logger | |||
| self.jobRegistry = .init() | |||
| self.options = options | |||
| Meter(label: "swift.jobs.worker.count").increment(by: Double(numWorkers)) | |||
There was a problem hiding this comment.
When does this number decrement?
There was a problem hiding this comment.
What happens when an app crashes?
There was a problem hiding this comment.
We should not decrement this gauge. We can use it for alerting purposes because once an app crashes or shutdown, after a while prometheus will return 0
There was a problem hiding this comment.
We should not decrement this gauge. We can use it for alerting purposes because once an app crashes or shutdown, after a while prometheus will return 0
Are you saying that prometheus will revert a Meter to zero if the connection from the app is lost due to a crash? What does that mean for all of the other Meters?
There was a problem hiding this comment.
I should have phrased the previous sentence better. Prometheus would not reset the meter to zero, there would be no data if one where to query for most recent update.
max_over_time(swift_jobs_worker_count{}[5m]))
To be safe, I will try to decrement during graceful shutdown. I can see the worker count might be misleading in a continuous deployment environment.
This PR is for the followings:
1 - fixed an issue where the queued meter would become negative during retries
2 - added number of workers to job metricsprocessing count can give the same metric
3 - added discarded jobs metrics
With the most recent commit:
Once can get prometheus UI or Grafana to update queue in realtime with the following query