Skip to content

Commit

Permalink
Refactors JoobQ module and enhances queue management
Browse files Browse the repository at this point in the history
Removes redundant documentation from JoobQ module for clarity
and updates queue handling by leveraging a shorthand method
to access queues by name. Enhances BaseQueue with additional
attributes for operational metrics and changes it to a module
for consistent behavior. Adjusts API server response with direct
queue name usage for consistent outputs.

These changes streamline the codebase, reduce duplication, and
improve the flexibility and readability of the queue management
logic in the job processing system.
  • Loading branch information
eliasjpr committed Nov 27, 2024
1 parent 748051b commit 2cfc3e7
Show file tree
Hide file tree
Showing 3 changed files with 13 additions and 59 deletions.
53 changes: 0 additions & 53 deletions src/joobq.cr
Original file line number Diff line number Diff line change
Expand Up @@ -7,55 +7,6 @@ require "cron_parser"
require "./joobq/store"
require "./joobq/**"

# ### Module `JoobQ`
#
# The main `JoobQ` module, which acts as the central orchestrator for a job
# queueing system. This module integrates various components like job queueing, scheduling, statistics tracking, and
# logging. Here's a detailed documentation of the `JoobQ` module:
#
# #### Overview
#
# The `JoobQ` module is the core module of a job queue system designed in Crystal. It sets up and manages the entire
# job processing environment, including configuration, queue management, scheduling, and statistics.
#
# #### Constants
#
# - `REDIS`: Initialized with the Redis client instance from `Configure.instance`, used for all Redis operations
# within the module.
#
# #### Initialization and Configuration
#
# - `Log.for("JoobQ")`: Initializes logging for the `JoobQ` system.
# - `Log.setup_from_env`: Sets up logging configuration from environment variables.
#
# #### Methods
#
# - `configure`: Provides a way to configure the `JoobQ` system. Yields to a block with `Configure.instance`
# for setting up configurations.
# - `config`: Returns the configuration instance (`Configure.instance`).
# - `queues`: Returns the hash of queues set up in the configuration.
# - `statistics`: Returns an instance of the `Statistics` for tracking and managing statistical data.
# - `push(job)`: Adds a job to its respective queue in Redis and logs the action. Returns the job's unique
# identifier (`jid`).
# - `scheduler`: Returns an instance of the `Scheduler` for managing job scheduling.
# - `[](name : String)`: A shorthand method to access a specific queue by name from the configured queues.
# - `reset`: Clears the Redis database and re-creates the statistical series.
# - `forge`: The main method to boot the `JoobQ` system. It initializes the statistics, starts the scheduler,
# starts all queues, and logs the initialization process.
#
# #### Usage
#
# - To initialize and start the `JoobQ` system, call `JoobQ.forge`.
# - Use `JoobQ.configure` to set up system configurations.
# - Jobs can be pushed to the queue using `JoobQ.add(job)`.
# - Access specific queues or the scheduler as needed.
#
# ### Notes
#
# - The `JoobQ` module brings together different components like queues, scheduler, and statistics into a cohesive system.
# - The use of a centralized Redis client ensures consistent database interactions.
# - Logging and statistics creation are integral parts of the module, facilitating monitoring and debugging.
# - The module's design allows for flexible configuration and easy management of job queues.
module JoobQ
CONFIG = Configure.new

Expand All @@ -75,10 +26,6 @@ module JoobQ
store.reset
end

def self.statistics
JoobQ::GlobalStats.instance.calculate_stats
end

def self.queues
config.queues
end
Expand Down
7 changes: 3 additions & 4 deletions src/joobq/api_server.cr
Original file line number Diff line number Diff line change
Expand Up @@ -160,16 +160,15 @@ module JoobQ
if request_body = context.request.body.try(&.gets_to_end)
payload = JSON.parse(request_body)
queue_name = payload["queue"].to_s
queue = ::JoobQ.queues[queue_name]

queue = JoobQ[queue_name]
return {error: "Invalid queue name"}.to_json unless queue

# Assuming there's a method to add a job to the queue
jid = queue.add(request_body)

response = {
status: "Job enqueued",
queue: queue.name,
queue: queue_name,
job_id: jid.to_s,
}
context.response.content_type = "application/json"
Expand Down Expand Up @@ -197,7 +196,7 @@ module JoobQ
def call(context : HTTP::Server::Context)
method = context.request.method
path = context.request.path
if handler = ENDPOINTS[{method: method, path: path}]
if handler = ENDPOINTS[{method: method, path: path}]?
handler.call(context)
else
call_next(context)
Expand Down
12 changes: 10 additions & 2 deletions src/joobq/queue.cr
Original file line number Diff line number Diff line change
@@ -1,13 +1,21 @@
module JoobQ
# BaseQueue remains as the abstract base class
abstract class BaseQueue
module BaseQueue
abstract def add(job : String)
abstract def start
abstract def stop!
abstract def name : String
abstract def size : Int64
abstract def job_type : String
abstract def total_workers : Int32
abstract def running_workers : Int32
abstract def status : String
abstract def throttle_limit : NamedTuple(limit: Int32, period: Time::Span)?
end

# The Queue class now focuses solely on queue operations
class Queue(T) < BaseQueue
class Queue(T)
include BaseQueue
getter store : Store = ::JoobQ.config.store
getter name : String
getter job_type : String = T.name
Expand Down

0 comments on commit 2cfc3e7

Please sign in to comment.