Common batch params

[dmitrypol]

Is there a way to store common job params?

For example, I want to import spreadsheet with thousands of records.
I convert each row into separate job and run them as a batch.
I store results (error or success) or each row in separate Redis List
Then when batch is done I go through my success and error Lists and email results to Batch Owner.

Anyone has experience with such issue? Ideally I would store batch_owner, batch_error and batch_success as common batch params.

I currently use my own solution but looking into switching to using this gem.

[cdale77]

Hi @dmitrypol. Sorry for the delayed response, I did not see this ticket.

A job batch is really just a Redis entry with a batch ID for a key and an array of job ids. I believe Redis supports a hash data type, so we could probably change to that and support job options. I'm not sure how that would affect compatibility with non-redis stores, though.

Could you use a database table for your needs? A table could store a batch or job id, and it's status and other details about the job. Note that the Redis values expire, so you'd need to move this data into some sort of long term storage if you need it for more than a couple of days.

[dmitrypol]

Hi @cdale77
Yes, using Redish Hash cold be one solution but that would tie active_job_status to Redis (and what about gems like DelayedJob?) Using DB is a problem because writing to disk each success/error would significantly slow down the process. And I only need to store this data for a limited time while the batch is running.

I have my custom solution built on top of Sidekiq but I also like this gem. I guess there is no easy answer here.

[cdale77]

Yes, I don't see an easy, simple extension here to do what you're looking for. I'll try to keep thinking about it. The original case for this gem was basically for UI -- to display a little "processing" notification to a user, for long running jobs.

In the past I've built systems that import database records from spreadsheets. It somewhat similar -- I go through each row in the sheet, tracking success or failure, and then email somebody the results. In those cases I made a db table, with one entry per sheet. I'd assume success, and only keep track of failing records (usually in a postgrss array or hash or something), then email the result when done.

Speed wasn't important to me. But by assuming success and only tracking the failures, and only doing a write at the end of the job, I wonder if a db-based solution might work? I also generally preferred a db approach because of record-keeping needs.