Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kuruko2 schedule job commands sometime runs twice #122

Open
algiecaballesbbo opened this issue Aug 23, 2018 · 8 comments
Open

Kuruko2 schedule job commands sometime runs twice #122

algiecaballesbbo opened this issue Aug 23, 2018 · 8 comments

Comments

@algiecaballesbbo
Copy link

When running testing executing a simple command execute: node -v and scheduled it to run every minute, there are instance that the job runs twice. please refer to the attached images for more details
screen shot 2018-08-23 at 12 26 19 pm
screen shot 2018-08-23 at 1 52 47 pm

Info:
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
Rails 5.2.1
rvm 1.29.4 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
node v8.4.0
RAILS_ENV=production
NUM_OF_WORKERS=1
@eagletmt
Copy link
Member

I never see such situation...
Could you paste log/job-scheduler.log and log/production.log?
Which RDB are you using for Kuroko2?

Kuroko2 currently does not support Rails 5.2 series, so you cannot use Kuroko2 with Rails 5.2.1.
https://github.com/cookpad/kuroko2/blob/v0.5.0/kuroko2.gemspec#L20
You can check installed gems by running bundle show command.

@algiecaballesbbo
Copy link
Author

For you question Kuroko2 currently does not support Rails 5.2 series, so you cannot use Kuroko2 with Rails 5.2.1.? , its currently working in our production using this version on rails. the only problem we have is it has duplicate process each commands when running scheduled jobs where we only use only 1 worker.
Second question Which RDB are you using for Kuroko2?, we are using MariaDB v5.5.56
Our installed gems in the production:

Gems included by the bundle:
  * actioncable (5.2.1)
  * actionmailer (5.2.1)
  * actionpack (5.2.1)
  * actionview (5.2.1)
  * activejob (5.2.1)
  * activemodel (5.2.1)
  * activerecord (5.2.1)
  * activestorage (5.2.1)
  * activesupport (5.2.1)
  * addressable (2.5.2)
  * archive-zip (0.11.0)
  * arel (9.0.0)
  * aws-sdk (2.11.96)
  * aws-sdk-core (2.11.96)
  * aws-sdk-resources (2.11.96)
  * aws-sigv4 (1.0.3)
  * bindex (0.5.0)
  * bootsnap (1.3.1)
  * builder (3.2.3)
  * bundler (1.16.3)
  * byebug (10.0.2)
  * capybara (3.4.2)
  * childprocess (0.9.0)
  * chromedriver-helper (1.2.0)
  * chrono (0.4.0)
  * coffee-rails (4.2.2)
  * coffee-script (2.3.0)
  * coffee-script-source (1.12.2)
  * commonmarker (0.16.8)
  * concurrent-ruby (1.0.5)
  * crass (1.0.4)
  * dalli (2.7.8)
  * dotenv (0.11.1)
  * dotenv-deployment (0.0.2)
  * dotenv-rails (0.11.1)
  * erubi (1.7.1)
  * execjs (2.7.0)
  * faraday (0.12.2)
  * ffi (1.9.25)
  * font-awesome-rails (4.7.0.4)
  * foreman (0.85.0)
  * globalid (0.4.1)
  * haml (5.0.4)
  * hashie (3.5.7)
  * hipchat (1.3.0)
  * html-pipeline (2.8.4)
  * http_accept_language (2.1.1)
  * httparty (0.16.2)
  * i18n (1.1.0)
  * io-like (0.3.0)
  * jbuilder (2.7.0)
  * jmespath (1.4.0)
  * jquery-rails (4.3.3)
  * json (2.1.0)
  * jwt (1.5.6)
  * kaminari (1.1.1)
  * kaminari-actionview (1.1.1)
  * kaminari-activerecord (1.1.1)
  * kaminari-core (1.1.1)
  * kuroko2 (0.4.6)
  * listen (3.1.5)
  * loofah (2.2.2)
  * mail (2.7.0)
  * marcel (0.3.2)
  * method_source (0.9.0)
  * mimemagic (0.3.2)
  * mini_mime (1.0.1)
  * mini_portile2 (2.3.0)
  * minitest (5.11.3)
  * momentjs-rails (2.20.1)
  * msgpack (1.2.4)
  * multi_json (1.13.1)
  * multi_xml (0.6.0)
  * multipart-post (2.0.0)
  * mysql2 (0.5.2)
  * nio4r (2.3.1)
  * nokogiri (1.8.4)
  * oauth2 (1.4.0)
  * oj (3.6.5)
  * omniauth (1.8.1)
  * omniauth-google-oauth2 (0.2.10)
  * omniauth-oauth2 (1.3.1)
  * passenger (5.3.3)
  * public_suffix (3.0.2)
  * rack (2.0.5)
  * rack-accept-default (0.0.2)
  * rack-store (0.0.4)
  * rack-test (1.1.0)
  * rails (5.2.1)
  * rails-dom-testing (2.0.3)
  * rails-html-sanitizer (1.0.4)
  * rails_bootstrap_sortable (2.0.3)
  * railties (5.2.1)
  * rake (12.3.1)
  * rb-fsevent (0.10.3)
  * rb-inotify (0.9.10)
  * redcarpet (3.4.0)
  * redis (4.0.1)
  * responders (2.4.0)
  * retryable (3.0.1)
  * rinku (2.0.4)
  * ruby-enum (0.7.2)
  * ruby_dep (1.5.0)
  * rubyzip (1.2.1)
  * sass (3.4.25)
  * sass-rails (5.0.7)
  * select2-rails (4.0.3)
  * selenium-webdriver (3.13.1)
  * serverengine (1.5.11)
  * sigdump (0.2.4)
  * slim (3.0.9)
  * slim-rails (3.1.3)
  * spring (2.0.2)
  * spring-watcher-listen (2.0.1)
  * sprockets (3.7.2)
  * sprockets-rails (3.2.1)
  * temple (0.8.0)
  * the_garage (2.4.3)
  * thor (0.19.4)
  * thread_safe (0.3.6)
  * tilt (2.0.8)
  * tzinfo (1.2.5)
  * uglifier (2.7.2)
  * visjs-rails (4.21.0.0)
  * weak_parameters (0.5.0)
  * web-console (3.6.2)
  * websocket-driver (0.7.0)
  * websocket-extensions (0.1.3)
  * xpath (3.1.0)

@riseshia
Copy link
Contributor

Could you paste log/job-scheduler.log and log/production.log?

Please check this one, too.

@joefrey25
Copy link

joefrey25 commented Aug 31, 2018 •
edited
Loading

Hi @riseshia san and @eagletmt san

This is lately the incident we have encountered in our production server.
log.txt

duplicate

@joefrey25
Copy link

hi @eagletmt san @riseshia san
This is just a follow up question. How do you properly restart kuroko2 services daemon in the server?
Do you have any proper commands that we can restart these services?

TIA

@riseshia
Copy link
Contributor

riseshia commented Sep 5, 2018

I read pasted logs, It seems that more than one job-scheduler is running

D, [2018-08-31T04:00:26.906791 #3825] DEBUG -- :    (0.2ms)  BEGIN
D, [2018-08-31T04:00:26.907739 #3825] DEBUG -- :   Kuroko2::Tick Load (0.4ms)  SELECT  `kuroko2_ticks`.* FROM `kuroko2_ticks` ORDER BY `kuroko2_ticks`.`id` ASC LIMIT 1
D, [2018-08-31T04:00:26.908511 #3825] DEBUG -- :   Kuroko2::Tick Update (0.2ms)  UPDATE `kuroko2_ticks` SET `at` = '2018-08-30 19:00:26' WHERE `kuroko2_ticks`.`id` = 1
D, [2018-08-31T04:00:26.909129 #3825] DEBUG -- :   Kuroko2::JobSchedule Load (0.2ms)  SELECT  `kuroko2_job_schedules`.* FROM `kuroko2_job_schedules` ORDER BY `kuroko2_job_schedules`.`id` ASC LIMIT 1000
D, [2018-08-31T04:00:26.909960 #3825] DEBUG -- :   Kuroko2::JobDefinition Load (0.2ms)  SELECT `kuroko2_job_definitions`.* FROM `kuroko2_job_definitions` WHERE `kuroko2_job_definitions`.`id` IN (2, 5)
D, [2018-08-31T04:00:26.910801 #3825] DEBUG -- :   Kuroko2::JobSuspendSchedule Load (0.2ms)  SELECT `kuroko2_job_suspend_schedules`.* FROM `kuroko2_job_suspend_schedules` WHERE `kuroko2_job_suspend_schedules`.`job_definition_id` IN (2, 5)
D, [2018-08-31T04:00:26.922660 #3825] DEBUG -- :    (0.9ms)  COMMIT
D, [2018-08-31T04:00:26.934955 #3855] DEBUG -- :    (0.1ms)  BEGIN
D, [2018-08-31T04:00:26.935703 #3855] DEBUG -- :   Kuroko2::Tick Load (0.3ms)  SELECT  `kuroko2_ticks`.* FROM `kuroko2_ticks` ORDER BY `kuroko2_ticks`.`id` ASC LIMIT 1
D, [2018-08-31T04:00:26.936373 #3855] DEBUG -- :   Kuroko2::Tick Update (0.2ms)  UPDATE `kuroko2_ticks` SET `at` = '2018-08-30 19:00:26' WHERE `kuroko2_ticks`.`id` = 1
D, [2018-08-31T04:00:26.936966 #3855] DEBUG -- :   Kuroko2::JobSchedule Load (0.2ms)  SELECT  `kuroko2_job_schedules`.* FROM `kuroko2_job_schedules` ORDER BY `kuroko2_job_schedules`.`id` ASC LIMIT 1000
D, [2018-08-31T04:00:26.937774 #3855] DEBUG -- :   Kuroko2::JobDefinition Load (0.2ms)  SELECT `kuroko2_job_definitions`.* FROM `kuroko2_job_definitions` WHERE `kuroko2_job_definitions`.`id` IN (2, 5)
D, [2018-08-31T04:00:26.938588 #3855] DEBUG -- :   Kuroko2::JobSuspendSchedule Load (0.2ms)  SELECT `kuroko2_job_suspend_schedules`.* FROM `kuroko2_job_suspend_schedules` WHERE `kuroko2_job_suspend_schedules`.`job_definition_id` IN (2, 5)
D, [2018-08-31T04:00:26.947863 #3855] DEBUG -- :    (0.7ms)  COMMIT

Base on your comment (#122 (comment)), It may fail to restart job scheduler, but just launch one more job scheduler.

we using systemd to manage these processes, except executor which need to be restarted when it's not working.
Please check example service unit files: https://github.com/cookpad/kuroko2/blob/master/docs/admin_guide.md#daemons

@joefrey25
Copy link

joefrey25 commented Sep 6, 2018 •
edited
Loading

@riseshia san

Thanks for the response,

Base on your comment (#122 (comment)), It may fail to restart job scheduler, but just launch one more job scheduler.

Yes we just launch 1 job in scheduler and it will only run everyday at 3am

we using systemd to manage these processes, except executor which need to be restarted when it's not working.
Please check example service unit files: https://github.com/cookpad/kuroko2/blob/master/docs/admin_guide.md#daemons

Yes we used this systemd file in our server and we also do have a cronjob to restart the services

BTW. may we know the versions of rails, ruby and web server(what you used) you guys running for this?
Thanks!

@joefrey25
Copy link

Hi @riseshia san

I just want to confirm this if this is another issue in kuroko2. I'm currently downgraded my rails version.
current version of rails 5.1.6
current version of ruby 2.5.1

new_job_definition_ _kuroko_2

As I can see there's a new table kuroko2_script_revisions and there's no documentation on how to do this script revisions.

Sorry for too much question.

TIA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@eagletmt @riseshia @joefrey25 @algiecaballesbbo