Task Scheduling

[vishalg]

I am trying to get a sensible task scheduling setup, but the current setup seems to be non-intuitive. I am in the UK timezone and struggling to get a schedule which would work correctly and finish early enough in the day so I can catch and correct any issues before going to bed.

1. run_daily_price_updates: I use the download_by_zone functionality with ASIA at 08:20, EMEA 18:20 and US at 20:20. This is for couple of reasons - I want the setup to work when the clocks are an hour off during the day light saving changeover (which happens at different times for US and Europe), secondly 20 mins past the hour instead of on the hour to allow for delayed feeds.
2. The crontab in the project folder has run_daily_update_multiple_adjusted_prices at 23:00 while run_systems kicks off at 20:30. If my understanding of the code is correct that sequencing does not seem correct - even with the process_configuration_previous_process dependency in the private_control_config run_systems will be kicked off before multiple adjusted prices have been updated for the day.

It seems that both the cron timing and the private_control_config need to be carefully synchronised - it would seem better to use a single scheduler which takes care of all PST processes and either runs all the time or is kicked off once a day via cron. If someone has a working solution or suggestions for an open source project to look into it would be very helpful.

[bug-or-feature]

1. Sounds fine
2. yes the default crontab is misleading for updating multiple and adjusted prices. Ideally that should start as soon as daily prices has finished. You can configure that to happen by setting it to start at say 20:20, and setting run_daily_update_multiple_adjusted_prices to wait for run_daily_prices_updates under process_configuration_previous_process

I would think of the scheduling stuff as a loose guide rather than anything else.

In terms of your general problem, why not just start the stack handler mid morning UK time? Or start it manually? That way you can resolve any issues first thing in the morning, instead of late in the evening.

I have a vague idea to set up my own instance of PST to be scheduled by Apache Airflow, but of course, have not got around to doing anything about it yet

[vishalg]

That makes sense. I had the multiple adjusted prices update set to go off at 2100. Then, I moved run_systems to 2030 as I wanted it to kick off as soon as possible - but that breaks the logic because the code checks whether the previous process has run within the last 24 hours. The solution I ended up with was to kick of the cronjobs sequentially in the order we need them executed.
Longer term - task scheduling may be an area that could be improved by for instance by defining a full daily cycle rather than loose linkage via previous process definitions.

[tgibson11]

Just to add my 2 cents...

I have always thought the previous process logic should be "has the previous process run today?" (i.e., since midnight).

I suggested it to Rob once, but he didn't like it because he thinks in terms of a 24-hour cycle rather than a calendar day. I've modified my fork and have been happy using it that way for a long time.

Stepping back a bit, I think this is all best managed by external job scheduling software though.

[vishalg]

I had the same dilemma, but I think a way around may be to make "start time" of the daily processes configurable. That said external scheduler may be the way to go, cron is easy and flexible but it is only suitable for jobs which don't run for too long.