DO backtests getting killed

[rorymac]

Hello All

I recently updated to 0c1033e

and in running a DO backtest have repeatedly got this:

22-08-24 22:33:20 {'type': 'base_system', 'stage': 'combForecast'}  Using pooled gross returns
2022-08-24 22:33:20 {'type': 'base_system', 'stage': 'combForecast'}  Using pooled turnover cost SR for CORN
2022-08-24 22:33:20 {'type': 'base_system', 'stage': 'combForecast'}  Applying cost multiplier of 2.000000
 [=------------------------------------------------------------------------------] 1.9% Optimising weights (0.0/0.0/0.0 secs left/elapsed/total)Killed

and no further calculations are done.

This is not always at the same instrument previously CAC , COPPER.
The config is attached as a text file.
config_yaml.txt

There are no error messages.
Prior to the update, this worked.

Any ideas?

[robcarver17]

The main recent change was that we don't calculate shrunk instrument correlation matricies, but instead shrink them when they are used for DO. This involves making a copy of the correlation matrix before shrinking it, so it will be increasing memory usage. It might be that this pushes you into memory issues. I'd advise trying with a smaller set of instruments to see if it will run.

If this turns out to be the problem then it might be worth thinking about some backtest memory optimisations.

[rorymac]

Thank you Rob.
Turns out that a reboot fixed the issue, but It took 3 hours to run.
So not sure if that is an issue for everyone, the computer is a fairly good one, with 6 cores and 15,5g of memory:
https://fit-iot.com/web/products/airtop3/

[oldlore]

I realize this is an old thread but I find myself running into it now that I've tried running backtests with many more instruments and estimating forecast/instrument weights. It always crashes in the first dozen optimizations. I have 32GB memory, intel i3-12100. If I run a much smaller list of instruments it'll usually finish running.

[tgibson11]

FWIW, I have had similar issues for a while. I used to run another application on my trading machine that used a lot of memory, so I just chalked it up to trying to do too much with an underpowered machine (8GB RAM), and I limited my system to around 40 instruments.

I also found that saving (pickling) the backtest basically doubles the amount of RAM required. I've disabled this, and instead have modified the code to immediately produce a strategy report using the system object that is already in memory. Of course, that means I don't have any saved backtests, so this may or may not work for others.

I have recently moved that memory-hogging application to a different machine. Just to give you a rough ballpark, with my 8GB RAM (and not saving the backtest), I'm currently running my system with about 85 instruments. I know that it definitely runs out of memory with 145 instruments, but I haven't tried to see how far I can push it between those numbers.

[oldlore]

The backtest was with 199 instruments and my machine can't handle it. My live system has 140-ish or so and runs fine. The memory requirements for the daily run_system method seems lighter too since it's not estimating weights.

[bug-or-feature]

Wouldn't it be better to do the estimation and optimisation separately? Do the estimation with a static system, then copy the config values into your DO setup

[oldlore]

This is probably ignorance on my part but when I run a backtest using just the base system with all instruments I still crash when estimating correlations.

[tgibson11]

I use static forecast & instrument weights, etc. But toward the end of the backtest, the are 3 or 4 progress bars about estimating correlations. Probably related to DO. That is where it crashes for me. (Sorry, not at my desk right now.)

This may or may not be the same place @oldlore is seeing the crash.

If I recall correctly, even Rob, with beefier machines than both of us, said he can't estimate forecast weight for all instruments at once.

[oldlore]

Maybe I'm doing this wrong. If so, perhaps someone can help. My current DO system has ~140 instruments with fixed FWs and IWs. I've been working on adding a bunch (~40) new instruments to DO and updating to the new autogroup methodology for calculating FW/IWs. However, I need new FDMs for the instruments. But whenever I run a backtest to estimate these (using the autogroup for FW/IWs) it gives me a bunch of FDMs that are almost all the same. I'm not sure if this is a mistake in how I'm running the backtest or a consequence of using the new FW methodology? So I tried estimating everything to see if the results would change but this gets us back to the crashing problem above.

forecast_div_multiplier:
  AEX: 2.239883445600876
  ALUMINIUM: 2.2398918765331546
  AUDJPY: 2.2356324679130366
  AUD_micro: 2.239883445600876
  BB3M: 2.2308961043430595
  BBCOMM: 2.239883445600876
  BITCOIN: 2.239883444957188
  BOBL: 2.239883445600876
  BONO: 2.239883445600581
  BOVESPA: 2.239878522461506
  BRE: 2.239883445600876
  BRENT-LAST: 2.239845372171262
  BRENT_W: 2.239883445600876
  BTP: 2.239883445600876
  BTP3: 2.239883445600876
  BUND: 2.239883445600876
  BUTTER: 2.239883445600876
  BUXL: 2.239883445600775
  CAC: 2.239883445600876
  CAD: 2.239883445600876
  CAD10: 2.239883445600876
  CAD2: 2.2393491513080885
  CAD5: 2.239883445600876
  CANOLA: 2.239883445600876
  ...

For comparison, these are some of my current FDMs

forecast_div_multiplier:
  ALUMINIUM: 1.3153539368408498
  AUD: 1.8418190876094975
  AUDJPY: 1.2755618049280335
  AUD_micro: 1.7188958681064284
  BB3M: 1.5251014858150365
  BBCOMM: 1.2883264918331379
  BITCOIN: 1.7442966838647695
  BOBL: 1.7615324345523744
  BONO: 1.2883267050110614
  BOVESPA: 1.28832649183386
  BRE: 1.7081121510331856
  BRENT-LAST: 1.8258330578490292
  BTP: 1.7069889890651984
  BTP3: 1.5387123864270917
  BUND: 1.6881057198145146
  BUTTER: 1.1707393882674082
  BUXL: 1.706988959823698
  CAC: 1.7359175335897101

Thanks.

[meldinman]

I am also getting this issue when trying estimate forecast weights for new instruments. I have 128GB ram and dual Xeon CPUs and it doesn't seem to be hitting any limits of my system processes yet I still get errors:

Any suggestion how to either reduce the sizes of the arrays being optimized with or to maybe disable the cache when running a full backtest for re-optimizing new weights?