Checking raw performance on optimised system

[meldinman]

Hello,
I am currently working through the step to implement pysystemtrade. I have been running some backtests after creating all the necessary mongo and arctic databases. While the most of the included systems seem to run in line with standard results, I am seeing something I did not expect when running the included rob_system / dynamic_small_system_optimise.
When running the dynamic optimisation system I get a neat looking curve, nicely in line with what rob was getting in his blogs.
from systems.provided.rob_system.run_system import futures_system
system=futures_system()

However when getting the portfolio object for the system...

Digging deeper to find what was causing this, it seems that ETHANOL is the main culprit

with all the rules deemed cheap enough to trade looking like this... (I wont bore you with all of them)

price looks to be ok...

notional and buffered positions seem to be ok... you can see the optimized never has a position

and same with position

It seems like the killer is SR costs...

I have tried this on both my dbFuturesData as well as csvFuturesData with similar results.

So I guess my question becomes

1. I havent seen this discussed on the blog or elitetrader forum but is this result in line with what other are seeing? Should system.accounts.portfolio() no longer be used in such case? When running a dynamic optimisation should we only be accessing the system.accounts.optimised_portfolio() object?
2. Are there other methods I am missing to help diagnose what is going on here or in future cases where an account curve seems to be off? I am trying to diagnose why all the rules seem to be broken for ethanol (I obviously wouldn't worry about a losing rule / instrument but this one seems to be due to something else).
   Appreciate any insight and thanks again for making this incredible project public!

[robcarver17]

" When running a dynamic optimisation should we only be accessing the system.accounts.optimised_portfolio() object?"

Yes.

Ethanol is f***** expensive

system.accounts.get_SR_cost_per_trade_for_instrument("ETHANOL") gives me 7.0

And it's even higher here

Basically if you use dynamic optimisation Ethanol is in the set of bad markets (explained here) so no position is ever taken in it even though we calculate a forecast which informs our position in other markets. But the non dynamic optimisation takes a position in it, and hence it costs you a fortune. If you weren't doing non dynamic optimisation you'd drop Ethanol entirely - this isn't 'in sample fitting', its fine to drop expensive instruments.

Your diagnostic skills are excellent.

[meldinman]

thanks for the explanation. I figured that was the reason once I started looking over the cost report
That complement means a lot coming from you. If only my python writing skills where on par :)