Realized vs. Backtested Performance in 2022

[73ze]

Hi Everyone,

If my goal is to backtest rob_system out of the box (with a fresh git clone of pysystemtrade, using the supplied CSV files, before any code or config change), is the following the expected way to do it, please?

    from sysdata.sim.csv_futures_sim_data import csvFuturesSimData
    from systems.provided.rob_system.run_system import futures_system

    data = csvFuturesSimData()
    system = futures_system(sim_data=data)

    system.config.notional_trading_capital = 1e6

    portfolio = system.accounts.optimised_portfolio()
    stats = portfolio.percent.stats()
    print(stats)
    portfolio.percent.curve()

It shows the following great stats:

  ('min', '-9.201'),
  ('max', '8.188'),
  ('median', '0.08106'),
  ('mean', '0.09956'),
  ('std', '1.131'),
  ('skew', '-0.06844'),
  ('ann_mean', '25.49'),
  ('ann_std', '18.1'),
  ('sharpe', '1.409'),
  ('sortino', '2.014'),
  ('avg_drawdown', '-7.155'),
  ('time_in_drawdown', '0.8995'),
  ('calmar', '0.8145'),
  ('avg_return_to_drawdown', '3.562'),
  ('avg_loss', '-0.7797'),
  ('avg_gain', '0.8497'),
  ('gaintolossratio', '1.09'),
  ('profitfactor', '1.278'),
  ('hitrate', '0.5398'),
  ('t_stat', '10.35'),
  ('p_value', '5.274e-25')],

The reason for asking is that the account curve generated this way shows negative cumulative returns in 2022 while Rob's realized returns were great in the same period. In particular the account curve does not seem to mimic the very large returns that Rob had in Feb/Mar.

So, I wondered whether the results out of the box should be expected to be closer (am I doing something wrong?) or if this is as expected?

Thank you!

[robcarver17]

The two most obvious differences between what I run and this are the capital (100k, rather than £400k or about $500k) which will reduce performance overall. Also, the .csv files don't cover all the markets I cover. Both of these effects will lead to a lack of diversification which may manifest in large tracking error for some periods. Does your backtest show big long positions in energy markets in Feb/March?

[meldinman]

In relation to this question, have you changed the setting your yaml file to reflect new instrument and forecast weights for any of the added instruments over the year?
How often would you go about re-estimating instrument and forecast weights etc?

[robcarver17]

I update instrument weights when I add new instruments. Haven't done it for a while, will add a load in April and probably won't do anymore unless / until some compelling new markets appear. I don't reestimate forecast weights. I just use manually handcrafted weights for both.

From around 2014 to 2020 I didn't really update my parameters at all.

[73ze]

Thank you so much for taking the time to answer my question.

To answer your question on energy positions as of Feb/March 2022, I exported system.optimisedPositions.get_optimised_weights_df as a CSV and attached it here. Correct me if that is not the intended way.

It is too large to display here for all periods but below are the positions for the last day of February. There are long positions in BRENT-LAST, CRUDE_W_mini, GASOILINE, HEATOIL; out of total of 31 positions (account size of 1 million).

	AEX	ALUMINIUM	AUD	BBCOMM	BITCOIN	BOBL	BONO	BOVESPA	BRE	BRENT-LAST	BTP	BTP3	BUND	BUTTER	BUXL	CAC	CAD	CH10	CHEESE	CHF	CNH	COPPER	CORN	COTTON	CRUDE_W_mini	CZK	DAX	DJSTX-SMALL	DOW	EDOLLAR	ETHANOL	ETHEREUM	EU-AUTO	EU-BANKS	EU-BASIC	EU-CHEM	EU-CONSTRUCTION	EU-DIV30	EU-DJ-TELECOM	EU-FOOD	EU-HEALTH	EU-HOUSE	EU-INSURE	EU-MEDIA	EU-MID	EU-OIL	EU-REALESTATE	EU-RETAIL	EU-TECH	EU-TRAVEL	EU-UTILS	EUR	EURCHF	EURO600	EUROSTX	EUROSTX-LARGE	EUROSTX-SMALL	EUROSTX200-LARGE	FEEDCOW	FTSECHINAA	FTSECHINAH	FTSEINDO	FTSETAIWAN	GAS-LAST	GASOILINE	GAS_US_mini	GBP	GBPEUR	GOLD_micro	HEATOIL	INR	IRON	IRS	JGB	JP-REALESTATE	JPY	KOSDAQ	KOSPI	KR10	KR3	KRWUSD	LEANHOG	LIVECOW	LUMBER	MILK	MILKDRY	MILKWET	MSCIASIA	MSCISING	MUMMY	MXP	NASDAQ_micro	NIFTY	NIKKEI	NIKKEI400	NOK	NZD	OAT	OATIES	OMX	PALLAD	PLAT	R1000	REDWHEAT	RICE	RUR	RUSSELL	SEK	SGD	SHATZ	SILVER	SMI	SMI-MID	SOYBEAN	SOYMEAL	SOYOIL	SP400	SP500_micro	TOPIX	US-DISCRETE	US-ENERGY	US-FINANCE	US-HEALTH	US-INDUSTRY	US-MATERIAL	US-PROPERTY	US-REALESTATE	US-STAPLES	US-TECH	US-UTILS	US10	US10U	US2	US20	US3	US30	US5	USIRS10	USIRS5	USIRS5ERIS	V2X	VIX	VNKI	WHEAT	YENEUR
2022-02-28	0	0	0	0.02291	0	0	0	0	0	0.0951	0	0.2537984394	0	0	0	0	0	0	0	0	-0.2005151607	0.11135	0.060725	0	0.0935	0	0	0	0	0	0	0	0	0	0.03686891205	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0.02513022825	0	0	0	0.0255237665	0	0	0	0	0	0	0.1200948	0	0	-0.2806857664	0.076028	0.1166634	0.066075	0.02908	0	-1.303145808	0	0	-0.01032146775	0	-0.1002149775	0.180131985	0	0.04537	-0.05508	0	0	0	0	0	0	0	0.024315	-0.028456	0	0	0	0	0	0	0	0	0	0	0.07959	0	0	0	-0.0102225	0	0	0.251425966	0	0	0	0.0717875	0.04425	0.042474	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	-0.02755	0	0	0

Anyone should be able to reproduce this output using a fresh git clone of pysystemtrade. I posted the code used to do it for reference in my post above and correct me if that is not the intended way. Thanks.

[robcarver17]

Ok that is what I would expect. It's very weird that you get negative returns in 2022 then, and in Feb and March when being long energy was a one way ticket to riches. I'm not sure what to suggest - could be a subtle bug in the accounting calculation.... are the negs due to costs or do you also seem them in gross returns?

[73ze]

It also show very slightly negative gross returns for 2022 (judged by system.accounts.optimised_portfolio().gross.percent.curve()).

[meldinman]

FWIW, heres the sharpe and equity curve when I run the futures system provided in Rob_system.run_system

from systems.provided.rob_system.run_system import futures_system
system=futures_system()
acc_curve_optimised = system.accounts.optimised_portfolio()
acc_curve_optimised.curve().tail(1000).plot()

In [11]:  acc_curve_optimised.sharpe()
Out [11]:  1.3473110053230832

[73ze]

Hi @meldinman, thank you so much for taking the time.

The curve I get looks pretty much the same shape as yours as you can see below (in percent terms) for the period 2019-05-01 to 2022-12-30 (the code used to generated it and the output as a CSV are provided in my first post):

If you focus on 2022 alone (from 2022-01-03 to 2022-12-30), it shows a slighly negative return:

Compared to Rob's 2022 positive realized returns:

[meldinman]

Looks good, I think some of the difference is in the capital size as well as Rob using GBP as his base capital. There was a lot of currency volatility last year that may have contributed to the difference there.

[meldinman]

interestingly if I change the config to use_instrument_weights_estimates = True 2022 looks a bit worse.

new_system = futures_system()
new_system.config.use_instrument_weight_estimates= True
new_acc_curve_optimised = new_system.accounts.optimised_portfolio()
new_acc_curve_optimised.percent.curve().tail(1000).plot()

new_acc_curve_optimised.sharpe()
1.306415918265575

[73ze]

Interesting, thanks for sharing this idea and for suggesting to look at the impact of the base currency.

To be clear, all my comments from the start of this discussion are for a capital of 1 million (system.config.notional_trading_capital = 1e6). DO would not propose as many as 30+ different instruments with only 100k of capital (see the positions CSV file attached earlier).

Regarding the impact of the base currency, here are % returns in 2022 with system.config.base_currency = 'USD':

and for comparison here are % returns in 2022 with system.config.base_currency = 'GBP':

There is indeed some material impact of the base currency as during the second half of the year the returns seem to decline more slowly with GBP than with USD. However, the two curves are very similar for 2022, both results in closer to flat returns for the year, slightly negative with USD, slightly positive with GBP, both much lower than Rob's realized returns in the same period:

To be clear, the backtested returns look fantastic over the entire history with either base currencies. Differences between backtested and realized are expected but given these seem somewhat important for 2022 (do they?), this seems an opportunity to look into it and understand better what we should expect with the system or find if we (or I?) might have a bug or misconfiguration perhaps.

Note: in case you wonder why the cumulative % return over the entire history is in the 1400% for USD vs. only 650% for GPB, that seems to be because with USD as the base currency, the backtesting starts doing investments ~1970 while with GBP investments don't start until 1991-03-06, which is the day after the file GBPUSD.csv has its first rate.

[robcarver17]

There might be some more colour on this when I do my april blog post on performance which will include live vs realised.

[73ze]

Exciting! Thanks for the heads up.

I wanted to ask if there was a way in PST, better intended than what I show above, to plot backtested vs. realized cumulative performance in a comparable way, please? (the simple code I use is in my first post)

In my understanding:

• The "Account curve report" daily email uses functions that seemed more designed to take "data" as input, which represents actually realized facts from mongodb and so they need to account for possible real world irregularities in sample dates, need to account for cash being actually injected or withdrawn from the account, etc...
• In contrast, the backtesting benchmarking infrastructure is based on the "system" in-memory representation, which is more idealized by design and perhaps does not need as much normalization and also controls the account multiplier by config rather than data. The backtesting guidelines emphasize more the use of portfolio.curve() or portfolio.percent.curve(), which is what I use.

I am looking at the details of both to understand if I need additional transformations to compare the curves, and it is interesting to look at the implementation details, but I thought it might be interesting for everyone to ask.

[robcarver17]

I wanted to ask if there was a way in PST, better intended than what I show above, to plot backtested vs. realized cumulative performance in a comparable way, please? (the simple code I use is in my first post)

In my understanding:

The "Account curve report" daily email uses functions that seemed more designed to take "data" as input, which represents actually realized facts from mongodb and so they need to account for possible real world irregularities in sample dates, need to account for cash being actually injected or withdrawn from the account, etc...
In contrast, the backtesting benchmarking infrastructure is based on the "system" in-memory representation, which is more idealized by design and perhaps does not need as much normalization and also controls the account multiplier by config rather than data. The backtesting guidelines emphasize more the use of portfolio.curve() or portfolio.percent.curve(), which is what I use.

You're overthinking it. They're just a series of daily returns

In fact if you do this:

>>> from sysproduction.data.capital import *
>>> d = dataCapital()
>>> d.get_percentage_returns_as_account_curve()
index
2014-04-09    0.000000
2014-04-10    0.006469
2014-04-11   -0.009541
2014-04-14   -0.003608
2014-04-15    0.011314
                ...   
2023-02-17    0.000000
2023-02-20    0.000000
2023-02-21    0.001858
2023-02-22   -0.006621
2023-02-23    0.001478
Freq: B, Length: 2317, dtype: float64
 Unweighted account curve; use object.stats() to see methods

(Your numbers will be different, this is from my system)

You get an object of type account curve with daily percentage returns, which is exactly the same as you get with futures_system.accounts.portfolio().percent

[meldinman]

If I try that I get an empty oject

>>> from sysproduction.data.capital import *
>>> d = dataCapital()
>>> data = d.get_percentage_returns_as_account_curve()
>>> data
accountCurve([], Freq: B, dtype: float64)
  Unweighted account curve; use object.stats() to see methods

Note that if I use the interactive_diagnostics menu I get the correct account curve.
I think it might have something to do with the totalCapitalCalculationData from new_capital.py. Seems to be some not implemented functions in there that aren't returning the proper values? Still trying to figure that part out.

[73ze]

So I take it that they are already directly comparable. Thank you!

[73ze]

Hi Rob,

May I ask if in your own backtesting you get the expected high returns for 2022?

[craigmediaservices]

Not to dredge up the dead, but I'm curious if this was ever sorted?

It looks like in the performance reports from April 2023, Rob did actually experience a loss as the simulation showed.

So it looks like this was a bug of some sort in the live reporting at the time?

[robcarver17]

I won't know until April

[BritvaBo]

nice, thanks for material