Live Performance Better than Backtests

[craigmediaservices]

Hi All,

I've only been running live since October - but noticed my live performance so far is better than the pickled back-test and backup YAML. This is an extremely short time period, so not really enough data.

The obvious reasons seem possibly costs, different instrument selection from the back-tested optimizer versus reality, and mostly accidents of realized price (e.g. mid-day versus next close?)

I wondered if rather and most likely it's a combination of all the above, and that the back-testing software tends to be rather "conservative" and contrary to most isn't really designed to optimize the displaying of fancy account curves.

I'm not complaining of course, but would like to understand when things look quite different from what's expected.

Does anyone else have a similar experience or a primary point of analysis to investigate?

The backtest:

2024-10-31    -5.109268
2024-11-30     2.142187
2024-12-31    -5.077474
Freq: M, dtype: object

Realized:

2024-10-31    0.507%
2024-11-30    4.688%
2024-12-31    4.121%

For the whole period the gains were concentrated in FX:

==================
P&L by asset class
==================

    codes  pandl
0  Metals  -1.93
1  Equity  -1.17
2  OilGas  -0.71
3    Bond  -0.66
4     Vol  -0.25
5  Sector   2.48
6     Ags   2.55
7      FX   9.90

Thank you

[craigmediaservices]

optimised_positions from the backtester are rather different from what I actually held

| Instrument       | Backtester Position | Aggregated Contract Position | Difference (Agg – BT) | Comment                                          |
|------------------|---------------------:|-----------------------------:|----------------------:|--------------------------------------------------|
| AUD              |                  -1  |                        -5    |                   -4  |                                                  |
| BITCOIN          |                   1  |                         2    |                    1  |                                                  |
| BRE              |                  -1  |                        -1    |                    0  |                                                  |
| BTP              |                   1  |                         1    |                    0  |                                                  |
| CHF              |                  -1  |                        -6    |                   -5  |                                                  |
| CORN             |                  -2  |                        -2    |                    0  |                                                  |
| DAX              |                   1  |                         2    |                    1  |                                                  |
| DOW              |                  -1  |                        -3    |                   -2  |                                                  |
| FTSETAIWAN       |                   1  |                         1    |                    0  |                                                  |
| GAS-PEN          |                  -1  |                        -1    |                    0  |                                                  |
| JPY              |                  -1  |                        -3    |                   -2  |                                                  |
| LEANHOG          |                   1  |                         1    |                    0  |                                                  |
| LIVECOW          |                   1  |                         1    |                    0  |                                                  |
| MSCISING         |                   2  |                         1    |                   -1  |                                                  |
| MXP              |                  -1  |                        -1    |                    0  |                                                  |
| REDWHEAT         |                  -1  |                        -2    |                   -1  |                                                  |
| SOYMEAL          |                  -1  |                        -1    |                    0  |                                                  |
| SOYOIL           |                  -1  |                        -1    |                    0  |                                                  |
|------------------|---------------------:|-----------------------------:|----------------------:|--------------------------------------------------|
| **Aggregated Only** |                  –  |                         –    |                      –  | These instruments appear only in the aggregate:  |
| BUND             |                  –   |                        -1    |                      –  |                                                  |
| EURO600          |                  –   |                         4    |                      –  |                                                  |
| PLAT             |                  –   |                        -1    |                      –  |                                                  |
| SEK              |                  –   |                        -1    |                      –  |                                                  |
| US-DISCRETE      |                  –   |                         1    |                      –  |                                                  |
|------------------|---------------------:|-----------------------------:|----------------------:|--------------------------------------------------|
| **Backtester Only**  |         NIKKEI, NIKKEI400, V2X, FEEDCOW, TOPIX, NZD, VIX, NASDAQ_micro, EU-BANKS, SOYBEAN_mini, US10, SP500_micro, EUR_micro, US-FINANCE, COPPER-micro, KR10, GOLD_micro, KOSDAQ, CRUDE_W, RUBBER |                         –    |                      –  | These instruments appear only in the backtester report |

[kylemarino22]

This is a bit of a strange result. I'd expect there to be some difference, but the last month has a discrepancy of 9% which seems significant. It seems to me that the backtesting software is conservative when it comes to costs and by delaying fills by a day, but I'm not sure if that's enough to explain it.

I'm also thinking maybe it's like a path dependence difference? Maybe the position your backtest had on the day you started live trading leads it on a less profitable trajectory than just starting from a zero position? Even still I'm not sure that would cause the difference. Maybe you could set up an experiment where you reset the position on that day.

Another thought I had was to reference some CTA firms that do a similar trend following strategy on futures markets as @vishalg mentioned on an earlier post of mine.

I'd also like to add that my own paper trading from around October has not been profitable. It seems to have caught a few swings, but has bled a lot on losing positions. I'm running a mostly unmodified version of rob's provided setup, at around 100k to get a conservative feel on how this system performs. I see you've been active here for a while, and I'm curious what changes you've made to achieve this performance? I'm currently suspicious of price spikes since I haven't really looked into how they're being handled too much. I saw in an old thread you set the threshold to 20 to get rid of false positives. Is that what you're currently doing?

[vishalg]

There is a part dependence as the optimisation tries to reduce churn but the positions should converge over a long period of time. You could run the backtest with different start dates to compare.