Backtest diagnosis

[emretezel]

Hi Everyone,

I ran my first backtest trial using dynamic optimised system, and something is going terribly wrong. Entire capital goes to zero quickly, please see the jpeg image.

I wanted to ask how one should go about diagnosing where the problem originates from. A particular instrument, particular rule, data issue, config issue?

I am attaching my config file, estimated weights as well as code showing how I ran the backtest in Jupyter. Any pointers greatly appreciated.

from systems.provided.rob_system.run_system import futures_system
from systems.diagoutput import systemDiag

system = futures_system(config_filename="private.system.config.yaml")
capital = system.accounts.get_actual_capital()
capital.plot(figsize=(15,7))
diag = systemDiag(system)
diag.yaml_config_with_estimated_parameters('config_out.yaml')
config = system.config
config.save("config.out.yaml")

config.out.txt
config_out.txt

[dr1ver1]

Take a look at the Backtesting doc - in particular the section that describes the objects for looking at results.
You can look at results for an individual instrument, or an instrument and Trading Rule:

    print(system.accounts.pandl_for_instrument("BUND").percent.stats()) ## produce % statistics 
    print(system.accounts.pandl_for_instrument_forecast("BUND", "carry").sharpe()) ## Sharpe for a specific trading rule variation

...you can display a few charts at once, for instance:

    instr = "BUND"
    fig, axes = plt.subplots(nrows=3, ncols=1)
    i=0
    system.accounts.pandl_for_instrument(instr).curve().plot(ax=axes[i])
    
    i=i+1
    system.data.get_raw_price(instr).plot(ax=axes[i])
    
    i=i+1
    system.forecastScaleCap.get_capped_forecast(instr,"carry").plot(ax=axes[i])

    fig.show()

I've cut and paste this from my own code, so you'll need to tweak it to work for you.

[dr1ver1]

I also wrote a function that dumps a whole of the results columns into csv which can then be analysed/ charted in Excel. I tried to write it in a way to easily add extra result columns... there's probably a better way of doing this - I'm not a Python expert:

import pandas as pd
import os

def createCSVs(instr):

    startDate = '2016-01-01'
    
    # Main: 
    results = system.positionSize.get_underlying_price(instr).loc[startDate:].to_frame()

    i=1
    
    results = pd.merge(results, system.positionSize.get_instrument_currency_vol(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='currencyVol'

    i+=1
    results = pd.merge(results, system.positionSize.get_instrument_value_vol(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='instrVol'

    i+=1
    results = pd.merge(results, system.positionSize.get_volatility_scalar(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='volScalar'

    i+=1
    results = pd.merge(results, system.positionSize.get_subsystem_position(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='subsysPos'

    i+=1
    results = pd.merge(results, system.accounts.get_notional_position(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='notionalPos'

    i+=1
    results = pd.merge(results, system.accounts.get_actual_position(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='actualPos'

    i+=1
    results = pd.merge(results, system.rawdata.daily_annualised_roll(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='dlyAnnRoll'

    i+=1
    results = pd.merge(results, system.positionSize.get_block_value(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='block'
    
    i+=1
    results = pd.merge(results, system.rawdata.raw_carry(instr).loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='rawCarry'

    i+=1
    results = pd.merge(results, system.positionSize.get_fx_rate(instr).loc[startDate:].resample('1B').last().to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='FXrate'
    
    i+=1
    results = pd.merge(results, system.rawdata.raw_futures_roll(instr).loc[startDate:].resample('1B').mean().to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='rawFuturesRoll'

    i+=1
    results = pd.merge(results, system.rawdata.annualised_roll(instr).loc[startDate:].resample('1B').last().to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='annRoll'
   
    # Carry
    i+=1
    results = pd.merge(results, system.forecastScaleCap.get_raw_forecast(instr, "carry").loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_RawFC'

    i+=1
    results = pd.merge(results, system.forecastScaleCap.get_forecast_scalar(instr, "carry").loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_FCScalar'

    i+=1
    results = pd.merge(results, system.forecastScaleCap.get_scaled_forecast(instr, "carry").loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_scaledFC'

    i+=1
    results = pd.merge(results, system.forecastScaleCap.get_capped_forecast(instr, "carry").loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_cappedFC'
    
    i+=1
    results = pd.merge(results, system.accounts.pandl_for_instrument_forecast(instr,"carry").loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_PLinstrFC'

    i+=1
    results = pd.merge(results, system.accounts.pandl_for_instrument_forecast(instr,"carry").percent.loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_PLinstrFCpct'

    i+=1
    results = pd.merge(results, 
        system.accounts.pandl_for_instrument_forecast(instr,"carry").percent.curve().loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_Curve_pct'
        
    i+=1
    results = pd.merge(results, 
        system.accounts.pandl_for_instrument_forecast(instr,"carry").curve().loc[startDate:].to_frame(), 
        left_index=True, right_index=True, how="left")
    results.columns.values[i]='Carry_Curve'


    results = pd.merge(results, system.rawdata.get_instrument_raw_carry_data(instr).loc[startDate:].resample('1B').last(), 
        left_index=True, right_index=True, how="left")    # 4 cols (+ index)

    results.columns.values[0]='price1'


    resultsFile = instr + '_results.csv'
    resultsPath = # ...your folder for results
    fullPath = os.path.join(resultsPath, resultsFile)

    results.to_csv(fullPath, date_format='%d/%m/%Y')

Again, I've cut & paste this from my own code - you'll need to tweak it to get it to work for you.

regards,
David

[emretezel]

Hi David,

Thanks a lot, I went through P&L contribution of each instrument, and the biggest issues arise from Ethanol and Euribor. On the cost reports, Rob also points out these two having massive SR costs. I thought I had excluded them in the backtest but apparently not. Now I am running without them. I used the following code to diagnose.

for instrument in system.get_instrument_list():
    plt.figure()
    inst_pl = system.accounts.pandl_for_instrument(instrument)
    inst_pl.curve().plot(title=instrument)

[meldinman]

I discussed a similar issue a few months ago. This discussion may shed some light.

[emretezel]

Thank you, I was using portfolio() method of accounts to see the P&L, let me try system.accounts.optimised_portfolio() and calculate sharp based on that.