IB delayed market data, perhaps useful for ICE?

[VMatthijs]

Given how expensive real time data subscriptions are for ICE while their products include pretty interesting liquid commodities such as COFFEE, COCOA, SUGAR11, COTTON2, GASOIL, etc., I am wondering how people approach trading ICE markets.

1. Do you just not trade them? (@robcarver17 seems to be in this camp, right?)
2. Do you just suck it up and pay the high fees?
3. Or is it good enough to use delayed data, which IB seems to provide for free (https://www.interactivebrokers.com/en/pricing/research-news-marketdata.php, https://tradersacademy.online/trading-lesson/receiving-market-data-and-historical-candlesticks, https://ibkr.info/ru/node/2966)? If not, then why not? Shouldn't data that is delayed by 15min be fine for slow trading strategies such as the ones that most of us use?

Has anyone tried option 3.?

[flipyou]

I'm firmly in the number 3 camp. 15 minute delayed is fine as the systems I currently trade live are based on daily bars, so I just use python to download the daily price bars more than 15 minutes after market close and calculate my setups and orders before the market opens.

[VMatthijs]

Very cool! Do you have any advice on how to do those downloads / any scripts you would be willing to share, @flipyou ? To be more concrete in my questions: do you just download this data from IB using their API using something like self.ib.reqMarketDataType(3) to get the delayed rather than real-time data? Did you modify update_historical_prices to do this daily? Does anything need to be changed about how orders are placed seeing that you do not have real time data?
Personally, I am inclined to just add a bunch of self.ib.reqMarketDataType(3)'s in ib_price_client.py to always retrieve delayed data (or first try real-time, and resort to delayed if that's not available), but I'm wondering whether that approach would be too naive and would break things. Thanks for the quick response!

[js190]

BTW there is an interesting discussion https://www.elitetrader.com/et/threads/fully-automated-futures-trading.289589/page-341 regarding using OrderType = "SNAP MID" for executing using delayed data. Please keep us informed of how you get on as this may be something I would adopt.

[tgibson11]

This doesn't seem clear in the OP, but with IB you have to distinguish between historical and streaming data. Streaming data can be either live or delayed. Delayed streaming data is free. Live streaming data can be expensive, depending on exchange and professional/non-professional classification.

Historical data is different: it is not available AT ALL unless you have a subscription to live data first. There is such a thing as delayed historical data, but I don't see what the point is, because you can't access it unless you also have access to "live" historical data. (I've tried.)

2. pysystemtrade out-of-the-box uses both kinds of data. Historical data is used to collect daily and hourly price data (but only once a day), and also to sample spread data periodically during the day. Streaming data is used to manage trades.

Sampling of spread data is easy to turn off via config, and only necessary if you don't want to rely on the pre-configured cost data. Delayed streaming data could also be used here, with some code changes.

Market orders or other IB order types can be used to get around not having live streaming data. Delayed data wouldn't be very useful for managing orders. Code changes are required to use market orders without live data though.

Historical data doesn't need to come from IB at all. There are several vendors that provide EOD data in the range of $200-500/year. Getting that data into pysystemtrade will of course be up to you.

[VMatthijs]

Thanks, @tgibson11 ! Very good point about the live data being needed for the current order execution strategy. So, you'd need some alternative like SNAP MID. I have managed to download lots of delayed historical data from IB though without market subscriptions for ICE, by using self.ib.reqMarketDataType(3).

[flipyou]

Very cool! Do you have any advice on how to do those downloads / any scripts you would be willing to share, @flipyou ? To be more concrete in my questions: do you just download this data from IB using their API using something like self.ib.reqMarketDataType(3) to get the delayed rather than real-time data? Did you modify update_historical_prices to do this daily? Does anything need to be changed about how orders are placed seeing that you do not have real time data? Personally, I am inclined to just add a bunch of self.ib.reqMarketDataType(3)'s in ib_price_client.py to always retrieve delayed data (or first try real-time, and resort to delayed if that's not available), but I'm wondering whether that approach would be too naive and would break things. Thanks for the quick response!

I use the the ib_insync package from https://github.com/erdewit/ib_insync
My use-case is a little messy because I use a blend of daily data I download from IBKR and my Norgate data subscription - I try to find the contract that matches the Norgate data continuous back adjusted price data by matching highs and lows (Norgate doesn't update their previous day's data until a few hours after most of the markets I track open, and if I want to have stop orders in the market before open I have to add the last day's prices that I grab using ib_insync.)

I build a list of instruments like this (Norgate and IBKR have different tickers for the same instrument and the factor is needed to make the prices match):

instruments = [
    {
        "IBticker": "EUR", "exchange": "GLOBEX", "NDticker": "6E", "factor": 1
    }, {
        "IBticker": "JPY", "exchange": "GLOBEX", "NDticker": "6J", "factor": 0.01
    }, {
        "IBticker": "GBP", "exchange": "GLOBEX", "NDticker": "6B", "factor": 1
    }, {
        "IBticker": "ZN", "exchange": "ECBOT", "NDticker": "ZN", "factor": 1
    },

Then I cycle through each instrument, to grab the data like this:

for instrument in instruments:
    print( instrument['IBticker'], instrument['exchange'] )
    if(instrument['exchange'] == 'NYBOT'):
        ib.reqMarketDataType( 3 ) # no subscription, so use delayed data
    else: 
        ib.reqMarketDataType( 1 )

    ibdf = get_df_IB(instrument['IBticker'], instrument['exchange'])
    nddf = get_df_ND(instrument['NDticker'], instrument['exchange'])

I have defined functions to actually grab the dataframes:

from ib_insync import IB, Future, ContFuture, util
import norgatedata
import pandas as pd
from datetime import datetime, timedelta
from pandas._testing import assert_frame_equal

def get_df_IB(ticker, exchange):
    contract = ContFuture(ticker, exchange)
    
    bars = ib.reqHistoricalData(
        contract, endDateTime='', durationStr='2 W',
        barSizeSetting='1 day', whatToShow='TRADES', useRTH=False)

    # convert to pandas dataframe:
    df = util.df(bars)
    df['source'] = "IBcont"
    df.set_index('date', inplace=True)
    df.index = pd.to_datetime(df.index)
    df = df.drop([ 'volume', 'average', 'barCount', ], axis = 1)
    return df