Trying to debug getting futures options chain (works sometimes, fails to load others)

[maqifrnswa]

When grabbing /ES option chains, sometimes the following code works, sometimes it doesn't. I think it's on the TWS side, but don't know how to address it (if possible)

symbol="ES" 
FOPexchange="CME"
underlyingExpire='20241220'


test=Future(symbol, exchange=FOPexchange, lastTradeDateOrContractMonth=underlyingExpire)
ib.qualifyContracts(test)
print(test)
chains = ib.reqSecDefOptParams(test.symbol, test.exchange, test.secType, test.conId)

I can't find the pattern when it does or doesn't work.

When it works, it returns the chain, and I can grab all the tickers.

When it does not work, I get the following error from python: Error 322, reqId 7: Error processing request. no derivatives returned which I believe is an error on the TWS side. The TWS API logs (logging level: Error) show

10:03:22:223 <- 78-7-ES-CME-FUT-495512557-
10:03:22:225 -> ---=4-2-7-322-Error processing request. no derivatives returned--

Is there some known bug in TWS, or is there some setting I'm missing?

I did come across this: https://groups.io/g/twsapi/topic/101318985
Maybe it's some corrupted cache/DB on the TWS side? Would using the IB Gateway be better than TWS?

[maqifrnswa]

update: I tried IB gateway and everything just worked. something wonky is happening in TWS.

[mattsta]

I've seen the exact same thing recently. The API just returned empty values for /ES chains and it refuses to qualify contracts for same-day /ES options even though they are tradable.

Things I've seen break the IBKR gateway API (without it actually crashing or any active error indication... just silent errors until manually restarted):

• too many client restarts connecting/disconnecting
• sometimes a single broken client connection on startup causes the gateway to never accept any new clients until it restarts (locking problem in the gateway)
• too much API data returned (like chains) (memory management problem in the gateway)
• some bugs across nightly refreshes/restarts randomly stops some tickers from receiving any new live quote price updates until the gateway restarts (lost data or locking problem in the gateway)

I attempted my lookups across multiple days, so I don't know if just restarting the gateway fixed it or if their backend APIs are down randomly. I saw the /ES chains problem fixed itself, but I'm still not sure about the 0DTE /ES FOP qualifications (I decided to just fetch all contracts 5+ days in advance and cache them so future lookups can't fail).

Also remember in the gateway app to uncheck "Expose entire trading schedule to API" under the Configuration -> API -> Settings parameters because it can cause chains to fail to load due to all the extra data returned when enabled (when using full details though).

Also remember to adjust your gateway memory far above the default limit. 4096 or 6144 MB are safer values than the defaults especially when fetching option chains with all the dates returned.

I also tried using reqSecDefOptParams() for some instruments, but it returns clearly invalid data for things like SPY and QQQ (it returns a union of all strikes across all future dates merged together, but weekly, monthly, and quarterly expiration strikes are different than daily strikes due to dividend/split adjustments over time, so the output is useless for many symbols. Instead of returning {day: [strikes-for-the-day]} it returns [all expiration dates], [all strikes merged across all expiration dates]. more IBKR weirdness.

[maqifrnswa]

Thanks! I set up the IB gateway, it worked fine - then I went back to TWS, and it started working again and worked all day. I probably can just cache them too. It started happening a week or two ago. Yeah, I noticed that it returns all strikes across all expirations too. I just ignore all those errors when it tries to fetch one. I followed your advice and unchecked "expose entire trading schedule." I already increased memory to 4096, but might as well bump it up some more to be safe.

Thanks again for the response

[maqifrnswa]

I've seen the exact same thing recently. The API just returned empty values for /ES chains and it refuses to qualify contracts for same-day /ES options even though they are tradable.

I think I found the culprit and solution... From #38 , one of the comments said

building an option chain for index (ES,MES) futures requires first to
qualify an Index
and then request ticker for FuturesOption

IBKR documentation says to reqSecDefOptParams on the Future, which is what I was doing. But the above says to qualify the Index. I gave that a try, and it works!

This fails sometimes, but works some other times:

test=Future(symbol, exchange="CME", lastTradeDateOrContractMonth="20241220")
ib.qualifyContracts(test)
chains = ib.reqSecDefOptParams(test.symbol, test.exchange, test.secType, test.conId)
util.df(chains)

This works, even when above is failing

test2 = Index("ES", exchange="CME")
ib.qualifyContracts(test2)
chains = ib.reqSecDefOptParams(test2.symbol, test2.exchange, test2.secType, test2.conId)
util.df(chains)

I actually can reproduce this directly in the TWS GUI and see try to deduce they coded up inside the GUI. Looking through the TWS GUI, it seems that each option has an underlying of the futures contract, and the futures contract has an underlying of the "index." However, options seems to actually be associated with the index, not the underlying futures contract. That's why when you choose "ES @ Dec 24" in the "options trading" mosaic tab, you still see the September '24 options - because it went to the "index" to grab the options data. Also, sometimes when I log out of TWS after looking up data on a specific option, when I log back in, a pop up tells me there are no contracts for the underlying. That's because it tried to find options for the specific delivery future and can't find them. But then if I just click on one on my portfolio, and they show up. They seem to have coded up a cludge to get around the confusion in the GUI to allow seeing the entire ES options chain across delivery dates by making options chains a child of the index. I'm guessing that whenever new option classes are added, something breaks in the inheritance in a way maintains the option chains in the index but not in the specific delivery date future. Might be some kind of infinite recursion where things become children of their parents maybe?

tldr; reqSecDefOptParams on the option Index (e.g., "ES IND") not the Future.