Not able to get all contracts for a given instrument

[TobiasAntiGravity]

I am not getting all of the contracts needed to generate the small multiple prices history missing from the updated repo data.

I have the same issue in all of the four instruments;
COTTON
MILKWET
MILKDRY
CHEESE

Example; MILKWET has 202207 as the forward contract at the last line of the multiple prices repo data. I have done the sysinit/futures/seed_price_data_from_IB.py, and afterwards done update_sampled_contracts - but I am not able to get prices for the 202207 contract.

for completeness;

from sysdata.arctic.arctic_futures_per_contract_prices import arcticFuturesContractPriceData
contract_prices = arcticFuturesContractPriceData()

contract_prices.contract_dates_with_price_data_for_instrument_code("MILKWET")
>>>>>
['20210500',
 '20210600',
 '20210800',
 '20210900',
 '20211100',
 '20220100',
 '20220200',
 '20220300',
 '20220500',
 '20220600',
 '20220800',
 '20221000',
 '20221100',
 '20230100',
 '20230200',
 '20230400',
 '20230500',
 '20230600',
 '20230800',
 '20231000',
 '20231100',
 '20240100',
 '20240200']

I thought about the HoldRollCycle and PricedRollCycle parameters in rollconfig.csv - but both have all months listed for MILKWET.

I have tried my darndest to find a class that I can use to pull the missing contract, but I have to admit that the code is on a higher level then my python skills. The abstraction is hard to see through. (honestly Rob's book excerpt with the punch line "apologies for my size" keeps popping up in my mind at the moment - size of my intellect that is)

I am keen og getting the paper trading up and running as fast as possible - and so I am going to attempt to import the needed prices from barchart - but this problem might well be a re-ocurring issue - and so I thought I would ask if anyone has any insight on the matter - would be very much appreciated!

[robcarver17]

Update sampled contracts just adds contracts to the database. If you want to update prices of individual contracts you need update_historical_prices. I appreciate this isn't a very helpful name, but it's hard to change.

This would also be done by run_daily_price_updates

[TobiasAntiGravity]

I forgot to mention it, but update_historical_prices is a script I a familiar with, and which I tried it first. The two options mentioned in the opening comment are actions taken which I took afterwards as an attempt to try to fix.

My working thesis is that I am not able to pull prices for certain contracts. The same seems to have happened for VNKI where I am not able to get contract prices for the forward contract, even though you, Mr. Carver, were able to roll VNKI and so obviously you are able to pull this data.

I will try to look more into this, but my debugging abilites have been disappointing (partly because I have not been able to connect pycharm to the container where the code runs, so that I can use the pycharm debug tool)

[TobiasAntiGravity]

A small FYI to anyone wanting to pull daily prices from barchart like I did - The barchart .csv time stamp only contained a date part, no time stamp. When I first importet I could not update multiple prices because of this. In addition to the date, the daily prices needs the 23:00 time prat. A small script to add the time part to the .csv did the trick.

I wrote a script for this, but my first try was to add this to the csv reader class, which was a hopeless thought. It should be handled separately on the .csv itself before the csv is read. By the time I realized this I did not want to spend more time on the matter (because, not sure if I mentioned this, but I really want to get up and running with pysystemtrade) and so I settled on a very hacky script that is not fit for publication. Once I have more time - I can get back to this and publish.

[robcarver17]

Does the update prices script try and get a price for say MILKWET 202207?

And when it does, what error does IB throw back?

For example

2022-04-18 10:38:22 {'type': 'Update-Historical-prices-manually', 'instrument_code': 'MILKWET', 'contract_date': '20220400', 'component': 'mongoIbBrokerClientIdData'}  Locked IB client ID 878
2022-04-18 10:38:31 {'type': 'Update-Historical-prices-manually', 'instrument_code': 'MILKWET', 'contract_date': '20220400', 'broker': 'IB', 'clientid': 878, 'component': 'ibFuturesContractPriceData'}  Reqid -1: 2106 HMDS data farm connection is OK:ushmds 



 Manually checking prices for MILKWET/20220700   #### <- do you see this line?


2022-04-18 10:38:47 {'type': 'Update-Historical-prices-manually', 'component': 'arcticFuturesContractPriceData', 'instrument_code': 'MILKWET', 'contract_date': '20220700'}  No new data  #### <- and if so, what do you get next?

If you don't see the first line, then there is a problem with the hold / roll cycle configuration (which seems unlikely) or update sampled contracts hasn't worked properly.

If you see the first line, but then get an error, then the problem is getting data from IB.

[TobiasAntiGravity]

Hi Mr. Carver

I solved the MILKWET issue by importing from barchart, so I have to look at another instrument. I did a check for VNKI which I cannot roll because of lacking price data for forward contract (july).

I looked at the echo logs from the daily runs. Output from update_historical_prices is the same for all contracts - here is the output concerning the july 2022 contract, which is the forward contract where there are no prices.

2022-04-15:2133.52 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 304, 'component': 'ibFuturesContractPriceData'}  Pausing 0.178609 seconds to avoid pacing violation
2022-04-15:2134.53 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 304, 'component': 'ibFuturesContractPriceData'} [Warning] No price data from IB
2022-04-15:2134.53 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 304, 'component': 'ibFuturesContractPriceData'} [Warning] Something went wrong getting IB price data for VNKI/20220700
2022-04-15:2134.53 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 304}  No prices from broker for VNKI/20220700

has been a problem for as far back as there are logs available:

2022-04-11:2126.15 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 270, 'component': 'ibFuturesContractPriceData'}  Pausing 0.276985 seconds to avoid pacing violation
2022-04-11:2127.16 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 270, 'component': 'ibFuturesContractPriceData'} [Warning] No price data from IB
2022-04-11:2127.16 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 270, 'component': 'ibFuturesContractPriceData'} [Warning] Something went wrong getting IB price data for VNKI/20220700
2022-04-11:2127.16 {'type': 'update_historical_prices', 'instrument_code': 'VNKI', 'contract_date': '20220700', 'broker': 'IB', 'clientid': 270}  No prices from broker for VNKI/20220700

So, obviously something wrong with my IB feed. I am sure I have the subscription, because it was so cheap compared to all the other data subscription plans - it was a no brainer. I will check with IBKR.

Looking at the update_sampled_contracts it looks like my subscription is working. One question though, if I may; I noticed that the update_sampled_contracts logs were giving the same messages about adding contracts for sampling and stopping sampling accross multiple daily logs;

2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Added contract VNKI 20220300
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Contract VNKI/20220300 now sampling
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Added contract VNKI 20220200
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Contract VNKI/20220200 now sampling
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Added contract VNKI 20220400
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Contract VNKI/20220400 now sampling
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData'} [Warning] Reqid 731: 200 No security definition has been found for the request  (NKVI/202202)
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=VNKI, ib_data=symbol='NKVI', exchange='OSE.JPN', currency='JPY', ibMultiplier='10000.0', priceMagnifier=1.00, ignoreWeekly='False, effective_multiplier=10000.00' ) could not resolve contracts: No contracts found matching pattern
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220200'} [Warning] Contract is missing can't get expiry
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Can't find expiry for VNKI/20220200, could be a connection problem but could be because contract has already expired
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData'} [Warning] Reqid 732: 200 No security definition has been found for the request  (NKVI/202203)
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=VNKI, ib_data=symbol='NKVI', exchange='OSE.JPN', currency='JPY', ibMultiplier='10000.0', priceMagnifier=1.00, ignoreWeekly='False, effective_multiplier=10000.00' ) could not resolve contracts: No contracts found matching pattern
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220300'} [Warning] Contract is missing can't get expiry
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Can't find expiry for VNKI/20220300, could be a connection problem but could be because contract has already expired
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData'} [Warning] Reqid 733: 200 No security definition has been found for the request  (NKVI/202204)
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=VNKI, ib_data=symbol='NKVI', exchange='OSE.JPN', currency='JPY', ibMultiplier='10000.0', priceMagnifier=1.00, ignoreWeekly='False, effective_multiplier=10000.00' ) could not resolve contracts: No contracts found matching pattern
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'component': 'ibFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220400'} [Warning] Contract is missing can't get expiry
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Can't find expiry for VNKI/20220400, could be a connection problem but could be because contract has already expired
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220500'}  No change to contract expiry VNKI/20220500 to 2022-05-10 00:00:00
2022-04-18:2007.00 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220600'}  No change to contract expiry VNKI/20220600 to 2022-06-07 00:00:00
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220700'}  No change to contract expiry VNKI/20220700 to 2022-07-12 00:00:00
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220800'}  No change to contract expiry VNKI/20220800 to 2022-08-09 00:00:00
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Added contract VNKI 20220200
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Contract VNKI/20220200 has expired so now stopped sampling
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Added contract VNKI 20220300
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Contract VNKI/20220300 has expired so now stopped sampling
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Added contract VNKI 20220400
2022-04-18:2007.01 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 309, 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Contract VNKI/20220400 has expired so now stopped sampling

2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Added contract VNKI 20220200
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Contract VNKI/20220200 now sampling
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Added contract VNKI 20220300
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Contract VNKI/20220300 now sampling
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Added contract VNKI 20220400
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Contract VNKI/20220400 now sampling
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData'} [Warning] Reqid 731: 200 No security definition has been found for the request  (NKVI/202202)
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=VNKI, ib_data=symbol='NKVI', exchange='OSE.JPN', currency='JPY', ibMultiplier='10000.0', priceMagnifier=1.00, ignoreWeekly='False, effective_multiplier=10000.00' ) could not resolve contracts: No contracts found matching pattern
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220200'} [Warning] Contract is missing can't get expiry
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Can't find expiry for VNKI/20220200, could be a connection problem but could be because contract has already expired
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData'} [Warning] Reqid 732: 200 No security definition has been found for the request  (NKVI/202203)
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=VNKI, ib_data=symbol='NKVI', exchange='OSE.JPN', currency='JPY', ibMultiplier='10000.0', priceMagnifier=1.00, ignoreWeekly='False, effective_multiplier=10000.00' ) could not resolve contracts: No contracts found matching pattern
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220300'} [Warning] Contract is missing can't get expiry
2022-04-15:2006.52 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Can't find expiry for VNKI/20220300, could be a connection problem but could be because contract has already expired
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData'} [Warning] Reqid 733: 200 No security definition has been found for the request  (NKVI/202204)
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData'} [Warning] futuresInstrumentWithIBConfigData(instrument=VNKI, ib_data=symbol='NKVI', exchange='OSE.JPN', currency='JPY', ibMultiplier='10000.0', priceMagnifier=1.00, ignoreWeekly='False, effective_multiplier=10000.00' ) could not resolve contracts: No contracts found matching pattern
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'component': 'ibFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220400'} [Warning] Contract is missing can't get expiry
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Can't find expiry for VNKI/20220400, could be a connection problem but could be because contract has already expired
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220500'}  No change to contract expiry VNKI/20220500 to 2022-05-10 00:00:00
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220600'}  No change to contract expiry VNKI/20220600 to 2022-06-07 00:00:00
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220700'}  No change to contract expiry VNKI/20220700 to 2022-07-12 00:00:00
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220800'}  No change to contract expiry VNKI/20220800 to 2022-08-09 00:00:00
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Added contract VNKI 20220200
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220200'}  Contract VNKI/20220200 has expired so now stopped sampling
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Added contract VNKI 20220300
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220300'}  Contract VNKI/20220300 has expired so now stopped sampling
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'component': 'mongoFuturesContractData', 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Added contract VNKI 20220400
2022-04-15:2006.53 {'type': 'update_sampled_contracts', 'broker': 'IB', 'clientid': 303, 'instrument_code': 'VNKI', 'contract_date': '20220400'}  Contract VNKI/20220400 has expired so now stopped sampling

EDIT: About the line Manually checking prices for - I searched for manual in the logs but came up empty - is there a different logging level for the echo logs compared to standard output?

[robcarver17]

'Manual mode' I ran interactive_manual_check_historical_prices because it's currently the only way to get prices for a single instrument.

[TobiasAntiGravity]

Ah, I should of course have thought of the interactive option. I was looking for a name == "main" clause or something to that effect.

[robcarver17]

As far as I know you can get contract dates for an instrument without having a price subscription.

You might want to double check it is an IB problem by trying to get the price in the underyling API: https://nbviewer.org/github/erdewit/ib_insync/blob/master/notebooks/bar_data.ipynb

[TobiasAntiGravity]

Thank you for all your pointer and help with this!

As mentioned this problem seems to have occured for several instruments, so it is strange.

I will try to get the prices directly from the API and take it from there

[TobiasAntiGravity]

I found the issue; IBKR was blocking the request due to a missing trading permission. I did not get any hints from the API error repsonse;

fut_contract = Contract()
fut_contract.secType = 'FUT'
fut_contract.exchange = 'OSE.JPN'
fut_contract.currency = 'JPY'
fut_contract.symbol = 'NKVI'
fut_contract.localSymbol = 'NVIK22'
ib.qualifyContracts(fut_contract)
>>>>
Error 200, reqId 8: No security definition has been found for the request, contract: Contract(secType='FUT', symbol='NKVI', exchange='OSE.JPN', currency='JPY', localSymbol='NVIK22')
Unknown contract: Contract(secType='FUT', symbol='NKVI', exchange='OSE.JPN', currency='JPY', localSymbol='NVIK22')

But logging into the web UI and entering the contract into the order registration form I noticed an alert box saying I did not have trading permissions for this instrument.

I set this up a while back and did not think to check Japan at the time. Had some other "holes" in the permissions as well. Requested permission, recieved them almost immediately, and now I am able to download the data.

So, to recap; Even though I had bought the data subscription, which should have allowed me to retrieve the price data, IBKR denied the data requests due to lacking trading permissions. I guess it is a bug, but might be a feature?!

This might very well explain the fact that I experienced this for other instruments as well, as there are multiple a lot of exchanges listed under the US, where I did not have any permissions.

[robcarver17]

I think if you looked at the logging on the IB gateway you would have seen the trading permission error.

[TobiasAntiGravity]

Right you are Mr. Carver - I see that now!