Crash in interactive_order_stack

[localempire]

Hello everyone,
(My paper account)
I had to switch to my backup system due to a failure.

First I updated the data.
Maybe by not first doing the "End of day process" (42) action with interactive_order_stack, I balanced the trades directly and got errors. After closing all with 42, still the same even with deleting all (40).

Today I have 2 trades from the system with ID 32 and 33 in the order stack.
After rebalancing the stack trace looks like this:

Your choice? 14
Use to fix breaks between instrument strategy and contract level positions
0: dynamic_system

Your choice? 0
Instrument code?V2X
Quantity -16
Filled price <RETURN for default 20.25>
Fill datetime: Enter date and time in format %Y-%m-%d eg '2024-08-09' OR '%Y-%m-%d %H:%M:%S' eg '2024-08-09 06:40:47'OR
(Order ID:no order ID) Type balance_trade for dynamic_system V2X, qty [-16], fill [-16]@ price, 20.25 Parent:no parent Children:no_children
Are you sure? (Y/other)Y
2024-08-09 06:43:33 DEBUG Interactive-Order-Stack {'strategy_name': 'dynamic_system', 'instrument_code': 'V2X', 'instrument_order_id': ''} Putting balancing order on instrument stack
2024-08-09 06:43:33 DEBUG Interactive-Order-Stack {'strategy_name': 'dynamic_system', 'instrument_code': 'V2X', 'instrument_order_id': ''} Marking balancing trades as completed and updating positions and historic order data
2024-08-09 06:43:33 DEBUG Interactive-Order-Stack {'strategy_name': 'dynamic_system', 'instrument_code': 'V2X', 'instrument_order_id': 34} Updated position of dynamic_system V2X from -81 to -97 because of trade (Order ID:34) Type balance_trade for dynamic_system V2X, qty [-16], fill [-16]@ price, 20.25 Parent:no parent Children:no_children 34 fill [-16]
2024-08-09 06:43:33 DEBUG connectionIB {'type': 'Interactive-Order-Stack', 'broker': 'IB', 'clientid': 509} Terminating {'ipaddress': '192.168.178.135', 'port': 4002, 'client': 509}
2024-08-09 06:43:33 DEBUG Interactive-Order-Stack {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 509
Traceback (most recent call last):
File "/root/pysystemtrade/sysproduction/linux/scripts/run.py", line 66, in
func(*args, **kwargs)
File "/root/pysystemtrade/sysproduction/interactive_order_stack.py", line 61, in interactive_order_stack
interactive_order_stack_with_ib_conn(ib_conn)
File "/root/pysystemtrade/sysproduction/interactive_order_stack.py", line 70, in interactive_order_stack_with_ib_conn
menu.run_menu()
File "/root/pysystemtrade/syscore/interactive/menus.py", line 64, in run_menu
method_chosen(*self._args, **self._kwargs)
File "/root/pysystemtrade/sysproduction/interactive_order_stack.py", line 368, in create_instrument_balance_trade
stack_handler.create_balance_instrument_trade(instrument_order)
File "/root/pysystemtrade/sysexecution/stack_handler/balance_trades.py", line 163, in create_balance_instrument_trade
self.handle_completed_instrument_order(instrument_order_id)
File "/root/pysystemtrade/sysexecution/stack_handler/completed_orders.py", line 60, in handle_completed_instrument_order
self.add_order_family_to_historic_orders_database(order_family)
File "/root/pysystemtrade/sysexecution/stack_handler/completed_orders.py", line 191, in add_order_family_to_historic_orders_database
order_data.add_historic_orders_to_data(
File "/root/pysystemtrade/sysproduction/data/orders.py", line 89, in add_historic_orders_to_data
self.add_historic_instrument_order_to_data(instrument_order)
File "/root/pysystemtrade/sysproduction/data/orders.py", line 98, in add_historic_instrument_order_to_data
self.db_strategy_historic_orders_data.add_order_to_data(instrument_order)
File "/root/pysystemtrade/sysdata/mongodb/mongo_historic_orders.py", line 73, in add_order_to_data
raise Exception(
Exception: Can't add order (Order ID:34) Type balance_trade for dynamic_system V2X, qty [-16], fill [-16]@ price, 20.25 Parent:no parent Children:no_children INACTIVE as order id 34 already exists!

The System is still unbalanced.

Your choice? 9
2024-08-09 06:44:49 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-08-09 06:44:49 DEBUG Interactive-Order-Stack {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 509
Optimal vs actual
current optimal breaks
dynamic_system CORN -30 -34.350/-31.078 True
dynamic_system SOFR 3 3.598/6.445 True
dynamic_system MXP 14 7.887/12.181 True
dynamic_system US10 -2 -1.338/-0.278 True
dynamic_system EUROSTX 3 -2.132/-0.663 True
dynamic_system V2X -97 -29.522/-25.349 True
Strategy positions
strategy_name instrument_code position
0 dynamic_system CORN -30
4 dynamic_system EUROSTX 3
2 dynamic_system MXP 14
1 dynamic_system SOFR 3
3 dynamic_system US10 -2
5 dynamic_system V2X -97

Contract level positions
instrument_code contract_date expiry_date position
2 CORN 20241213 2024-12-13 00:00:00 -30.0
0 EUROSTX 20240920 2024-09-20 00:00:00 3.0
1 MXP 20240916 2024-09-16 00:00:00 14.0
3 US10 20240919 2024-09-19 00:00:00 -2.0

BREAKS between strategy and contract positions: [V2X, SOFR]

Broker positions
instrument_code contract_date expiry_date position
1 CORN 20241213 2024-12-13 00:00:00 -30.0
2 EUROSTX 20240920 2024-09-20 00:00:00 1.0
4 MXP 20240916 2024-09-16 00:00:00 17.0
0 US10 20240919 2024-09-19 00:00:00 -2.0
3 V2X 20241016 2024-10-16 00:00:00 -16.0

BREAKS between broker and DB stored contract positions: [EUROSTX/20240920, MXP/20240916, V2X/20241016]

I corrected V2X back from -97 to -26 using the same error.
What can I do to get synched with the broker Stack?.
Thanks Ralf

[bug-or-feature]

Its hard to know what happened exactly, because you didn't say at what point you had your 'failure', or what steps you took to switch over to your backup. But, the error you're seeing is because of an Order ID conflict. You're trying to create a Balance Instrument trade, the system thinks the next order ID should be 34, but an order with ID 34 already exists, so it falls over.

In the normal day to day schedule, all orders are cleared from each stack when the stack handler shuts down. Maybe that didn't happen because of your 'failure'. That might account for what you're seeing. So you could view the stacks before trying to create the balance trade, maybe deleting any orders that shouldn't be there.

[oldlore]

Also, each of the stack databases has a counter to let it know what the next order ID should be. You could try incrementing that by one or the number after the last order ID.

[localempire]

Thank you bug-or-features and oldlore,
I guess I fixed it by looking at the max_order_id used in mongo_order_stack.py and corrected the value.
The stack between local and broker looks ok now.