IMPORTANT: Adding parquet functionality in

[robcarver17]

As discussed here, this is to ensure that dependencies on arctic db can be removed, and so that the required packages and python version can be updated to the current versions.

IF YOU WANT TO USE PARQUET (recommended):

• add a directory to your private config: parquet_store: '/home/rob/data/parquet/' (you do not need to create it, or subdirectories this is done automatically)
• install pyarrow
• backup all your data just in case. Stop any processes using adjusted price data.
• pull and install the latest commit
• run python3 ~/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py
• you should now be able to get data from parquet rather than arctic
• your arctic data will still be there, but will no longer be updated

IF YOU DO NOT WANT TO USE PARQUET BUT WANT TO GET THE LATEST COMMIT

• Modify the classes assigned in sysproduction.data.production_data_objects to point back to arctic
• Modify the classes assigned in sysdata.sim.db_futures_sim_data to point back to arctic

WHAT WILL HAPPEN IF YOU DON'T DO EITHIER AND PULL THE LATEST COMMIT

• You will find you can't certain data and many things will break. To fix just follow one of the two choices above.

[yk2k]

Thank you for the great tool!
I needed to add this to backup_capital() to make it run when parquet data directory is empty.

try:                                                                   
    parquet_data = data.parquet_capital.get_capital_pd_df_for_strategy(
        strategy_name                                                  
    )                                                                  
except missingData:                                                    
    parquet_data = [] 

if len(parquet_data) >= len(strategy_capital_data):
   ...

[rorymac]

yk2k

A question if you would.

There are two backup_capital( ) functions in PST - I assume you are talking about the one in ~/pysystemtrade/sysinit/transfer/backup_ arctic_to_parquet.py ?

Thank you

[oldlore]

yes that is the file. line 488 or so.

[t3chap]

Rob, et al:
If I'm installing from scratch, do I need to go through the steps to install arctic and mongodb and then run the scripts to transition to parquet?
I have build my own methods of importing the data that work with the previous (pre-parquet, mongodb using arctic) regime but I was trying a fresh install so I could use the latest versions of ubuntu, pandas, etc.

I guess another way of phrasing the question is can I go straight from a clean install ov 1.80 straight into parquet without ever touching mongodb or arctic?

Thanks.

[tgibson11]

Regarding MongoDB, I've been running version 7 (the latest) on my development machine, along with pymongo 3.11.3, which is documented as compatible, only that it doesn't support newer MongoDB features that pysystemtrade wouldn't be using anyway. This has been working fine for me.

TL;DR - you shouldn't need to worry about installing an older version of MongoDB if that was a concern.

[oldlore]

This is a sort of related question. I'm running Linux Mint 20.3. Adding parquet seems straightforward enough, but is it worth the trouble of upgrading to Mint 21.x and upgrading python from 3.8.10? To maintain compatibility with future libraries perhaps?

[rorymac]

Todd
Question if you would be so kind.
I have all the data backed up in the standard csv files, if I attempt to do a clean install How do I get that into parquet?
I am not having any luck trying the transition, my environment is handing out module not found errors.
Would like to attempt a clean install

[littlebigfis]

There is code in the futures workflow process (barchart_futures_contract_prices ) and contract_prices_from_csv_to_arctic that have dependencies that appear to rely on arctic. I'm not sure how deep the rabbit hole goes with these dependencies and so I've been a bit paralyzed. Is it just easier to rewrite my own code for this part of the process?

[bug-or-feature]

@littlebigfis no you shouldn't need to write your own code. Start a new discussion and describe your problem, I'll try to talk you through it. It's a bit confusing as some classes and functions didn't get renamed properly

[rorymac]

A question Rob, when you say pull and install the latest commit, I assume you mean the droparctic branch?

[robcarver17]

No, I mean develop.

[rorymac]

I have done a full new install of the latest version. [05184ea] The Master branch.
As well as Pyenv
All looked good the test worked - copied over the DB
BUT on running python3 ~/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py
Got the following error - which is puzzling
rorym@rorym-VirtualBox:~/pysystemtrade$ python3 ~/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py Configuring sim logging Traceback (most recent call last): File "/home/rorym/pysystemtrade/sysinit/transfer/backup_arctic_to_parquet.py", line 40, in <module> from sysdata.arctic.arctic_futures_per_contract_prices import ( File "/home/rorym/pysystemtrade/sysdata/arctic/arctic_futures_per_contract_prices.py", line 8, in <module> from sysdata.arctic.arctic_connection import arcticData File "/home/rorym/pysystemtrade/sysdata/arctic/arctic_connection.py", line 2, in <module> from arctic import Arctic ModuleNotFoundError: No module named 'arctic'

As you can see the test bed is a Virtual Machine (which I believe should not make a difference.)
Any ideas out there?

Regards

[craigmediaservices]

It appears your machine thinks the Python artic extension is not installed? Were you able to fix this?

[rorymac]

Yes , I did. You are correct , on the new install , somehow I messed up the requirements.txt file and had not installed arctic.
I hacked it though by doing a mongodump of the production DB and restored it on the new machine.
I also had to fiddle with the csv backup files for the pnl took the back up csv and wrote it to parquet then the system worked.
I took the long way around, created a VM for a test and once I had that working, then installed everything on my server.
I still have a small glitch in that my pnl report won't run. But that is not that much of an issue for me.

[craigmediaservices]

VM is nice for isolation. Right now I've built out everything in Docker so I could theoretically clone the project rapidly on any machine.

[oldlore]

So I've tried to do this as well in a VM box.

1. Ran into the same problem with 'ModuleNotFoundError: No module named 'syscore' error. This occurs when I try to run something that goes through the run.py like interactive_diagnostics. I suspect it has something to do with the virtual environment because I can import modules fine in the python command line or in VS Code. After getting past that, I'm stuck in the parquet config setup.

PermissionError: [Errno 13] Permission denied: '/home/me'

I have the parquet_store variable set up in the private_config with the correct directory and even changed the default config but for some reason it won't read it and keeps looking in /home/me.

[rorymac]

oldlore
I finally figured it out (The module not found error) - you need to check that your system path in python has the correct paths.
procedure is as follows:
open python on the command line
`
rorym@rorym-VirtualBox:~/pysystemtrade$ python3
Python 3.8.10 (default, Nov 22 2023, 10:22:35)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
import pprint
pprint.pprint(sys.path)
['',
'/usr/lib/python38.zip',
'/usr/lib/python3.8',
'/usr/lib/python3.8/lib-dynload',
'/home/rorym/.local/lib/python3.8/site-packages',
'/usr/local/lib/python3.8/dist-packages',
'/home/rorym/pysystemtrade',
'/usr/local/lib/python3.8/dist-packages/statsmodels-0.12.2-py3.8-linux-x86_64.egg',
'/usr/lib/python3/dist-packages',
'/usr/lib/python3.8/dist-packages']

`
So if the sys.path does not have your pysystemtrade path correct you will get the error. Also another complication if your python version changes then some of the pip packages will not be found.
I have no idea how I messed up the pysys path as that is done when you run the setup.py - but there it is.

I did mange to get the DB transferred to parquet - all except Capital, but that is enough to get by.

The procedure.
Is exactly as described by Rob above BUT do not install - (by that think he means run the setup.py ) if you do that then you get the pip package errors, module not found.

As to your last comment I had something similar on one of the iterations - I do think that has something to do with the change of the python version.

[littlebigfis]

Apologies if this is a bad question: I am in the early stages of sourcing data from Barchart using bcutils and whatever remains through IB. My goal is to bring multiple prices up to date with as many contracts as possible to select from in dynamic optimization. In examining the code in pysystemtrade to help with the "Futures Workflow Process" there appears to be an embedded reliance on Arctic in many of these scripts. How challenging will this be to run down all the Arctic dependencies in the code and replace with Parquet or is that even necessary? Has anyone started this process fresh with Parquet instead of migrating from Arctic?

[Spacious08]

Hi,

Rob has already adjusted the code to use Parquet instead of Arctic, although he kept the term "Arctic" in some file names, which can be confusing. For example, the file \sysinit\futures\rollcalendars_from_arcticprices_to_csv.py actually uses Parquet prices.

To add Barchart data into the system using Parquet, I follow these steps:

• Run \sysinit\futures\contract_prices_from_csv_to_arctic.py to read the Barchart CSV and populate prices to Parquet.
• Validate the imported prices and contract volume to determine when to roll. I have created my own script for this validation.
• Create CSV roll calendars using \sysinit\futures\rollcalendars_from_arcticprices_to_csv.py.
• Create and store multiple prices using \sysinit\futures\multipleprices_from_db_prices_and_csv_calendars_to_db.py.
• Create and store adjusted prices using \sysinit\futures\adjustedprices_from_db_multiple_to_db.py.

Following these steps, the instrument should be integrated into the system. You still need to add weights, adjust sampled contracts, roll contracts, etc.

[littlebigfis]

I’ve been working on all the scripts to update historical futures prices as well as multiple and adjusted prices, and I’m unsure where this data is ultimately being stored given the codebase vs changelog. The dataBlob object seems to handle data management and versatile storage options, but it’s unclear whether historical prices, multiple prices, and adjusted prices are being stored in Parquet, CSV, or MongoDB or some combination of the three

Fro example, PST has moved static config data (e.g., /data/futures/csvconfig/) from Mongo to CSV files, but when it comes to storing historical prices, it seems that dataBlob might leave open the option to use any of these formats. In update_historical_prices, I’m having trouble tracking if data ends up in Parquet, or if it’s still going to MongoDB as well?

Can anyone clarify the intended storage location for historical, multiple, and adjusted prices in this context?

Also, can someone clarify what the specific role of mongo is for all price data at this point? Is there any metadata being stored in mongo for price or contract data?

[tgibson11]

For production purposes:

• Time series data (which includes all price data, and also a few other things) is stored in Parquet
• Non-time series data is stored in MongoDB
• Config data is either in yaml or csv files, except for spread cost data, which is in MongoDB

As you point out, the dataBlob class is designed to be very flexible to allow for different data sources. So you could easily use CSV data for backtesting, for example.

[littlebigfis]

Very clear and now i'm feeling better having that clarity, thank you!