Ideas for improvements

[bug-or-feature]

Ideas for improvements:

• adopt semantic versioning
• adopt pyproject.toml fully for building and dependencies. Deprecate setup.py and requirements.txt
• fix all the FutureWarnings from pandas
• improve and update the docs

Thoughts, ideas, objections?

[emretezel]

Hi Andy,

Firstly, thank you for taking the mantle for this project.

Long time ago, I proposed to Rob to have separate master and develop branches. With hindsight I am of the view that this wasn't a good idea. Going forward I would recommend that we treat master branch as the production branch. One should get latest production quality code by pulling master. If we are working on substantial new features these should first be developed in their own branches and later merged to master. In other words we should not shy away from many development branches named after the new features.

Secondly, ever since parquet functionality has been added, I could not migrate my system to using parquet. There are two open items for this issue, #1364 and #1377. If this issue could be patched, it would be much appreciated.

Thanks again for stepping up!

Best regards,
Emre

[bug-or-feature]

Hi @emretezel - the idea of having a develop and master branch was a good one at the time; it reduced the impact of Rob's commit style into the master branch. Unfortunately it wasn't always implemented perfectly, which led to other issues. Hopefully, things will be a bit more stable now

The two parquet capital issues are easy to fix, I'll push a change soon

[bug-or-feature]

parquet capital issues fix on develop #1423

[dr1ver1]

Hi Andy,

Thanks for taking the lead on this project and for opening up the discussion for improvements! I’ve spent some time with the codebase in the past and, while I deeply appreciate all the work Rob put into it, I’ve found myself struggling to fully grasp the flow due to the depth of the inheritance structure.

While I understand that inheritance can be a powerful tool, in this particular codebase, it seems to create challenges when trying to step through the code, especially when the call stack grows quite deep. This can make it difficult to build a mental model of what’s happening during execution, particularly for people who are trying to understand the system or extend it.

Would it make sense to explore ways of reducing the level of inheritance or perhaps consider alternatives like composition where possible? I’ve noticed that there’s been a shift towards favoring composition over deep inheritance in modern Python development, as it can sometimes improve code readability and maintainability.

That said, if the inheritance is serving a necessary purpose, perhaps some additional documentation could help clarify how the different layers interact. Even a high-level overview of the class hierarchy and their relationships could make it easier to follow for contributors who aren’t as familiar with the design patterns at play.

Looking forward to hearing your thoughts!

Best,
David

[bug-or-feature]

Hi @dr1ver1 - yes composition should be favoured over inheritance generally. I can't imagine there are too many software engineering principles with their own wiki page. Rob previously acknowledged the complexity of the inheritance hierarchy for the Accounts classes, and I think it's more complex now. It would be a mammoth task to change anything. I wouldn't fancy trying to unpick it!

I'd like to improve the docs though, so I'll start an ideas discussion about that. Maybe you can make some suggestions for improvements there?

[dr1ver1]

Hi Andy,

Thanks for the thoughtful response, and I appreciate the acknowledgment around the complexity of the inheritance structure. I agree — I think it would be a big effort to simplify the inheritance, so focusing on documentation seems like a good way forwards. The class diagram you've put together in that discussion with Rob is very helpful!

It’s been a little while since I last looked at the code, so I’ll need some time to get back up to speed. I also have a lot of other commitments at the moment, so it might be a few months before I can meaningfully contribute, but I’ll certainly join in as soon as I can.

From what I remember, there were times in VS Code when the call stack seemed to get incredibly deep — sometimes dozens or possibly hundreds of calls down. This made understanding the flow of the code pretty challenging, but I’d need to dive back in to refresh my memory before commenting further on specifics.

Looking forward to contributing when I can.

Kind regards,
David

[dr1ver1]

https://diataxis.fr/ - they have put a lot of thought into structuring documentation

[vishalg]

A few ideas that are a mix of useful improvements and aspirations

• Bug fixes (I have a couple of fixes to contribute here - will create patches and submit)
• Better and more extensive error catching and handling - particularly in the interactive reports and live trading/broker connections
• Dashboard development - (again a few quality of life improvements I can contribute but this can be developed a lot further to incorporate all the interactive_diagnostics and interactive control functionality)
• Ability to send email reports in html format (much more readable on a mobile device)
• Refactor backtest persistence - pickling is really only suitable for smaller amounts of data, we could persist the overnight backtest data in a database. Pickling uses a lot of memory and not suited for a cloud deployment. This may in due course also facilitate incremental trade/signal/backtest generations - unless I am missing something there is no reason why we need to rerun the entire backtest instead of updating previous backtest with most recent data.

[bug-or-feature]

@vishalg - thanks for the ideas. For small fixes, error handling etc, create a PR or an issue. For bigger changes, or feature requests, I suggest creating an Ideas discussion first. Then everyone (including Rob?) can add their input

[vishalg]

@bug-or-feature Thanks for taking on the mantle. A brief guidelines/tutorial for patch submission and contributions will also be helpful especially for those of us who are not terribly familiar with git/github workflows.

[bug-or-feature]

there is a contributing guide already. I'll ad some notes with a git workflow - I saw a good one recently I can steal from

[bug-or-feature]

The contributing guide has been updated with developer setup

[vishalg]

thank you very much

[adnan-trader]

As I am new to pysystemtrade and this community. I have just recently tried to set up a 'test' production environment and it has been quite the task to read and understand all the docs and steps. It is clear a lot of work has been done, but also that it has become outdated.

My vote would be to improve the following:

1. Documentation: Production as this is the most important section for those that want to use Rob's system as they move from backtesting to production. In particular, how to run a daily process with guide on common errors/issues and how to resolve (Operations Manual).
2. Documentation Production How-to-Setup for Windows environment.
   a. I am happy to contribute as I have just gone through the process of setting up on Windows...still have some more to do. Not everyone wants to be on Linux!.
3. Process Monitoring Reporting: As noted by many, the email reports are very difficult to read. I would suggest an exception based reporting where relevant. That is, what is wrong and not what has gone right, e.g., which market prices are not updated. Build a better output structure that can be more easily processed by destination: email or database.
4. Reporting (Perf, Risk, Trade, Position, etc): I believe this type of reporting should be database driven where output is stored in a data warehouse (proper dimensions/fact tables). Then, any front-end can be used (PowerBI, Looker, etc.) I can contribute here as well.
5. Scheduling: Should we consider replacing the self-coded scheduling module with an open-source equivalent, if available, with all the dependency mapping, logging, error handling features? Active members of community with experience with solutions could be leveraged.
   a. Also, better documentation on how handle errors and how to recover using the STOP/GO/NO RUN and 'close' process. For newbies, this is quite confusing.

Thanks

[vcaldas]

@adnan-trader Im on the same page. Trying to get all up and running and then work on improving the docs - and hopefully to the project as well.

[vcaldas]

I just deployed this POC for the documentation: https://vcaldas.github.io/aptrade/introduction/
Interested in feedback and, if this is a good path forward, will make into a proper PR.

[oldlore]

I personally haven't gotten around to doing this yet, but how about adding Crude Palm Oil Futures and FTSE Bursa Malaysia KLCI Futures to the system? Not a high priority I imagine but it'd be nice to have it done in the repo so that we're all using the same instrument name, etc.