Skip to content

Commit

Permalink
Merge branch 'develop' of https://github.com/robcarver17/pysystemtrade …
Browse files Browse the repository at this point in the history 
…into develop
  • Loading branch information
robcarver17 committed Jan 18, 2024
2 parents 8d5a170 + 1bbd443 commit 132570b
Show file tree
Hide file tree
Showing 11 changed files with 65 additions and 283 deletions.
4 changes: 2 additions & 2 deletions docs/backtesting.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3851,7 +3851,7 @@ These functions are used internally whenever a file name is passed in, so feel f

### Basic logging

pysystemtrade uses the [Python logging module](https://docs.python.org/3.8/library/logging.html). The system, data, config and each stage object all have a .log attribute, to allow the system to report to the user; as do the functions provided to estimate correlations and do optimisations.
pysystemtrade uses the [Python logging module](https://docs.python.org/3.10/library/logging.html). The system, data, config and each stage object all have a .log attribute, to allow the system to report to the user; as do the functions provided to estimate correlations and do optimisations.

By default, log messages will print out to the console (`std.out`) at level DEBUG. This what you get in sim. This is configured by function `_configure_sim()` in `syslogging.logger.py`.

Expand Down Expand Up @@ -3888,7 +3888,7 @@ I strongly encourage the use of logging, rather than printing, since printing on

### Advanced logging

In my experience wading through long log files is a rather time-consuming experience. On the other hand it's often more useful to use a logging approach to monitor system behaviour than to try and create quantitative diagnostics. For this reason I'm a big fan of logging with *attributes*. This project uses a custom version of [logging.LoggerAdapter](https://docs.python.org/3.8/library/logging.html#loggeradapter-objects) for that purpose:
In my experience wading through long log files is a rather time-consuming experience. On the other hand it's often more useful to use a logging approach to monitor system behaviour than to try and create quantitative diagnostics. For this reason I'm a big fan of logging with *attributes*. This project uses a custom version of [logging.LoggerAdapter](https://docs.python.org/3.10/library/logging.html#loggeradapter-objects) for that purpose:

```python
from syslogging.logger import *
Expand Down
117 changes: 38 additions & 79 deletions docs/production.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -659,7 +659,7 @@ Note: the configuration variable echo_extension will need changing in `private_c

### Logging

pysystemtrade uses the [Python logging module](https://docs.python.org/3.8/library/logging.html). See the [user guide for more detail](/docs/backtesting.md#logging) about logging in sim. Python logging is powerful and flexible, and log messages can be [formatted as you like, and sent virtually anywhere](https://docs.python.org/3.8/howto/logging.html#logging-advanced-tutorial) by providing your own config. But this section describes the default provided production setup.
pysystemtrade uses the [Python logging module](https://docs.python.org/3.10/library/logging.html). See the [user guide for more detail](/docs/backtesting.md#logging) about logging in sim. Python logging is powerful and flexible, and log messages can be [formatted as you like, and sent virtually anywhere](https://docs.python.org/3.10/howto/logging.html#logging-advanced-tutorial) by providing your own config. But this section describes the default provided production setup.

In production, the requirements are more complex than in sim. As well as the context relevant attributes (that we have with sim), we also need
- ability to log to the same file from different processes
Expand Down Expand Up @@ -752,88 +752,47 @@ There is a special SMTP handler, for CRITICAL log messages only. This handler us

#### Adding logging to your code

Here is an example of logging code (needs to adjusted for new style logging):
See the [logging docs](https://docs.python.org/3.10/library/logging.html) for usage examples. There are four ways to manage context attributes:
* *overwrite* - passed attributes are merged with any existing, overwriting duplicates (the default)
* *preserve* - passed attributes are merged with any existing, preserving duplicates
* *clear* - existing attributes are cleared, passed ones added
* *temp* - passed attributes will only be used for one invocation

```python
from syslogging.logger import *


def top_level_function():
"""
This is a function that's called as the top level of a process
"""

# logger setup
log = get_logger("top-level-function")

# note use of log.setup when passing log to other components, this creates a copy of the existing log with an additional attribute set - TODO transition to sysloggging
conn = connectionIB(client=100, log=log.setup(component="IB-connection"))

# - TODO transition to sysloggging
ibfxpricedata = ibFxPricesData(conn, log=log.setup(component="ibFxPricesData"))

# - TODO transition to sysloggging
arcticfxdata = arcticFxPricesData(log=log.setup(component="arcticFxPricesData"))

list_of_codes_all = ibfxpricedata.get_list_of_fxcodes() # codes must be in .csv file /sysbrokers/IB/ibConfigSpotFx.csv
log.debug("FX Codes: %s" % str(list_of_codes_all))
for fx_code in list_of_codes_all:

# Using log.label permanently adds the labelled attribute (although in this case it will be replaced on each iteration of the loop - TODO transition to sysloggging
log.label(currency_code=fx_code)
new_fx_prices = ibfxpricedata.get_fx_prices(fx_code)

if len(new_fx_prices) == 0:
log.error("Error trying to get data for %s" % fx_code)
continue
```

#### Refactoring logging

There is an ongoing project (June 2023) to migrate [legacy logging](/syslogdiag/pst_logger.py) to the built-in Python logging module. Currently, lots of methods are marked as deprecated - they will be refactored away in time. But if you are working on some code and want to make a change now:
- `log.msg()` - > `log.debug()`
- `log.terse()` - > `log.info()`
- `log.warn()` - > `log.warning()`

For other methods, like `label()`, `setup()`, each should be taken on a case by case basis. Under the hood, a call to `get_logger()` creates an instance of `DynamicAttributeLogger` which has an instance of a [Python logger](https://docs.python.org/3.8/library/logging.html#logging.Logger). From the docs:

> Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.
So our outer object handles the context attributes, and the inner `logging.Logger` object does the rest. We cannot copy logger instances as we did with the legacy system. Instead, we can manage the attributes with four ways to merge: *overwrite* (the default), *preserve*, *clear*, and *temp*.
#### Examples

```python
# merging attributes: method 'overwrite' (default if no method supplied)
overwrite = get_logger("Overwrite", {"type": "first"})
overwrite.info("overwrite, type 'first'")
overwrite.info(
"overwrite, type 'second', stage 'one'",
method="overwrite",
type="second",
stage="one",
)

# merging attributes: method 'preserve'
preserve = get_logger("Preserve", {"type": "first"})
preserve.info("preserve, type 'first'")
preserve.info(
"preserve, type 'first', stage 'one'", method="preserve", type="second", stage="one"
)

# merging attributes: method 'clear'
clear = get_logger("Clear", {"type": "first", "stage": "one"})
clear.info("clear, type 'first', stage 'one'")
clear.info("clear, type 'second', no stage", method="clear", type="second")
clear.info("clear, no attributes", method="clear")
# merging attributes: method 'temp'
temp = get_logger("temp", {"type": "first"})
temp.info("type should be 'first'")
temp.info(
"type should be 'second' temporarily",
method="temp",
type="second",
)
temp.info("type should be back to 'first'")
overwrite = get_logger("Overwrite", {"type": "first"})
overwrite.info("overwrite, type 'first'")
overwrite.info(
"overwrite, type 'second', stage 'one'",
method="overwrite",
type="second",
stage="one",
)

# merging attributes: method 'preserve'
preserve = get_logger("Preserve", {"type": "first"})
preserve.info("preserve, type 'first'")
preserve.info(
"preserve, type 'first', stage 'one'", method="preserve", type="second", stage="one"
)

# merging attributes: method 'clear'
clear = get_logger("Clear", {"type": "first", "stage": "one"})
clear.info("clear, type 'first', stage 'one'")
clear.info("clear, type 'second', no stage", method="clear", type="second")
clear.info("clear, no attributes", method="clear")

# merging attributes: method 'temp'
temp = get_logger("temp", {"type": "first"})
temp.info("type should be 'first'")
temp.info(
"type should be 'second' temporarily",
method="temp",
type="second",
)
temp.info("type should be back to 'first'")
```

#### Cleaning old logs
Expand Down
138 changes: 0 additions & 138 deletions examples/logging/logging_help.md
View file

This file was deleted.

7 changes: 0 additions & 7 deletions examples/logging/poc.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -83,13 +83,6 @@
level.info("does not print")
level.warning("does print")


# alias 'setup'
setup = get_logger("Setup", {"stage": "one", "type": "first"})
setup.info("stage one, type first")
setup = setup.setup(stage="two")
setup.info("stage two, no type")

# replacing log.label() - we want to update the log attributes permanently - same as
# overwrite
label = get_logger("label", {"stage": "whatever"})
Expand Down
36 changes: 16 additions & 20 deletions sysdata/data_blob.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,9 @@ def __init__(
.... sets up the following equivalencies:
data.broker_contract_price = ibFuturesContractPriceData(ib_conn, log=log.setup(component="IB-price-data"))
data.db_futures_contract_price = arcticFuturesContractPriceData(mongo_db=mongo_db,
log=log.setup(component="arcticFuturesContractPriceData"))
data.db_futures_contract = mongoFuturesContractData(mongo_db=mongo_db,
log = log.setup(component="mongoFuturesContractData"))
data.broker_contract_price = ibFuturesContractPriceData(ib_conn)
data.db_futures_contract_price = arcticFuturesContractPriceData(mongo_db=mongo_db)
data.db_futures_contract = mongoFuturesContractData(mongo_db=mongo_db)
This abstracts the precise data source
Expand All @@ -50,11 +48,9 @@ def __init__(
.... sets up the following equivalencies. This is useful if you are copying from one source to another
data.ib_contract_price = ibFuturesContractPriceData(ib_conn, log=log.setup(component="IB-price-data"))
data.arctic_futures_contract_price = arcticFuturesContractPriceData(mongo_db=mongo_db,
log=log.setup(component="arcticFuturesContractPriceData"))
data.mongo_futures_contract = mongoFuturesContractData(mongo_db=mongo_db,
log = log.setup(component="mongoFuturesContractData"))
data.ib_contract_price = ibFuturesContractPriceData(ib_conn)
data.arctic_futures_contract_price = arcticFuturesContractPriceData(mongo_db=mongo_db)
data.mongo_futures_contract = mongoFuturesContractData(mongo_db=mongo_db)
Expand Down Expand Up @@ -133,9 +129,9 @@ def _add_ib_class(self, class_object):
except Exception as e:
class_name = get_class_name(class_object)
msg = (
"Error %s couldn't evaluate %s(self.ib_conn, self, log = self.log.setup(component = %s)) This might be because (a) IB gateway not running, or (b) import is missing\
"Error %s couldn't evaluate %s(self.ib_conn, self) This might be because (a) IB gateway not running, or (b) import is missing\
or (c) arguments don't follow pattern"
% (str(e), class_name, class_name)
% (str(e), class_name)
)
self._raise_and_log_error(msg)

Expand All @@ -148,10 +144,10 @@ def _add_mongo_class(self, class_object):
except Exception as e:
class_name = get_class_name(class_object)
msg = (
"Error '%s' couldn't evaluate %s(mongo_db=self.mongo_db, log = self.log.setup(component = %s)) \
"Error '%s' couldn't evaluate %s(mongo_db=self.mongo_db) \
This might be because import is missing\
or arguments don't follow pattern"
% (str(e), class_name, class_name)
% (str(e), class_name)
)
self._raise_and_log_error(msg)

Expand All @@ -164,10 +160,10 @@ def _add_arctic_class(self, class_object):
except Exception as e:
class_name = get_class_name(class_object)
msg = (
"Error %s couldn't evaluate %s(mongo_db=self.mongo_db, log = self.log.setup(component = %s)) \
"Error %s couldn't evaluate %s(mongo_db=self.mongo_db) \
This might be because import is missing\
or arguments don't follow pattern"
% (str(e), class_name, class_name)
% (str(e), class_name)
)
self._raise_and_log_error(msg)

Expand All @@ -182,10 +178,10 @@ def _add_parquet_class(self, class_object):
except Exception as e:
class_name = get_class_name(class_object)
msg = (
"Error '%s' couldn't evaluate %s(parquet_access = self.parquet_access, log = self.log.setup(component = %s)) \
"Error '%s' couldn't evaluate %s(parquet_access = self.parquet_access) \
This might be because import is missing\
or arguments don't follow pattern or parquet_store is undefined"
% (str(e), class_name, class_name)
% (str(e), class_name)
)
self._raise_and_log_error(msg)

Expand All @@ -200,10 +196,10 @@ def _add_csv_class(self, class_object):
except Exception as e:
class_name = get_class_name(class_object)
msg = (
"Error %s couldn't evaluate %s(datapath = datapath, log = self.log.setup(component = %s)) \
"Error %s couldn't evaluate %s(datapath = datapath) \
This might be because import is missing\
or arguments don't follow pattern"
% (str(e), class_name, class_name)
% (str(e), class_name)
)
self._raise_and_log_error(msg)

Expand Down
Loading

0 comments on commit 132570b

Please sign in to comment.