Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Account tests #528

Draft
wants to merge 116 commits into
base: basic_buy_bot
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
482403c
Expose `.accounting.load_account()`
goodboy Jun 28, 2023
c780164
Fix test to use new `load_account()` location
goodboy Jun 28, 2023
87d6115
Add src asset name ignore via `MktPair._fqme_without_src: bool`
goodboy Jun 29, 2023
75f01e2
Drop `Position.expiry`, delegate to `.mkt: MktPair`
goodboy Jun 29, 2023
3ff9fb3
clearing._messages: add todo to drop the `BrokedPosition` msg
goodboy Jun 29, 2023
9748b22
Always include the src asset for (parquet file names) for fiat pairs
goodboy Jun 29, 2023
c0929c0
ib: fix `Client.trades()` return type annot
goodboy Jun 29, 2023
10ebc85
ib: fully handle `MktPair.src` and `.dst` in ledger loading
goodboy Jun 29, 2023
745c144
ib.feed: handle fiat (forex) pairs with `Asset`
goodboy Jun 29, 2023
05af2b3
Rework `.accounting.Position` calcs to prep for `polars`
goodboy Jul 3, 2023
0050232
Add a symbology cache subsys
goodboy Jul 5, 2023
c8c28df
Much (much) better symbology cache refinements
goodboy Jul 6, 2023
309b916
Finally, support full `MktPair` + `Asset` msgs
goodboy Jul 7, 2023
13f231b
Decode cached mkts and assets back to structs B)
goodboy Jul 7, 2023
3994fd8
Also handle `Decimal` interchange in `MktPair` msg-ification
goodboy Jul 7, 2023
ddc5f2b
Use `MktPair.from_msg()` in symcache
goodboy Jul 7, 2023
520414a
Oof, fix `.size` tick msg encode..
goodboy Jul 7, 2023
0e94e89
Finally, just drop `Transaction.sym`
goodboy Jul 7, 2023
f5d4f58
`Account` api update and refine
goodboy Jul 8, 2023
8f1983f
Move df loading into `calc.load_ledger_dfs()`
goodboy Jul 10, 2023
3704e2c
Call `open_ledger_dfs()` for `disect` sub-cmd
goodboy Jul 10, 2023
749401e
.accounting: expose new names at pkg top level
goodboy Jul 10, 2023
ff26789
Change cached-client hit msg to runtime level
goodboy Jul 10, 2023
87185cf
Drop `config` get/set/del apis..
goodboy Jul 10, 2023
8f40e52
Add handy `DiffDump`ing for our `.types.Struct`
goodboy Jul 10, 2023
c9681d0
.nativedb: ignore an `expired/` subdir
goodboy Jul 10, 2023
3c84ac3
binance.venues: add pair-type specific asset keying
goodboy Jul 10, 2023
19be834
binance.api: add venue qualified symcache support
goodboy Jul 10, 2023
a2c6749
binance.feed: use `Client.get_assets()` for mkt pairs
goodboy Jul 10, 2023
55c3d61
brokers.core: open cached client before hitting `.get_mkt_info()`
goodboy Jul 10, 2023
4123c97
Add symcache support to paper eng
goodboy Jul 10, 2023
243821a
Bleh! Ok make `open_symcache()` and `@acm`..
goodboy Jul 10, 2023
8330b36
User/return explicit `symcache` var name in sync case
goodboy Jul 10, 2023
14d5b3c
Be pedantic in `open_trade_ledger()` from sync code
goodboy Jul 10, 2023
ddcdbce
Use `acnt` instead of `table` for ref name B)
goodboy Jul 10, 2023
108e8c7
.accounting: expose `open_account()` at subsys pkg level
goodboy Jul 10, 2023
06c581b
Async enter/open the symcache in paper engine
goodboy Jul 10, 2023
8b94942
Don't verify the history step period for now in `tsdb_backfill()`
goodboy Jul 10, 2023
c30d8ac
ib: port to new `.accounting` APIs
goodboy Jul 10, 2023
7f4884a
data.types.Struct.to_dict(): discard non-member struct by default
goodboy Jul 12, 2023
da206f5
Store "namespace path" for each backend's pair struct
goodboy Jul 13, 2023
494b3fa
Formalize transaction normalizer func signature
goodboy Jul 14, 2023
803f4a6
Add first account cumsize test; known to fail Bo
goodboy Jul 14, 2023
b9fec09
Allow accounting (file) dir override via kwarg
goodboy Jul 15, 2023
69314e9
Passthrough all **kwargs `Struct.to_dict()`
goodboy Jul 15, 2023
3d20490
Move cum-calcs to `open_ledger_dfs()`, always parse `str`->`Datetime`
goodboy Jul 15, 2023
d794afc
Adjust `.clearing._paper_engine.norm_trade()` to new sig
goodboy Jul 15, 2023
a5821ae
binance: spec `.ns_path: str` on pair structs
goodboy Jul 15, 2023
4c55073
kraken: be symcache compatible!
goodboy Jul 16, 2023
430309b
.accounting: type `Transaction.etype` as a `Literal`
goodboy Jul 17, 2023
e8025d0
.data.types.Struct: by default include non-members from `.to_dict()`..
goodboy Jul 17, 2023
2dab0e2
Expose `.data._symcache` stuff at subpkg toplevel
goodboy Jul 17, 2023
8fb6676
Open symcaches as part of per-backend search spawning
goodboy Jul 17, 2023
71d0097
Switch to `Position.cumsize` in tracker and order mode mods
goodboy Jul 17, 2023
82fd785
Adjust default `[binance]` config to use paper and disable testnets
goodboy Jul 17, 2023
912f1bc
.kraken: start new `.symbols` submod and move symcache and search stu…
goodboy Jul 17, 2023
dfa13af
Allow backends to "bypass" symcache loading
goodboy Jul 17, 2023
e4731ef
Fix `Position.expiry == None` bug
goodboy Jul 17, 2023
a05a824
Log a warning on no symcache support in a backend
goodboy Jul 17, 2023
9e87b65
ib: be symcache compat by using bypass attr
goodboy Jul 17, 2023
fe78277
ib: add new `.symbols` sub-mod
goodboy Jul 17, 2023
8a10cbf
Change `Position.clearsdict()` -> `.clearsitems()`
goodboy Jul 19, 2023
5eb310c
ib: more fixes to try and get positioning correct..
goodboy Jul 19, 2023
5e7916a
Start `piker.toolz` subpkg for all our tooling B)
goodboy Jul 20, 2023
e88913e
.data._pathops: drop profiler imports, fix some naming to appease `ruff`
goodboy Jul 20, 2023
759ebe7
Allow disabling symcache load via kwarg as well
goodboy Jul 20, 2023
897c20b
Moar `.accounting` tweaks
goodboy Jul 22, 2023
50b221f
ib: rework client-internal contract caching
goodboy Jul 25, 2023
b33be86
ib: fill out contract tables in `.get_mkt_info()`
goodboy Jul 25, 2023
e344bdb
ib: rework trade handling, take ib position sizes as gospel
goodboy Jul 25, 2023
1d35747
Always clear `Position._events` in `.from_msg()`..
goodboy Jul 26, 2023
bebc817
Partition ledger data frames by `bs_mktid`
goodboy Jul 26, 2023
1885085
Utilize the new `_mktmap_table` input in paper engine
goodboy Jul 26, 2023
d0f72bf
Wrap symcache loading into `.from_scratch()`
goodboy Jul 26, 2023
9fbb75c
Remove piker.trionics; already factored into `tractor`
goodboy Jul 26, 2023
58cf7ce
Add `norm_trade()` ep to validator warnings
goodboy Jul 26, 2023
4436342
Change ui stuff to use new `Position.cumsize` attr name
goodboy Jul 26, 2023
c00cf41
kraken: `norm_trade()` now much accept an optional symcache
goodboy Jul 26, 2023
618c461
binance: always upper case venue and expiry tokens
goodboy Jul 26, 2023
bd0af7a
kucoin: facepalm, use correct pair fields for price/size ticks
goodboy Jul 26, 2023
64329d4
Flip `tractor.breakpoint()`s to new `.pause()`
goodboy Jul 26, 2023
7802feb
Backfill history gaps with pre-gap close
goodboy Jul 26, 2023
f1289cc
ib: Oof, right need to create ledger entries too..
goodboy Jul 26, 2023
2c6ae5d
Drop the `gap_dt_unit: str` column
goodboy Jul 26, 2023
08e8990
Do single `ShmArray.array` read on zero-time filtering
goodboy Jul 26, 2023
d94ab9d
order_mode: Only send cancels for dialogs that still exist
goodboy Jul 26, 2023
3855612
Add gap detection into the `store ldshm` cmd
goodboy Jul 26, 2023
b1edaf0
First draft position accounting with `polars`
goodboy Jul 30, 2023
b37a447
Implement PPU and BEP and inject the ledger frames
goodboy Jul 31, 2023
a088ebf
Use inf row/col repr for debugging atm
goodboy Jul 31, 2023
100be54
data.history: add TODO for non-zero epochs and some typing
goodboy Jul 31, 2023
5d24b5d
Swap branch order for enter/exit
goodboy Jul 31, 2023
85ae180
Factor df conversion into lone routine: `ledger_to_dfs()`
goodboy Jul 31, 2023
29bab02
Pass sync code flag in flex report processor
goodboy Aug 1, 2023
b6a7058
Handle txn costs in BEP, factor enter/exit blocks and df row assignme…
goodboy Aug 1, 2023
1e3a4ca
Drop commented, now deprecated edge case notes :surfer:
goodboy Aug 1, 2023
7ecf2bd
Guess exit transaction costs for BEP prediction
goodboy Aug 2, 2023
fff610f
Fix `PositionTracker.pane` attr resolve bug..
goodboy Aug 2, 2023
94ebe1e
Add some new hotkey maps for chart zoom and pane hiding
goodboy Aug 3, 2023
a51a610
Drop `virt_cost: str` from df output
goodboy Aug 3, 2023
ae444d1
Add note about `xonsh.main.main()` attempted usage
goodboy Aug 3, 2023
e9dfd28
ib: add back `src/dst` parsing for fiat pairs
goodboy Aug 3, 2023
60751ac
Officially drop `Position.size`
goodboy Aug 3, 2023
e4ea7d6
Lul, fix `open_ledger_dfs()` to `yield` when ledger passed in..
goodboy Aug 3, 2023
5d86d33
Parametrize account names for offline ledger tests
goodboy Aug 3, 2023
5ed8544
Bleh, move `.data.types` back up to top level pkg
goodboy Aug 5, 2023
eba6a77
Add paper-engine cost simulation support
goodboy Aug 7, 2023
85a38d0
Factor cumsize sign to var
goodboy Aug 7, 2023
ff2bbd5
ib: handle order errors via `reqid` lookup
goodboy Aug 7, 2023
562d027
Relay brokerd errors to client side, correctly..
goodboy Aug 10, 2023
f66a1f8
ib: relay submission errors, allow adhoc mkt overrides
goodboy Aug 10, 2023
78178c2
Add example mtr prober from `mtrpacket`
goodboy Aug 10, 2023
077d9bf
Better commenting around order-mode error block
goodboy Aug 10, 2023
c5ed6e6
Facepalm: remove now unused `CostModel` idea..
goodboy Aug 11, 2023
4aa04e1
Add note about broadcast when no `.symbol` found
goodboy Aug 11, 2023
6e8d078
Pkg with `poetry`, `poetry2nix` and a `flake.nix`
goodboy Aug 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions config/brokers.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@
# ---- CEXY ----
################
[binance]
accounts.paper = 'paper'

accounts.usdtm = 'futes'
futes.use_testnet = true
futes.use_testnet = false
futes.api_key = ''
futes.api_secret = ''

accounts.spot = 'spot'
spot.use_testnet = true
spot.use_testnet = false
spot.api_key = ''
spot.api_secret = ''

Expand Down
85 changes: 85 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# NOTE: to convert to a poetry2nix env like this here are the
# steps:
# - install poetry in your system nix config
# - convert the repo to use poetry using `poetry init`:
# https://python-poetry.org/docs/basic-usage/#initialising-a-pre-existing-project
# - then manually ensuring all deps are converted over:
{
description = "piker: trading gear for hackers (pkged with poetry2nix)";

inputs.flake-utils.url = "github:numtide/flake-utils";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
inputs.poetry2nix = {
url = "github:nix-community/poetry2nix";
inputs.nixpkgs.follows = "nixpkgs";
};

outputs = {
self,
nixpkgs,
flake-utils,
poetry2nix,
}:
flake-utils.lib.eachDefaultSystem (system:
let
# use PWD as sources
projectDir = ./.;
pyproject = ./pyproject.toml;
poetrylock = ./poetry.lock;

# TODO: port to 3.11 and support both versions?
python = "python3.10";

# see https://github.com/nix-community/poetry2nix/tree/master#api
# for more functions and examples.
# inherit
# (poetry2nix.legacyPackages.${system})
# mkPoetryApplication;
# pkgs = nixpkgs.legacyPackages.${system};
pkgs = nixpkgs.legacyPackages.x86_64-linux;

in
{
# let
# devEnv = poetry2nix.mkPoetryEnv {
# projectDir = ./.;
# };

packages = {
piker = poetry2nix.mkPoetryEditablePackage {
# env = poetry2nix.mkPoetryEnv {

# NOTE: taken from surrounding inputs
# projectDir = projectDir;
editablePackageSources = { piker = ./piker; };

# override msgspec to include setuptools as input
# https://github.com/nix-community/poetry2nix/blob/master/docs/edgecases.md#modulenotfounderror-no-module-named-packagenamed
overrides = poetry2nix.defaultPoetryOverrides.extend
(self: super: {
msgspec = super.msgspec.overridePythonAttrs
(
old: {
buildInputs = (old.buildInputs or [ ]) ++ [ super.setuptools ];
}
);
}
);

};
};


# boot xonsh inside the poetry virtualenv when
# define the custom entry point via an expected
# output-attr that `nix-develop` scans for:
# https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-develop.html#flake-output-attributes
devShells.default = pkgs.mkShell {
# packages = [ poetry2nix.packages.${system}.poetry ];
packages = [ poetry2nix.packages.x86_64-linux.poetry ];
shellHook = "poetry run xonsh";
# shellHook = "poetry shell";
};
}
);
}
16 changes: 16 additions & 0 deletions piker/accounting/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.accounting
-----------
A subsystem for transaction processing, storage and historical
measurement.


.pnl
----
BEP, the break even price: the price at which liquidating
a remaining position results in a zero PnL since the position was
"opened" in the destination asset.

PPU: price-per-unit: the "average cost" (in cumulative mean terms)
of the "entry" transactions which "make a position larger"; taking
a profit relative to this price means that you will "make more
profit then made prior" since the position was opened.
40 changes: 15 additions & 25 deletions piker/accounting/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,21 @@
'''
from ..log import get_logger

from ._ledger import (
from .calc import (
iter_by_dt,
)
from ._ledger import (
Transaction,
TransactionLedger,
open_trade_ledger,
)
from ._pos import (
load_pps_from_ledger,
Account,
load_account,
load_account_from_ledger,
open_pps,
open_account,
Position,
PpTable,
)
from ._mktinfo import (
Asset,
Expand All @@ -47,22 +51,25 @@
Allocator,
)


log = get_logger(__name__)

__all__ = [
'Account',
'Allocator',
'Asset',
'MktPair',
'Position',
'PpTable',
'Symbol',
'Transaction',
'TransactionLedger',
'dec_digits',
'digits_to_dec',
'iter_by_dt',
'load_pps_from_ledger',
'load_account',
'load_account_from_ledger',
'mk_allocator',
'open_account',
'open_pps',
'open_trade_ledger',
'unpack_fqme',
Expand All @@ -82,7 +89,7 @@ def get_likely_pair(

'''
try:
src_name_start = bs_mktid.rindex(src)
src_name_start: str = bs_mktid.rindex(src)
except (
ValueError, # substr not found
):
Expand All @@ -93,25 +100,8 @@ def get_likely_pair(
# log.warning(
# f'No src fiat {src} found in {bs_mktid}?'
# )
return
return None

likely_dst = bs_mktid[:src_name_start]
likely_dst: str = bs_mktid[:src_name_start]
if likely_dst == dst:
return bs_mktid


if __name__ == '__main__':
import sys
from pprint import pformat

args = sys.argv
assert len(args) > 1, 'Specifiy account(s) from `brokers.toml`'
args = args[1:]
for acctid in args:
broker, name = acctid.split('.')
trans, updated_pps = load_pps_from_ledger(broker, name)
print(
f'Processing transactions into pps for {broker}:{acctid}\n'
f'{pformat(trans)}\n\n'
f'{pformat(updated_pps)}'
)
10 changes: 4 additions & 6 deletions piker/accounting/_allocate.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@

from ._pos import Position
from . import MktPair
from ..data.types import Struct
from piker.types import Struct


_size_units = bidict({
Expand Down Expand Up @@ -118,9 +118,9 @@ def next_order_info(
ld: int = mkt.size_tick_digits

size_unit = self.size_unit
live_size = live_pp.size
live_size = live_pp.cumsize
abs_live_size = abs(live_size)
abs_startup_size = abs(startup_pp.size)
abs_startup_size = abs(startup_pp.cumsize)

u_per_slot, currency_per_slot = self.step_sizes()

Expand Down Expand Up @@ -213,8 +213,6 @@ def next_order_info(
slots_used = self.slots_used(
Position(
mkt=mkt,
size=order_size,
ppu=price,
bs_mktid=mkt.bs_mktid,
)
)
Expand All @@ -241,7 +239,7 @@ def slots_used(
Calc and return the number of slots used by this ``Position``.

'''
abs_pp_size = abs(pp.size)
abs_pp_size = abs(pp.cumsize)

if self.size_unit == 'currency':
# live_currency_size = size or (abs_pp_size * pp.ppu)
Expand Down
Loading