[WIP] The Everything Refactor #2212

- message transmission routed through refactored handling - still work to be done, but now to make it compile at least

- almost all errors have been commented out for refactor or already refactored - committing this prior to sorting out the cmake structure - upcoming include-what-you-use application

- Get rid of CryptoManager. - Get rid of Crypto. - Move all the Crypto instance methods to llarp::crypto functions. (None of them needed to be methods at all, so this is simple). - Move sodium/ntru initialization into static initialization. - Add llarp::csrng, which is an available llarp::CSRNG instance which is a bit easier than needing to construct a `CSRNG rng{};` in various places. - Various related small simplifications/cleanups.

- includes are now sorted in consistent, logical order; first step in an attempt to fix the tomfoolery (no relation to Tom) brought in by include-what-you-use - shuffled around some cmake linking to simplify dependency graph - superfluous files removed

As long as it's loaded before oxen-mq or libquic, this should end up being the one that gets used in oxen-mq/libquic, and so we don't have to update those when we want something in a newer oxen-encoding.

- cleaned up include-what-you-use errors (mostly quoted includes) - misc fixes

They aren't useful right now.

Also deliberately don't provide the function at all if we aren't compiled with libcrypt so that we can't link if we try to call it when not available.

There are some bugfix releases, but also the upstream hash was wrong for the current release anyway (upstream changed it?).

- some weird function call business - string formatting - etc

This was causing cmake to double-invoke ccache for compilation units, and broke oxen-mq's local zmq build for the same reason.

Use backports for cmake because ngtcp2 needs 3.20+.

- Local and Remote RC's now implemented with discrete functionalities and uses

- RemoteRC supplants most of the functionality throughout the code of RouterContact - Next step will be to sort out CI issues, then see if we can get rid of either LocalRC (and therefore RouterContact entirely)

- .snodes don't need to support SRV records, so remove that - untangle the mess of captured lambdas capturing other lambdas capturing other lambdas; we still need a chain of nested lambdas because we have a chain of callbacked events, but hiding the nesting by capturing them in other lambdas didn't improve anything.

- the one addr to rule them all, and its name was oxen::quic::Address - no more vectors of inbound/outbound junk

there are a few TODOs which merit further discussion

- control messages can be sent along a path - the path owner onion-encrypts the "inner" message for each hop in the path - relays on the path will onion the payload in both directions, such that the terminal relay will get the plaintext "inner" message and the client will get the plaintext "response" to that. - control messages have (mostly, see below) been changed to be invokable either over a path or directly to a relay, as appropriate. TODO: - exit messages need looked at, so they have not yet been changed for this - path transfer messages (traffic from client to client over 2 paths with a shared "pivot") are not yet implemented

It seems RC refactor will obviate the need for a "get individual RC" method, so this comments out some usage of that to sidestep build errors, rather than correcting them in a way that will just be wasted.

change path control message inner message response to take just a string, which will be a bt-encoded response with an early key for status. If there is a timeout we pass a bt dict that only has that as the status, else the response we de-onioned should have either an OK status or some other error. change messages to use new status key correctly call Path::EnterState on path build response

Lots of code was using 32-byte nonces for xchacha20 symmetric encryption, but this just means 8 extra bytes per packet wasted as chacha is only using the first 24 bytes of that nonce anyway. Changing this resulted in a lot of dead/dying code breaking, so this commit also removes a lot of that (and comments a couple places with TODO instead) Also nounce -> nonce where it came up.

It's unnecessary abstraction that barely simplifies anything, and is now only used in one single place anyway, which is easily replaced with a (unabstracted) lambda.

also move messages' statuses into their own namespace

path build frames should be onioned at each hop to avoid a bad actor controlling two nodes in a path being able to know (with certainty, temporal correlation is hard to avoid) that they're hops on the same path. This is desirable as in the worst case someone could be your edge hop and terminal hop on a path, and now the terminal hop knows your IP making the path basically pointless.

also deserialize to unsigned string where possible/useful so to not have unnecessary reinterpret_casts all over the place.

RC "lookup" is being replaced with "gimme all recently updated RCs". As such, doing a lookup on a specific RC is going away, as is network exploration, so a lot of what RCLookupHandler was doing will no longer be relevant. Functionality from it which was kept has moved to NodeDB, as it makes sense for that functionality to live where the RCs live.

We're removing the notion of find/lookup a singular RC, so this gets rid of all functions which did that and replaces their usages with something sensible.

TODO: refactor or remove RCGossiper and revisit RC regen and when-to-gossip logic.

We will want some notion of "when did we receive it" for RCs (or RouterIDs, details tbd), but that will be per-source as a means to form some metric of consensus/trust on which relays are *actually* on the network. Clients don't have a blockchain daemon to pull this from, so they have to ask many relays for the full list of relays and form a trust model on that (bootstrapping problem notwithstanding).

Relays will now re-sign and gossip their RCs every 6 hours (minus a couple random minutes) using the new gossip_rc message. Removes the old RCGossiper concept

This command will be called periodically by clients to maintain a list of RCs of active relay nodes. It will require another command (future commit) to fetch the RouterIDs from many nodes and reconcile those so we have some notion of good-ness of the RCs we're getting; if we get what seems to be a bad set of RCs (this concept not yet implemented), we will choose a different relay to fetch RCs from. These are left as TODOs for now.

Periodically clients will fetch the set of RouterIDs for all relays on the network. It will request this list from a number (12, currently) of relays, but as we are likely to be requesting from more relays than we want to have edge connections, this request will itself be relayed to the target source via one of our edges. As we can't trust our edge to do this honestly, the responses are signed by the source relay. TODO: the responses from all (12) relays are collected, then processed together. The reconciliation of their responses is not yet implemented. TODO: the source selection for this method obviously requires sources to begin with, but this is the method by which we learn of those...bootstrapping is still a bit in-progress, and will need to be finished for this. TODO: make Router call this periodically, as with RC fetching.

- When receiving a request to fetch RouterID's, the remote endpoint fulfilling the request stores them in an unordered set. When the request caller receives that payload, it is loaded into a vector in the same order. However, we should just load it directly into an unordered set to enforce both the order and that none appear twice - The trust model will have to operate on multiple large lists of RouterID's and RC's efficiently, and maintaining a sort order ensures the values are workable immediately after deserialization

- initial/subsequent fetching combined for RouterContacts and RouterIDs - bootstraps fallback implemented and looped into fetch logic

- greedy evaluation of returned rid's, simplifying post-processing logic to simple frequency comparison per rid against a constant threshold - tidied up link_manager request/response handling - TODO: - review and decide thresholds - evaluate necessity and potential implementation of rc comparison

- bootstrap cooldown implemented with 1min timer in case all bootstraps fail - set comparison implemented in non-initial and non-bootstrap rc fetching; set comparison in rid fetching is done every fetch - nodedb get_random functions refactored into conditional/non-conditional methods. Conditional search implements reservoir sampling for one-pass accumulation of n random rcs

- disable reachability testing with config option; required to be done on testnet - reachability testing pipeline through link_manager executes pings similar to storage server. connection established hook reports successful reachability, while connection closed callback (with non-default error code) reports unsuccessful testing

- Once we have our set of returned rc's and accepted rid's (ones that were found locally), the remainder are placed in an "unconfirmed" state - Once there, they have five subsequent successful fetches to be found in request response, at which point their verification counter is incremented and their attempt counter is reset - If they appear three times, they are "promoted" and moved to our "known_{rid,rc}" list

- redoing link_manager functions again to implement previously ignored review comments on several PRs - conceptually merging "whitelist_routers" and new "known_{rids,rcs}", s.t. we can completely eliminate white/red/gray/green/etc lists in favor of something that isn't dumb

- Up and running locally, no connections yet - Next: flip testnet and do the gosh dang thing

- now pointing to jason/void-listen

- straightened out setting of public addr and public port in config vs listen ("bind") addr - fixed small bug in router contact writing and saving

Some of these were wrong (trying to use formats that didn't work); this updates them all to new-style logging.

This doesn't really have a purpose.

`deregister_peer` does the exact same thing as `close_connection` so just remove it. Also removes an unnecessary loop dispatch call (because we *have* to be in the logic thread already to be able to touch the variables we are touching before the dispatch).

- TODO: discuss authentication for nodes connecting to bootstrap seed - crit log num connected/RC's, conn open/closed, etc

- nodes now send their RC to the bootstrap seed on making a request - allows the bootstrap seed to distribute RCs

We are reinterpret_cast'ing the version to a string to send it as raw bytes, but it was sending \x00\x00\x09 instead of \x00\x09\x0a because the version constant was actually a uint16_t array. This just makes the version constant a uint8_t array instead so that it works (and I am not at all worried about any version component getting larger than 255).

from_string was trying to parse it as base32z.snode Also leave FIXMEs behind for the badly named methods (both in RouterID itself and in ancestor classes).

It is now called get_best_public_address, and takes (bool, port) argument to return an optional quic::Address to make life easier: the caller now can just give the default port to set, and we keep the C sockaddr* more constrained.

This is, apparently, the only way to move an element out of a std::set.

- pending conns container stops them from being counted towards active conns in the interim - un-abstracted pendingmessages vs pendingdatamessages vs pendingcontrolmessages (gross) - fixed bootstrap fetching and storage!

- implemented ngtcp2 ping to keep connections alive - fixed weird lambda captures - fetch logic - lets see what happens

- new btdc method used to ensure no junk at the end of our bt data - DRYed out the RC code - check inbound bootstraps against all registered routers, not just whitelist - libquic vbump

- try closing the connection via link::Endpoint like we probably should be doing.. - testing out dropping stream constructor, godspeed

It's a gross implementation, and even if it wasn't, using it takes more code than not using it.

If given and true then return nullopt if we don't find the requested number; otherwise return them even if there aren't as many as we requested.

- laying the groundwork for functional client->service node connections. this requires ALPNs verification as a secondary method of identification to the remote key - refactored btreq stream creation to use improved stream creation logic in libquic

- pending_conns removed in favor of direct creation of link::Connection objects in link::Endpoint::{service,client}_conn containers - conn lookup maps removed, they were pointless

- rework bootstrap loading to move all logic into BootstrapList object - ability to parse lists and dicts of bootstraps implemented - netid parsing refers to the correct fallback - cross your fingers boys here we go

If you start without -DWITH_SETCAP=OFF, but then later re-run cmake, SETCAP is still set (and so still gets tried). This fixes it.

- removed superfluous typedefs obfuscating what is actually happening - Builder -> PathBuilder; next is moving PathSet into PathBuilder - enum -> enum class where appropriate - ran linter

- misc libquic fixes squashed into this - thread-safe wrapping

- When two relays are repeatedly attempting connections to one another simultaneously, the connection initiated by the RouterID that appears first (in lexicographical order) is deferred to. The connection initiated by the other endpoint is marked to close quietly (w/o executing callbacks), and is rejected in the TLS verification hook - Bypassing callback execution is critical, as it will clean-up the link::Connection object for the connection that is being deferred to; this results in BOTH connections being destroyed.

- Deprecate pathset, smashed into PathBuilder (renamed to PathHandler) - Re-abstraction of sessions and PathHandlers - Renamed PathBuilder -> PathHandler to more accurately reflect purpose - {Service,Exit}Handler will remain as PathHandlers, though currently no path-sharing amongst sessions being managed is to be implemented. Handlers will maintain their own paths for both lookups and initiating sessions, while sessions will manage their paths independantly. - Session object handling necessitates the differentiation between outbound and inbound sessions. Initiators of sessions are entirely responsible for dictating the chosen path on which the session communicates, and must therefore continually build and manage paths for the negotiated session. - Outbound sessions are now {Service,Exit}Sessions - Inbound sessions are implemented with InboundSession, which is agnostic to the type of service being operated locally (service vs exit, client vs relay, etc). When the Session initiator signals a switch to a different path, it will be assigned to the InboundSession object by {Service,Exit}Endpoint, which manages local services and exits

- brought up libev loop implementation from libquic - this will streamline deprecating the old IP_packet, UDPHandle, and other types

- old types almost entirely deprecated (ip_packet, sockaddr, etc) - need to touch base with the team to discuss packet routing and the like

- C++ bumped 17 -> 20 - clang-format (plus all clangs) -> 17

- eliminated {client,relay}address - better loops in new key types

- there is one key difference between the address classes: ClientAddress can be constructed with an ONS name, and can therefore exist in one of 2 dual states (pubkey vs ONS)

- cleaned up config parsing using the better Address/IP types and quic::Address where appropriate - created new templated address_map using concepts constraining local types to quic::Address and IPRange, and remote types to {Client,Relay}Address - now the correct local addresses and ranges can be passed to the correct handlers and endpoints for remote and local exit/service operation - basically fully configured ::RemoteHandler - deleted deprecated object files previously removed from compilation

- combined {exit,service}::Endpoint into handlers::LocalEndpoint - combined {exit,service}::Handler into handlers::RemoteHandler - BaseSession subsumed {Client,Relay}Session -> OutboundSession - Session initialization code underway

- It became increasingly tedious to partition the address types between {Client,Relay}Address, as a client can operate an exit or hidden service while a relay could also be operating a hidden service. The client's hidden service would terminate in a `.loki` while the relay's would terminate in a `.snode`, complicating the ONS logic implemented in ClientAddress - Address types were partitioned into {Network,Relay}Address; NetworkAddress conceptually encapsulates all remote hidden services (agnostic of client vs relay) and exit nodes being operated on the network: RelayAddress specifically designates an addressible relay to be used in scopes excluding hidden services and exit nodes

- Session initiation code laid out from path build to successful session object creation - NullEndpoint renamed to more apt descriptor

- Path building logic in ::PathHandler is broken up into 3 functions. This allows paths built outside of the usual PathHandler::Tick() logic (ex: paths for initiating sessions) to implement custom logic and path handling - RemoteHandler now uses segmented build logic to create a path upon session initiation without storing the created path internally

- if a hop can generate the symmetric key and decrypt its respective payload, then the concept of checking a hash is somewhat pointless - hmac generation and validation removed

- DRYed out symm. enc. in path-building and session initiation - onwards to unifying session initiation/path-build into one-shot

- correctly looks up introset locally, stores on successful request-response if not found locally - fetched introset used for recursive path-build through pivots - serialize/deserialize logic DRYs out implementation - session init handling laid out

- LocalEndpoint and RemoteHandler properly configured using values from ::NetworkConfig - ONS range mappings passed to config now resolved by RemoteHandler at earliest possible phase in Router::run()

- LocalEndpoint logic for regenerating and publishing introset - 'publish_intro' performed over path messages: to be revised upon evaluating determinism in dht key comparison with every relay holding the entire network's RC's at all time - surrounding logic left unchanged to be revised as per previous bullet -- can also likely deprecate "order" field with more massaging - futher deprecation of "is_relayed" bool field would require different endpoints for relays to call

- TunEndpoint and LocalEndpoint prefiguring - Fully removed service::Address and all uses of AddressVariant_t in favor of the improved NetworkAddress type - Fully removed exit/* and most of service/*

- Fixed final bencode.h dependencies in AlignedBuffer that was pending refactor of other types - Last traces of any non-oxenc bencode are done - Next is TCP listeners!

- Was able to repurpose a lot of the code from the ill-fated ev-dns implementation - TCP{Handle,Socket} implemented to parallelize UDP{Handle,Socket} in functionality - TCP implementation takes in a callback that receives an IPPacket; can also be easily reconfigured to hanle a UDPPacket by simply changing the callback type and the constructor called in llarp/ev/tcp.cpp::tcp_read_cb()

- link::TunnelManager will be re-implemented in link::QUICTunnel to make and accept TCP connections - Will be simplified using Libevent and hooked into the new LocalEndpoint/RemoteHandler model - TCP{Handle,Socket} ownership model will be revised to more monadic model; this will allow the TCPSockets (sockets representing accepted connections) to function entirely independantly of the bound socket

- manual routing - Network ctor from existing Loop

- EmbeddedEndpoint removed for solo implementation of TunEndpoint for all instances except non-embedded clients - Deprecated TransitHopInfo, TransitHopID; absorbed into TransitHop - Reworked PathContext internal storage of TransitHops, both mutators/setters/getters - Straightened out success conditions of path building - Session initiation and path respective path building code is almost done being laid out - Full steam ahead buddy

- Session initiation/handling session initiation request logic finished; still to loop in TCP logic and handlers - remove SessionAuthPolicy from OutboundSession - auth token mapping passed to RemoteHandler; ONS auth mappings resolved with ONS range mappings - auth_static_tokens passed to LocalEndpoint; received auth verify passed tokens against that

- TCP handles and connections are now looped into session initiation and reception - I'm tired

- TCP backend is differentiated in Inbound vs Outbound sessions, as one listens while the other connects - These pathways are mroe discretely implemented, and the logic may move into the derived classes instead - Now to add hooks to connect TCP server/client to backend (app or system TUN)

- LocalEndpoint/RemoteHandler combined into SessionEndpoint for ambidextrous session management - Data message routing in/out of paths implemented using callbacks for receiving datagrams differently for embedded vs full clients - Tunneled QUIC endpoints (no-UDP endpoints) implemented with TCP client/server endpoints on embedded/full clients - Lots of surrounding work on the above points

- libquic bumped for needed ip methods - bumped nlohmann, cpr, CLI11, oxen-encoding, pybind11, sqlite_orm - span-lite, ghc-filesystem, and uvw removed after shift to C++20

- Tunnel IP assignment requires robust IP assignment - With the correct implementation of max_ip/next_ip retrieval from ip and ip_range objects, we can now implement IPRangeIterator to manage this

- TUN endpoint assigns a local IP to route traffic for inbound/outbound sessions - Config mappings pre-loaded and superside ip iteration

- Packets coming in and out of full clients need their IP src/dest addresses re-written - Lokinet's TUN device sucks up all packets aimed at the if range, correctly addressing them to the proper outbound - On reception, lokinet's TUN device addresses correctly for the mapped application receiving the packet - Brought a lil somethin' somethin' from libquic for CI - Bump oxenmq to cpp20 branch - Revert cpr to 1.9 stable for subproject curl dep bug

… bumps

- libevent loop ticker now handles all execution of `Router::tick()`

- Formatting concept to enable oxen-logging printing for custom objects was updated with libquic vbump to require boolean member variable - Members added in base objects to be inherited through

Commits on Feb 4, 2024

libquic testing out verbose bparser stuff

dr7ana committed Feb 4, 2024

Configuration menu

View commit details

Copy full SHA for 5004fbc

Browse repository at this point

Copy the full SHA

5004fbc View commit details

Browse the repository at this point in the history

Commits on Feb 6, 2024

deprecated old types from config

dr7ana committed Feb 6, 2024

Configuration menu

View commit details

Copy full SHA for e495637

Browse repository at this point

Copy the full SHA

e495637 View commit details

Browse the repository at this point in the history

Commits on Feb 8, 2024

bump libquic to expose ev loop

dr7ana committed Feb 8, 2024

Configuration menu

View commit details

Copy full SHA for f541d0b

Browse repository at this point

Copy the full SHA

f541d0b View commit details

Browse the repository at this point in the history

Commits on Mar 4, 2024

libquic bump

dr7ana committed Mar 4, 2024

Configuration menu

View commit details

Copy full SHA for 44e7c1b

Browse repository at this point

Copy the full SHA

44e7c1b View commit details

Browse the repository at this point in the history

Commits on Mar 27, 2024

libquic c++20/ev-loop/ip bump

dr7ana committed Mar 27, 2024

Configuration menu

View commit details

Copy full SHA for 6a20b3d

Browse repository at this point

Copy the full SHA

6a20b3d View commit details

Browse the repository at this point in the history

Commits on Jun 4, 2024

testing something on CI

dr7ana committed Jun 4, 2024

Configuration menu

View commit details

Copy full SHA for d455a36

Browse repository at this point

Copy the full SHA

d455a36 View commit details

Browse the repository at this point in the history

Commits on Jun 19, 2024

libquic bump

dr7ana committed Jun 19, 2024

Configuration menu

View commit details

Copy full SHA for 52ceb26

Browse repository at this point

Copy the full SHA

52ceb26 View commit details

Browse the repository at this point in the history

Commits on Aug 28, 2024

so much better!

dr7ana committed Aug 28, 2024

Configuration menu

View commit details

Copy full SHA for ab30bf2

Browse repository at this point

Copy the full SHA

ab30bf2 View commit details

Browse the repository at this point in the history

Commits on Aug 29, 2024

Misc (squash)

dr7ana committed Aug 29, 2024

Configuration menu

View commit details

Copy full SHA for b794430

Browse repository at this point

Copy the full SHA

b794430 View commit details

Browse the repository at this point in the history

Commits on Aug 30, 2024

kick

dr7ana committed Aug 30, 2024

Configuration menu

View commit details

Copy full SHA for ca7fc07

Browse repository at this point

Copy the full SHA

ca7fc07 View commit details

Browse the repository at this point in the history

Commits on Sep 9, 2024

testing this out

dr7ana committed Sep 9, 2024

Configuration menu

View commit details

Copy full SHA for 104ba6d

Browse repository at this point

Copy the full SHA

104ba6d View commit details

Browse the repository at this point in the history

Commits on Sep 12, 2024

libquic -> dr7ana/0rtt, etc

dr7ana committed Sep 12, 2024

Configuration menu

View commit details

Copy full SHA for ea6d62c

Browse repository at this point

Copy the full SHA

ea6d62c View commit details

Browse the repository at this point in the history

Commits on Sep 13, 2024

updated, may kill testnet but yolo

dr7ana committed Sep 13, 2024

Configuration menu

View commit details

Copy full SHA for e3d80ea

Browse repository at this point

Copy the full SHA

e3d80ea View commit details

Browse the repository at this point in the history

Commits on Sep 17, 2024

squash, but more the zucchini type

dr7ana committed Sep 17, 2024

Configuration menu

View commit details

Copy full SHA for ec32167

Browse repository at this point

Copy the full SHA

ec32167 View commit details

Browse the repository at this point in the history

Commits on Sep 18, 2024

maybe a little green beans too

dr7ana committed Sep 18, 2024

Configuration menu

View commit details

Copy full SHA for 5b522e6

Browse repository at this point

Copy the full SHA

5b522e6 View commit details

Browse the repository at this point in the history

Commits on Sep 19, 2024

any beets on the side?

dr7ana committed Sep 19, 2024

Configuration menu

View commit details

Copy full SHA for c874387

Browse repository at this point

Copy the full SHA

c874387 View commit details

Browse the repository at this point in the history

Commits on Sep 24, 2024

whatsup paths

dr7ana committed Sep 24, 2024

Configuration menu

View commit details

Copy full SHA for 8495517

Browse repository at this point

Copy the full SHA

8495517 View commit details

Browse the repository at this point in the history

Commits on Sep 25, 2024

Better linux fd poller

dr7ana committed Sep 25, 2024

Configuration menu

View commit details

Copy full SHA for 3103194

Browse repository at this point

Copy the full SHA

3103194 View commit details

Browse the repository at this point in the history

[WIP] The Everything Refactor #2212

Are you sure you want to change the base?

[WIP] The Everything Refactor #2212

Commits on Jan 31, 2024

Commits on Feb 1, 2024

Commits on Feb 2, 2024

Commits on Feb 4, 2024

Commits on Feb 5, 2024

Commits on Feb 6, 2024

Commits on Feb 7, 2024

Commits on Feb 8, 2024

Commits on Feb 12, 2024

Commits on Feb 14, 2024

Commits on Mar 4, 2024

Commits on Mar 17, 2024

Commits on Mar 18, 2024

Commits on Mar 19, 2024

Commits on Mar 22, 2024

Commits on Mar 27, 2024

Commits on Mar 29, 2024

Commits on Apr 2, 2024

Commits on Apr 3, 2024

Commits on Apr 4, 2024

Commits on Apr 8, 2024

Commits on Apr 9, 2024

Commits on Apr 10, 2024

Commits on Apr 12, 2024

Commits on Apr 17, 2024

Commits on Apr 18, 2024

Commits on Apr 19, 2024

Commits on Apr 22, 2024

Commits on Apr 24, 2024

Commits on May 28, 2024

Commits on May 29, 2024

Commits on Jun 2, 2024

Commits on Jun 3, 2024

Commits on Jun 4, 2024

Commits on Jun 14, 2024

Commits on Jun 19, 2024

Commits on Jun 25, 2024

Commits on Aug 14, 2024

Commits on Aug 16, 2024

Commits on Aug 23, 2024

Commits on Aug 28, 2024

Commits on Aug 29, 2024

Commits on Aug 30, 2024

Commits on Sep 3, 2024

Commits on Sep 4, 2024

Commits on Sep 9, 2024

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 24, 2024

Commits on Sep 25, 2024