| uip | title | description | author | status | type | category | created |
|---|---|---|---|---|---|---|---|
0121 |
%pine Request at Latest |
Add a way to request the latest scry path |
~rovnys-ricfer |
Draft |
Standards Track |
Arvo |
2024-03-21 |
Urbit needs to support some way for a client to efficiently obtain the most recent version of some value over the network. This functionality is a basic piece of most HTTP interactions -- the client sends a GET request for some data at a path, and the server responds using its latest state corresponding to that path -- but this is not yet supported in the scry namespace, and because the scry namespace is immutable, paths cannot be re-bound to new values, so request-at-latest is not an intrinsic feature.
This document proposes adding a non-referentially-transparent request type, called %pine, to Urbit's network protocol. A %pine request includes a scry path with no $case (revision number or date), since due to the ever-present potential for clock skew between any two machines, only the publishing ship knows its own latest date. A %pine request is resolved by a %pine response, which wraps a [path data] scry binding at a fully qualified scry path, whose $case was resolved by the publishing ship.
A %pine request is only valid if it uses a %w care (request type), whose response lists the relevant cases for a path (date and revision number).
There are many ways to design a %pine request system. This proposal is intended to satisfy the following properties:
- It requires only a small addition to existing protocols.
- It builds on existing Directed Messaging routing infrastructure and Pending Interest Tables.
- It does not introduce any potential for Byzantine faults in the scry namespace, i.e. conflicting bindings at the same path.
- It does not require any formal state on publishers (meaning no Arvo events).
- It establishes a clear distinction between mutable and immutable requests.
- It minimizes ambiguity regarding whether a
%pineresponse will persist long enough to be downloaded by a client.
A %pine request packet is labeled as such in the 32-bit header. This disambiguates it from %peek request packets, which include a fully qualified scry path and must be treated differently over the network.
When a publisher hears a %pine request packet, it interpolates the current date into the request's case-less scry path to obtain a fully qualified scry path. It peforms the scry request, then wraps the first packet of the scry response in a %pine response envelope to indicate that the response is intended to resolve a %pine request, rather than resolving a fully qualified %peek request. Note that a %pine scry response message will always fit in a single UDP packet with a 1500-byte MTU.
The %pine enveloping on request and response is used to prevent ambiguity between mutable and immutable responses on the publisher and relays. When a relay hears a %pine request packet, it adds that to a separate pending interest table just for %pines, not the PIT for immutable %peek requests. This way, when the %pine response arrives at the relay, the relay knows to use that response to resolve the %pine.
The %pine response packet is used to establish the case for a %pine request. When the requesting ship hears the %pine response packet, it remembers the case and uses that to request the value as an immutable %peek request. This memory could be transient in the runtime or injected into Arvo as an event. Either approach should be legitimate; injecting the event into Arvo is simpler.
Relays should not cache %pine responses, since they immediately become stale as soon as they are received. Any kind of time-to-live (TTL) system for %pine packets would complicate the model. It is better to have only packets that can always be cached and packets that can never be cached -- anything in between reestablishes the difficulties with cache invalidation that plague computer science.
To understand why the separation between %pine packets and other packets is needed, consider an alternative design that did not distinguish %pine responses from fully qualified responses, and the scenario in which two ships make requests from the same publisher. One is a %peek request for an old piece of data at a historical date, say a month in the past. The other request is a %pine. Both requests arrive at a relay in quick succession, before either one can be resolved. The first response packet that arrives is the response to the %peek at the old date. The relay would end up using this response to satisfy the %pine, yielding a misleading, stale result on the requesting ship.
Since the relay would have no way of knowing that the response packet was not intended to resolve the %pine request, it would just check for a simple caseless match on the paths and conclude that it should send the stale response back to the transport address that had sent the %pine request, and the sending ship would also have no way of knowing it was stale.
A previous version of this proposal would allow any piece of data from anywhere within Arvo to be retrievable over the network, as any path can be made referentially-transparent by including the current date.
One problem with this approach is that it places increased, informal responsibility for data availability on the runtime. A datum that is available at time T but not at time T' must then be "fully resolved" at time T. If such a datum is requested over the network via %pine, and it is too large to fit into a single packet (or larger than the conventional batch of packets), the runtime must make special efforts to retrieve and store all the packets of that datum in order to successfully serve the sequence of requests for it.
If %pine is instead restricted to only perform "path canonicalization", the responsibility for data availability stays with Arvo. A non-referentially-transparent "request-for-latest" merely results in the referentially-transparent identifier for the latest version, which can then be resolved normally. In practical terms this reduces the scope of %pine to "frontier discovery" of data explicitly published in the namespace.
New ships would need to be able to perform an explicit protocol negotiation to discover whether peer ships support %pine. This should not be particularly complex or unusual compared to other protocol upgrades.
A mechanism for rate-limiting %pine requests could be implemented to mitigate denial of service vulnerability from unauthenticated %pine requests. This could even implement an informal TTL without breaking the model.
Copyright and related rights waived via CC0.