Build guide: validation

.cspell/holochain-words.txt

@@ -10,6 +10,7 @@ Holo's
 Holochain
 Holochain's
 Holonix
+kitsune
 Lightningrod
 memproof
 memproofs

.cspell/words-that-should-exist.txt

@@ -2,6 +2,8 @@
 affordance
 affordances
 automations
+backlink
+backlinks
 birthdate
 chrono
 composability

src/pages/_data/navigation/mainNav.json5

@@ -38,6 +38,9 @@
           { title: "Entries", url: "/build/entries/" },
           { title: "Links, Paths, and Anchors", url: "/build/links-paths-and-anchors/" },
         ]},
+        { title: "Validation", url: "/build/validation/", children: [
+          { title: "Genesis Self-Check Callback", url: "/build/genesis-self-check-callback/" },
+        ]},
       ]
     },
     { title: "Resources", url: "/resources/", children: [

src/pages/build/callbacks-and-lifecycle-hooks.md

@@ -21,44 +21,16 @@ The `validate` callback is called at two times:
 1. When an agent tries to author an [action](/build/working-with-data/#entries-actions-and-records-primary-data), and
 2. When an agent receives a [DHT operation](/concepts/4_dht/#a-cloud-of-witnesses) to store and serve as part of the shared database.
 
-The nature of validation is out of scope for this page (we'll write a page on it soon), but here's a very basic example of a validation callback that approves everything: <!-- TODO: remove this example when the validation page is written -->
-
-```rust
-use hdi::prelude::*;
-
-#[hdk_extern]
-pub fn validate(_: Op) -> ExternResult<ValidateCallbackResult> {
-    Ok(ValidateCallbackResult::Valid)
-}
-```
+The nature of validation is [a topic of its own](/build/validation/). Read the [`validate` callback page](/build/validate-callback/) to see examples.
 
 
 ### Define a `genesis_self_check` callback
 
-If your network needs to control who can and can't join it, you can write your DNA to require a [**membrane proof**](/resources/glossary/#membrane-proof), a small chunk of bytes that the app receives from the user at installation time and stores on the source chain. You can then write validation rules for this record just like any other record.
-
-There's one challenge with this: validation requires access to the network (a membrane proof may reference DHT data that contains a list of public keys authorized to grant access to newcomers, for instance), but the membrane proof is written at [**genesis**](/resources/glossary/#genesis-records) time when the cell doesn't have network access yet. So Holochain can't do the usual self-validation before publishing it.
-
-This means someone could accidentally type or paste a malformed membrane proof and be rejected from the network. To guard against this, you can define a `genesis_self_check` function that runs at genesis time and checks the content of the membrane proof before it's written.
+There's a moment in a cell's life, after it's been instantiated but before it's connected to its network, where it's published data that it can't fully validate. This data is their [**membrane proof**](/concepts/3_source_chain/#source-chain-your-own-data-store), an optional chunk of data that can serve as a joining code for the network. You can write validation rules for it just like any other record, allowing existing agents to reject newcomers with invalid credentials.
 
-`genesis_self_check` must take a single argument of type [`GenesisSelfCheckData`](https://docs.rs/hdi/latest/hdi/prelude/type.GenesisSelfCheckData.html) and return a value of type [`ValidateCallbackResult`](https://docs.rs/hdi/latest/hdi/prelude/enum.ValidateCallbackResult.html) wrapped in an `ExternResult`.
+This record can't be validated, however, because it's written to the source chain before the agent joins the network, and the `validate` callback can only be run after they've joined.
 
-Here's an example that checks that the membrane proof exists and is the right length: <!-- TODO: move this to the validation page too -->
-
-```rust
-use hdi::prelude::*;
-
-#[hdk_extern]
-pub fn genesis_self_check(data: GenesisSelfCheckData) -> ExternResult<ValidateCallbackResult> {
-    if let Some(membrane_proof) = data.membrane_proof {
-        if membrane_proof.bytes().len() == 32 {
-            return Ok(ValidateCallbackResult::Valid);
-        }
-        return Ok(ValidateCallbackResult::Invalid("Membrane proof is not the right length. Please check it and enter it again.".into()));
-    }
-    Ok(ValidateCallbackResult::Invalid("This network needs a membrane proof to join.".into()))
-}
-```
+However, if you write a `genesis_self_check` callback, it can guard against some basic user entry errors that would get an honest agent banned from a network. Read the [Genesis Self-Check Callback](/build/genesis-self-check-callback/) page for more info.
 
 ## Coordinator zomes
 

src/pages/build/dht-operations.md

@@ -0,0 +1,56 @@
+---
+title: "DHT operations"
+---
+
+::: intro
+A DHT operation is what an agent receives when they're being asked to store and serve a piece of data. It's a request for an agent to transform [their slice of the DHT](/concepts/4_dht/#finding-peers-and-data-in-a-distributed-database) into a new state. For this reason it's the input parameter to the [`validate` callback](/build/validate-callback/): if an agent is going to change their state at someone else's request, they need to know that the new state is correct.
+
+:::
+
+!!! info Validating actions instead of DHT operations
+In general, it's quite complex to validate your data using DHT operations, so we recommend letting the [scaffolding tool](/get-started/3-forum-app-tutorial/) generate a `validate` callback for you, which calls out to [stub functions](/build/validate-callback/#create-boilerplate-code-with-the-scaffolding-tool) that validate the _actions_ that create, update, or delete your application's entry and link types rather than the _operations_ that communicate those actions, and makes opinionated but sensible choices about which groups of authorities should validate what portion of the op data. The following information is presented for people who have a specific need to further reduce overall validation overhead across the network.
+!!!
+
+An [action](/build/working-with-data/#entries-actions-and-records-primary-data) on an agent's [source chain](/concepts/3_source_chain/) yields multiple DHT operations, each of which goes to an [**authority**](/resources/glossary/#validation-authority) for that operation's [**basis address**](/resources/glossary/#basis-address) (a DHT address that the authority is responsible for).
+
+Here are all the DHT operations produced for all the actions:
+
+* All actions
+    * [`RegisterAgentActivity`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.RegisterAgentActivity)
+        * Basis address: author's public key
+        * Contents: action (and optionally entry, if applicable)
+        * Effect: Store the action. Because one agent's actions will all go to the same set of authorities, their job is to check for [source chain forks](/resources/glossary/#fork-source-chain), which is a system-level rule that you don't have to implement.
+    * [`StoreRecord`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.StoreRecord)
+        * Basis address: action hash
+        * Contents: action (and optionally entry, if applicable)
+        * Effect: Store the action, along with any entry data.
+* [`Create`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.Create)
+    * [`StoreEntry`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.StoreEntry)
+        * Basis address: entry hash
+        * Contents: entry and action that wrote it
+        * Effect: Store the entry, if an identical entry hasn't been created yet, and add the action to the the list of actions associated with its creation. An entry can be created by multiple authors, and each creation action paired with its entry [can be treated as an independent piece of data](/build/entries/#entries-and-actions). **This operation isn't produced for private entries.**
+* [`Update`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.Update)
+    * `StoreEntry` (see above)
+    * [`RegisterUpdate`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.RegisterUpdate)
+        * Basis addresses: entry and action hashes of the _old_ entry being updated
+        * Contents: action and entry
+        * Effect: Mark an entry creation action as being replaced by a new one, pointing the the entry and action that replace it. **An entry and its creation action can have multiple actions updating them.**
+* [`Delete`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.Delete)
+    * [`RegisterDelete`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.RegisterDelete)
+        * Basis addresses: entry and action hashes of the entry being deleted
+        * Contents: action
+        * Effect: Mark an entry creation action as deleted, without removing the actual data. Because an entry can be created by multiple creation actions, the entry itself isn't marked as deleted until a `RegisterDelete` has been integrated for _all_ of its creation actions.
+* [`CreateLink`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.CreateLink)
+    * [`RegisterCreateLink`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.RegisterCreateLink)
+        * Basis address: link's [base address](/build/links-paths-and-anchors/#define-a-link-type)
+        * Contents: action
+        * Effect: Add a link to the list of links pointing from the base to other locations
+* [`DeleteLink`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.DeleteLink)
+    * [`RegisterDeleteLink`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/op/enum.Op.html#variant.RegisterCreateLink)
+        * Basis addresses: old link's [base address](/build/links-paths-and-anchors/#define-a-link-type) and action hash
+        * Contents: action
+        * Effect: Mark a link as deleted, without removing the actual data.
+
+### Choosing who should validate what
+
+In practice, it's usually okay to have all groups of authorities validate all the data in an action. However, if your validation logic is highly complex and computationally costly, it can sometimes be useful to choose different validation tasks for the different DHT operations. For instance, a `RegisterAgentActivity` authority may not need to validate entry data; it could just focus on the action in the context of the entire source chain. Or a `StoreEntry` authority could focus on entry contents and leave checking of create or update privileges to `StoreRecord` authorities.

src/pages/build/genesis-self-check-callback.md

@@ -0,0 +1,110 @@
+---
+title: "Genesis Self-Check Callback"
+---
+
+::: intro
+To enforce access control for a network, a DNA can require a [**membrane proof**](/concepts/3_source_chain/#source-chain-your-own-data-store), which is a piece of data that gets entered by the user and written to their [**source chain**](/concepts/3_source_chain/). The `genesis_self_check` function can guard against user entry error and help prevent them from being banned from the network accidentally.
+:::
+
+## Membrane proof: a per-agent joining code for a network
+
+If your network needs to enforce membership control, your [`validate` callback](/build/validate-callback/) can check the contents of the [`AgentValidationPkg`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.AgentValidationPkg) record. This [**genesis record**](/resources/glossary/#genesis-records) contains data that's entered by the user on app installation. If validators find an invalid proof, they can ban the unauthorized agent from joining the network.
+
+A membrane proof can be basic, like per-agent invite codes, or it can be something complex like a [JSON Web Token (JWT)](https://jwt.io/) signed by an agent that has authority to admit members.
+
+!!! info Membership control isn't implemented yet
+This feature is not fully implemented. Currently, validators merely record validation failure for a membrane proof and supply it on request. Our plan is to use membrane proof validation outcomes to admit or block communications from peers.
+!!!
+
+## The need for basic pre-validation
+
+Most useful membrane proofs will require access to network data in order to fully validate them. But an agent can't self-validate their own membership proof at genesis time, because they haven't joined the network yet. This creates a minor problem; they may accidentally type or paste their membrane proof wrong, but won't find out until they try to join the network and get ejected.
+
+To reduce the risk, you can define a `genesis_self_check` function that checks the membrane proof before network communications start. This function is limited --- it naturally doesn't have access to DHT data -- but it can be a useful guard against basic data entry errors.
+
+## Define a `genesis_self_check` callback
+
+`genesis_self_check` must take a single argument of type [`GenesisSelfCheckData`](https://docs.rs/hdi/latest/hdi/prelude/type.GenesisSelfCheckData.html) and return a value of type [`ValidateCallbackResult`](https://docs.rs/hdi/latest/hdi/prelude/enum.ValidateCallbackResult.html) wrapped in an `ExternResult`.
+
+Here's an example that checks that the membrane proof exists and is the right length:
+
+```rust
+use hdi::prelude::*;
+
+#[hdk_extern]
+pub fn genesis_self_check(data: GenesisSelfCheckData) -> ExternResult<ValidateCallbackResult> {
+    if let Some(membrane_proof) = data.membrane_proof {
+        if membrane_proof.bytes().len() == 32 {
+            return Ok(ValidateCallbackResult::Valid);
+        }
+        return Ok(ValidateCallbackResult::Invalid("Membrane proof is not the right length. Please check it and enter it again.".into()));
+    }
+    Ok(ValidateCallbackResult::Invalid("This network needs a membrane proof to join.".into()))
+}
+```
+
+This more complex example deserializes a JWT, checks that its [`sub` (subject) field](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.2) matches the agent's public key, and checks that its [`iss` (issuer) field](https://datatracker.ietf.org/doc/html/rfc7519#section-4.1.1) matches an authorized key hard-coded into the DNA's [`properties` block](/build/dnas/#use-dna-properties). {#jwt-membrane-proof}
+
+```rust
+use hdi::prelude::*;
+
+// A type for deserializing a JWT payload from our membrane proof.
+#[derive(Serialize, Deserialize)]
+pub struct MembraneProofJwtPayload {
+    pub iss: AgentPubKeyB64,
+    pub sub: AgentPubKeyB64,
+}
+
+// A type for deserializing the DNA properties that this integrity zome needs.
+#[dna_properties]
+pub struct DnaProperties {
+    membrane_proof_signing_authority: AgentPubKeyB64,
+}
+
+#[hdk_extern]
+pub fn genesis_self_check(data: GenesisSelfCheckData) -> ExternResult<ValidateCallbackResult> {
+    if let Some(membrane_proof) = data.membrane_proof {
+        // A JWT is three JSON objects, Base64 urlencoded, and concatenated
+        // with periods. Let's break it apart, decode, and deserialize it.
+        let parts: Vec<&str> = std::str::from_utf8(membrane_proof.bytes())
+            .map_err(|e| wasm_error!(e.to_string()))?
+            .split(".")
+            .collect();
+        let payload_encoded = parts[1];
+        let payload_decoded = base64_url::decode(payload_encoded)
+            .map_err(|e| wasm_error!(e.to_string()))?;
+        let payload_string = std::str::from_utf8(&payload_decoded)
+            .map_err(|e| wasm_error!(e.to_string()))?;
+        let payload: MembraneProofJwtPayload = serde_json::from_str(payload_string)
+            .map_err(|e| wasm_error!(e.to_string()))?;
+
+        // Now that we've got that, compare it against the public key of the
+        // agent submitting the membrane proof.
+        if payload.sub != data.agent_key.into() {
+            return Ok(ValidateCallbackResult::Invalid("Author's public key doesn't match membrane proof".into()));
+        }
+
+        // And against the DNA's authorized membrane proof signer.
+        if payload.iss != DnaProperties::try_from_dna_properties()?.membrane_proof_signing_authority {
+            return Ok(ValidateCallbackResult::Invalid("Membrane proof issuer is unrecognized".into()));
+        }
+    }
+    Ok(ValidateCallbackResult::Invalid("This network needs a membrane proof to join.".into()))
+}
+```
+
+## Fully validating a membrane proof
+
+Your `validate` callback also needs to validate a membrane proof in order to actually enforce network access. At the very least, it should apply the same rules as `genesis_self_check` does, and add on any checks that require DHT data. We'll explore this in full on the [`validate` callback](/build/validate-callback/#validate-agent-joining) page.
+
+## Reference
+
+* [`holochain_integrity_types::action::AgentValidationPkg`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/action/enum.Action.html#variant.AgentValidationPkg)
+* [`holochain_integrity_types::genesis::GenesisSelfCheckData`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/genesis/type.GenesisSelfCheckData.html)
+* [`holochain_integrity_types::validate::ValidateCallbackResult`](https://docs.rs/holochain_integrity_types/latest/holochain_integrity_types/genesis/type.GenesisSelfCheckData.html)
+* [`holochain_derive::dna_properties`](https://docs.rs/hdk_derive/latest/hdk_derive/attr.dna_properties.html)
+
+## Further reading
+
+* [Core Concepts: Genesis self-check](/concepts/7_validation/#genesis-self-check)
+* [Build Guide: Validation](/build/validation/)

src/pages/build/index.md

@@ -40,4 +40,11 @@ Now that you've got some basic concepts and the terms we use for them, it's time
 * [Identifiers](/build/identifiers) --- working with hashes and other unique IDs
 * [Entries](/build/entries/) --- defining, creating, reading, updating, and deleting data
 * [Links, Paths, and Anchors](/build/links-paths-and-anchors/) --- creating relationships between data
-:::
+:::
+
+## Validation
+
+::: topic-list
+* [Overview](/build/validation/) --- The purpose of validation in a hApp, abilities, requirements and constraints, determinism
+    * [Genesis Self-Check Callback](/build/genesis-self-check-callback/) --- Purpose and use cases
+    * [Validate Callback](/build/validate-callback/) --- Writing validation routines for various needs