Merge pull request #74 from confio/docs-extension-development

WIP: Document extension development

docs/_static/css/custom.css

@@ -1,3 +1,7 @@
+.highlight {
+    background-color: #ffffff;
+}
+
 .rst-content code.literal {
     color: #1569C7;
     font-size: 80%;

docs/design/extensions.rst

@@ -1,73 +1,82 @@
-----------------------
-Extension Design (WIP)
-----------------------
-
-**State: Proposal**
-
-This is a basic set of information on how to design an extension.
-
-External Interface (Transactions)
-=================================
-
-Define a set of transactions that the module supports.
-These should be generic as well, so another module may
-support the same external calls, even with a very
-different implementation.
-
-**TODO**
-
-Internal Calls
-==============
-
-Extensions need to call between each other, to trigger actions
-in other "smart contracts", or query other state. This can be
-done by importing them directly and linking them directly,
-which is simple but rigid. Instead, we recommend to encapsulate
-all internal calls we wish to make in an interface, which should
-be passed in the constructor of the handler. And exporting
-an object with methods that expose all functions we wish to
-provide to other extensions.
-
-When composing the application (in the main loop), we can
-wire the extensions together as needed, while allowing the
-compiler to verify that all needed functionality is properly
-provided. A middleground between static-compile time resolution,
-and dynamic run-time resolution.
-
-**TODO** (This is like Controller???)
-
-Persistent Models
-=================
-
-We suggest to create a ``.proto`` file defining the ``Msg`` structure
-for any messages that this extension supports, as well as all data
-structures that are supposed to be persisted in the merkle store.
-
-**TODO: talk about ORM**
-
-Wiring it up
-============
-
-We have 4 ways to connect this logic to the framework:
-
-Handler, Decorator (Context), Ticker, Init
-
-**TODO**
-
-Flexible Composition
-====================
-
-How to connect them together?
-
-We discussed a `flexible auth framework <./extensions.rst>`.
-Here are some thoughts on how to flexibly tie together.
-
-Using interfaces and inheritance... Allowing extensions to demand
-functionality for what they link to, but not restricting the
-implementation. This is a pattern that can be layered upon
-the description in "Internal Calls" to allows for even more
-flexibility in system composition.
-
-**TODO: with examples**
-
-
+----------------
+Extension Design
+----------------
+
+`Confio weave <https://github.com/confio/weave>`_ doesn't just
+produce a ``mycoind`` executable, but was designed from the
+group up to be extremely flexible for many common use cases.
+One can easily extend the functionality of ``mycoind``
+by adding more *extensions* on top of it, like
+`bcp-demo <https://github.com/iov-one/bcp-demo>`_
+or you can chose not to import any of the modules of
+mycoind and just use the building blocks to build an
+entirely different system (like utxo chain).
+
+Note that even pieces as fundamental as
+`signature valdation <https://github.com/confio/weave/tree/master/x/sigs>`_
+or `isolating failed transactions <https://github.com/confio/weave/tree/blob/x/utils/savepoint.go>`_ are implemented as importable modules and wired up
+together when you `construct the application <https://github.com/confio/weave/tree/blob/examples/mycoind/app/app.go#L39-54>`_.
+
+Extension Functionality
+=======================
+
+Most code to extend the basic framework is packaged as *extensions* and
+stored in packages under the `x` directory. One can ignore all code
+under `x` and still build a complete weave-compatible application,
+these are just some pieces that many chains would like to reuse.
+You can import these extensions, or write you own, with the same
+power and functionality.
+
+When you write a weave-based application (covered in the next section),
+you will likely want to create a few new extensions to add new
+functionality tied into why your chain is unique. What types of
+behavior can you customize?
+
+* Handler - process transactions, maintain local state, control state transitions. Sending coins is an example of a Handler
+* Decorator (aka Middleware) - do some pre-processing and update the context with information to influence eventual Handler. Signature validation is an example of a Decorator.
+* Initer - Provide a function to initialize the data store based on a section of the app_state in the genesis file
+* Handle ABCI calls - ``app.BaseApp`` implements ``abci.Application`` and you
+can wrap it with a pure ABCI Middleware to do things like record timing information on the ``Commit`` or ``Info`` calls, without forking the code.
+
+App Framework
+=============
+
+If ``weave`` allows you to customize everything, what does it provide?
+
+**ORM** - the ``orm`` package wraps up the merkle tree, provable kv-store provided by `tendermint iavl <https://github.com/tendermint/iavl>`_ and adds convenient features on top, like ``CacheWrap`` to isolate read/writes of a transaction before deciding to Write or Discard, and provides type-safe data storage and secondary indexes if you write to an ``orm.Bucket``. In fact, even if you want to build your own framework from scratch, take a look about using ``orm`` and ``iavl`` together to provide storage
+
+**ABCI Adapter** the ``app`` package builds on top of the ``orm`` to
+provide default implementations for many of the abci calls, and parses
+the other ones to allow us to handle requests easier with internal functions.
+One one hand, it adapts the format, so we can do things like locally
+return changes to the validator set during ``DeliverTx`` (when we calculate
+it), but return the final change on ``EndBlock`` (when tendermint
+expects the response). It also handles routing transactions and queries
+to various modules rather than one monolith. This package demonstrates
+where most of the main interfaces are used for.
+
+**Handlers and Decorators** A *Handler* defines what actions to perform
+in response to a transaction in either CheckTx or DeliverTx. The ``app``
+package also allows us to ``ChainDecorators`` and register multiple
+``Handlers`` on a ``Router`` to separate processing logic based
+on the contents of the transaction.
+
+**Error Handling** The ``errors`` package provides some nice helpers
+to produce error return values that conform to both ``pkg/errors``
+(allowing a full stack trace in testing or deployment using
+``fmt.Printf("%+v", err)``), as well as maintaining an ABCI error code
+to be returned on the result. We can pass around typical ``error``
+objects in the code, which work well with debugging, logging,
+and the ABCI interface.
+
+**Serialization Standard** Weave defines simple
+`Marshaller <https://github.com/confio/weave/blob/master/tx.go#L21-L28>`_ and
+`Persistent <https://github.com/confio/weave/blob/master/tx.go#L30-L41>`_ interface standards. These interfaces are automatically
+implemented by any code autogenerated by protoc from protobuf files.
+This allows us to easily define serialization in an efficient and
+extremely portable manner (protobuf has support in every major
+language and platform). However, the interfaces don't force protobuf
+and you can define these two methods on any object to provide
+a custom serialization format as desired. This should allow interoperability
+with the Application and Handler code with any serialization library
+you wish to use.

docs/design/permissions.rst

@@ -63,7 +63,7 @@ multiple authentication schemes. For example, if we want to
 design HTLC, we could add an optional "Preimage" to the
 Tx structure. We add a "Hasher" middleware that hashes
 this preimage, and then grants the condition of something like
-``preimage/<hash of preimage>``. This is stored in the context
+``hash/sha256/<hash of preimage>``. This is stored in the context
 and the "Hasher" exports an Authenticator that allows
 access to this.
 

docs/design/queries.rst

@@ -23,9 +23,15 @@ ABCI Format
         Prove  bool
     }
 
-The request uses ``Height`` to select which tree to query and ``Prove``
-to determine if we should also return merkle proofs for the
-response. The actual data that we wish to read is declared in ``Path``
+The request uses ``Height`` to select which block height to query and
+``Prove`` to determine if we should also return merkle proofs for the
+response. Note that "block height to query" is shorthand for "query the
+state produced after executing all transactions included in all blocks
+up to *and including* ``Height``". For various reasons internal to
+tendermint, the root hash of this state is included as ``AppHash``
+in the block at ``Height+1``.
+
+The actual data that we wish to read is declared in ``Path``
 and ``Data``. ``Path`` defines what kind of query this is, much like the
 path in an http get request. ``Data`` is an arbitrary argument. In
 the typical case, ``Path = /key`` and ``Data = <key bytes>`` to directly
@@ -52,8 +58,8 @@ non-zero only when there is an error in processing the query.
 info. ``Index`` *may* be the location of this key in the merkle tree,
 but that is not well defined.
 
-Now to the important ones. ``Height``, as above, the the version of
-the tree we queried and is always set, even if the query had 0 to
+Now to the important ones. ``Height``, as above, the the block height
+of the tree we queried and is always set, even if the query had 0 to
 request "most recent". ``Key`` is the key of the merkle tree we got,
 ``Value`` is the value stored at that key (may be empty if there
 is no value), and ``Proof`` is a merkle proof in an undefined format.
@@ -107,7 +113,9 @@ Path: ``/wallets?range``, Data: ``complex type to be defined``:
 
 Note that if we have a numeric index, the range query could be
 easily be used to generate ``<``, ``<=``, ``>``, ``>=``, and
-``BETWEEN`` queries over those values.
+``BETWEEN`` queries over those values. Range is currently
+not implemented as of weave v0.4.1, but will be added as
+soon as there is a clear use case.
 
 Weave Response Types
 ====================
@@ -153,6 +161,12 @@ another method ``RegisterQueries``.
 Proofs
 ======
 
+**Proofs are not yet implemented as of weave v0.4.1**
+This is both due to prioritization of other features,
+and also as we wish to provide a solid proof format that is
+useful for IBC as well, and watching cosmos-sdk development
+so we can maintain some compatibility.
+
 As a primative to build up proofs, we define a generic ``ProofPath``
 data type that contains a merkle proof from a ``key:value`` pair to
 a root hash. That root hash can be tied externally to a hash

docs/index.rst

@@ -85,14 +85,32 @@ and the various components you will use
    design/extensions.rst
 
 
-Backend Development
--------------------
+Backend Development Tutorial
+----------------------------
+
+To make this theory more tangible, we will build a sample
+application alongside this tutorial, to demonstrate dealing
+with real-world constraints. The application is located in the
+`examples/tutorial package <https://github.com/confio/weave/tree/master/examples/tutorial>`_
+in the weave repository, but it is designed to be self-contained
+and could just as easily live in an external repo that imported
+weave.
+
+In this tutorial, you will learn how to serialize and model
+you data strucutres, define messages and handlers, expose
+queries, and read initial configuration from the genesis file.
+You will be able to build a new extension and tie it together
+with other extensions into a complete blockchain application.
 
-A step by step example of writing your first
-application built on top of mycoind.
-This is all about writing go code that runs as an ABCI app.
-We will write a new extension and compile an application
-that builds upon ``mycoind``.
+.. toctree::
+   :maxdepth: 2
 
-**TODO**
+   tutorial/protobuf.rst
+   tutorial/datamodel.rst
+   tutorial/messages.rst
+   tutorial/handler.rst
+   tutorial/queries.rst
+   tutorial/init.rst
+   tutorial/integration.rst
+   tutorial/advanced.rst
 

docs/tutorial/advanced.rst

@@ -0,0 +1,11 @@
+-----------------------
+Advanced Customizations
+-----------------------
+
+**TODO**
+
+Describe some possible advanced use cases:
+
+* BeginBlock - triggering delayed actions
+* Validator Diffs - how to manage that
+* Custom Decorators - why and how