From b70f286f79cd160c4b26301183b26ff10cdd777e Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Sun, 17 Jun 2018 14:36:09 +0200 Subject: [PATCH 01/19] A bit of cleanup --- docs/design/permissions.rst | 2 +- docs/design/queries.rst | 26 ++++++++++++++++++++------ 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/design/permissions.rst b/docs/design/permissions.rst index f64385a2..30574d07 100644 --- a/docs/design/permissions.rst +++ b/docs/design/permissions.rst @@ -63,7 +63,7 @@ multiple authentication schemes. For example, if we want to design HTLC, we could add an optional "Preimage" to the Tx structure. We add a "Hasher" middleware that hashes this preimage, and then grants the condition of something like -``preimage/``. This is stored in the context +``hash/sha256/``. This is stored in the context and the "Hasher" exports an Authenticator that allows access to this. diff --git a/docs/design/queries.rst b/docs/design/queries.rst index 873c9f77..342d7bf2 100644 --- a/docs/design/queries.rst +++ b/docs/design/queries.rst @@ -23,9 +23,15 @@ ABCI Format Prove bool } -The request uses ``Height`` to select which tree to query and ``Prove`` -to determine if we should also return merkle proofs for the -response. The actual data that we wish to read is declared in ``Path`` +The request uses ``Height`` to select which block height to query and +``Prove`` to determine if we should also return merkle proofs for the +response. Note that "block height to query" is shorthand for "query the +state produced after executing all transactions included in all blocks +up to *and including* ``Height``". For various reasons internal to +tendermint, the root hash of this state is included as ``AppHash`` +in the block at ``Height+1``. + +The actual data that we wish to read is declared in ``Path`` and ``Data``. ``Path`` defines what kind of query this is, much like the path in an http get request. ``Data`` is an arbitrary argument. In the typical case, ``Path = /key`` and ``Data = `` to directly @@ -52,8 +58,8 @@ non-zero only when there is an error in processing the query. info. ``Index`` *may* be the location of this key in the merkle tree, but that is not well defined. -Now to the important ones. ``Height``, as above, the the version of -the tree we queried and is always set, even if the query had 0 to +Now to the important ones. ``Height``, as above, the the block height +of the tree we queried and is always set, even if the query had 0 to request "most recent". ``Key`` is the key of the merkle tree we got, ``Value`` is the value stored at that key (may be empty if there is no value), and ``Proof`` is a merkle proof in an undefined format. @@ -107,7 +113,9 @@ Path: ``/wallets?range``, Data: ``complex type to be defined``: Note that if we have a numeric index, the range query could be easily be used to generate ``<``, ``<=``, ``>``, ``>=``, and -``BETWEEN`` queries over those values. +``BETWEEN`` queries over those values. Range is currently +not implemented as of weave v0.4.1, but will be added as +soon as there is a clear use case. Weave Response Types ==================== @@ -153,6 +161,12 @@ another method ``RegisterQueries``. Proofs ====== +**Proofs are not yet implemented as of weave v0.4.1** +This is both due to prioritization of other features, +and also as we wish to provide a solid proof format that is +useful for IBC as well, and watching cosmos-sdk development +so we can maintain some compatibility. + As a primative to build up proofs, we define a generic ``ProofPath`` data type that contains a merkle proof from a ``key:value`` pair to a root hash. That root hash can be tied externally to a hash From 1784b02684bbebeec8dcb6f9f5a29d3a78a829f9 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Sun, 17 Jun 2018 15:38:29 +0200 Subject: [PATCH 02/19] Write intro to extension design --- docs/design/extensions.rst | 155 ++++++++++++++++++++----------------- 1 file changed, 82 insertions(+), 73 deletions(-) diff --git a/docs/design/extensions.rst b/docs/design/extensions.rst index 07210067..3985408e 100644 --- a/docs/design/extensions.rst +++ b/docs/design/extensions.rst @@ -1,73 +1,82 @@ ----------------------- -Extension Design (WIP) ----------------------- - -**State: Proposal** - -This is a basic set of information on how to design an extension. - -External Interface (Transactions) -================================= - -Define a set of transactions that the module supports. -These should be generic as well, so another module may -support the same external calls, even with a very -different implementation. - -**TODO** - -Internal Calls -============== - -Extensions need to call between each other, to trigger actions -in other "smart contracts", or query other state. This can be -done by importing them directly and linking them directly, -which is simple but rigid. Instead, we recommend to encapsulate -all internal calls we wish to make in an interface, which should -be passed in the constructor of the handler. And exporting -an object with methods that expose all functions we wish to -provide to other extensions. - -When composing the application (in the main loop), we can -wire the extensions together as needed, while allowing the -compiler to verify that all needed functionality is properly -provided. A middleground between static-compile time resolution, -and dynamic run-time resolution. - -**TODO** (This is like Controller???) - -Persistent Models -================= - -We suggest to create a ``.proto`` file defining the ``Msg`` structure -for any messages that this extension supports, as well as all data -structures that are supposed to be persisted in the merkle store. - -**TODO: talk about ORM** - -Wiring it up -============ - -We have 4 ways to connect this logic to the framework: - -Handler, Decorator (Context), Ticker, Init - -**TODO** - -Flexible Composition -==================== - -How to connect them together? - -We discussed a `flexible auth framework <./extensions.rst>`. -Here are some thoughts on how to flexibly tie together. - -Using interfaces and inheritance... Allowing extensions to demand -functionality for what they link to, but not restricting the -implementation. This is a pattern that can be layered upon -the description in "Internal Calls" to allows for even more -flexibility in system composition. - -**TODO: with examples** - - +---------------- +Extension Design +---------------- + +`Confio weave `_ doesn't just +produce a ``mycoind`` executable, but was designed from the +group up to be extremely flexible for many common use cases. +One can easily extend the functionality of ``mycoind`` +by adding more *extensions* on top of it, like +`bcp-demo `_ +or you can chose not to import any of the modules of +mycoind and just use the building blocks to build an +entirely different system (like utxo chain). + +Note that even pieces as fundamental as +`signature valdation `_ +or `isolating failed transactions `_ are implemented as importable modules and wired up +together when you `construct the application `_. + +Extension Functionality +======================= + +Most code to extend the basic framework is packaged as *extensions* and +stored in packages under the `x` directory. One can ignore all code +under `x` and still build a complete weave-compatible application, +these are just some pieces that many chains would like to reuse. +You can import these extensions, or write you own, with the same +power and functionality. + +When you write a weave-based application (covered in the next section), +you will likely want to create a few new extensions to add new +functionality tied into why your chain is unique. What types of +behavior can you customize? + +* Handler - process transactions, maintain local state, control state transitions. Sending coins is an example of a Handler +* Decorator (aka Middleware) - do some pre-processing and update the context with information to influence eventual Handler. Signature validation is an example of a Decorator. +* Initer - Provide a function to initialize the data store based on a section of the app_state in the genesis file +* Handle ABCI calls - ``app.BaseApp`` implements ``abci.Application`` and you +can wrap it with a pure ABCI Middleware to do things like record timing information on the ``Commit`` or ``Info`` calls, without forking the code. + +App Framework +============= + +If ``weave`` allows you to customize everything, what does it provide? + +**ORM** - the ``orm`` package wraps up the merkle tree, provable kv-store provided by `tendermint iavl `_ and adds convenient features on top, like ``CacheWrap`` to isolate read/writes of a transaction before deciding to Write or Discard, and provides type-safe data storage and secondary indexes if you write to an ``orm.Bucket``. In fact, even if you want to build your own framework from scratch, take a look about using ``orm`` and ``iavl`` together to provide storage + +**ABCI Adapter** the ``app`` package builds on top of the ``orm`` to +provide default implementations for many of the abci calls, and parses +the other ones to allow us to handle requests easier with internal functions. +One one hand, it adapts the format, so we can do things like locally +return changes to the validator set during ``DeliverTx`` (when we calculate +it), but return the final change on ``EndBlock`` (when tendermint +expects the response). It also handles routing transactions and queries +to various modules rather than one monolith. This package demonstrates +where most of the main interfaces are used for. + +**Handlers and Decorators** A *Handler* defines what actions to perform +in response to a transaction in either CheckTx or DeliverTx. The ``app`` +package also allows us to ``ChainDecorators`` and register multiple +``Handlers`` on a ``Router`` to separate processing logic based +on the contents of the transaction. + +**Error Handling** The ``errors`` package provides some nice helpers +to produce error return values that conform to both ``pkg/errors`` +(allowing a full stack trace in testing or deployment using +``fmt.Printf("%+v", err)``), as well as maintaining an ABCI error code +to be returned on the result. We can pass around typical ``error`` +objects in the code, which work well with debugging, logging, +and the ABCI interface. + +**Serialization Standard** Weave defines simple +`Marshaller `_ and +`Persistent `_ interface standards. These interfaces are automatically +implemented by any code autogenerated by protoc from protobuf files. +This allows us to easily define serialization in an efficient and +extremely portable manner (protobuf has support in every major +language and platform). However, the interfaces don't force protobuf +and you can define these two methods on any object to provide +a custom serialization format as desired. This should allow interoperability +with the Application and Handler code with any serialization library +you wish to use. From 0b0a9b73e59da2a0aa804efa82d929728045f041 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Sun, 17 Jun 2018 16:03:07 +0200 Subject: [PATCH 03/19] Started adding developer doc section --- docs/development/advanced.rst | 11 +++++++++++ docs/development/datamodel.rst | 10 ++++++++++ docs/development/handler.rst | 9 +++++++++ docs/development/init.rst | 5 +++++ docs/development/integration.rst | 5 +++++ docs/development/messages.rst | 9 +++++++++ docs/development/protobuf.rst | 10 ++++++++++ docs/development/queries.rst | 5 +++++ docs/index.rst | 12 ++++++++++++ 9 files changed, 76 insertions(+) create mode 100644 docs/development/advanced.rst create mode 100644 docs/development/datamodel.rst create mode 100644 docs/development/handler.rst create mode 100644 docs/development/init.rst create mode 100644 docs/development/integration.rst create mode 100644 docs/development/messages.rst create mode 100644 docs/development/protobuf.rst create mode 100644 docs/development/queries.rst diff --git a/docs/development/advanced.rst b/docs/development/advanced.rst new file mode 100644 index 00000000..06d963f9 --- /dev/null +++ b/docs/development/advanced.rst @@ -0,0 +1,11 @@ +----------------------- +Advanced Customizations +----------------------- + +**TODO** + +Describe some possible advanced use cases: + +* BeginBlock - triggering delayed actions +* Validator Diffs - how to manage that +* Custom Decorators - why and how diff --git a/docs/development/datamodel.rst b/docs/development/datamodel.rst new file mode 100644 index 00000000..f979aeea --- /dev/null +++ b/docs/development/datamodel.rst @@ -0,0 +1,10 @@ +----------------------- +Defining the Data Model +----------------------- + +**TODO** + +* Key-Value pairs -> select primary key +* Using a Bucket to store an object +* Adding validation and helpers +* Adding secondary indexes diff --git a/docs/development/handler.rst b/docs/development/handler.rst new file mode 100644 index 00000000..eca84891 --- /dev/null +++ b/docs/development/handler.rst @@ -0,0 +1,9 @@ +---------------- +Message Handlers +---------------- + +**TODO** + +* Writing Handler for one message +* Routing messages to Handler +* Check vs Deliver diff --git a/docs/development/init.rst b/docs/development/init.rst new file mode 100644 index 00000000..8a5bc839 --- /dev/null +++ b/docs/development/init.rst @@ -0,0 +1,5 @@ +------------- +Initial State +------------- + +**TODO** diff --git a/docs/development/integration.rst b/docs/development/integration.rst new file mode 100644 index 00000000..949a8268 --- /dev/null +++ b/docs/development/integration.rst @@ -0,0 +1,5 @@ +----------------- +Tying it together +----------------- + +**TODO** diff --git a/docs/development/messages.rst b/docs/development/messages.rst new file mode 100644 index 00000000..cdbc2c9f --- /dev/null +++ b/docs/development/messages.rst @@ -0,0 +1,9 @@ +----------------- +Defining Messages +----------------- + +**TODO** + +* Messages vs. Transactions, what is the distinction? +* Defining message types - needed data, non-maleability +* Validation of Messages diff --git a/docs/development/protobuf.rst b/docs/development/protobuf.rst new file mode 100644 index 00000000..6620bbbf --- /dev/null +++ b/docs/development/protobuf.rst @@ -0,0 +1,10 @@ +--------------------- +Using Protobuf Codecs +--------------------- + +**TODO** + +* How to create .proto files +* How to compile them (with weave Makefile) +* How to use compiled code +* Comment about oneof... diff --git a/docs/development/queries.rst b/docs/development/queries.rst new file mode 100644 index 00000000..ead4d116 --- /dev/null +++ b/docs/development/queries.rst @@ -0,0 +1,5 @@ +------------------ +Processing Queries +------------------ + +**TODO** diff --git a/docs/index.rst b/docs/index.rst index 8c3d93f5..404fc839 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -96,3 +96,15 @@ that builds upon ``mycoind``. **TODO** +.. toctree:: + :maxdepth: 2 + + development/protobuf.rst + development/datamodel.rst + development/messages.rst + development/handler.rst + development/queries.rst + development/init.rst + development/integration.rst + development/advanced.rst + From 9bd986eaede485dd1e6ffe9b1ce0e472bc3cf983 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Tue, 19 Jun 2018 21:31:48 +0200 Subject: [PATCH 04/19] detailed protobuf usage instructions --- docs/development/protobuf.rst | 299 +++++++++++++++++++++++++++++++++- 1 file changed, 294 insertions(+), 5 deletions(-) diff --git a/docs/development/protobuf.rst b/docs/development/protobuf.rst index 6620bbbf..8162de5a 100644 --- a/docs/development/protobuf.rst +++ b/docs/development/protobuf.rst @@ -2,9 +2,298 @@ Using Protobuf Codecs --------------------- -**TODO** +Weave defines simple +`Marshaller `_ and +`Persistent `_ interface standards. These interfaces are automatically +implemented by any code autogenerated by protoc from protobuf files, +and we recommend using .proto files to specify the serialization +format for any persistent data in our application (internal state +as well as transactions and messages). However, if you have never +worked with protobuf, this might be a bit of a challenge, so we +explain a simple workflow that we use in weave based projects. + +Create Proto File +================= + +The first thing is to imagine the shape of your classes. +These should be defined in `proto3 (TODO) `_ syntax. +There are a number of different int encodings, byte slices, +strings, and nested structures. And fields may be repeated. +So forget complex types with methods now and just focus on +the actual data structure. The `x/codec.proto `_ file defines the Coin type rather simply, +once you remove the comments, this is all that is left: + +:: + + syntax = "proto3"; + + package x; + + message Coin { + int64 whole = 1; // default: 0 + int64 fractional = 2; // default: 0 + string ticker = 3; + string issuer = 4; // optional + } + +Or the `app/results.proto `_ file, that defines an array of byte slices: + +:: + + syntax = "proto3"; + + package app; + + // ResultSet contains a list of keys or values + message ResultSet { + repeated bytes results = 1; + } + +Note that the package defined in the protobuf file must match the +package name used by the golang code in the same directory. + +You can also import types from one proto file into another. +Make sure to use the full github path in order that the generated +go code has properly working imports. The package name above is +also used as a namespace for the imported protobuf definitions. +This is how `x/cash `_ creates a wallet that contains an +array of tokens of different currencies. + +:: + + syntax = "proto3"; + + package cash; + + import "github.com/confio/weave/x/codec.proto"; + + // Set may contain Coin of many different currencies. + // It handles adding and subtracting sets of currencies. + message Set { + repeated x.Coin coins = 1; + } + +Compiling Proto Files +===================== + +To compile protobuf files, you need to have the `protoc (TODO) `_ +binary installed and a language-specific translator +(`gogo-protobuf `_ in this case). +This can be a bit of a pain, especially the first time, so the default +weave `Makefile < https://github.com/confio/weave/blob/master/Makefile>`_ +contains some helpers for you. + +* ``make prototools`` will install all the needed tools, perform once to set up your machine +* ``make protoc`` will compile all the _.proto_ files in the repo + +If you are building a repo based on weave, you are invited to copy the +bottom part of the Makefile and just copy the ``make prototools`` logic +verbatim. Let's take a look at the second phase, as this is the one you +will have to modify when you add a new protobuf file, either to weave +or to your own repo. + +:: + + protoc: + protoc --gogofaster_out=. app/*.proto + protoc --gogofaster_out=. crypto/*.proto + protoc --gogofaster_out=. orm/*.proto + protoc --gogofaster_out=. x/*.proto + protoc --gogofaster_out=. -I=. -I=$(GOPATH)/src x/cash/*.proto + protoc --gogofaster_out=. -I=. -I=$(GOPATH)/src x/sigs/*.proto + +First, you notice that we need the protoc executable that we installed +with ``prototools``. Next you notice the ``--gogofaster_out=.`` flag. +This indicated that we should use ``protoc-gen-gogofaster`` to generate +the code (we installed the driver in ``$GOBIN`` during the ``prototools`` +step). Also, that the output file will be placed in the same directory as +the input file. So ``app/results.proto`` produces ``app/results.pb.go``. + +The first few lines should make sense now, but what is with the +``-I=. -I=$(GOPATH)/src`` flags used in the last two lines? These +_.proto_ files import other _.proto_ files and _protoc_ needs to know +where to find them. Since we want the generated code to use absolute +paths, we have to import them with their absolute path from the +root of our _GOPATH_, thus: ``-I=$(GOPATH)/src``. If you just add +that one, it will fail with the following message, which can be +resolved by adding ``-I=.`` as well: + +:: + + x/sigs/codec.proto: File does not reside within any path specified using + --proto_path (or -I). You must specify a --proto_path which encompasses + this file. Note that the proto_path must be an exact prefix of the + .proto file names -- protoc is too dumb to figure out when two paths + (e.g. absolute and relative) are equivalent (it's harder than you think). + +You are welcome to use other codecs than ``gogofaster``, you can also +try the standard golang protobuf compiler. What this mode goes is +auto-generate static code for serialization and deserialization of the +type. It performs the introspection one time to generate efficient code +allowing us to avoid the use of reflection at runtime and get ~10x +speed ups in the serialization/deserialization. I like this, but +this may vary based on your preference or aversion of auto-generated code. + +Using Autogenerated Structs +=========================== + +The first time through the above process may appear tedious, but once you +get the hang of it, you just have to add a few lines to a _.proto_ file +and type ``make protoc``. Et viola! You have a bunch of fresh ``*.pb.go`` +files that provide efficient, portable serialization for your code. + +But how do you use those structs? Taking ``Coin`` from ``x/codec.proto`` +as an example, we see a ``x/codec.pb.go`` file with ``type Coin struct {...}`` +that very closely mirrors the content of the ``codec.proto`` file, as +well as a number of methods. There are some auto-generated getters, +which can be useful to fulfill interfaces or to query field +of _nil_ objects without panicking. And then there are some (very long) +Marshal and Unmarshal methods. These are the meat of the matter. +They fulfill the `Persistent `_ +interface and let us write code like this: + +:: + + orig := Coin{Whole: 123, Ticker: "CASH"} + bz, err := orig.Marshal() + parsed := Coin{} + err = parsed.Unmarshal(bz) + +This is fine, but what happens when I want to add custom logic to +my ``Coin`` struct, perhaps adding validation logic, or code +to add two coins? Luckily for us, go allows you two write methods +for your structs in _any file in the same package_. That means that +we can just inherit the struct definition and all the serialization +logic and just append the methods we care about. +`coin.go `_ +is a great example of extending the functionality, with code like: + +:: + + func (c Coin) Add(o Coin) (Coin, error) { + if !c.SameType(o) { + err := ErrInvalidCurrency(c.Ticker, o.Ticker) + return Coin{}, err + } + c.Whole += o.Whole + c.Fractional += o.Fractional + return c.normalize() + } + + func (c Coin) Validate() error { + if !IsCC(c.Ticker) { + return ErrInvalidCurrency(c.Ticker) + } + if c.Whole < MinInt || c.Whole > MaxInt { + return ErrOutOfRange(c) + } + if c.Fractional < MinFrac || c.Fractional > MaxFrac { + return ErrOutOfRange(c) + } + // make sure signs match + if c.Whole != 0 && c.Fractional != 0 && + ((c.Whole > 0) != (c.Fractional > 0)) { + return ErrMismatchedSign(c) + } + + return nil + } + +This is a quite productive workflow and I recommend trying it out. +You may find it doesn't work for you and you can try other approaches, +like copying the protobuf generated structs into some custom-writen +structs you like and then copying back into protobuf structs for +serialization. You can also try playing with special +`gogo-protobuf (TODO) `_ flags in your +protobuf files to shape the autogenerated code into the exact shape +you want. + +Notes about oneof +================= + +**oneof** is a powerful feature to produce union/sum types in your +protobuf structures. For example, you may have a public key which +may be one of many different algorithms, and can define cases for each, +which can be swtiched upon in runtime. We also use this for the +transaction to enumerate a set of possible messages that can be +embedded in the transaction. A transaction may have any one of them +and serialize and deserialize properly. Type-safety is enforced +in compile-time and we can switch on the kind on runtime, quite nice. +(Example from `bcp-demo `_): + +:: + + oneof sum{ + cash.SendMsg send_msg = 1; + namecoin.NewTokenMsg new_token_msg = 2; + namecoin.SetWalletNameMsg set_name_msg = 3; + escrow.CreateEscrowMsg create_escrow_msg = 4; + escrow.ReleaseEscrowMsg release_escrow_msg = 5; + escrow.ReturnEscrowMsg return_escrow_msg = 6; + escrow.UpdateEscrowPartiesMsg update_escrow_msg = 7; + } + +The only problem is that the generated code is ugly to some people's eyes. +This lies in the fact that there is no clean way to express sum types in +golang, and you have to force an interface with private methods in order +to close the set of possible types. Although some people have been +so revolted by this code that they prefered to +`write their own serialization library `_, +I would suggest just taking the breath and getting to know it. +Here are the relevant pieces: + +:: + + type Tx struct { + // msg is a sum type over all allowed messages on this chain. + // + // Types that are valid to be assigned to Sum: + // *Tx_SendMsg + // *Tx_NewTokenMsg + // *Tx_SetNameMsg + // *Tx_CreateEscrowMsg + // *Tx_ReleaseEscrowMsg + // *Tx_ReturnEscrowMsg + // *Tx_UpdateEscrowMsg + Sum isTx_Sum `protobuf_oneof:"sum"` + ... + } + + type isTx_Sum interface { + isTx_Sum() + MarshalTo([]byte) (int, error) + Size() int + } + + type Tx_SendMsg struct { + SendMsg *cash.SendMsg `protobuf:"bytes,1,opt,name=send_msg,json=sendMsg,oneof"` + } + type Tx_NewTokenMsg struct { + NewTokenMsg *namecoin.NewTokenMsg `protobuf:"bytes,2,opt,name=new_token_msg,json=newTokenMsg,oneof"` + } + +We now have some intermediate structs that give us a layer of indirection +in order to enforce the fact we can now securely switch over all +possible ``tx.Sum`` fields, with +`code like this `_: + +:: + + sum := tx.GetSum() + switch t := sum.(type) { + case *Tx_SendMsg: + return t.SendMsg, nil + case *Tx_SetNameMsg: + return t.SetNameMsg, nil + case *Tx_NewTokenMsg: + return t.NewTokenMsg, nil + case *Tx_CreateEscrowMsg: + return t.CreateEscrowMsg, nil + case *Tx_ReleaseEscrowMsg: + return t.ReleaseEscrowMsg, nil + case *Tx_ReturnEscrowMsg: + return t.ReturnEscrowMsg, nil + case *Tx_UpdateEscrowMsg: + return t.UpdateEscrowMsg, nil + } -* How to create .proto files -* How to compile them (with weave Makefile) -* How to use compiled code -* Comment about oneof... From afbba21b3582c656ab618fe6d5b02f89f3ca6db3 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 17:16:51 +0200 Subject: [PATCH 05/19] Filled in TODOs in protobuf links --- docs/development/protobuf.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/development/protobuf.rst b/docs/development/protobuf.rst index 8162de5a..dec4edb9 100644 --- a/docs/development/protobuf.rst +++ b/docs/development/protobuf.rst @@ -16,7 +16,7 @@ Create Proto File ================= The first thing is to imagine the shape of your classes. -These should be defined in `proto3 (TODO) `_ syntax. +These should be defined in `proto3 `_ syntax. There are a number of different int encodings, byte slices, strings, and nested structures. And fields may be repeated. So forget complex types with methods now and just focus on @@ -76,7 +76,8 @@ array of tokens of different currencies. Compiling Proto Files ===================== -To compile protobuf files, you need to have the `protoc (TODO) `_ +To compile protobuf files, you need to have the +`protoc `_ binary installed and a language-specific translator (`gogo-protobuf `_ in this case). This can be a bit of a pain, especially the first time, so the default @@ -204,7 +205,7 @@ You may find it doesn't work for you and you can try other approaches, like copying the protobuf generated structs into some custom-writen structs you like and then copying back into protobuf structs for serialization. You can also try playing with special -`gogo-protobuf (TODO) `_ flags in your +`gogo-protobuf `_ flags in your protobuf files to shape the autogenerated code into the exact shape you want. From 6494ab4221abd91076765d78f619c77bfa1aef6f Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 17:17:44 +0200 Subject: [PATCH 06/19] Renamed docs/development -> docs/tutorial --- docs/index.rst | 20 +++++++++---------- docs/{development => tutorial}/advanced.rst | 0 docs/{development => tutorial}/datamodel.rst | 0 docs/{development => tutorial}/handler.rst | 0 docs/{development => tutorial}/init.rst | 0 .../{development => tutorial}/integration.rst | 0 docs/{development => tutorial}/messages.rst | 0 docs/{development => tutorial}/protobuf.rst | 0 docs/{development => tutorial}/queries.rst | 0 9 files changed, 10 insertions(+), 10 deletions(-) rename docs/{development => tutorial}/advanced.rst (100%) rename docs/{development => tutorial}/datamodel.rst (100%) rename docs/{development => tutorial}/handler.rst (100%) rename docs/{development => tutorial}/init.rst (100%) rename docs/{development => tutorial}/integration.rst (100%) rename docs/{development => tutorial}/messages.rst (100%) rename docs/{development => tutorial}/protobuf.rst (100%) rename docs/{development => tutorial}/queries.rst (100%) diff --git a/docs/index.rst b/docs/index.rst index 404fc839..93acadf5 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -85,8 +85,8 @@ and the various components you will use design/extensions.rst -Backend Development -------------------- +Backend Development Tutorial +---------------------------- A step by step example of writing your first application built on top of mycoind. @@ -99,12 +99,12 @@ that builds upon ``mycoind``. .. toctree:: :maxdepth: 2 - development/protobuf.rst - development/datamodel.rst - development/messages.rst - development/handler.rst - development/queries.rst - development/init.rst - development/integration.rst - development/advanced.rst + tutorial/protobuf.rst + tutorial/datamodel.rst + tutorial/messages.rst + tutorial/handler.rst + tutorial/queries.rst + tutorial/init.rst + tutorial/integration.rst + tutorial/advanced.rst diff --git a/docs/development/advanced.rst b/docs/tutorial/advanced.rst similarity index 100% rename from docs/development/advanced.rst rename to docs/tutorial/advanced.rst diff --git a/docs/development/datamodel.rst b/docs/tutorial/datamodel.rst similarity index 100% rename from docs/development/datamodel.rst rename to docs/tutorial/datamodel.rst diff --git a/docs/development/handler.rst b/docs/tutorial/handler.rst similarity index 100% rename from docs/development/handler.rst rename to docs/tutorial/handler.rst diff --git a/docs/development/init.rst b/docs/tutorial/init.rst similarity index 100% rename from docs/development/init.rst rename to docs/tutorial/init.rst diff --git a/docs/development/integration.rst b/docs/tutorial/integration.rst similarity index 100% rename from docs/development/integration.rst rename to docs/tutorial/integration.rst diff --git a/docs/development/messages.rst b/docs/tutorial/messages.rst similarity index 100% rename from docs/development/messages.rst rename to docs/tutorial/messages.rst diff --git a/docs/development/protobuf.rst b/docs/tutorial/protobuf.rst similarity index 100% rename from docs/development/protobuf.rst rename to docs/tutorial/protobuf.rst diff --git a/docs/development/queries.rst b/docs/tutorial/queries.rst similarity index 100% rename from docs/development/queries.rst rename to docs/tutorial/queries.rst From 1bdf417063310acd76d0c77f481744501f5c8e8b Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 17:36:28 +0200 Subject: [PATCH 07/19] Tutorial introduction --- docs/index.rst | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 93acadf5..820c4f8b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -88,13 +88,19 @@ and the various components you will use Backend Development Tutorial ---------------------------- -A step by step example of writing your first -application built on top of mycoind. -This is all about writing go code that runs as an ABCI app. -We will write a new extension and compile an application -that builds upon ``mycoind``. - -**TODO** +To make this theory more tangible, we will build a sample +application alongside this tutorial, to demonstrate dealing +with real-world constraints. The application is located in the +[examples/tutorial package](https://github.com/confio/weave/tree/master/examples/tutorial) +in the weave repository, but it is designed to be self-contained +and could just as easily live in an external repo that imported +weave. + +In this tutorial, you will learn how to serialize and model +you data strucutres, define messages and handlers, expose +queries, and read initial configuration from the genesis file. +You will be able to build a new extension and tie it together +with other extensions into a complete blockchain application. .. toctree:: :maxdepth: 2 From e1c7bab618b8912a8ca88f1bfc6057f1404e6051 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 18:16:08 +0200 Subject: [PATCH 08/19] Define domain models --- docs/tutorial/datamodel.rst | 94 ++++++++++++++++++++++++++-- examples/tutorial/x/blog/state.proto | 25 ++++++++ 2 files changed, 115 insertions(+), 4 deletions(-) create mode 100644 examples/tutorial/x/blog/state.proto diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index f979aeea..a4b08caa 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -2,9 +2,95 @@ Defining the Data Model ----------------------- +The first thing we consider is the data we want to store +(the state). After that we can focus on the messages, +which trigger state transitions. All blockchain state must +be stored in our merkle-ized store, that can provide +validity hashes and proofs. This is exposed to the application +as a basic key-value store, which also allows in-order +iteration over the keys. On top of this, we have built some +tools like secondary indexes and sequences, to produce +an API [similar to boltdb](https://github.com/boltdb/bolt#using-buckets). + +Define the Domain +----------------- + +Let us build a simple blog application. We will allow multiple +blogs to exist, each one registering a unique name, and each blog +may have a series of posts. The blog may contain rules as to who +(which public keys) may post on that blog. We will also allow +people to optionally register a profile tied to their public key +to present themselves. We will not add comments, likes, or other +features in order to keep the scope manageable. But we do +immediately see that there are some 1:N relationship and secondary +key lookups needed, so this is non-trivial and can provide a +decent example for a real application. + +What data do we need to store? + +* **Blog**: Unique name (slug), Full title, List of allowed authors +* **Post**: Link to blog (with sequence), Title, Text, Author, Date +* **Profile**: Link to author, Name, Description, Link to Posts + +Select Primary Keys +------------------- + +Some of this data belongs in the primary key, the rest in the value. +Weave introduces the concept of an [Object](https://github.com/confio/weave/blob/master/orm/interfaces.go#L8-L21) +which contains a Key (`[]byte`) and Value (`Persistent` struct). +It can be cloned and validated. When we query we will receive +this object, so we can place some critical information in the Key +and expect it to always be present. + +The primary key must be a unique identifier and it should be the +main way we want to access the data. Let's break down the four +models above into keys and [protobuf models](https://github.com/confio/weave/blob/master/examples/tutorial/x/blog/state.proto): + +Blog +~~~~ + +Key: Use the unique name `(slug)` as the primary key. + +.. literalinclude:: ../../examples/tutorial/x/blog/state.proto + :language: proto + :lines: 5-10 + +Post +~~~~ + +Key: Use `(blog slug, index)` as composite primary key. This allows +us to guarantee uniqueness and efficiently paginate through all +posts on a given blog. + +.. literalinclude:: ../../examples/tutorial/x/blog/state.proto + :language: proto + :lines: 12-20 + +Profile +~~~~~~~ + +Key: Use `(author address)` as primary key. + +.. literalinclude:: ../../examples/tutorial/x/blog/state.proto + :language: proto + :lines: 22-25 + +Using Buckets +-------------- + **TODO** -* Key-Value pairs -> select primary key -* Using a Bucket to store an object -* Adding validation and helpers -* Adding secondary indexes +Validating Models +----------------- + +**TODO** + +Secondary Indexes +------------------ + +**TODO** + +Sequences +--------- + +**TODO** diff --git a/examples/tutorial/x/blog/state.proto b/examples/tutorial/x/blog/state.proto new file mode 100644 index 00000000..fbc6df7d --- /dev/null +++ b/examples/tutorial/x/blog/state.proto @@ -0,0 +1,25 @@ +syntax = "proto3"; + +package blog; + +message Blog { + string title = 1; + // Author bytes to be interpreted as weave.Address + repeated bytes authors = 2; + int64 num_articles = 3; +} + +message Post { + string title = 1; + bytes author = 2; + // a timestamp would differ between nodes and be + // non-deterministic when replaying blocks. + // block height is the only constant + int64 creation_block = 3; + string text = 4; +} + +message Profile { + string name = 1; + string description = 2; +} From 8b4cb76ce981195c91a1d36512d0ff4527a4515a Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 18:23:21 +0200 Subject: [PATCH 09/19] Compile state protobuf file --- docs/tutorial/datamodel.rst | 13 + examples/tutorial/Makefile | 7 + examples/tutorial/x/blog/state.pb.go | 841 +++++++++++++++++++++++++++ 3 files changed, 861 insertions(+) create mode 100644 examples/tutorial/Makefile create mode 100644 examples/tutorial/x/blog/state.pb.go diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index a4b08caa..9013d50e 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -75,6 +75,19 @@ Key: Use `(author address)` as primary key. :language: proto :lines: 22-25 +Compile Protobuf +---------------- + +We add the compilation steps into our [Makefile](https://github.com/confio/weave/blob/master/examples/tutorial/Makefile): + +.. literalinclude:: ../../examples/tutorial/Makefile + :language: Makefile + :lines: 3-4 + +Now we run ``make protoc`` to generate the golang objects. (You +will have to add and run the `prototools` section if you are +using your own repo, we inherit that from root weave Makefile). + Using Buckets -------------- diff --git a/examples/tutorial/Makefile b/examples/tutorial/Makefile new file mode 100644 index 00000000..446f83a9 --- /dev/null +++ b/examples/tutorial/Makefile @@ -0,0 +1,7 @@ +.PHONY: protoc deps + +protoc: + protoc --gogofaster_out=. x/blog/*.proto + +deps: + @echo "add custom dependencies here" diff --git a/examples/tutorial/x/blog/state.pb.go b/examples/tutorial/x/blog/state.pb.go new file mode 100644 index 00000000..471466cd --- /dev/null +++ b/examples/tutorial/x/blog/state.pb.go @@ -0,0 +1,841 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: x/blog/state.proto + +/* + Package blog is a generated protocol buffer package. + + It is generated from these files: + x/blog/state.proto + + It has these top-level messages: + Blog + Post + Profile +*/ +package blog + +import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" +import math "math" + +import io "io" + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + +type Blog struct { + Title string `protobuf:"bytes,1,opt,name=title,proto3" json:"title,omitempty"` + // Author bytes to be interpreted as weave.Address + Authors [][]byte `protobuf:"bytes,2,rep,name=authors" json:"authors,omitempty"` + NumArticles int64 `protobuf:"varint,3,opt,name=num_articles,json=numArticles,proto3" json:"num_articles,omitempty"` +} + +func (m *Blog) Reset() { *m = Blog{} } +func (m *Blog) String() string { return proto.CompactTextString(m) } +func (*Blog) ProtoMessage() {} +func (*Blog) Descriptor() ([]byte, []int) { return fileDescriptorState, []int{0} } + +func (m *Blog) GetTitle() string { + if m != nil { + return m.Title + } + return "" +} + +func (m *Blog) GetAuthors() [][]byte { + if m != nil { + return m.Authors + } + return nil +} + +func (m *Blog) GetNumArticles() int64 { + if m != nil { + return m.NumArticles + } + return 0 +} + +type Post struct { + Title string `protobuf:"bytes,1,opt,name=title,proto3" json:"title,omitempty"` + Author []byte `protobuf:"bytes,2,opt,name=author,proto3" json:"author,omitempty"` + // a timestamp would differ between nodes and be + // non-deterministic when replaying blocks. + // block height is the only constant + CreationBlock int64 `protobuf:"varint,3,opt,name=creation_block,json=creationBlock,proto3" json:"creation_block,omitempty"` + Text string `protobuf:"bytes,4,opt,name=text,proto3" json:"text,omitempty"` +} + +func (m *Post) Reset() { *m = Post{} } +func (m *Post) String() string { return proto.CompactTextString(m) } +func (*Post) ProtoMessage() {} +func (*Post) Descriptor() ([]byte, []int) { return fileDescriptorState, []int{1} } + +func (m *Post) GetTitle() string { + if m != nil { + return m.Title + } + return "" +} + +func (m *Post) GetAuthor() []byte { + if m != nil { + return m.Author + } + return nil +} + +func (m *Post) GetCreationBlock() int64 { + if m != nil { + return m.CreationBlock + } + return 0 +} + +func (m *Post) GetText() string { + if m != nil { + return m.Text + } + return "" +} + +type Profile struct { + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + Description string `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"` +} + +func (m *Profile) Reset() { *m = Profile{} } +func (m *Profile) String() string { return proto.CompactTextString(m) } +func (*Profile) ProtoMessage() {} +func (*Profile) Descriptor() ([]byte, []int) { return fileDescriptorState, []int{2} } + +func (m *Profile) GetName() string { + if m != nil { + return m.Name + } + return "" +} + +func (m *Profile) GetDescription() string { + if m != nil { + return m.Description + } + return "" +} + +func init() { + proto.RegisterType((*Blog)(nil), "blog.Blog") + proto.RegisterType((*Post)(nil), "blog.Post") + proto.RegisterType((*Profile)(nil), "blog.Profile") +} +func (m *Blog) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Blog) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Title) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintState(dAtA, i, uint64(len(m.Title))) + i += copy(dAtA[i:], m.Title) + } + if len(m.Authors) > 0 { + for _, b := range m.Authors { + dAtA[i] = 0x12 + i++ + i = encodeVarintState(dAtA, i, uint64(len(b))) + i += copy(dAtA[i:], b) + } + } + if m.NumArticles != 0 { + dAtA[i] = 0x18 + i++ + i = encodeVarintState(dAtA, i, uint64(m.NumArticles)) + } + return i, nil +} + +func (m *Post) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Post) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Title) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintState(dAtA, i, uint64(len(m.Title))) + i += copy(dAtA[i:], m.Title) + } + if len(m.Author) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintState(dAtA, i, uint64(len(m.Author))) + i += copy(dAtA[i:], m.Author) + } + if m.CreationBlock != 0 { + dAtA[i] = 0x18 + i++ + i = encodeVarintState(dAtA, i, uint64(m.CreationBlock)) + } + if len(m.Text) > 0 { + dAtA[i] = 0x22 + i++ + i = encodeVarintState(dAtA, i, uint64(len(m.Text))) + i += copy(dAtA[i:], m.Text) + } + return i, nil +} + +func (m *Profile) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *Profile) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Name) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintState(dAtA, i, uint64(len(m.Name))) + i += copy(dAtA[i:], m.Name) + } + if len(m.Description) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintState(dAtA, i, uint64(len(m.Description))) + i += copy(dAtA[i:], m.Description) + } + return i, nil +} + +func encodeVarintState(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *Blog) Size() (n int) { + var l int + _ = l + l = len(m.Title) + if l > 0 { + n += 1 + l + sovState(uint64(l)) + } + if len(m.Authors) > 0 { + for _, b := range m.Authors { + l = len(b) + n += 1 + l + sovState(uint64(l)) + } + } + if m.NumArticles != 0 { + n += 1 + sovState(uint64(m.NumArticles)) + } + return n +} + +func (m *Post) Size() (n int) { + var l int + _ = l + l = len(m.Title) + if l > 0 { + n += 1 + l + sovState(uint64(l)) + } + l = len(m.Author) + if l > 0 { + n += 1 + l + sovState(uint64(l)) + } + if m.CreationBlock != 0 { + n += 1 + sovState(uint64(m.CreationBlock)) + } + l = len(m.Text) + if l > 0 { + n += 1 + l + sovState(uint64(l)) + } + return n +} + +func (m *Profile) Size() (n int) { + var l int + _ = l + l = len(m.Name) + if l > 0 { + n += 1 + l + sovState(uint64(l)) + } + l = len(m.Description) + if l > 0 { + n += 1 + l + sovState(uint64(l)) + } + return n +} + +func sovState(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozState(x uint64) (n int) { + return sovState(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *Blog) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Blog: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Blog: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Title", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Title = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Authors", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Authors = append(m.Authors, make([]byte, postIndex-iNdEx)) + copy(m.Authors[len(m.Authors)-1], dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 3: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field NumArticles", wireType) + } + m.NumArticles = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.NumArticles |= (int64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + default: + iNdEx = preIndex + skippy, err := skipState(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthState + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *Post) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Post: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Post: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Title", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Title = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Author", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Author = append(m.Author[:0], dAtA[iNdEx:postIndex]...) + if m.Author == nil { + m.Author = []byte{} + } + iNdEx = postIndex + case 3: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field CreationBlock", wireType) + } + m.CreationBlock = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.CreationBlock |= (int64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + case 4: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Text", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Text = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipState(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthState + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *Profile) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: Profile: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: Profile: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Name", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Name = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Description", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowState + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthState + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Description = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipState(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthState + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipState(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowState + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowState + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowState + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + iNdEx += length + if length < 0 { + return 0, ErrInvalidLengthState + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowState + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipState(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthState = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowState = fmt.Errorf("proto: integer overflow") +) + +func init() { proto.RegisterFile("x/blog/state.proto", fileDescriptorState) } + +var fileDescriptorState = []byte{ + // 247 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x74, 0x90, 0x41, 0x4a, 0xc3, 0x40, + 0x14, 0x86, 0x9d, 0x36, 0xb6, 0xf4, 0x35, 0x8a, 0x0c, 0x22, 0xb3, 0x0a, 0x63, 0x40, 0xc8, 0xca, + 0x2e, 0x3c, 0x80, 0x98, 0x13, 0x94, 0xd9, 0xb9, 0x2a, 0x93, 0x38, 0xd6, 0xc1, 0x49, 0x5e, 0x99, + 0x79, 0x81, 0x1e, 0xc3, 0x63, 0xb9, 0xf4, 0x08, 0x12, 0x2f, 0x22, 0x99, 0x26, 0xe0, 0xc6, 0xdd, + 0xff, 0x7f, 0x0f, 0xbe, 0x1f, 0x1e, 0xf0, 0xe3, 0xa6, 0x72, 0xb8, 0xdf, 0x04, 0xd2, 0x64, 0xee, + 0x0f, 0x1e, 0x09, 0x79, 0x32, 0x90, 0xfc, 0x19, 0x92, 0xd2, 0xe1, 0x9e, 0x5f, 0xc3, 0x39, 0x59, + 0x72, 0x46, 0x30, 0xc9, 0x8a, 0x95, 0x3a, 0x15, 0x2e, 0x60, 0xa9, 0x3b, 0x7a, 0x43, 0x1f, 0xc4, + 0x4c, 0xce, 0x8b, 0x54, 0x4d, 0x95, 0xdf, 0x42, 0xda, 0x76, 0xcd, 0x4e, 0x7b, 0xb2, 0xb5, 0x33, + 0x41, 0xcc, 0x25, 0x2b, 0xe6, 0x6a, 0xdd, 0x76, 0xcd, 0xd3, 0x88, 0x72, 0x84, 0x64, 0x8b, 0x81, + 0xfe, 0x51, 0xdf, 0xc0, 0xe2, 0xe4, 0x12, 0x33, 0xc9, 0x8a, 0x54, 0x8d, 0x8d, 0xdf, 0xc1, 0x65, + 0xed, 0x8d, 0x26, 0x8b, 0xed, 0xae, 0x72, 0x58, 0xbf, 0x8f, 0xea, 0x8b, 0x89, 0x96, 0x03, 0xe4, + 0x1c, 0x12, 0x32, 0x47, 0x12, 0x49, 0x74, 0xc6, 0x9c, 0x3f, 0xc2, 0x72, 0xeb, 0xf1, 0xd5, 0x3a, + 0x33, 0x9c, 0x5b, 0xdd, 0x4c, 0x93, 0x31, 0x73, 0x09, 0xeb, 0x17, 0x13, 0x6a, 0x6f, 0x0f, 0x83, + 0x26, 0xce, 0xae, 0xd4, 0x5f, 0x54, 0x5e, 0x7d, 0xf6, 0x19, 0xfb, 0xea, 0x33, 0xf6, 0xdd, 0x67, + 0xec, 0xe3, 0x27, 0x3b, 0xab, 0x16, 0xf1, 0x57, 0x0f, 0xbf, 0x01, 0x00, 0x00, 0xff, 0xff, 0xe4, + 0xe8, 0x20, 0xef, 0x41, 0x01, 0x00, 0x00, +} From bebe4abe16258cf606ae0b607ecc7ff1f0c51cfd Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 19:16:00 +0200 Subject: [PATCH 10/19] Add model validation --- docs/tutorial/datamodel.rst | 91 +++++++++++++++++++++++++--- docs/tutorial/protobuf.rst | 4 +- examples/tutorial/x/blog/models.go | 96 ++++++++++++++++++++++++++++++ 3 files changed, 182 insertions(+), 9 deletions(-) create mode 100644 examples/tutorial/x/blog/models.go diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index 9013d50e..58d9e6be 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -9,8 +9,13 @@ be stored in our merkle-ized store, that can provide validity hashes and proofs. This is exposed to the application as a basic key-value store, which also allows in-order iteration over the keys. On top of this, we have built some -tools like secondary indexes and sequences, to produce -an API [similar to boltdb](https://github.com/boltdb/bolt#using-buckets). +tools like secondary indexes and sequences, in a similar +manner to show +[storm adds a orm](https://github.com/asdine/storm#simple-crud-system) +on top of +[boltdb's kv store](https://github.com/boltdb/bolt#using-buckets). +We have avoided struct tags and tried to type as strictly as +we can (without using generics). Define the Domain ----------------- @@ -84,19 +89,91 @@ We add the compilation steps into our [Makefile](https://github.com/confio/weave :language: Makefile :lines: 3-4 -Now we run ``make protoc`` to generate the golang objects. (You -will have to add and run the `prototools` section if you are +Now we run ``make protoc`` to generate the +[golang objects](https://github.com/confio/weave/blob/master/examples/tutorial/x/blog/state.pb.go). +(You will have to add and run the `prototools` section if you are using your own repo, we inherit that from root weave Makefile). Using Buckets -------------- -**TODO** +When running your handlers, you get access to the root +[KVStore](https://godoc.org/github.com/confio/weave#KVStore), +which is an abstraction level similar to boltdb or leveldb. +An extenstion can opt-in to using one or more +[Buckets](https://godoc.org/github.com/confio/weave/orm#Bucket) +to store the data. Buckets offer the following advantages: + +* Isolation between extensions (each Bucket has a unique prefix that is transparently prepended to the keys) +* Type safety (enforce all data stored in a Bucket is the same type, to avoid parse errors later on) +* Indexes (Buckets are well integrated with the secondary indexes and keep them in sync every time data is modified) +* Querying (Buckets can easily register query handlers including prefix queries and secondary index queries) + +All extensions from weave use Buckets, so for compatibility as +well as the features, please use Buckets in your app, unless you +have a very good reason not to (and know what you are doing). + +To do so, you will have to wrap your state data structures into +[Objects](https://godoc.org/github.com/confio/weave/orm#Object). +The simplest way is to use ``SimpleObj``: + +.. literalinclude:: ../../orm/object.go + :language: golang + :lines: 14-17 + +And extend your protobuf objects to implement +[CloneableData](https://godoc.org/github.com/confio/weave/orm#CloneableData): + +.. literalinclude:: ../../orm/interfaces.go + :language: golang + :lines: 35-39 + +This basically consists of adding `Copy()` and `Validate()` +to the objects in ``state.pb.go``. Just create a +[models.go](https://github.com/confio/weave/blob/master/examples/tutorial/x/blog/models.go) +file and add extra methods to the auto-generated structs. +If we don't care about validation, this can be as simple as: + +.. code:: golang + + // enforce that Post fulfils desired interface compile-time + var _ orm.CloneableData = (*Post)(nil) + + // Validate enforces limits of text and title size + func (p *Post) Validate() error { + // TODO + return nil + } + + // Copy makes a new Post with the same data + func (p *Post) Copy() orm.CloneableData { + return &Post{ + Title: p.Title, + Author: p.Author, + Text: p.Text, + CreationBlock: p.CreationBlock, + } + } + Validating Models ------------------ +~~~~~~~~~~~~~~~~~ + +We will want to fill in these Validate methods to enforce +any invariants we demand of the data to keep our database clean. +Anyone who has spent much time dealing with production +applications knows how "invalid data" can start creeping in +without a strict database schema, this is what we do in code. + + + +Errors +~~~~~~ + + +Custom Bucket +~~~~~~~~~~~~~ -**TODO** Secondary Indexes ------------------ diff --git a/docs/tutorial/protobuf.rst b/docs/tutorial/protobuf.rst index dec4edb9..594cd0c1 100644 --- a/docs/tutorial/protobuf.rst +++ b/docs/tutorial/protobuf.rst @@ -23,7 +23,7 @@ So forget complex types with methods now and just focus on the actual data structure. The `x/codec.proto `_ file defines the Coin type rather simply, once you remove the comments, this is all that is left: -:: +.. code:: proto syntax = "proto3"; @@ -163,7 +163,7 @@ interface and let us write code like this: This is fine, but what happens when I want to add custom logic to my ``Coin`` struct, perhaps adding validation logic, or code to add two coins? Luckily for us, go allows you two write methods -for your structs in _any file in the same package_. That means that +for your structs in *any file in the same package*. That means that we can just inherit the struct definition and all the serialization logic and just append the methods we care about. `coin.go `_ diff --git a/examples/tutorial/x/blog/models.go b/examples/tutorial/x/blog/models.go new file mode 100644 index 00000000..56e89ed4 --- /dev/null +++ b/examples/tutorial/x/blog/models.go @@ -0,0 +1,96 @@ +package blog + +import "github.com/confio/weave/orm" + +//----- Blog ------- + +const MaxAuthors = 10 +const MaxTitleLength = 100 +const MaxTextLength = 20 * 1000 +const MaxNameLength = 30 +const MaxDescriptionLength = 280 + +// enforce that Blog fulfils desired interface compile-time +var _ orm.CloneableData = (*Blog)(nil) + +// Validate enforces limits of title size and number of authors +func (b *Blog) Validate() error { + if len(b.Title) > MaxTitleLength { + return ErrTitleTooLong() + } + if len(b.Authors) > MaxAuthors || len(b.Authors) == 0 { + return ErrInvalidAuthorCount(len(b.Authors)) + } + if b.NumArticles < 0 { + return ErrNegativeArticles() + } + return nil +} + +// Copy makes a new blog with the same data +func (b *Blog) Copy() orm.CloneableData { + // copy into a new slice to allow modifications + authors := make([][]byte, len(b.Authors)) + copy(authors, b.Authors) + return &Blog{ + Title: b.Title, + Authors: authors, + NumArticles: b.NumArticles, + } +} + +//------- Post ------ + +// enforce that Post fulfils desired interface compile-time +var _ orm.CloneableData = (*Post)(nil) + +// Validate enforces limits of text and title size +func (p *Post) Validate() error { + if len(p.Title) > MaxTitleLength { + return ErrTitleTooLong() + } + if len(p.Text) > MaxTextLength { + return ErrTextTooLong() + } + if len(p.Author) == 0 { + return ErrNoAuthor() + } + if p.CreationBlock < 0 { + return ErrNegativeCreation() + } + return nil +} + +// Copy makes a new Post with the same data +func (p *Post) Copy() orm.CloneableData { + return &Post{ + Title: p.Title, + Author: p.Author, + Text: p.Text, + CreationBlock: p.CreationBlock, + } +} + +//-------- Profile ------ + +// enforce that Profile fulfils desired interface compile-time +var _ orm.CloneableData = (*Profile)(nil) + +// Validate enforces limits of text and title size +func (p *Profile) Validate() error { + if len(p.Name) > MaxNameLength { + return ErrNameTooLong() + } + if len(p.Description) > MaxDescriptionLength { + return ErrDescriptionTooLong() + } + return nil +} + +// Copy makes a new Profile with the same data +func (p *Profile) Copy() orm.CloneableData { + return &Profile{ + Name: p.Name, + Description: p.Description, + } +} From f2c16fc14ac389504353243b34a36e7711fd43b3 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 19:41:36 +0200 Subject: [PATCH 11/19] Demonstrate error usage --- docs/tutorial/datamodel.rst | 58 ++++++++++++++++++++++++++ examples/tutorial/x/blog/errors.go | 65 ++++++++++++++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100644 examples/tutorial/x/blog/errors.go diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index 58d9e6be..4ef1bf52 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -165,11 +165,69 @@ Anyone who has spent much time dealing with production applications knows how "invalid data" can start creeping in without a strict database schema, this is what we do in code. +We can do some basic checks and return an error if none of them +pass: +.. literalinclude:: ../../examples/tutorial/x/blog/models.go + :language: golang + :lines: 16-28 Errors ~~~~~~ +What is with these ``ErrXYZ()`` calls you may think? Well, we +could return a "normal" error like ``errors.New("fail")``, +but we wanted two more features. First of all, it helps +debugging enormously to have a stack trace of where the error +originally occured. For this we use +`pkg/errors `_ +that attaches a stacktrace to the error that can optionally +be printed later with a ``Printf("%+v", err)``. +We also want to return a unique abci error code, which may be +interpretted by client applications, either programmatically +or to provide translations of the error message client side. + +For these reasons, weave provides some utility methods +and common error types in the +`errors `_ +package. The ABCI Code attached to the error is then +`returned in the DeliverTx Result `_. + +Every package can define it's own custom error types and +error codes, generally in a file called +`errors.go `_. The key elements are: + +.. code:: golang + + // ABCI Response Codes + // tutorial reserves 400 ~ 420. + const ( + CodeInvalidText uint32 = 400 + ) + + var ( + errTitleTooLong = fmt.Errorf("Title is too long") + errInvalidAuthorCount = fmt.Errorf("Invalid number of blog authors") + ) + + // Error code with no arguments, check on code not particular type + func ErrTitleTooLong() error { + return errors.WithCode(errTitleTooLong, CodeInvalidText) + } + func IsInvalidTextError(err error) bool { + return errors.HasErrorCode(err, CodeInvalidText) + } + + // You can also prepend a variable message using WithLog + func ErrInvalidAuthorCount(count int) error { + msg := fmt.Sprintf("authors=%d", count) + return errors.WithLog(msg, errInvalidAuthorCount, CodeInvalidAuthor) + } + +Take a deeper look at the file and if you start using that pattern +you will see the nicer debug messages, usable error codes, and +the ability to check the type of error in your test code without +resorting to string comparisons. Custom Bucket ~~~~~~~~~~~~~ diff --git a/examples/tutorial/x/blog/errors.go b/examples/tutorial/x/blog/errors.go new file mode 100644 index 00000000..f4fdfe3b --- /dev/null +++ b/examples/tutorial/x/blog/errors.go @@ -0,0 +1,65 @@ +package blog + +import ( + "fmt" + + "github.com/confio/weave/errors" +) + +// ABCI Response Codes +// tutorial reserves 400 ~ 420. +const ( + CodeInvalidText uint32 = 400 + CodeInvalidAuthor uint32 = 401 + CodeNegativeNumber uint32 = 402 +) + +var ( + errTitleTooLong = fmt.Errorf("Title is too long") + errTextTooLong = fmt.Errorf("Text is too long") + errNameTooLong = fmt.Errorf("Name is too long") + errDescriptionTooLong = fmt.Errorf("Description is too long") + + errNoAuthor = fmt.Errorf("No author for post") + errInvalidAuthorCount = fmt.Errorf("Invalid number of blog authors") + + errNegativeArticles = fmt.Errorf("Article count is negative") + errNegativeCreation = fmt.Errorf("Creation block is negative") +) + +func ErrTitleTooLong() error { + return errors.WithCode(errTitleTooLong, CodeInvalidText) +} +func ErrTextTooLong() error { + return errors.WithCode(errTextTooLong, CodeInvalidText) +} +func ErrNameTooLong() error { + return errors.WithCode(errNameTooLong, CodeInvalidText) +} +func ErrDescriptionTooLong() error { + return errors.WithCode(errDescriptionTooLong, CodeInvalidText) +} +func IsInvalidTextError(err error) bool { + return errors.HasErrorCode(err, CodeInvalidText) +} + +func ErrNoAuthor() error { + return errors.WithCode(errNoAuthor, CodeInvalidAuthor) +} +func ErrInvalidAuthorCount(count int) error { + msg := fmt.Sprintf("authors=%d", count) + return errors.WithLog(msg, errInvalidAuthorCount, CodeInvalidAuthor) +} +func IsInvalidAuthorError(err error) bool { + return errors.HasErrorCode(err, CodeInvalidAuthor) +} + +func ErrNegativeArticles() error { + return errors.WithCode(errNegativeArticles, CodeNegativeNumber) +} +func ErrNegativeCreation() error { + return errors.WithCode(errNegativeCreation, CodeNegativeNumber) +} +func IsNegativeNumberError(err error) bool { + return errors.HasErrorCode(err, CodeNegativeNumber) +} From aa6a20bbd465e65152cf9c506512768c678f43e9 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 19:44:54 +0200 Subject: [PATCH 12/19] Fix md -> rst links --- docs/index.rst | 2 +- docs/tutorial/datamodel.rst | 32 +++++++++++++++++--------------- 2 files changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 820c4f8b..2992d560 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -91,7 +91,7 @@ Backend Development Tutorial To make this theory more tangible, we will build a sample application alongside this tutorial, to demonstrate dealing with real-world constraints. The application is located in the -[examples/tutorial package](https://github.com/confio/weave/tree/master/examples/tutorial) +`examples/tutorial package `_ in the weave repository, but it is designed to be self-contained and could just as easily live in an external repo that imported weave. diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index 4ef1bf52..bcd08794 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -10,10 +10,10 @@ validity hashes and proofs. This is exposed to the application as a basic key-value store, which also allows in-order iteration over the keys. On top of this, we have built some tools like secondary indexes and sequences, in a similar -manner to show -[storm adds a orm](https://github.com/asdine/storm#simple-crud-system) +manner to how +`storm adds a orm `_ on top of -[boltdb's kv store](https://github.com/boltdb/bolt#using-buckets). +`boltdb's kv store `_. We have avoided struct tags and tried to type as strictly as we can (without using generics). @@ -41,7 +41,8 @@ Select Primary Keys ------------------- Some of this data belongs in the primary key, the rest in the value. -Weave introduces the concept of an [Object](https://github.com/confio/weave/blob/master/orm/interfaces.go#L8-L21) +Weave introduces the concept of an +`Object `_ which contains a Key (`[]byte`) and Value (`Persistent` struct). It can be cloned and validated. When we query we will receive this object, so we can place some critical information in the Key @@ -49,12 +50,13 @@ and expect it to always be present. The primary key must be a unique identifier and it should be the main way we want to access the data. Let's break down the four -models above into keys and [protobuf models](https://github.com/confio/weave/blob/master/examples/tutorial/x/blog/state.proto): +models above into keys and +`protobuf models `_: Blog ~~~~ -Key: Use the unique name `(slug)` as the primary key. +Key: Use the unique name ``(slug)`` as the primary key. .. literalinclude:: ../../examples/tutorial/x/blog/state.proto :language: proto @@ -63,7 +65,7 @@ Key: Use the unique name `(slug)` as the primary key. Post ~~~~ -Key: Use `(blog slug, index)` as composite primary key. This allows +Key: Use ``(blog slug, index)`` as composite primary key. This allows us to guarantee uniqueness and efficiently paginate through all posts on a given blog. @@ -74,7 +76,7 @@ posts on a given blog. Profile ~~~~~~~ -Key: Use `(author address)` as primary key. +Key: Use ``(author address)`` as primary key. .. literalinclude:: ../../examples/tutorial/x/blog/state.proto :language: proto @@ -90,18 +92,18 @@ We add the compilation steps into our [Makefile](https://github.com/confio/weave :lines: 3-4 Now we run ``make protoc`` to generate the -[golang objects](https://github.com/confio/weave/blob/master/examples/tutorial/x/blog/state.pb.go). -(You will have to add and run the `prototools` section if you are +`golang objects `_. +(You will have to add and run the ``prototools`` section if you are using your own repo, we inherit that from root weave Makefile). Using Buckets -------------- When running your handlers, you get access to the root -[KVStore](https://godoc.org/github.com/confio/weave#KVStore), +`KVStore `_, which is an abstraction level similar to boltdb or leveldb. An extenstion can opt-in to using one or more -[Buckets](https://godoc.org/github.com/confio/weave/orm#Bucket) +`Buckets `_ to store the data. Buckets offer the following advantages: * Isolation between extensions (each Bucket has a unique prefix that is transparently prepended to the keys) @@ -114,7 +116,7 @@ well as the features, please use Buckets in your app, unless you have a very good reason not to (and know what you are doing). To do so, you will have to wrap your state data structures into -[Objects](https://godoc.org/github.com/confio/weave/orm#Object). +`Objects `_. The simplest way is to use ``SimpleObj``: .. literalinclude:: ../../orm/object.go @@ -122,7 +124,7 @@ The simplest way is to use ``SimpleObj``: :lines: 14-17 And extend your protobuf objects to implement -[CloneableData](https://godoc.org/github.com/confio/weave/orm#CloneableData): +`CloneableData `_: .. literalinclude:: ../../orm/interfaces.go :language: golang @@ -130,7 +132,7 @@ And extend your protobuf objects to implement This basically consists of adding `Copy()` and `Validate()` to the objects in ``state.pb.go``. Just create a -[models.go](https://github.com/confio/weave/blob/master/examples/tutorial/x/blog/models.go) +`models.go `_ file and add extra methods to the auto-generated structs. If we don't care about validation, this can be as simple as: From ab1e27c7ab3e517afe2fb97a8d1f1a14eff061e3 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 20:15:36 +0200 Subject: [PATCH 13/19] Fix language in code blocks --- docs/tutorial/datamodel.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index bcd08794..01940dcf 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -92,7 +92,7 @@ We add the compilation steps into our [Makefile](https://github.com/confio/weave :lines: 3-4 Now we run ``make protoc`` to generate the -`golang objects `_. +`go objects `_. (You will have to add and run the ``prototools`` section if you are using your own repo, we inherit that from root weave Makefile). @@ -120,14 +120,14 @@ To do so, you will have to wrap your state data structures into The simplest way is to use ``SimpleObj``: .. literalinclude:: ../../orm/object.go - :language: golang + :language: go :lines: 14-17 And extend your protobuf objects to implement `CloneableData `_: .. literalinclude:: ../../orm/interfaces.go - :language: golang + :language: go :lines: 35-39 This basically consists of adding `Copy()` and `Validate()` @@ -136,7 +136,7 @@ to the objects in ``state.pb.go``. Just create a file and add extra methods to the auto-generated structs. If we don't care about validation, this can be as simple as: -.. code:: golang +.. code:: go // enforce that Post fulfils desired interface compile-time var _ orm.CloneableData = (*Post)(nil) @@ -171,7 +171,7 @@ We can do some basic checks and return an error if none of them pass: .. literalinclude:: ../../examples/tutorial/x/blog/models.go - :language: golang + :language: go :lines: 16-28 Errors @@ -199,7 +199,7 @@ Every package can define it's own custom error types and error codes, generally in a file called `errors.go `_. The key elements are: -.. code:: golang +.. code:: go // ABCI Response Codes // tutorial reserves 400 ~ 420. From 7855e3ed2673afc66f3c55764042f9f1179fc752 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Thu, 21 Jun 2018 20:41:31 +0200 Subject: [PATCH 14/19] Add buckets and secondary indexes --- docs/tutorial/datamodel.rst | 42 ++++++++++- examples/tutorial/x/blog/models.go | 108 ++++++++++++++++++++++++++++- 2 files changed, 147 insertions(+), 3 deletions(-) diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index 01940dcf..ad358efe 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -234,13 +234,51 @@ resorting to string comparisons. Custom Bucket ~~~~~~~~~~~~~ +We want to enforce the data consistency on the buckets. All +data is validated before saving, but we also need to make sure +that all data is the proper type of object before saving. +Unfortunately, this is quite difficult to do compile-time +without generic, so a typical apporach is to embed the +`orm.Bucket <>`_ +in another struct and just force validation of the object type +runtime before save. + +.. literalinclude:: ../../examples/tutorial/x/blog/models.go + :language: go + :lines: 101-128 Secondary Indexes ------------------ -**TODO** +Sometimes we need another index for the data. Generally, we +will look up a post from the blog it belongs to and it's +index in the blog. But what if we want to list all posts by +one author over all blogs? For this, we need to add a secondary +index on the posts to query by author. This is a typical case +and weave provides nice support for this functionality. + +.. literalinclude:: ../../examples/tutorial/x/blog/models.go + :language: go + :lines: 141-165 + +We add a indexing method to take any object, enforce the type +to be a proper Post, then extract the index we want. This +can be a field, or any deterministic transformation of +one (or multiple) fields. The output of the index becomes a +key in another query. Bucket provides a simple +`method to query by index `_. You can query by name like: + +.. code:: go + + posts, err := bucket.GetIndexed(db, "author", address) + +This will return a (possibly empty) list of Objects +(keys and values) that have an author index matching the query. Sequences --------- -**TODO** +You can also add an auto-incrementing sequence to a bucket. +That isn't so important in this case, but if you are curious +how to use it, take a look at the +`escrow bucket in bcp-demo `_. diff --git a/examples/tutorial/x/blog/models.go b/examples/tutorial/x/blog/models.go index 56e89ed4..39d96583 100644 --- a/examples/tutorial/x/blog/models.go +++ b/examples/tutorial/x/blog/models.go @@ -1,6 +1,11 @@ package blog -import "github.com/confio/weave/orm" +import ( + "errors" + + "github.com/confio/weave" + "github.com/confio/weave/orm" +) //----- Blog ------- @@ -94,3 +99,104 @@ func (p *Profile) Copy() orm.CloneableData { Description: p.Description, } } + +//------ Blog Bucket + +const BlogBucketName = "blogs" + +// BlogBucket is a type-safe wrapper around orm.Bucket +type BlogBucket struct { + orm.Bucket +} + +// NewBlogBucket initializes a BlogBucket with default name +// +// inherit Get and Save from orm.Bucket +// add run-time check on Save +func NewBlogBucket() BlogBucket { + bucket := orm.NewBucket(BlogBucketName, + orm.NewSimpleObj(nil, new(Blog))) + return BlogBucket{ + Bucket: bucket, + } +} + +// Save enforces the proper type +func (b BlogBucket) Save(db weave.KVStore, obj orm.Object) error { + if _, ok := obj.Value().(*Blog); !ok { + return orm.ErrInvalidObject(obj.Value()) + } + return b.Bucket.Save(db, obj) +} + +//------ Post Bucket + +const PostBucketName = "posts" + +// PostBucket is a type-safe wrapper around orm.Bucket +type PostBucket struct { + orm.Bucket +} + +// NewPostBucket initializes a PostBucket with default name +// +// inherit Get and Save from orm.Bucket +// add run-time check on Save +func NewPostBucket() PostBucket { + bucket := orm.NewBucket(PostBucketName, + orm.NewSimpleObj(nil, new(Post))). + WithIndex("author", idxAuthor, false) + return PostBucket{ + Bucket: bucket, + } +} + +func idxAuthor(obj orm.Object) ([]byte, error) { + // these should use proper errors, but they never occur + // except in case of developer error (wrong data in wrong bucket) + if obj == nil { + return nil, errors.New("Cannot take index of nil") + } + post, ok := obj.Value().(*Post) + if !ok { + return nil, errors.New("Can only take index of Post") + } + return post.Author, nil +} + +// Save enforces the proper type +func (b PostBucket) Save(db weave.KVStore, obj orm.Object) error { + if _, ok := obj.Value().(*Post); !ok { + return orm.ErrInvalidObject(obj.Value()) + } + return b.Bucket.Save(db, obj) +} + +//------ Profile Bucket + +const ProfileBucketName = "profiles" + +// ProfileBucket is a type-safe wrapper around orm.Bucket +type ProfileBucket struct { + orm.Bucket +} + +// NewProfileBucket initializes a ProfileBucket with default name +// +// inherit Get and Save from orm.Bucket +// add run-time check on Save +func NewProfileBucket() ProfileBucket { + bucket := orm.NewBucket(ProfileBucketName, + orm.NewSimpleObj(nil, new(Profile))) + return ProfileBucket{ + Bucket: bucket, + } +} + +// Save enforces the proper type +func (b ProfileBucket) Save(db weave.KVStore, obj orm.Object) error { + if _, ok := obj.Value().(*Profile); !ok { + return orm.ErrInvalidObject(obj.Value()) + } + return b.Bucket.Save(db, obj) +} From 04ed3f3b2783ec313d67e5c88d676c1cd90a2bfc Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Fri, 22 Jun 2018 21:07:27 +0200 Subject: [PATCH 15/19] Describe and add messages to the tutorial --- docs/tutorial/messages.rst | 96 ++ examples/tutorial/x/blog/messages.pb.go | 1286 +++++++++++++++++++++++ examples/tutorial/x/blog/messages.proto | 51 + 3 files changed, 1433 insertions(+) create mode 100644 examples/tutorial/x/blog/messages.pb.go create mode 100644 examples/tutorial/x/blog/messages.proto diff --git a/docs/tutorial/messages.rst b/docs/tutorial/messages.rst index cdbc2c9f..a4c6d586 100644 --- a/docs/tutorial/messages.rst +++ b/docs/tutorial/messages.rst @@ -2,6 +2,102 @@ Defining Messages ----------------- +We just discussed messages, which are persistent objects +requiring validation, which are stored in our local +key-value store. Messages are requests for a change in the +state, the action part of a transaction. They also need +to be persisted (to be sent over the wire and stored on the +blockchain), and must also be validated. They later are passed +into `Handlers `_ +to be processed and effect change in the blockchain state. + +Messages vs. Transactions +------------------------- + +A message is a request to make change and this is the basic +element of a blockchain. A transaction contains a message +along with metadata and authorization information, such +as fees, signatures, nonces, and time-to-live. + +A `Transaction `_ +is fundamentally defined as anything persistent that holds a message: + +.. code:: go + + type Tx interface { + Persistent + // GetMsg returns the action we wish to communicate + GetMsg() (Msg, error) + } + +And every application can extend it with additional functionality, +such as +`Signatures `_, +`Fees `_, +or anything else your application needs. The data placed in the +Transaction is meant to be anything that applies to all modules, and +is processed by a Middleware. + +A `Message `_ +is also persistent and can be pretty much anything that an +extension defines, as it also defines the +`Handler `_ +to process it. The only necessary feature of a Message is +that it can return a ``Path() string`` which allows us to +route it to the proper Handler. + +When we define a concrete transaction type for one application, +we define it in protobuf with a set of possible messages that +it can contain. Every application can add optional field to the +transaction and allow a different set of messages, and the +Handlers and Decorators work orthogonally to this, regardless +of the concrete Transaction type. + +Defining Messages +----------------- + +Messages are similar to the ``POST`` endpoints in a typical +API. They are the only way to effect a change in the system. +Ignoring the issue of authentication and rate limitation, +which is handled by the Decorators / Middleware, when we design +Messages, we focus on all possible state transitions and the +information they need to proceed. + +In the blog example, we can imagine: + +* Create Blog +* Update Blog Title +* Add/Remove Blog Author +* Create Post +* Create Profile +* Modify Profile (which may be merged with above) + +We can create a protobuf message for each of these types: + +.. literal-include + +And then add a ``Path`` method that returns a constant based on +the type: + +.. literal-include + +Validation +---------- + +While validation for data models is much more like SQL constraints: +"max length 20", "not null", "constaint foo > 3", validation for +messages is validating potentially malicious data coming in from +external sources and should be validated more thoroughly. +One may want to use regexp to avoid control characters or null bytes +in a "string" input. Maybe restrict it to alphanumeric or ascii +characters, strip out html, or allow full utf-8. Addresses must be +checked to be the valid length. Amount being sent to be positive +(else I send you -5 ETH and we have a TakeMsg, instead of SendMsg). + +The validation on Messages should be a lot more thorough and well +tested than the validation on data models, which is as much documentation +of acceptable values as it is runtime security. + **TODO** * Messages vs. Transactions, what is the distinction? diff --git a/examples/tutorial/x/blog/messages.pb.go b/examples/tutorial/x/blog/messages.pb.go new file mode 100644 index 00000000..4f39cc76 --- /dev/null +++ b/examples/tutorial/x/blog/messages.pb.go @@ -0,0 +1,1286 @@ +// Code generated by protoc-gen-gogo. DO NOT EDIT. +// source: x/blog/messages.proto + +/* + Package blog is a generated protocol buffer package. + + It is generated from these files: + x/blog/messages.proto + x/blog/state.proto + + It has these top-level messages: + CreateBlogMsg + RenameBlogMsg + ChangeBlogAuthorsMsg + CreatePostMsg + SetProfileMsg + Blog + Post + Profile +*/ +package blog + +import proto "github.com/gogo/protobuf/proto" +import fmt "fmt" +import math "math" + +import io "io" + +// Reference imports to suppress errors if they are not otherwise used. +var _ = proto.Marshal +var _ = fmt.Errorf +var _ = math.Inf + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the proto package it is being compiled against. +// A compilation error at this line likely means your copy of the +// proto package needs to be updated. +const _ = proto.GoGoProtoPackageIsVersion2 // please upgrade the proto package + +// CreateBlogMsg starts a new blog with a set of authors +type CreateBlogMsg struct { + // slug is a short, unique string used as primary key + Slug string `protobuf:"bytes,1,opt,name=slug,proto3" json:"slug,omitempty"` + // title is longer text used for display + Title string `protobuf:"bytes,2,opt,name=title,proto3" json:"title,omitempty"` + // initial set of authors (must be 1 - MaxAuthors) + Authors [][]byte `protobuf:"bytes,3,rep,name=authors" json:"authors,omitempty"` +} + +func (m *CreateBlogMsg) Reset() { *m = CreateBlogMsg{} } +func (m *CreateBlogMsg) String() string { return proto.CompactTextString(m) } +func (*CreateBlogMsg) ProtoMessage() {} +func (*CreateBlogMsg) Descriptor() ([]byte, []int) { return fileDescriptorMessages, []int{0} } + +func (m *CreateBlogMsg) GetSlug() string { + if m != nil { + return m.Slug + } + return "" +} + +func (m *CreateBlogMsg) GetTitle() string { + if m != nil { + return m.Title + } + return "" +} + +func (m *CreateBlogMsg) GetAuthors() [][]byte { + if m != nil { + return m.Authors + } + return nil +} + +// RenameBlogMsg updates the title of an existing blog +type RenameBlogMsg struct { + // slug is a short, unique string used as primary key + Slug string `protobuf:"bytes,1,opt,name=slug,proto3" json:"slug,omitempty"` + // title is longer text used for display + Title string `protobuf:"bytes,2,opt,name=title,proto3" json:"title,omitempty"` +} + +func (m *RenameBlogMsg) Reset() { *m = RenameBlogMsg{} } +func (m *RenameBlogMsg) String() string { return proto.CompactTextString(m) } +func (*RenameBlogMsg) ProtoMessage() {} +func (*RenameBlogMsg) Descriptor() ([]byte, []int) { return fileDescriptorMessages, []int{1} } + +func (m *RenameBlogMsg) GetSlug() string { + if m != nil { + return m.Slug + } + return "" +} + +func (m *RenameBlogMsg) GetTitle() string { + if m != nil { + return m.Title + } + return "" +} + +// ChangeBlogAuthorsMsg adds or removes an author from the blog's +// authorized author list +type ChangeBlogAuthorsMsg struct { + // whether we add or remove them + Add bool `protobuf:"varint,1,opt,name=add,proto3" json:"add,omitempty"` + // author to add or remove + Author []byte `protobuf:"bytes,2,opt,name=author,proto3" json:"author,omitempty"` +} + +func (m *ChangeBlogAuthorsMsg) Reset() { *m = ChangeBlogAuthorsMsg{} } +func (m *ChangeBlogAuthorsMsg) String() string { return proto.CompactTextString(m) } +func (*ChangeBlogAuthorsMsg) ProtoMessage() {} +func (*ChangeBlogAuthorsMsg) Descriptor() ([]byte, []int) { return fileDescriptorMessages, []int{2} } + +func (m *ChangeBlogAuthorsMsg) GetAdd() bool { + if m != nil { + return m.Add + } + return false +} + +func (m *ChangeBlogAuthorsMsg) GetAuthor() []byte { + if m != nil { + return m.Author + } + return nil +} + +// CreatePostMsg adds a post to a blog +type CreatePostMsg struct { + // blog is the slug of the blog this post belongs to + Blog string `protobuf:"bytes,1,opt,name=blog,proto3" json:"blog,omitempty"` + Title string `protobuf:"bytes,2,opt,name=title,proto3" json:"title,omitempty"` + Text string `protobuf:"bytes,3,opt,name=text,proto3" json:"text,omitempty"` + // author is optional, by default the first signer, + // only needed if it is multisig + Author []byte `protobuf:"bytes,4,opt,name=author,proto3" json:"author,omitempty"` +} + +func (m *CreatePostMsg) Reset() { *m = CreatePostMsg{} } +func (m *CreatePostMsg) String() string { return proto.CompactTextString(m) } +func (*CreatePostMsg) ProtoMessage() {} +func (*CreatePostMsg) Descriptor() ([]byte, []int) { return fileDescriptorMessages, []int{3} } + +func (m *CreatePostMsg) GetBlog() string { + if m != nil { + return m.Blog + } + return "" +} + +func (m *CreatePostMsg) GetTitle() string { + if m != nil { + return m.Title + } + return "" +} + +func (m *CreatePostMsg) GetText() string { + if m != nil { + return m.Text + } + return "" +} + +func (m *CreatePostMsg) GetAuthor() []byte { + if m != nil { + return m.Author + } + return nil +} + +// SetProfileMsg will create or update a profile +type SetProfileMsg struct { + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + Description string `protobuf:"bytes,2,opt,name=description,proto3" json:"description,omitempty"` + // author is optional, by default the first signer, + // only needed if it is multisig + Author []byte `protobuf:"bytes,3,opt,name=author,proto3" json:"author,omitempty"` +} + +func (m *SetProfileMsg) Reset() { *m = SetProfileMsg{} } +func (m *SetProfileMsg) String() string { return proto.CompactTextString(m) } +func (*SetProfileMsg) ProtoMessage() {} +func (*SetProfileMsg) Descriptor() ([]byte, []int) { return fileDescriptorMessages, []int{4} } + +func (m *SetProfileMsg) GetName() string { + if m != nil { + return m.Name + } + return "" +} + +func (m *SetProfileMsg) GetDescription() string { + if m != nil { + return m.Description + } + return "" +} + +func (m *SetProfileMsg) GetAuthor() []byte { + if m != nil { + return m.Author + } + return nil +} + +func init() { + proto.RegisterType((*CreateBlogMsg)(nil), "blog.CreateBlogMsg") + proto.RegisterType((*RenameBlogMsg)(nil), "blog.RenameBlogMsg") + proto.RegisterType((*ChangeBlogAuthorsMsg)(nil), "blog.ChangeBlogAuthorsMsg") + proto.RegisterType((*CreatePostMsg)(nil), "blog.CreatePostMsg") + proto.RegisterType((*SetProfileMsg)(nil), "blog.SetProfileMsg") +} +func (m *CreateBlogMsg) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *CreateBlogMsg) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Slug) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Slug))) + i += copy(dAtA[i:], m.Slug) + } + if len(m.Title) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Title))) + i += copy(dAtA[i:], m.Title) + } + if len(m.Authors) > 0 { + for _, b := range m.Authors { + dAtA[i] = 0x1a + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(b))) + i += copy(dAtA[i:], b) + } + } + return i, nil +} + +func (m *RenameBlogMsg) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *RenameBlogMsg) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Slug) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Slug))) + i += copy(dAtA[i:], m.Slug) + } + if len(m.Title) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Title))) + i += copy(dAtA[i:], m.Title) + } + return i, nil +} + +func (m *ChangeBlogAuthorsMsg) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *ChangeBlogAuthorsMsg) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if m.Add { + dAtA[i] = 0x8 + i++ + if m.Add { + dAtA[i] = 1 + } else { + dAtA[i] = 0 + } + i++ + } + if len(m.Author) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Author))) + i += copy(dAtA[i:], m.Author) + } + return i, nil +} + +func (m *CreatePostMsg) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *CreatePostMsg) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Blog) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Blog))) + i += copy(dAtA[i:], m.Blog) + } + if len(m.Title) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Title))) + i += copy(dAtA[i:], m.Title) + } + if len(m.Text) > 0 { + dAtA[i] = 0x1a + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Text))) + i += copy(dAtA[i:], m.Text) + } + if len(m.Author) > 0 { + dAtA[i] = 0x22 + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Author))) + i += copy(dAtA[i:], m.Author) + } + return i, nil +} + +func (m *SetProfileMsg) Marshal() (dAtA []byte, err error) { + size := m.Size() + dAtA = make([]byte, size) + n, err := m.MarshalTo(dAtA) + if err != nil { + return nil, err + } + return dAtA[:n], nil +} + +func (m *SetProfileMsg) MarshalTo(dAtA []byte) (int, error) { + var i int + _ = i + var l int + _ = l + if len(m.Name) > 0 { + dAtA[i] = 0xa + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Name))) + i += copy(dAtA[i:], m.Name) + } + if len(m.Description) > 0 { + dAtA[i] = 0x12 + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Description))) + i += copy(dAtA[i:], m.Description) + } + if len(m.Author) > 0 { + dAtA[i] = 0x1a + i++ + i = encodeVarintMessages(dAtA, i, uint64(len(m.Author))) + i += copy(dAtA[i:], m.Author) + } + return i, nil +} + +func encodeVarintMessages(dAtA []byte, offset int, v uint64) int { + for v >= 1<<7 { + dAtA[offset] = uint8(v&0x7f | 0x80) + v >>= 7 + offset++ + } + dAtA[offset] = uint8(v) + return offset + 1 +} +func (m *CreateBlogMsg) Size() (n int) { + var l int + _ = l + l = len(m.Slug) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Title) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + if len(m.Authors) > 0 { + for _, b := range m.Authors { + l = len(b) + n += 1 + l + sovMessages(uint64(l)) + } + } + return n +} + +func (m *RenameBlogMsg) Size() (n int) { + var l int + _ = l + l = len(m.Slug) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Title) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + return n +} + +func (m *ChangeBlogAuthorsMsg) Size() (n int) { + var l int + _ = l + if m.Add { + n += 2 + } + l = len(m.Author) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + return n +} + +func (m *CreatePostMsg) Size() (n int) { + var l int + _ = l + l = len(m.Blog) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Title) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Text) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Author) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + return n +} + +func (m *SetProfileMsg) Size() (n int) { + var l int + _ = l + l = len(m.Name) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Description) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + l = len(m.Author) + if l > 0 { + n += 1 + l + sovMessages(uint64(l)) + } + return n +} + +func sovMessages(x uint64) (n int) { + for { + n++ + x >>= 7 + if x == 0 { + break + } + } + return n +} +func sozMessages(x uint64) (n int) { + return sovMessages(uint64((x << 1) ^ uint64((int64(x) >> 63)))) +} +func (m *CreateBlogMsg) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: CreateBlogMsg: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: CreateBlogMsg: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Slug", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Slug = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Title", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Title = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Authors", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Authors = append(m.Authors, make([]byte, postIndex-iNdEx)) + copy(m.Authors[len(m.Authors)-1], dAtA[iNdEx:postIndex]) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipMessages(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthMessages + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *RenameBlogMsg) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: RenameBlogMsg: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: RenameBlogMsg: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Slug", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Slug = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Title", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Title = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipMessages(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthMessages + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *ChangeBlogAuthorsMsg) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: ChangeBlogAuthorsMsg: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: ChangeBlogAuthorsMsg: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field Add", wireType) + } + var v int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + v |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + m.Add = bool(v != 0) + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Author", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Author = append(m.Author[:0], dAtA[iNdEx:postIndex]...) + if m.Author == nil { + m.Author = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipMessages(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthMessages + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *CreatePostMsg) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: CreatePostMsg: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: CreatePostMsg: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Blog", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Blog = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Title", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Title = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Text", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Text = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 4: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Author", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Author = append(m.Author[:0], dAtA[iNdEx:postIndex]...) + if m.Author == nil { + m.Author = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipMessages(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthMessages + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func (m *SetProfileMsg) Unmarshal(dAtA []byte) error { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + preIndex := iNdEx + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + fieldNum := int32(wire >> 3) + wireType := int(wire & 0x7) + if wireType == 4 { + return fmt.Errorf("proto: SetProfileMsg: wiretype end group for non-group") + } + if fieldNum <= 0 { + return fmt.Errorf("proto: SetProfileMsg: illegal tag %d (wire type %d)", fieldNum, wire) + } + switch fieldNum { + case 1: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Name", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Name = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 2: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Description", wireType) + } + var stringLen uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + stringLen |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + intStringLen := int(stringLen) + if intStringLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + intStringLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Description = string(dAtA[iNdEx:postIndex]) + iNdEx = postIndex + case 3: + if wireType != 2 { + return fmt.Errorf("proto: wrong wireType = %d for field Author", wireType) + } + var byteLen int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return ErrIntOverflowMessages + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + byteLen |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + if byteLen < 0 { + return ErrInvalidLengthMessages + } + postIndex := iNdEx + byteLen + if postIndex > l { + return io.ErrUnexpectedEOF + } + m.Author = append(m.Author[:0], dAtA[iNdEx:postIndex]...) + if m.Author == nil { + m.Author = []byte{} + } + iNdEx = postIndex + default: + iNdEx = preIndex + skippy, err := skipMessages(dAtA[iNdEx:]) + if err != nil { + return err + } + if skippy < 0 { + return ErrInvalidLengthMessages + } + if (iNdEx + skippy) > l { + return io.ErrUnexpectedEOF + } + iNdEx += skippy + } + } + + if iNdEx > l { + return io.ErrUnexpectedEOF + } + return nil +} +func skipMessages(dAtA []byte) (n int, err error) { + l := len(dAtA) + iNdEx := 0 + for iNdEx < l { + var wire uint64 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowMessages + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + wire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + wireType := int(wire & 0x7) + switch wireType { + case 0: + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowMessages + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + iNdEx++ + if dAtA[iNdEx-1] < 0x80 { + break + } + } + return iNdEx, nil + case 1: + iNdEx += 8 + return iNdEx, nil + case 2: + var length int + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowMessages + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + length |= (int(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + iNdEx += length + if length < 0 { + return 0, ErrInvalidLengthMessages + } + return iNdEx, nil + case 3: + for { + var innerWire uint64 + var start int = iNdEx + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return 0, ErrIntOverflowMessages + } + if iNdEx >= l { + return 0, io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + innerWire |= (uint64(b) & 0x7F) << shift + if b < 0x80 { + break + } + } + innerWireType := int(innerWire & 0x7) + if innerWireType == 4 { + break + } + next, err := skipMessages(dAtA[start:]) + if err != nil { + return 0, err + } + iNdEx = start + next + } + return iNdEx, nil + case 4: + return iNdEx, nil + case 5: + iNdEx += 4 + return iNdEx, nil + default: + return 0, fmt.Errorf("proto: illegal wireType %d", wireType) + } + } + panic("unreachable") +} + +var ( + ErrInvalidLengthMessages = fmt.Errorf("proto: negative length found during unmarshaling") + ErrIntOverflowMessages = fmt.Errorf("proto: integer overflow") +) + +func init() { proto.RegisterFile("x/blog/messages.proto", fileDescriptorMessages) } + +var fileDescriptorMessages = []byte{ + // 273 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x94, 0x91, 0xb1, 0x4e, 0xc3, 0x30, + 0x14, 0x45, 0x09, 0x09, 0x05, 0x1e, 0x8d, 0x54, 0x59, 0x05, 0x79, 0x8a, 0xa2, 0x4c, 0x9d, 0xe8, + 0xc0, 0xc4, 0x06, 0xed, 0x8c, 0x54, 0xb9, 0x33, 0x83, 0x4b, 0x1e, 0x6e, 0x24, 0x37, 0xae, 0xec, + 0x57, 0xa9, 0x9f, 0xc1, 0x67, 0x31, 0xf2, 0x09, 0x28, 0xfc, 0x08, 0xb2, 0x93, 0x48, 0x61, 0x60, + 0xe8, 0x76, 0xdf, 0x55, 0x72, 0x8e, 0xae, 0x0c, 0xb7, 0xc7, 0xf9, 0x46, 0x1b, 0x35, 0xdf, 0xa1, + 0x73, 0x52, 0xa1, 0xbb, 0xdf, 0x5b, 0x43, 0x86, 0x25, 0xbe, 0x2c, 0xd6, 0x90, 0x2e, 0x2d, 0x4a, + 0xc2, 0x85, 0x36, 0xea, 0xc5, 0x29, 0xc6, 0x20, 0x71, 0xfa, 0xa0, 0x78, 0x94, 0x47, 0xb3, 0x6b, + 0x11, 0x32, 0x9b, 0xc2, 0x05, 0x55, 0xa4, 0x91, 0x9f, 0x87, 0xb2, 0x3d, 0x18, 0x87, 0x4b, 0x79, + 0xa0, 0xad, 0xb1, 0x8e, 0xc7, 0x79, 0x3c, 0x1b, 0x8b, 0xfe, 0x2c, 0x1e, 0x21, 0x15, 0x58, 0xcb, + 0xdd, 0xe9, 0xd0, 0xe2, 0x09, 0xa6, 0xcb, 0xad, 0xac, 0x55, 0xf8, 0xf5, 0xb9, 0xe5, 0x79, 0xc2, + 0x04, 0x62, 0x59, 0x96, 0x01, 0x70, 0x25, 0x7c, 0x64, 0x77, 0x30, 0x6a, 0x7d, 0x01, 0x30, 0x16, + 0xdd, 0x55, 0x60, 0xbf, 0x68, 0x65, 0x1c, 0x75, 0x72, 0x3f, 0xb5, 0x97, 0xfb, 0xfc, 0xcf, 0x22, + 0x06, 0x09, 0xe1, 0x91, 0x78, 0xdc, 0x7e, 0xe9, 0xf3, 0x40, 0x93, 0xfc, 0xd1, 0xbc, 0x42, 0xba, + 0x46, 0x5a, 0x59, 0xf3, 0x5e, 0x69, 0xec, 0x34, 0x7e, 0x72, 0xaf, 0xf1, 0x99, 0xe5, 0x70, 0x53, + 0xa2, 0x7b, 0xb3, 0xd5, 0x9e, 0x2a, 0x53, 0x77, 0xb2, 0x61, 0x35, 0xc0, 0xc7, 0x43, 0xfc, 0x62, + 0xf2, 0xd9, 0x64, 0xd1, 0x57, 0x93, 0x45, 0xdf, 0x4d, 0x16, 0x7d, 0xfc, 0x64, 0x67, 0x9b, 0x51, + 0x78, 0xb6, 0x87, 0xdf, 0x00, 0x00, 0x00, 0xff, 0xff, 0xb0, 0x47, 0x61, 0x31, 0xcf, 0x01, 0x00, + 0x00, +} diff --git a/examples/tutorial/x/blog/messages.proto b/examples/tutorial/x/blog/messages.proto new file mode 100644 index 00000000..5b621530 --- /dev/null +++ b/examples/tutorial/x/blog/messages.proto @@ -0,0 +1,51 @@ +syntax = "proto3"; + +package blog; + +// CreateBlogMsg starts a new blog with a set of authors +message CreateBlogMsg { + // slug is a short, unique string used as primary key + string slug = 1; + // title is longer text used for display + string title = 2; + // initial set of authors (must be 1 - MaxAuthors) + repeated bytes authors = 3; +} + +// RenameBlogMsg updates the title of an existing blog +message RenameBlogMsg { + // slug is a short, unique string used as primary key + string slug = 1; + // title is longer text used for display + string title = 2; +} + +// ChangeBlogAuthorsMsg adds or removes an author from the blog's +// authorized author list +message ChangeBlogAuthorsMsg { + // whether we add or remove them + bool add = 1; + // author to add or remove + bytes author = 2; +} + +// CreatePostMsg adds a post to a blog +message CreatePostMsg { + // blog is the slug of the blog this post belongs to + string blog = 1; + string title = 2; + string text = 3; + // author is optional, by default the first signer, + // only needed if it is multisig + bytes author = 4; + // creation height is added by handler +} + +// SetProfileMsg will create or update a profile +message SetProfileMsg { + string name = 1; + string description = 2; + // author is optional, by default the first signer, + // only needed if it is multisig + bytes author = 3; +} From a200faa34bb315c3c25703138da98daaa1025317 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Fri, 22 Jun 2018 21:42:24 +0200 Subject: [PATCH 16/19] Document message validation and paths --- docs/tutorial/datamodel.rst | 8 +- docs/tutorial/messages.rst | 24 +++-- examples/tutorial/x/blog/errors.go | 6 +- examples/tutorial/x/blog/models.go | 8 +- examples/tutorial/x/blog/msgs.go | 144 +++++++++++++++++++++++++++++ 5 files changed, 166 insertions(+), 24 deletions(-) create mode 100644 examples/tutorial/x/blog/msgs.go diff --git a/docs/tutorial/datamodel.rst b/docs/tutorial/datamodel.rst index ad358efe..9b617fa0 100644 --- a/docs/tutorial/datamodel.rst +++ b/docs/tutorial/datamodel.rst @@ -172,7 +172,7 @@ pass: .. literalinclude:: ../../examples/tutorial/x/blog/models.go :language: go - :lines: 16-28 + :lines: 15-27 Errors ~~~~~~ @@ -239,13 +239,13 @@ data is validated before saving, but we also need to make sure that all data is the proper type of object before saving. Unfortunately, this is quite difficult to do compile-time without generic, so a typical apporach is to embed the -`orm.Bucket <>`_ +`orm.Bucket `_ in another struct and just force validation of the object type runtime before save. .. literalinclude:: ../../examples/tutorial/x/blog/models.go :language: go - :lines: 101-128 + :lines: 99-124 Secondary Indexes ------------------ @@ -259,7 +259,7 @@ and weave provides nice support for this functionality. .. literalinclude:: ../../examples/tutorial/x/blog/models.go :language: go - :lines: 141-165 + :lines: 139-159 We add a indexing method to take any object, enforce the type to be a proper Post, then extract the index we want. This diff --git a/docs/tutorial/messages.rst b/docs/tutorial/messages.rst index a4c6d586..198ab258 100644 --- a/docs/tutorial/messages.rst +++ b/docs/tutorial/messages.rst @@ -72,14 +72,20 @@ In the blog example, we can imagine: * Create Profile * Modify Profile (which may be merged with above) -We can create a protobuf message for each of these types: +We can create a protobuf message for +`each of these types `_: -.. literal-include +.. literalinclude:: ../../examples/tutorial/x/blog/messages.proto + :language: proto + :lines: 5-13 -And then add a ``Path`` method that returns a constant based on -the type: +And then add a ``Path`` method that +`returns a constant `_ +based on the type: -.. literal-include +.. literalinclude:: ../../examples/tutorial/x/blog/msgs.go + :language: go + :lines: 9-10,25,32-38 Validation ---------- @@ -98,8 +104,6 @@ The validation on Messages should be a lot more thorough and well tested than the validation on data models, which is as much documentation of acceptable values as it is runtime security. -**TODO** - -* Messages vs. Transactions, what is the distinction? -* Defining message types - needed data, non-maleability -* Validation of Messages +.. literalinclude:: ../../examples/tutorial/x/blog/msgs.go + :language: go + :lines: 27-30,40-61 diff --git a/examples/tutorial/x/blog/errors.go b/examples/tutorial/x/blog/errors.go index f4fdfe3b..538ce7f7 100644 --- a/examples/tutorial/x/blog/errors.go +++ b/examples/tutorial/x/blog/errors.go @@ -17,7 +17,7 @@ const ( var ( errTitleTooLong = fmt.Errorf("Title is too long") errTextTooLong = fmt.Errorf("Text is too long") - errNameTooLong = fmt.Errorf("Name is too long") + errInvalidName = fmt.Errorf("Name is too long") errDescriptionTooLong = fmt.Errorf("Description is too long") errNoAuthor = fmt.Errorf("No author for post") @@ -33,8 +33,8 @@ func ErrTitleTooLong() error { func ErrTextTooLong() error { return errors.WithCode(errTextTooLong, CodeInvalidText) } -func ErrNameTooLong() error { - return errors.WithCode(errNameTooLong, CodeInvalidText) +func ErrInvalidName() error { + return errors.WithCode(errInvalidName, CodeInvalidText) } func ErrDescriptionTooLong() error { return errors.WithCode(errDescriptionTooLong, CodeInvalidText) diff --git a/examples/tutorial/x/blog/models.go b/examples/tutorial/x/blog/models.go index 39d96583..eb81e772 100644 --- a/examples/tutorial/x/blog/models.go +++ b/examples/tutorial/x/blog/models.go @@ -9,12 +9,6 @@ import ( //----- Blog ------- -const MaxAuthors = 10 -const MaxTitleLength = 100 -const MaxTextLength = 20 * 1000 -const MaxNameLength = 30 -const MaxDescriptionLength = 280 - // enforce that Blog fulfils desired interface compile-time var _ orm.CloneableData = (*Blog)(nil) @@ -84,7 +78,7 @@ var _ orm.CloneableData = (*Profile)(nil) // Validate enforces limits of text and title size func (p *Profile) Validate() error { if len(p.Name) > MaxNameLength { - return ErrNameTooLong() + return ErrInvalidName() } if len(p.Description) > MaxDescriptionLength { return ErrDescriptionTooLong() diff --git a/examples/tutorial/x/blog/msgs.go b/examples/tutorial/x/blog/msgs.go new file mode 100644 index 00000000..2b837802 --- /dev/null +++ b/examples/tutorial/x/blog/msgs.go @@ -0,0 +1,144 @@ +package blog + +import ( + "regexp" + + "github.com/confio/weave" +) + +const ( + PathCreateBlogMsg = "blog/create" + PathRenameBlogMsg = "blog/rename" + PathChangeBlogAuthorsMsg = "blog/authors" + PathCreatePostMsg = "blog/post" + PathSetProfileMsg = "blog/profile" + + MinAuthors = 1 + MaxAuthors = 10 + MinTitleLength = 8 + MaxTitleLength = 100 + MinTextLength = 200 + MaxTextLength = 20 * 1000 + MinNameLength = 6 + MaxNameLength = 30 + MaxDescriptionLength = 280 +) + +var ( + // IsValidName is the RegExp to ensure valid profile and blog names + IsValidName = regexp.MustCompile(`^[a-zA-Z0-9_\-\.]{6,30}$`).MatchString +) + +// Ensure we implement the Msg interface +var _ weave.Msg = (*CreateBlogMsg)(nil) + +// Path returns the routing path for this message +func (CreateBlogMsg) Path() string { + return PathCreateBlogMsg +} + +// Validate makes sure that this is sensible +func (s *CreateBlogMsg) Validate() error { + // validate the strings + if !IsValidName(s.Slug) { + return ErrInvalidName() + } + if len(s.Title) < MinTitleLength || len(s.Title) > MaxTitleLength { + return ErrTitleTooLong() + } + // check the number of authors + authors := len(s.Authors) + if authors < MinAuthors || authors > MaxAuthors { + return ErrInvalidAuthorCount(authors) + } + // and validate all of them are valid addresses + for _, a := range s.Authors { + if err := weave.Address(a).Validate(); err != nil { + return err + } + } + return nil +} + +// Ensure we implement the Msg interface +var _ weave.Msg = (*RenameBlogMsg)(nil) + +// Path returns the routing path for this message +func (RenameBlogMsg) Path() string { + return PathRenameBlogMsg +} + +// Validate makes sure that this is sensible +func (s *RenameBlogMsg) Validate() error { + if !IsValidName(s.Slug) { + return ErrInvalidName() + } + if len(s.Title) < MinTitleLength || len(s.Title) > MaxTitleLength { + return ErrTitleTooLong() + } + return nil +} + +// Ensure we implement the Msg interface +var _ weave.Msg = (*ChangeBlogAuthorsMsg)(nil) + +// Path returns the routing path for this message +func (ChangeBlogAuthorsMsg) Path() string { + return PathChangeBlogAuthorsMsg +} + +// Validate makes sure that this is sensible +func (s *ChangeBlogAuthorsMsg) Validate() error { + // Validate if this is a valid Address + return weave.Address(s.Author).Validate() +} + +// Ensure we implement the Msg interface +var _ weave.Msg = (*CreatePostMsg)(nil) + +// Path returns the routing path for this message +func (CreatePostMsg) Path() string { + return PathCreatePostMsg +} + +// Validate makes sure that this is sensible +func (s *CreatePostMsg) Validate() error { + if !IsValidName(s.Blog) { + return ErrInvalidName() + } + if len(s.Title) < MinTitleLength || len(s.Title) > MaxTitleLength { + return ErrTitleTooLong() + } + if len(s.Text) < MinTextLength || len(s.Text) > MaxTextLength { + return ErrTextTooLong() + } + + // if an author is present, validate it is a valid address + if len(s.Author) > 0 { + return weave.Address(s.Author).Validate() + } + return nil +} + +// Ensure we implement the Msg interface +var _ weave.Msg = (*SetProfileMsg)(nil) + +// Path returns the routing path for this message +func (SetProfileMsg) Path() string { + return PathSetProfileMsg +} + +// Validate makes sure that this is sensible +func (s *SetProfileMsg) Validate() error { + if !IsValidName(s.Name) { + return ErrInvalidName() + } + if len(s.Description) > MaxDescriptionLength { + return ErrDescriptionTooLong() + } + // if an author is present, validate it is a valid address + if len(s.Author) > 0 { + return weave.Address(s.Author).Validate() + } + return nil +} From cbb3cfdd8585a90e229f73b1d872a0c54fcdc7f9 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Fri, 22 Jun 2018 22:31:34 +0200 Subject: [PATCH 17/19] Text description of how to build handlers (no code) --- docs/tutorial/handler.rst | 137 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 133 insertions(+), 4 deletions(-) diff --git a/docs/tutorial/handler.rst b/docs/tutorial/handler.rst index eca84891..4eb7126b 100644 --- a/docs/tutorial/handler.rst +++ b/docs/tutorial/handler.rst @@ -2,8 +2,137 @@ Message Handlers ---------------- -**TODO** +A message is a statement of intention, and wrapped in a transaction, +while provides authorization to this intention. Once this message +ends up in the ABCI application and is to be processed, we send it +to a `Handler `_, +which we have registered for this application. + +Check vs Deliver +---------------- + +If you look at the definiton of a *Handler*, you will see it is +responsible for *Check* and *Deliver*. These are similar logic, but +there is an important distinction. *Check* is performed when +a client proposes the transaction to the mempool, before it is +added to a block. It is meant as a quick filter to weed out garbage +transactions before writing them to the blockchain. The state it +provides is a scratch buffer around the last committed state and +will be discarded next block, so any writes here are never writen +to disk. + +*Deliver* is performed after the transaction was writen to +the block. Upon consensus, every node will processes the block +by calling *BeginBlock*, *Deliver* for every transaction in the block, +and finally *EndBlock* and *Commit*. *Deliver* will be called in +the same order on every node and must make the **exact same changes** +on every node, both now and in the future when the blocks are +replayed. Even the slightest deviation will cause the merkle root +of the store at the end of the block to differ with other nodes, +and thus kick the deviating nodes out of consensus. +(Note that *Check* may actually vary between nodes without breaking +consensus rules, although we generally keep this deterministic as well). + +Writing a Handler +----------------- + +We usually can write a separate handler for each message type, +although you can register multiple messages with the same +handler if you reuse most of the code. Let's focus on the +simplest case, and the handler for +`adding a Post `_ +to an existing blog. + +Remember that we have to fulfill both *Check* and *Deliver* methods, +and they share most of the same validation logic. A typical +approach is to define a *validate* method that parses the +proper message out of the transaction, verify all authorization +preconditions are fulfilled by the transaction, and possibly +check the current state of the blockchain to see if the action +is allowed. If the *validate* method doesn't return an error, +then *Check* will return the expected cost of the transaction, +while *Deliver* will actually peform the action and update +the blockchain state accordingly. + +Note that we can generally assume that *Handlers* are wrapped +by a `Savepoint Decorator `_, +and that if *Deliver* returns an error after updating some +objects, those update will be discarded. This means you can +treat *Handlers* as atomic actions, all or none, and not worry +too much about cleaning up partially finished state changes +if a later portion fails. + +In the case of adding a post, we must first ``validate`` +that the transaction hold the proper message, the message +passes all internal validation checks, the blog named +in the message exists in our state, and the author both +signed this transaction and belongs to authorized authors +for this blog... What a mouthful. Since *validate* must load +the relevant blog for authorization, which we may want to use +elsewhere in the Handler as well, we return it from the *validate* +call as well to avoid loading it twice. + +**TODO** include validate code +Like: https://github.com/iov-one/bcp-demo/blob/master/x/escrow/handler.go#L108-L144 + + +Once ``validate`` is implemented, ``Check`` must ensure it is valid +and then return a rough cost of the message, which may be based +on the storage cost of the text of the post. This return value +is similar to the concept of *gas* in ethereum, although it doesn't +count to the fees yet, but rather is used by tendermint to prioritize +the transactions to fit in a block. + +**TODO** include check code +Like: https://github.com/iov-one/bcp-demo/blob/master/x/escrow/handler.go#L47-L61 + +``Deliver`` also makes use of ``validate`` to perform the original +checks, then it increments the article count on the *Blog*, and +calculates the key of the *Post* based on the *Blog* slug and the count +of this article. It then saves both the *Post* and the updated *Blog*. +Note how the *Handler* has access to the height of the current block +being processes (which is deterministic in contrast to a timestamp), +and can attach that to the *Post* to allow a client to get a timestamp +from the relevant header. (Actually the *Handler* has access to the full +header, which contains a timestamp, +`which may or may not be reliable `_.) + +**TODO** include deliver code +Like: https://github.com/iov-one/bcp-demo/blob/master/x/escrow/handler.go#L62-L107 + +Routing Messages to Handler +--------------------------- + +After defining all the *Messages*, along with *Handlers* for them all, +we need to make sure the application knows about them. When we +instantiate an application, we define a +`Router and then register all handlers `_ +we are interested in. This allows the application +to explicitly state, not only which messages it supports +(in the Tx struct), but also which business logic will process +each message. + +.. literalinclude:: ../../examples/mycoind/app/app.go + :language: go + :lines: 56-62 + +In order to make it easy for applications to register our extension as +one piece and not worry about attaching every *Handler* we provide, +it is common practice for an extension to provide a ``RegisterRoutes`` +function that will take a *Router* (or the more permissive *Registry* +interface), and any information it needs to construct instances +of all the handlers. This ``RegisterRoutes`` function is responsible +for instantiating all the *Handlers* with the desired configuration +and attaching them to the *Router* to process the matching +*Message* type (identified by it's *Path*): + +**TODO** include our code + +.. code:: go + + // RegisterRoutes will instantiate and register + // all handlers in this package + func RegisterRoutes(r weave.Registry, auth x.Authenticator) { + r.Handle(pathSendMsg, NewSendHandler(auth)) + } -* Writing Handler for one message -* Routing messages to Handler -* Check vs Deliver From acdb56b9473785eaed55eb0ca4698e90fb0e7cb9 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Fri, 22 Jun 2018 22:59:06 +0200 Subject: [PATCH 18/19] Explain how to register QueryHandlers --- docs/tutorial/messages.rst | 4 ++-- docs/tutorial/queries.rst | 38 +++++++++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/docs/tutorial/messages.rst b/docs/tutorial/messages.rst index 198ab258..621466e0 100644 --- a/docs/tutorial/messages.rst +++ b/docs/tutorial/messages.rst @@ -85,7 +85,7 @@ based on the type: .. literalinclude:: ../../examples/tutorial/x/blog/msgs.go :language: go - :lines: 9-10,25,32-38 + :lines: 9-10,25-26,32-38 Validation ---------- @@ -106,4 +106,4 @@ of acceptable values as it is runtime security. .. literalinclude:: ../../examples/tutorial/x/blog/msgs.go :language: go - :lines: 27-30,40-61 + :lines: 27-31,40-61 diff --git a/docs/tutorial/queries.rst b/docs/tutorial/queries.rst index ead4d116..9f2e31e9 100644 --- a/docs/tutorial/queries.rst +++ b/docs/tutorial/queries.rst @@ -2,4 +2,40 @@ Processing Queries ------------------ -**TODO** +We don't only want to modify data, but allow the clients +to query the current state. Clients can call ``/abci_query`` +to tendermint which will make a +`Query `_ +request on the weave application. + +Note how it uses a +`QueryRouter `_ +to send queries to different +`QueryHandlers `_ +based on their *Path*? It just happens that *Buckets* implement +the *QueryHandler* interface, and now that we understand how +*RegisterRoutes* work, this should be quite simple. + +When constructing the application, we register QueryHandlers from +every extension we support onto a main QueryRouter that handles +all requests. Each extension is responsible for registering it's +*Bucket* (or *Buckets*) under appropriate paths. Here we see how +the escrow extension +`registers its bucket `_ +to handle all querys for the ``/escrows`` path: + +.. code:: go + + // RegisterQuery will register this bucket as "/escrows" + func RegisterQuery(qr weave.QueryRouter) { + NewBucket().Register("escrows", qr) + } + +**TODO** demo with code from tutorial. + +Notice that this automatically handles prefix queries, +with paths like ``/escrows?prefix`` as well as queries +on secondary indexes like ``/escrows/recipient``. All +you have to do is set up the bucket properly and attach it +to the *QueryRouter*. + From b7e8e2eb4d900869d991d2a507580eae170667d1 Mon Sep 17 00:00:00 2001 From: Ethan Frey Date: Tue, 26 Jun 2018 15:22:49 +0200 Subject: [PATCH 19/19] White background for code samples --- docs/_static/css/custom.css | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css index 6fa852cb..fefdf9d6 100644 --- a/docs/_static/css/custom.css +++ b/docs/_static/css/custom.css @@ -1,3 +1,7 @@ +.highlight { + background-color: #ffffff; +} + .rst-content code.literal { color: #1569C7; font-size: 80%;