Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use the IPFS core object to post block data during proposal #178

Merged
merged 27 commits into from
Mar 15, 2021
Merged

Use the IPFS core object to post block data during proposal #178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
eb421c8
initial commit for posting data to ipfs using an ifps core object, an…
evan-forbes Mar 4, 2021
09cfd50
remove unused hash calc
evan-forbes Mar 5, 2021
8bb43e9
use NodeAdder instead of the full IPFS API object
evan-forbes Mar 5, 2021
7778e00
refactor flattening and adding the namespaces of extended data squares
evan-forbes Mar 5, 2021
b5f6c85
only post data to IPFS when an ipfs node is started
evan-forbes Mar 5, 2021
50b2a03
refactor and export NodeCollector
evan-forbes Mar 5, 2021
f13043d
export CidFromNamespacedSha256
evan-forbes Mar 8, 2021
c99bc38
switch to using NmtNodeAdder instead of using the old NmtNodeCollector
evan-forbes Mar 8, 2021
4ad27a8
change the default hash function to sha256 instead of sha3
evan-forbes Mar 8, 2021
e5c2e17
get out of dependency hell by adding an extra replace directive
evan-forbes Mar 8, 2021
2534b07
use a more minimal api
evan-forbes Mar 8, 2021
e74ed4e
add an extra block decoder register, as some of the CI tests don't re…
evan-forbes Mar 9, 2021
e9ca0b9
update default ifps config to reflect what the plugin uses
evan-forbes Mar 9, 2021
742b62a
update testing infra for TestPutBlock
evan-forbes Mar 9, 2021
9dfa413
switch e2e test ipfs dir
evan-forbes Mar 9, 2021
11c6313
use passed timeout instead of hard coded one
evan-forbes Mar 10, 2021
b560b90
don't post data to ipfs if there's not any block data
evan-forbes Mar 10, 2021
441e158
don't post the data to ipfs until after the proposal is made. Set Tim…
evan-forbes Mar 10, 2021
8e31785
add test cases
evan-forbes Mar 10, 2021
417a817
update ADR to reflect implementation changes
evan-forbes Mar 10, 2021
d53622f
review feedback: add docs
evan-forbes Mar 10, 2021
52fdd1b
defer the cancel for the timeout
evan-forbes Mar 10, 2021
4464a08
review feedback: add go-verifcid replace directive comment
evan-forbes Mar 10, 2021
c07235d
fix comment position
evan-forbes Mar 11, 2021
3bf7bca
register nmt codec to the global ipfs codec maps
evan-forbes Mar 15, 2021
8cbaa42
Merge branch 'master' into evan/put-ipfs-api-object-compute-twice
evan-forbes Mar 15, 2021
c0f3d1c
change NmtCodecName to "nmt-node"
evan-forbes Mar 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion blockchain/msgs_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ func TestBlockchainMessageVectors(t *testing.T) {
BlockRequest: &bcproto.BlockRequest{Height: math.MaxInt64}}},
"0a0a08ffffffffffffffff7f"},
{"BlockResponseMessage", &bcproto.Message{Sum: &bcproto.Message_BlockResponse{
BlockResponse: &bcproto.BlockResponse{Block: bpb}}}, "1ac0020abd020a5b0a02080b1803220b088092b8c398feffffff012a0212003a20ba28ef83fed712be8a128587469a4effda9f4f4895bd5638dab1f10775c9bed66a20e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85512130a0b48656c6c6f20576f726c6412001a0022001ac8010a300000000000000001000000000000000119251d276ad8a1831db7b86ead3f42c4e03093d50ecf026da7ecc3b0da8ec87d0a30ffffffffffffffffffffffffffffffff12d55aea72367d0d6a7899103a437c913ee5a6f9e86f42e0fe8743b0d8d3a1e812300000000000000001000000000000000119251d276ad8a1831db7b86ead3f42c4e03093d50ecf026da7ecc3b0da8ec87d1230ffffffffffffffffffffffffffffffff12d55aea72367d0d6a7899103a437c913ee5a6f9e86f42e0fe8743b0d8d3a1e8"},
BlockResponse: &bcproto.BlockResponse{Block: bpb}}}, "1ac0020abd020a5b0a02080b1803220b088092b8c398feffffff012a0212003a2016106406aededa633a265efd5833af53775af5a27c803518313c9b77e68875926a20e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85512130a0b48656c6c6f20576f726c6412001a0022001ac8010a3000000000000000010000000000000001266fcca850585e31b54074399b0dbb5b04b408b77d5f357a528ed3e04b63c3c10a30ffffffffffffffffffffffffffffffffe04e02811b28234428f1c1219d5627f61c02c415c175bdabd9e7934519baef07123000000000000000010000000000000001266fcca850585e31b54074399b0dbb5b04b408b77d5f357a528ed3e04b63c3c11230ffffffffffffffffffffffffffffffffe04e02811b28234428f1c1219d5627f61c02c415c175bdabd9e7934519baef07"},
{"NoBlockResponseMessage", &bcproto.Message{Sum: &bcproto.Message_NoBlockResponse{
NoBlockResponse: &bcproto.NoBlockResponse{Height: 1}}}, "12020801"},
{"NoBlockResponseMessage", &bcproto.Message{Sum: &bcproto.Message_NoBlockResponse{
Expand Down
2 changes: 1 addition & 1 deletion config/ipfs_config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ type IPFSConfig struct {
// locally for testing purposes.
func DefaultIPFSConfig() *IPFSConfig {
return &IPFSConfig{
ConfigRootPath: "ipfs/",
ConfigRootPath: ".ipfs/",
API: "/ip4/127.0.0.1/tcp/5002",
Gateway: "/ip4/127.0.0.1/tcp/5002",
Swarm: []string{"/ip4/0.0.0.0/tcp/4002", "/ip6/::/tcp/4002"},
Expand Down
18 changes: 17 additions & 1 deletion consensus/state.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package consensus

import (
"bytes"
"context"
"errors"
"fmt"
"io/ioutil"
Expand All @@ -11,7 +12,7 @@ import (
"time"

"github.com/gogo/protobuf/proto"

ipfsapi "github.com/ipfs/interface-go-ipfs-core"
cfg "github.com/lazyledger/lazyledger-core/config"
cstypes "github.com/lazyledger/lazyledger-core/consensus/types"
"github.com/lazyledger/lazyledger-core/crypto"
Expand Down Expand Up @@ -92,6 +93,8 @@ type State struct {
// store blocks and commits
blockStore sm.BlockStore

IpfsAPI ipfsapi.CoreAPI

// create and execute blocks
blockExec *sm.BlockExecutor

Expand Down Expand Up @@ -1100,6 +1103,19 @@ func (cs *State) defaultDecideProposal(height int64, round int32) {
} else if !cs.replayMode {
cs.Logger.Error("enterPropose: Error signing proposal", "height", height, "round", round, "err", err)
}

// post data to ipfs
// TODO(evan): don't hard code context and timeout
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's track minor todos like this this in a follup issue before we merge.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this case, we can pass-through context already, having an issue for that seems like an overmanagement :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not about management. But from my experience todos can be forgotten and hence it is good to capture them somewhere more visible. It does not need to be one issue per todo. We can collect all minor things like this in one issue (see for instance: #179).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tracked these todos in #204 in order to prevent github calling me out on twitter. 😄
https://twitter.com/github/status/1367885997527171073?s=20

if cs.IpfsAPI != nil {
// longer timeouts result in block proposers failing to propose blocks in time.
ctx, cancel := context.WithTimeout(context.Background(), time.Millisecond*1500)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a special context.TODO() for such cases.

Also, what's the rationale behind having timeout here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I didn't know that. In this case, we need the timeout because PutBlock is synchronous and will take too long, stopping the block proposer from proposing a block in time.

defer cancel()
// TODO: post data to IPFS in a goroutine
err := block.PutBlock(ctx, cs.IpfsAPI.Dag().Pinning())
if err != nil {
cs.Logger.Error(fmt.Sprintf("failure to post block data to IPFS: %s", err.Error()))
}
}
}

// Returns true if the proposal block is complete &&
Expand Down
11 changes: 1 addition & 10 deletions docs/lazy-adr/adr-002-ipld-da-sampling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ func RetrieveBlockData(ctx contex.Context, dah *DataAvailabilityHeader) (types.D
// the row to the Merkle Dag, in our case a Namespaced Merkle Tree.
// Note, that this method could also fill the DA header.
// The data will be pinned by default.
func (b *Block) PutBlock(ctx contex.Context) error
func (b *Block) PutBlock(ctx contex.Context, nodeAdder ipld.NodeAdder) error
```

We now describe the lower-level library that will be used by above methods.
Expand All @@ -185,15 +185,6 @@ func GetLeafData(
leafIndex uint32,
totalLeafs uint32, // this corresponds to the extended square width
) ([]byte, error)

// PutLeaves takes the namespaced leaves, a row of the from the extended data square,
// and calls nodes.DataSquareRowOrColumnRawInputParser of the ipld plugin.
// The resulting ipld nodes are passed to a Batch calling AddMany:
// https://github.com/ipfs/go-ipld-format/blob/d2e09424ddee0d7e696d01143318d32d0fb1ae63/batch.go#L29
// Note, that this method could also return the row and column roots.
// Tha caller is responsible for making sure that the leaves are sorted by namespace ID.
// The data will be pinned by default.
func PutLeaves(ctx contex.Context, namespacedLeaves [][]byte) error
```

`GetLeafData` can be used by above `ValidateAvailability` and `RetrieveBlock` and
Expand Down
3 changes: 3 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ require (
github.com/hdevalence/ed25519consensus v0.0.0-20201207055737-7fde80a9d5ff
github.com/ipfs/go-ipfs v0.8.0
github.com/ipfs/go-ipfs-config v0.12.0
github.com/ipfs/go-ipld-format v0.2.0
github.com/ipfs/interface-go-ipfs-core v0.4.0
github.com/lazyledger/lazyledger-core/p2p/ipld/plugin v0.0.0-20210219190522-0eccfb24e2aa
github.com/lazyledger/nmt v0.2.0
Expand Down Expand Up @@ -46,5 +47,7 @@ require (

replace (
github.com/ipfs/go-ipfs => github.com/lazyledger/go-ipfs v0.8.0-lazypatch
// adding an extra replace statement here enforces usage of our fork of go-cerifcid
github.com/ipfs/go-verifcid => github.com/lazyledger/go-verifcid v0.0.1-lazypatch
liamsi marked this conversation as resolved.
Show resolved Hide resolved
github.com/lazyledger/lazyledger-core/p2p/ipld/plugin => ./p2p/ipld/plugin
)
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,6 +623,8 @@ github.com/lazyledger/go-ipfs v0.8.0-lazypatch h1:8Dkw7Or6d0BmpFYFcxwgqWZ047BPGC
github.com/lazyledger/go-ipfs v0.8.0-lazypatch/go.mod h1:CE4cJkjUmwW5LwJP26KKEAZ11ZED0DxzSryfv5RMf6E=
github.com/lazyledger/go-leopard v0.0.0-20200604113236-298f93361181 h1:mUeCGuCgjZVadW4CzA2dMBq7p2BqaoCfpnKjxMmSaSE=
github.com/lazyledger/go-leopard v0.0.0-20200604113236-298f93361181/go.mod h1:v1o1CRihQ9i7hizx23KK4aR79lxA6VDUIzUCfDva0XQ=
github.com/lazyledger/go-verifcid v0.0.1-lazypatch h1:jAVwUw+DhuCzx5IcYpFh6d6HWxRRz8nhJ3rQo+vlFAc=
github.com/lazyledger/go-verifcid v0.0.1-lazypatch/go.mod h1:kXPYu0XqTNUKWA1h3M95UHjUqBzDwXVVt/RXZDjKJmQ=
github.com/lazyledger/merkletree v0.0.0-20201214195110-6901c4c3c75f h1:jbyPAH6o6hGte4RtZBaqWs2n4Fl6hS7qJGXX3qnjiy4=
github.com/lazyledger/merkletree v0.0.0-20201214195110-6901c4c3c75f/go.mod h1:10PA0NlnYtB8HrtwIDQAyTKWp8TEZ0zBZCGlYC/7+QE=
github.com/lazyledger/nmt v0.2.0 h1:jKpC+XMSc4P1asLvutSwyMlgUDnAvv1HAVyzal21MbA=
Expand Down
8 changes: 8 additions & 0 deletions node/node.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (
"time"

ipfscore "github.com/ipfs/go-ipfs/core"
coreapi "github.com/ipfs/go-ipfs/core/coreapi"
"github.com/ipfs/go-ipfs/core/node/libp2p"
"github.com/ipfs/go-ipfs/plugin/loader"
"github.com/ipfs/go-ipfs/repo/fsrepo"
Expand Down Expand Up @@ -960,6 +961,13 @@ func (n *Node) OnStart() error {
if err != nil {
return fmt.Errorf("failed to create IPFS node: %w", err)
}

ipfsAPI, err := coreapi.NewCoreAPI(n.ipfsNode)
if err != nil {
return fmt.Errorf("failed to create an instance of the IPFS core API: %w", err)
}

n.consensusState.IpfsAPI = ipfsAPI
}

return nil
Expand Down
Loading