Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node in committee not part of topology - need to assert #1505

Open
IdoZilberberg opened this issue Dec 17, 2019 · 1 comment
Open

Node in committee not part of topology - need to assert #1505

IdoZilberberg opened this issue Dec 17, 2019 · 1 comment
Assignees

Comments

@IdoZilberberg
Copy link
Contributor

Describe the bug
This is a result of our research on 17-DEC-2019 into problems on vchain 5111.
The committee contained a node with public address "461280" (shown on a "RECEIVED COMMITTEE" log message of Lean Helix).
But, the node was not in the topology, so trying to send a message to it resulted in "unknown public address" error.

The symptom in this specific case was failure of the leader to send a NEW_VIEW message because the missing node was the first in the list of target nodes.

But the deeper problem here is the presence of a node in the committee, without it being part of the topology.
We need some assertion (error) to logs that alerts that a node which entered the committee is not part of the topology.

Steps To Reproduce
vchain 5111, around 17-DEC-2019 11:34 IST.

See logz.io:

https://app.logz.io/#/dashboard/kibana/discover?_a=(columns:!(message,node),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:%5BlogzioCustomerIndex%5DYYMMDD,key:vcid,negate:!f,params:(query:5111,type:phrase),type:phrase,value:'5,111'),query:(match:(vcid:(query:5111,type:phrase))))),index:%5BlogzioCustomerIndex%5DYYMMDD,interval:auto,query:(language:lucene,query:%22COMMITTEE%22),sort:!('@timestamp',desc))&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-12h,mode:quick,to:now))&notFound=search&accountIds=65212

@noambergIL
Copy link
Contributor

with new management file...

  1. in public do we consider this non-issue ?
  2. in private it will generate a lot of logs but if there are enough memebers it won't cause a problem
  3. do we want to somehow not allow it via code in ONG managemt ?
    @gadcl @OdedWx @talkol

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants