Add small & stable identifiers

[moretea]

Zookeeper needs small identifiers, in the range (1...255). In the context of Habitat, they should be consistently mapped to a supervisor instance across all instances and preserved over restarts.

At the moment @IxDay and I do not see any way to generate those in a proper way that will ensure that zookeeper runs correctly.

Also see habitat-sh/core-plans#497

[reset]

@moretea do these need to be unique among the cluster as well?

[moretea]

I'm not sure what you mean by cluster lifetime, are you referring to the Zookeeper ensemble, or something else?

I assume one could remove a certain zookeeper instance (with a certain ID) from the ensemble, and then re-add another instance with that same ID later on, but this probably requires a custom state machine + logic to do it properly. I'd like to keep that discussion in habitat-sh/core-plans#497

The minimum thing we need to be able to do to make some progress is to get a unique ID in some range, that is not ever assigned again by Habitat. The next step would be to come up with a strategy to reclaim old ID's (potentially via the HTTP API)

[reset]

The minimum thing we need to be able to do to make some progress is to get a unique ID in some range, that is not ever assigned again by Habitat. The next step would be to come up with a strategy to reclaim old ID's (potentially via the HTTP API)

This is exactly what I needed, thank you!

[moretea]

To add a bit to this, I think that it' might not be enough to link these values directly to the lifetime of a gossip member. In our case, we might have to do some cleanup action in zookeeper before that number can be re-used again.

[eeyun]

@reset and @moretea is this resolved in the current state of the supervisor? I haven't looked too deeply into the memberID logic, wondering if we reuse member IDs or have some kind of garbage collection planned for old IDs.

[reset]

@eeyun no, this is a feature we still need to add

[adamhjk]

I don’t think this is possible without a significant increase in complexity. We could give a counter that increases, but not that’s guaranteed to never re-use a number, or to never give the same result to two members at once. To solve this in a way that’s not prone to bad race conditions, we would need to have a true consensus protocol (raft, paxos).

As an alternative, might I recommend we write a small, centralized, state full number service? We could then bind zookeeper to it, and use it for bootstrapping. It will be easy, it would be small, and it would be safer

[christophermaier]

Going to close this out in favor of the workaround @adamhjk suggested.