[zookeeper] multi-node setup issue

[IxDay]

I am currently trying to add support for multi-node in zookeeper plan.
All nodes have to be named in the config in the form of

server.{{id}}={{host}}:{{port}} // to simplify

I wanted to use @index from the svc context, but the id value must be between 1 and 255, and index starts at 0.
as http://handlebarsjs.com/ does not support expression and evaluation, I get stuck here unless I write some kind of helper, which I think will be complicated to add as it complexify the template language.
Does someone have a idea how I can bypass this restriction ?

[bdangit]

Would it be possible to take advantage of dynamic configuration in zookeeper instead? https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html#ch_reconfig_dyn

You would basically not template out the config but design your hooks to make API calls to add and remove nodes.

[moretea]

I believe that there are still some issues here that are hard to solve. Like the fact that there should be a stable mapping of the habitat supervisor -> zookeeper id.

[moretea]

@bdangit see the two issues I opened up on habitat-sh/habitat

Furthermore, there are several misconfigurations possible according to https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html, which I'm not sure that we can prevent from happening. We'd need some shared distributed memory with proper atomic operations to perform those re-configurations.

[bdangit]

Hrmm. I see. Wouldn't we run into an issue where all the nodes would have to be reconfigured because the template is changing? If done via dynamic reconfigure, a node could gather cluster health from Zookeeper and let Zookeeper routines manage addition/removal of nodes.

[moretea]

To make sure that we're on the same page: we definitely should use the reconfigure hook to keep zookeeper in the air at all times, restarting zookeeper is not an option IMHO.

We'll check if we can implement this once habitat-sh/habitat#2224 is implemented.

[bdangit]

@moretea yep agree with reconfigure and then we should be able to do a rolling restart. At least that's what we needed to do to get the configs to take place right?

[eeyun]

We hit this in triage today! This is still a feature we want and the two referenced issues on the habitat core repo are probably the best place to track any early work on this. Without those two features we can't sanely implement this feature in the plan.

[moretea]

If you add special cased helper functions, you'll eventually hit another use case that you had not envisioned before, and will be unable to implement.

There are two options to solve this problem IMHO.

Proper programming language

For example, https://github.com/dhall-lang/, which gives you strong termination guarantees, and does not allow you to write random programs doing random IO (unlike how Ruby is used for Chef).

Enable external dynamic behavior

This idea is close to how Kubernetes' Controllers work, except that you'd elect a single controller in one of the supervisors in a ring. This Controller would read a stream of events on stdin, and write commands to the Supervisor on stdout.
The events it gets notified about would be that a peer joined, left, got restarted, etc.