Failed awaiting on state barrier

[lwmonster]

My program has running for several days, and then it crushed.
Checking the log, I found this error:
2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership 2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership 2016-12-28/02:31:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:31:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703 2016-12-28/02:32:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]

This error repeated 6 times and panic.

I did not run other consumers, and it just comes out.

I have checked #139 and it doesn't work for me, any clue?

[lwmonster]

Full log:

2016-12-28/02:30:47 [INFO] [zk] Received zkEvent Type: EventNodeChildrenChanged State: Unknown Path: /consumers/queryengine/ids
2016-12-28/02:30:47 [INFO] [zk] Trying to renew watcher for consumer registry
2016-12-28/02:30:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] rebalance triggered for 2f33e845-4de4-e780-956d-40e9950b01da
2016-12-28/02:30:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership
2016-12-28/02:30:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership
2016-12-28/02:30:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:30:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:31:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]
2016-12-28/02:31:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership
2016-12-28/02:31:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership
2016-12-28/02:31:17 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:31:17 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
GC forced
gc 1115 @47606.472s 0%: 1.5+10+0.61+53+5.6 ms clock, 3.1+10+0+0/25/84+11 ms cpu, 244->244->226 MB, 454 MB goal, 2 P
2016-12-28/02:31:47 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]
2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership
2016-12-28/02:31:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership
2016-12-28/02:31:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:31:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:32:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]
2016-12-28/02:32:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership
2016-12-28/02:32:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership
2016-12-28/02:32:17 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:32:17 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:32:47 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]
2016-12-28/02:32:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership
2016-12-28/02:32:47 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership
2016-12-28/02:32:47 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:32:47 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:33:08 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Starting coordinator cleanup of API reqeusts for queryengine
scvg317: inuse: 261, idle: 599, sys: 860, released: 479, consumed: 380 (MB)
2016-12-28/02:33:17 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]
2016-12-28/02:33:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Releasing partition ownership
2016-12-28/02:33:17 [INFO] [2f33e845-4de4-e780-956d-40e9950b01da] Successfully released partition ownership
2016-12-28/02:33:17 [INFO] [zk] Joining state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:33:17 [INFO] [zk] Successfully joined state barrier /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703
2016-12-28/02:33:47 [ERROR] [zk] Failed awaiting on state barrier c5f1ca43bdfe93122679741a84646703 [Timed out waiting for consensus on barrier path /consumers/queryengine/api/rebalance/c5f1ca43bdfe93122679741a84646703]
panic: Could not reach consensus on state barrier.
goroutine 23971565 [running]:
github.com/stealthly/go_kafka_client.(*Consumer).rebalance.func1()
/home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/consumer.go:776 +0x1038
github.com/stealthly/go_kafka_client.inLock(0xc82065c028, 0xc84dcadf68)
/home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/utils.go:44 +0x52
github.com/stealthly/go_kafka_client.(*Consumer).rebalance(0xc82065c000)
/home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/consumer.go:801 +0x5d
created by github.com/stealthly/go_kafka_client.(*Consumer).subscribeForChanges.func1
/home/liangwei/gowork/src/github.com/stealthly/go_kafka_client/consumer.go:704 +0x489`