Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

关于Comsumer重连问题 #108

Open
lnj2050 opened this issue Aug 28, 2014 · 5 comments
Open

关于Comsumer重连问题 #108

lnj2050 opened this issue Aug 28, 2014 · 5 comments

Comments

@lnj2050
Copy link

lnj2050 commented Aug 28, 2014

在生产环境运行过程中，偶尔因为网络问题，导致Consumer一直报下面的错误
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]

中间也进行了rebalance，从日志看也成功了。
2014-08-28 13:21:43 INFO ConsumerZooKeeper - begin rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.

但从实际日志打印来看，一直在包这种问题，网络异常到恢复也就半个小时时间，但是这个异常从上周五一直打印到现在，说明Consumer重连还是存在一些问题
@lnj2050
Copy link
Author

lnj2050 commented Aug 28, 2014

另外，下面这个日志打印信息也有问题吧：
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]

这块我理解不应该是 Send message timeout in 10000 mills
而应该是 Receive message timeout in 10000 mills

@allwmh
Copy link
Collaborator

allwmh commented Aug 28, 2014

亲 您有没有即时通讯之类的联系方式呢？

------------------ 原始邮件 ------------------
发件人: "lnj2050";[email protected];
发送时间: 2014年8月28日(星期四) 下午3:26
收件人: "killme2008/Metamorphosis"[email protected];

主题: [Metamorphosis] 关于Comsumer重连问题 (#108)

在生产环境运行过程中，偶尔因为网络问题，导致Consumer一直报下面的错误
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]
2014-08-28 13:21:53 ERROR SimpleFetchManager - 获取消息失败,topic=Disc,partition=0
com.taobao.metamorphosis.exception.MetaOpeartionTimeoutException: Send message timeout in 10000 mills
at com.taobao.metamorphosis.client.consumer.SimpleMessageConsumer.fetch(SimpleMessageConsumer.java:372) ~[metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.processRequest(SimpleFetchManager.java:227) [metamorphosis-client-1.4.6.2.jar:na]
at com.taobao.metamorphosis.client.consumer.SimpleFetchManager$FetchRequestRunner.run(SimpleFetchManager.java:215) [metamorphosis-client-1.4.6.2.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_45]

中间也进行了rebalance，从日志看也成功了。
2014-08-28 13:21:43 INFO ConsumerZooKeeper - begin rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - Consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 with {Config=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525677558-3], StatusPoll=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525679271-4], Disc=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676090-1], Event=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6], Perform=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986104-5], CollectSched=[IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408525676825-2]} doesn't need to be rebalanced.
2014-08-28 13:21:44 WARN ConsumerZooKeeper - end rebalancing consumer IDCP-SHUOZHOU_sigmam-probe_10_210_19_52_10.210.19.52-7212-1408526986226-6 try #0
2014-08-28 13:21:44 WARN ConsumerZooKeeper - rebalance success.

但从实际日志打印来看，一直在包这种问题，网络异常到恢复也就半个小时时间，但是这个异常从上周五一直打印到现在，说明Consumer重连还是存在一些问题

—
Reply to this email directly or view it on GitHub.

@lnj2050
Copy link
Author

lnj2050 commented Aug 28, 2014

qq: 24614577

@allwmh
Copy link
Collaborator

allwmh commented Aug 28, 2014

云 你+我吧 497055733

------------------ 原始邮件 ------------------
发件人: "lnj2050"[email protected];
发送时间: 2014年8月28日(星期四) 下午3:49
收件人: "killme2008/Metamorphosis"[email protected];
抄送: "allwmh"[email protected];
主题: Re: [Metamorphosis] 关于Comsumer重连问题 (#108)

qq: 24614577

—
Reply to this email directly or view it on GitHub.

@muyunisawesome
Copy link

muyunisawesome commented Jun 22, 2016 •
edited
Loading

这个问题我也遇到了，请问怎么解决啊。 打出log4j日志，显示
[2016-06-22 16:08:57,559] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2016-06-22 16:08:57,559] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2016-06-22 16:08:57,560] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2016-06-22 16:08:57,560] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2016-06-22 16:08:57,560] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2016-06-22 16:08:57,560] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
[2016-06-22 16:08:57,560] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient)
consumer那边就是各种getmessage超时

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lnj2050 @allwmh @muyunisawesome