Is it possible to really launch this project on differents physical servers?

[ghost]

Hello,
I have read in your documentation "This repository makes it easy to run distributed, multiple server (multiple Zookeeper, multiple Storm Supervisor)".
But by "multiple server" you mean "multiple docker"?

I try to run this on multiple Rackspace instance and it's impossible. One thing is that you use docker link between dockers. And 2 servers can't communicate with link until you use a Docker ambassador (https://docs.docker.com/articles/ambassador_pattern_linking/)

So I start my zookeeper first, work fine:

./start-storm.sh zookeeper

Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in ./venv/lib/python2.7/site-packages (from -r requirements.txt (line 1))
Cleaning up...

docker run -p 2181:2181 -p 2888:2888 -p 2888:2888/udp -p 3888:3888 --expose 2181 --expose 2888 --expose 3888 -p 127.0.0.1:49122:22 -h zookeeper --name zookeeper -d viki_data/zookeeper --my-ip-address 172.
17.42.1 --my-ip-address XX.XX.XX.237 --my-ip-address XXX.XXX.XXX.49
ed12a53c9cc1af767eb3cc811789309ede5c99eee595295f9e877e305ec17788

But after when I go to my second instance and try to launch nimbus, I got this "normal" errror:
./start-storm.sh nimbus
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in ./venv/lib/python2.7/site-packages (from -r requirements.txt (line 1))
Cleaning up...
docker run -p 6627:6627 -p 6772:6772 -p 6773:6773 --expose 6627 --expose 6772 --expose 6773 --name nimbus --link zookeeper:zk -h nimbus -d viki_data/storm-nimbus --my-ip-address 172.17.42.1 --my-ip-address XX.XX.XX.111 --my-ip-address XXX.XXX.XXX.232
3cba56650f16aefa0325ff1e16f545e140306183c3aaa4995607130ba2b4559f
2014/09/29 16:03:13 Error response from daemon: Could not find entity for zookeeper

Maybe I'm wrong and do something wrong with my configuration. Do I really need a ambassador or should I hack to use IP? I setup my zookeeper's IP inside storm-setup.yaml.

Thanks for the support.

[yanhan]

Hi Damien, it is indeed possible to run this across multiple physical servers. That is the main reason why we even released this project - otherwise it'll be no different from the other offerings out there. And no, ambassador is not required if you set things up the way we intended. Really.

The command line flags you see when you run the start-storm.sh script are not entirely hardcoded - a lot of these flags, including the --link flags are generated using a set of Python scripts in the docker_python_helpers folder, which the start-storm.sh calls indirectly. These flags are generated based on your configuration in the config/storm-setup.yaml file.

Initially, I wrote some things that didn't answer your question, but I looked through the code a bit and found this:

https://github.com/viki-org/storm-docker/blob/master/start-storm.sh#L28

Which is relevant to the issue your are facing. So you are right. There is indeed this link to a Zookeeper container. So there has to be a Zookeeper Docker container named zookeeper running on the same physical machine as the nimbus container. There may be reasons why we required the Nimbus container to run on the same machine as some Zookeeper container. I've forgotten the reasons though, if there were any. Perhaps this is our oversight. Currently I do not see too much benefit in running Nimbus on a different machine than some Zookeeper. Feel free to point out any advantages that may be gained, because I am not some expert in Storm. Regardless, this is something you'll have to live with unless you edit the code. Patches are welcome. If you can convince me of the advantage and if Viki allows me to use their AWS account, I may even find some time to come up with a patch myself but that is highly unlikely for the time being because time is not a luxury I can afford...

Try editing your setup and run the Nimbus container on the same machine as some Zookeeper container and see if that solves the problem?

If this doesn't resolve the issue, do you mind sharing your config/storm-setup.yaml file here? It'll help a lot towards resolving the issue.

Thanks for trying out storm-docker =)

[ghost]

Yes I found that disabling docker links in "start-storm.sh" let me launch the nimbus and the UI. So I have already done that. But I have others problems ... The UI have a "org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused".
I need to check all the stack manually and configure it ot check what is wrong.

Here is my config/storm-setup.yaml:
is_localhost_setup: false
all_machines_are_ec2_instances: false
storm.supervisor.hosts:

ip: "10.11.7.177"
aliases:
  - "storm-supervisor-01"

• ip: "10.11.8.80"
  aliases:
  • "storm-supervisor-02"
    zookeeper.multiple.setup:
    follower.port: 2888
    election.port: 3888
    storm.yaml:
    storm.local.dir: "/mnt/storm"
    storm.zookeeper.servers:
  • "10.11.4.49"
    storm.zookeeper.port: 2181
    nimbus.host: "10.11.6.232"
    nimbus.thrift.port: 6627
    ui.port: 8080
    nimbus.childopts: "-Djava.net.preferIPv4Stack=true"
    supervisor.childopts: "-Djava.net.preferIPv4Stack=true"
    worker.childopts: "-Djava.net.preferIPv4Stack=true"
    storm.messaging.transport: "backtype.storm.messaging.netty.Context"
    storm.messaging.netty.server_worker_threads: 1
    storm.messaging.netty.client_worker_threads: 1
    storm.messaging.netty.buffer_size: 16384
    storm.messaging.netty.max_retries: 10
    storm.messaging.netty.min_wait_ms: 1000
    storm.messaging.netty.max_wait_ms: 5000
    drpc.servers:
  • "10.11.6.232"
    drpc.port: 6772
    drpc.invocations.port: 6773
    logviewer.port: 8000
    supervisor.slots.ports:
  • 6700
  • 6701
  • 6702
  • 6703

[yanhan]

It's quite safe to say that your Nimbus is not able to talk to your Zookeeper (and vice versa). As you said in your first post, you will need ambassador. Since you don't want to run the Nimbus container on the same physical machine as some Zookeeper container, you are on your own. Good luck!

[yanhan]

Hi @zaide , I know it's been quite a while and I'm not sure if you're still interested, but I've given this issue some thought a while back and these few days I finally had some time to perform some experiments with using ambassador. The results have been positive.

I will be working on some patches to enable running the (Nimbus and UI) container on a separate physical machine from any Zookeeper server. To enable this may be a bit clunky and I have to put some thought into this. Hopefully the patches will be done next week.