forked from nathanmarz/storm
-
Notifications
You must be signed in to change notification settings - Fork 3
Using non JVM languages with Storm
nathanmarz edited this page Sep 17, 2011
·
7 revisions
- two pieces: creating topologies and implementing spouts and bolts in other languages
- creating topologies in another language is easy since topologies are just thrift structures (link to storm.thrift)
- implementing spouts and bolts in another language is called a "multilang components" or "shelling"
- currently you can only define multilang bolts, but this will be remedied soon
- the thrift structure lets you define multilang components explicitly as a program and a script (e.g., python and the file implementing your bolt)
- In Java, you override ShellBolt or ShellSpout to create multilang components
- note that output fields declarations happens in the thrift structure, so in Java you create multilang components like the following:
- declare fields in java, processing code in the other language by specifying it in constructor of shellbolt
- note that output fields declarations happens in the thrift structure, so in Java you create multilang components like the following:
- multilang uses json messages over stdin/stdout to communicate with the subprocess
- storm comes with ruby, python, and fancy adapters that implement the protocol. show an example of python
- python supports emitting, anchoring, acking, and logging
- "storm shell" command makes constructing jar and uploading to nimbus easy
- makes jar and uploads it
- calls your program with host/port of nimbus and the jarfile id