Skip to content

Using non JVM languages with Storm

nathanmarz edited this page Sep 17, 2011 · 7 revisions
  • two pieces: creating topologies and implementing spouts and bolts in other languages
  • creating topologies in another language is easy since topologies are just thrift structures (link to storm.thrift)
  • implementing spouts and bolts in another language is called a "multilang components" or "shelling"
    • currently you can only define multilang bolts, but this will be remedied soon
    • the thrift structure lets you define multilang components explicitly as a program and a script (e.g., python and the file implementing your bolt)
    • In Java, you override ShellBolt or ShellSpout to create multilang components
      • note that output fields declarations happens in the thrift structure, so in Java you create multilang components like the following:
        • declare fields in java, processing code in the other language by specifying it in constructor of shellbolt
    • multilang uses json messages over stdin/stdout to communicate with the subprocess
    • storm comes with ruby, python, and fancy adapters that implement the protocol. show an example of python
      • python supports emitting, anchoring, acking, and logging
  • "storm shell" command makes constructing jar and uploading to nimbus easy
    • makes jar and uploads it
    • calls your program with host/port of nimbus and the jarfile id