7670173442_b35f5584f2_b

Twitter Storm – Real time computation (on top of zeromq)

Twitter storm open source product could be compared to a hadoop like map reduce fro realtime.

It build on top of zeromq message processing. zeromq is great but it on a lower level. Storm make a lot off off additional job and fundamental for your application:
  • deploy workers and queues
  • make sure queues and workers stays up
  • manage worker topology
  • easily scale with less effort
  • message distribution guaranted

When use storm ?

  • stream processing
  • parallel computing

On Storm you run “topologies”. It runs all the time (conituous stream).

Component

  • Nimbus get all messages and distribute them to workers
  • Supervisor == Node responsible the execute a part of the topology
  • Zookoper maintain all the system coherency (orchestra..)

storm-cluster.png

Streams and topology

  • stream a continuous sequence of tuples
  • spouts a source of stream
  • bolts stream transformation

topology.png

Storm guarantees that every message flowing through a topology will be processed, even if a machine goes down and the messages it was processing get dropped. How Storm accomplishes this without any intermediate queuing is the key to how it works and what makes it so fast.

Benefits of storm

After all, what benefits toward zeromq and some hand made code ? This tricky contraints are managed:

  • Guaranteed message processing
  • Robust process management workers are monitored
  • Fault detection and automatic reassignment
  • Efficient message passing zeromq rocks !

More infos:

http://engineering.twitter.com/2011/08/storm-is-coming-more-details-and-plans.html

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">