Twitter storm open source product could be compared to a hadoop like map reduce fro realtime.
It build on top of zeromq message processing. zeromq is great but it on a lower level. Storm make a lot off off additional job and fundamental for your application:
- deploy workers and queues
- make sure queues and workers stays up
- manage worker topology
- easily scale with less effort
- message distribution guaranted
When use storm ?
- stream processing
- parallel computing
On Storm you run “topologies”. It runs all the time (conituous stream).
Component
- Nimbus get all messages and distribute them to workers
- Supervisor == Node responsible the execute a part of the topology
- Zookoper maintain all the system coherency (orchestra..)
Streams and topology
- stream a continuous sequence of tuples
- spouts a source of stream
- bolts stream transformation
Storm guarantees that every message flowing through a topology will be processed, even if a machine goes down and the messages it was processing get dropped. How Storm accomplishes this without any intermediate queuing is the key to how it works and what makes it so fast.
Benefits of storm
After all, what benefits toward zeromq and some hand made code ? This tricky contraints are managed:
- Guaranteed message processing
- Robust process management workers are monitored
- Fault detection and automatic reassignment
- Efficient message passing zeromq rocks !
More infos:
