Open Data made simple – couchdb read only + elasticsearch river

 

A friend of mine, Pierre, owner of project openbeelab gave me a read only access to its couchdb database.

Couchdb has a map / reduce view system that is precompiled. Great but every view need to be compiled and so coded before using it.

There is not so much parameter you can give to couch to customize the result like summing, aggregating by date on the fly and so on.

So i decided to run an elasticsearch couchdb river to import couchdb database onto elasticsearch.

couchdb => elasticsearch

1 ) elasticsearch import all existing documents

2 ) elasticsearch watch for new couch document with the _changes feed and add them

3 ) couchdb db doc can be mapped to your es mapping. you can change index name and so on on the  fly for data partitionning by day for example.

Benefits ?

  • now i can request the db locally without load to openbeelab production
  • i can make make my own request with elasticsearch without querying Pierre to add more couch views
  • i can use new elasticsearch agregations for example to compute my dataviz data
  • data is up to date ! so my app is near real time !
  • i can use kibana to debug the model (as long as data is time series based like openbeelab data)

Next: now i’m ready for a better dataviz. and maybe i will help openbeelab to handle a tiny more advanced api .. 3 hour of work with node. js maybe !

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *