Elastically adding and removing nodes using akka cluster

Friday, August 7, 2015

This post explores a pull-based master/worker architecture – one that is suitable for anyone who is looking to elastically provision nodes when the load is higher than normal and under-provision when the load is below normal. In this post, the master accepts RSS links from frontend which can be a user submitting links and worker accept one RSS link and extract information such as article’s published date, title and a brief description. All this information is indexed into ELASTICSEARCH.

RSS links are submitted to the master and the workers register themselves with the master. Once the worker registers itself, it asks for work from the master. If the master has enough work, then it sends an RSS link to the worker.

The master/worker works according to the following protocol:

Elastically Adding Nodes

Once the PROTOCOL is established, we can add new nodes based on the load. Akka cluster membership helps here to figure which node is up. Once the node is up, the node registers itself as shown in the protocol described above. Adding a node happens dynamically without altering the normal working of the cluster.

Removing node(s)

Removing a node is also dynamic. When a node is removed manually or if it went down due to a failure, the master will automatically down that worker after a while based on the Akka cluster configuration. Dynamically removing a node doesn’t alter the normal working of the cluster. If currently a work being performed by a node which goes down due to failure, then the work assigned to that node will be marked as pending by the master since the master doesn’t get an acknowledgement back by the worker. This undone work will be put back into the queue by the master and will be assigned to some other worker.

Elasticsearch Indexing

Once the workers start indexing, the Elasticsearch indexing status can be viewed through marvel plugin. Marvel is an excellent plugin to view the status of the Elasticsearch’s cluster health and the number of documents indexed in near-real time. Once indexed it looks like as shown below:

Elasticssearch Indexing


Adding and removing nodes is essential for dynamically changing load without altering the normal working of a cluster. Not many frameworks has such flexibility with regards to building a distributed system and an Akka cluster provides a great platform to build such frameworks without the need to worry about handling the internals of building a distributed systems. The advantage of a pull based architecture is that there is implicit throttling and each worker will asynchronously ask for more work only when it has finished its current work on hand. On the other hand, if the master pushes work to each worker, then that would lead to disastrous results unless the master implements throttling explicitly. There is no way to know how much load a worker already has if the master keeps pushing to worker. Some of the niceties available in a pull based approach would be lost in a push mechanism. Dynamically adding/removing nodes would not be as easy in a push based mechanism.