It is a common problem in all interlinked systems that if not node is out of order the entire chain breaks and the complete system slows down. It is a common problem in Hadoop cluster that if a faulty node or a slow node is there the entire system slows down. How can we work around this problem?
This problem can be approached in two ways one of which is using a token aware client system like Astyanax (https://github.com/Netflix/astyanax). Client like these are designed to bypass the nodes that are slow. In addition to this we can baseline the response time of requests on the server nodes and look into the nodes that are slowing down. Identifying the node and taking it down would solve the problem instantly as mapping system would distribute the data to be hosted among the nodes available.
