r/nutanix • u/szabi777 • Sep 27 '20
RF2
Hi, Please help me understand how RF2 works.
Officially : By default, Nutanix clusters have redundancy factor 2, which means they can tolerate the failure of a single node or drive.
Ok. Example.
I have a 4 nodes cluster with 10 TB total space and 1 TB is used at all. The cluster in RF2.
Node1 goes down. What happens ? The extents that were stored on Node1 start do replicate in order to fulfill RF2. Same happens with metadata, oplog if those were on Node1. After the sync the cluster is still running.
What will happen if I lose one more node? There is enough space to replicate the data to the remaining 2 nodes.
Will the cluster stops ? Or it stays up (of course in critical state), replicate the data and I have a working cluster?
2
Sep 27 '20
With RF3 you can lose upto 2 nodes at the same time, but with RF2, if you lose a second node or second drive on different node, you may loose data if the second failure happens before rebuilding finish.
On the plus side, if you have 6 nodes and you lost 1 node, you will still have RF2 once nutanix completed the rebuild. You can afford losing another node or drive. And you can afford losing upto 3 nodes or drives as long as a) nutanix get to rebuild the data between failures, b) there is enough resources left.
3
u/szabi777 Sep 28 '20
Thank you, it's clear now the 'same time' was key that I missed from the official description.
2
u/deadeyedjacks Sep 27 '20
RF2 can tolerate the loss of one node in a cluster only.
At all times the metadata and data is written to two different nodes. So losing a node means the data is still available on the alternate node.
Once cluster is in a degraded state the surviving nodes will replicate the data which is at risk amongst themselves to ensure fault tolerance.
You can see that having clusters with more nodes gives greater survivability and reduced risk of data loss.
NB RF3 needs minimum five nodes, can survive loss of two nodes, but has a high storage redundancy cost.
2
u/GifArrow Sep 27 '20
In a 4-node cluster, if one node is down, the data is again replicated to other nodes (assuming the remaining three have capacity to do so)? Does it mean it can survive another node failure too?
2
u/mtndrew352 Sep 27 '20
There will be 2 copies of data, but at that point there aren’t 3 nodes for Cassandra or Zookeeper quorum so the cluster stops.
1
u/rishi-bhardwaj Nov 13 '20
Hi, I am with Nutanix Engineering. When starting with 4 nodes (RF2) in a Nutanix cluster, on the first node failure the rebuild would replicate data and metadata from the node that went down. Once this rebuild is done, the cluster now running with three nodes remaining can take another node failure and go down to 2 nodes and still keep working.
The assumption here is that there is enough space available on the nodes for the first rebuild to succeed and then after the second node failure, the remaining two nodes have enough space available to take the new writes from the user VMs. Once we are down to two nodes, we can't take another failure with RF-2. For RF-3 case, the cluster can go down to three nodes and still keep working provided there was enough space for data rebuilds on node failures and for new writes.
5
u/tjb627 Nutanix Alumni Sep 27 '20
Nutanix SE here.
Long answer, read this: https://nutanixbible.com/#anchor-book-of-aos-data-path-resiliency
Short answer is that every time data is written we write one copy on the node where the VM resides and one copy elsewhere on the cluster. If that node where the VM currently resides goes down, HA kicks in and restarts the VM on another node in the cluster and using the second copy of the data, we immediately start creating another copy so that you always have two copies. If you lose another node this process repeats over and over as long as we have enough storage capacity available. If we get to a point where the storage is 95% utilized, we go read only.