<div dir="ltr"><div>Hello,</div><div><br></div><div>I've found GlusterFS to be an interesting project. Not so much experience of it</div><div>(although from similar usecases with DRBD+NFS setups) so I setup some</div><div>
testcase to try out failover and recovery.</div><div><br></div><div>For this I have a setup with two glusterfs servers (each is a VM) and one client (also a VM).</div><div>I'm using GlusterFS 3.4 btw.</div><div><br></div>
<div>The servers manages a gluster volume created as:</div><div><br></div><div>gluster volume create testvol rep 2 transport tcp gs1:/export/vda1/brick gs2:/export/vda1/brick</div><div>gluster volume start testvol</div><div>
gluster volume set testvol network.ping-timeout<span class="" style="white-space:pre"> </span>5</div><div><br></div><div>Then the client mounts this volume as:</div><div>mount -t glusterfs gs1:/testvol /import/testvol</div>
<div><br></div><div>Everything seems to work good in normal usecases, I can write/read to the volume, take servers down and up again etc.</div><div><br></div><div>As a fault scenario, I'm testing a fault injection like this:</div>
<div><br></div><div>1. continuesly writing timestamps to a file on the volume from the client. It is automated in a smaller testscript like:</div><div><div>per@hallsmark.se:~/glusterfs-test$ cat scripts/test-gfs-client.sh </div>
<div>#!/bin/sh</div><div><br></div><div>gfs=/import/testvol</div><div><br></div><div>while true; do</div><div><span class="" style="white-space:pre"> </span>date +%s >> $gfs/timestamp.txt</div><div><span class="" style="white-space:pre"> </span>ts=`tail -1 $gfs/timestamp.txt`</div>
<div><span class="" style="white-space:pre"> </span>md5sum=`md5sum $gfs/timestamp.txt | cut -f1 -d" "`</div><div><span class="" style="white-space:pre"> </span>echo "Timestamp = $ts, md5sum = $md5sum"</div>
<div><span class="" style="white-space:pre"> </span>sleep 1</div><div>done</div><div>per@hallsmark.se:~/glusterfs-test$</div></div><div><br></div><div><div>As can be seen, the client is a quite simple user of the glusterfs volume. Low datarate and single user for example.</div>
</div><div><br></div><div><br></div><div>2. disabling ethernet in one of the VM (ifconfig eth0 down) to simulate like a broken network</div><div><br></div><div>3. After a short while, the failed server is brought alive again (ifconfig eth0 up)</div>
<div><br></div><div>Step 2 and 3 is also automated in a testscript like:</div><div><br></div><div><div>per@hallsmark.se:~/glusterfs-test$ cat scripts/fault-injection.sh </div><div>#!/bin/sh</div><div><br></div><div># fault injection script tailored for two glusterfs nodes named gs1 and gs2</div>
<div><br></div><div>if [ "$HOSTNAME" == "gs1" ]; then</div><div><span class="" style="white-space:pre"> </span>peer="gs2"</div><div>else</div><div><span class="" style="white-space:pre"> </span>peer="gs1"</div>
<div>fi</div><div><br></div><div>inject_eth_fault() {</div><div><span class="" style="white-space:pre"> </span>echo "network down..."</div><div><span class="" style="white-space:pre"> </span>ifconfig eth0 down</div>
<div><span class="" style="white-space:pre"> </span>sleep 10</div><div><span class="" style="white-space:pre"> </span>ifconfig eth0 up</div><div><span class="" style="white-space:pre"> </span>echo "... and network up again."</div>
<div>}</div><div><br></div><div>recover() {</div><div><span class="" style="white-space:pre"> </span>echo "recovering from fault..."</div><div><span class="" style="white-space:pre"> </span>service glusterd restart</div>
<div>}</div><div><br></div><div>while true; do</div><div><span class="" style="white-space:pre"> </span>sleep 60</div><div><span class="" style="white-space:pre"> </span>if [ ! -f /tmp/nofault ]; then</div><div><span class="" style="white-space:pre"> </span>if ping -c 1 $peer; then</div>
<div><span class="" style="white-space:pre"> </span>inject_eth_fault</div><div><span class="" style="white-space:pre"> </span>recover</div><div><span class="" style="white-space:pre"> </span>fi</div><div><span class="" style="white-space:pre"> </span>fi</div>
<div>done</div><div>per@hallsmark.se:~/glusterfs-test$</div></div><div><br></div><div><br></div><div>I then see that:</div><div><br></div><div>A. This goes well first time, one server leaves the cluster and the client hang for like 8 seconds before beeing able to write to the volume again.</div>
<div><br></div><div>B. When the failed server comes back, I can check that from both servers they see each other and "gluster peer status" shows they believe the other is in connected state.</div><div><br></div>
<div>C. When the failed server comes back, it is not automatically seeking active participation on syncing volume etc (the local storage timestamp file isn't updated).</div><div><br></div><div>D. If I do restart of glusterd service (service glusterd restart) the failed node seems to get back like it was before. Not always though... The chance is higher if I have long time between fault injections (long = 60 sec or so, with a forced faulty state of 10 sec)</div>
<div>With a period time of some minutes, I could have the cluster servicing the client OK for up to 8+ hours at least.</div><div>Shortening the period, I'm easily down to like 10-15 minutes.</div><div><br></div><div>E. Sooner or later I enter a state where the two servers seems to be up, seeing it's peer (gluster peer status) and such but none is serving the volume to the client.</div>
<div>I've tried to "heal" the volume in different way but it doesn't help. Sometimes it is just that one of the timestamp copies in each of</div><div>the servers is ahead which is simpler but sometimes both the timestamp files have added data at end that the other doesnt have.</div>
<div><br></div><div>To the questions: </div><div><br></div><div>* Is it so that from a design point of perspective, the choice in the glusterfs team is that one shouldn't rely soley on glusterfs daemons beeing able to recover from a faulty state? There is need for cluster manager services (like heartbeat for example) to be part? That would make experience C understandable and one could then take heartbeat or similar packages to start/stop services.</div>
<div><br></div><div>* What would then be the recommended procedure to recover from a faulty glusterfs node? (so that experience D and E is not happening)</div><div><br></div><div>* What is the expected failover timing (of course depending on config, but say with a give ping timeout etc)?</div>
<div> and expected recovery timing (with similar dependency on config)?</div><div><br></div><div>* What/how is glusterfs team testing to make sure that the failover, recovery/healing functionality etc works?</div><div><br>
</div><div>Any opinion if the testcase is bad is of course also very welcome.</div><div><br></div><div>Best regards,</div><div>Per</div></div>