<div dir="ltr"><div><div><div><div><div><div><div><div><div>Hi list,<br></div>I would like to describe an issue I had today with Gluster and ask for opinion:<br><br>I have a replicated mount with 2 replica. There is about 1TB of production data in there in around 100.000 files. They sit on 2x Supermicro x9dr3-ln4f machines with a RAID array of 18TB each, 64gb of ram, 2x Xeon CPUs, as recommended in Red Hat hardware guidelines for storage server. They have a 10gb link between each other. I am running gluster 3.4.2 on centos 6.5<br>
<br></div><div>This storage is NFS-mounted to a lot of production servers. A very little part of this data is actually useful, the rest is legacy.<br></div><div><br></div>Due to some unrelated issue with one of the supermicro server (faulty memory), I had to take one of the nodes offline for 3 days.<br>
<br></div>When I brought it back up, some files and directories ended up in heal-failed state (but no split-brain). Unfortunately that were the critical files that had been edited in the last 3 days. On the NFS mounts, attempts to read these files resulted in I/O error.<br>
<br></div><div>I was able to fix a few of these files by manually removing them in each brick and then copying them to the mounted volume again. But I did not know what to do when full directories were unreachable because of "heal failed".<br>
</div><div><br></div>I later read that healing could take time and that heal-failed may be a transient state (is that correct? <a href="http://stackoverflow.com/questions/19257054/is-it-normal-to-get-a-lot-of-heal-failed-entries-in-a-gluster-mount">http://stackoverflow.com/questions/19257054/is-it-normal-to-get-a-lot-of-heal-failed-entries-in-a-gluster-mount</a>), but at the time I thought that was beyond recovery, so I proceeded to destroy the gluster volume. Then on one of the replicas I moved the content of the brick to another directory, created another volume with the same name, then copied the content of the brick to the mounted volume. This took around 2 hours. Then I had to reboot all my NFS-mounted machines which were in "stale NFS file handle" state.<br>
<br></div>Few questions :<br></div>- I realize that I cannot expect 1TB of data to heal instantly, but is there any way for me to know if the system would have recovered eventually despite being shown as "heal failed" ? <br>
</div><div>- if yes, what amount of files and filesize should I clean-up from my volume to make this time go under 10 minutes ?<br></div><div>- would native gluster mounts instead of NFS have been of help here ?<br></div>
- would any other course of action have resulted in faster recovery time ?<br></div>- is there a way in such situation to make one replica have authority about the correct status of the filesystem ?<br></div><br>Thanks in advance for your replies.<br>
<div><br></div></div>