<div dir="ltr"><div>Thanks, it's always good to know I'm not alone with problem! Also good to know I haven't missed something blindingly obvious in the config/setup. <br><br></div><div>WE had our VPN drop between the DCs yesterday afternoon, which resulted in high load on 1 gluster server at a time for about 10 minutes once the VPN was back up, so unless anyone else has any ideas, I think looking at alternatives is our only way forward. I had a quick look the other day and Ceph was one of the possibilities that stood out for me.<br>
<br></div><div>Thanks.<br></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 14 May 2013 03:21, Toby Corkindale <span dir="ltr"><<a href="mailto:toby.corkindale@strategicdata.com.au" target="_blank">toby.corkindale@strategicdata.com.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 11/05/13 00:40, Matthew Day wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
I'm pretty new to Gluster, and the company I work for uses it for<br>
storage across 2 data centres. An issue has cropped up fairly recently<br>
with regards to the self-heal mechanism.<br>
<br>
Occasionally the connection between these 2 Gluster servers breaks or<br>
drops momentarily. Due to the nature of the business it's highly likely<br>
that files have been written during this time. When the self-heal daemon<br>
runs it notices a discrepancy and gets the volume up to date. The<br>
problem we've been seeing is that this appears to cause the CPU load to<br>
increase massively on both servers whilst the healing process takes place.<br>
<br>
After trying to find out if there were any persistent network issues I<br>
tried recreating this on a test system and can now re-produce at will.<br>
Our test system set up is made up of 3 VMs, 2 Gluster servers and a<br>
client. The process to cause this was:<br>
Add in an iptables rule to block one of the Gluster servers from being<br>
reached by the other server and the client.<br>
Create some random files on the client.<br>
Flush the iptables rules out so the server is reachable again.<br>
Force a self heal to run.<br>
Watch as the load on the Gluster servers goes bananas.<br>
<br>
The problem with this is that whilst the self-heal happens one the<br>
gluster servers will be inaccessible from the client, meaning no files<br>
can be read or written, causing problems for our users.<br>
<br>
I've been searching for a solution, or at least someone else who has<br>
been having the same problem and not found anything. I don't know if<br>
this is a bug or config issue (see below for config details). I've tried<br>
a variety of different options but none of them have had any effect.<br>
</blockquote>
<br>
<br></div></div>
For what it's worth.. I get this same behaviour, and our gluster servers aren't even in separate data centres. It's not always the self-heal daemon that triggers it -- sometimes the client gets in first.<br>
<br>
Either way -- while recovery occurs, the available i/o to clients drops to effectively nothing, and they stall until recovery completes.<br>
<br>
I believe this problem is most visible when your architecture contains a lot of small files per directory. If you can change your filesystem layout to avoid this, then you may not be hit as hard.<br>
(eg. Take an MD5 hash of the path and filename, then store the file under a subdirectory named after the first few characters in the hash. (2 chars will divide the files-per-directory by ~1300, three by ~47k) eg. "folder/file.dat" becomes "66/folder/file.dat")<br>
<br>
<br>
I've given up on GlusterFS though; have a look at Ceph and RiakCS if your systems suit Swift/S3 style storage.<br>
<br>
-Toby<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
</blockquote></div><br></div></div></div></div>