Was there anything in dmesg on the servers? If you are able to reproduce the hang, can you get the output of 'gluster volume status <name> callpool' and 'gluster volume status <name> nfs callpool' ?<div>
<br></div><div>How big is the 'log/secure' file? Is it so large the the client was just busy writing it for a very long time? Are there any signs of disconnections or ping tmeouts in the logs?</div><div><br></div>
<div>Avati<br><br><div class="gmail_quote">On Sat, Jun 16, 2012 at 10:48 AM, Sean Fulton <span dir="ltr"><<a href="mailto:sean@gcnpublishing.com" target="_blank">sean@gcnpublishing.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I do not mean to be argumentative, but I have to admit a little frustration with Gluster. I know an enormous emount of effort has gone into this product, and I just can't believe that with all the effort behind it and so many people using it, it could be so fragile.<br>
<br>
So here goes. Perhaps someone here can point to the error of my ways. I really want this to work because it would be ideal for our environment, but ...<br>
<br>
Please note that all of the nodes below are OpenVZ nodes with nfs/nfsd/fuse modules loaded on the hosts.<br>
<br>
After spending months trying to get 3.2.5 and 3.2.6 working in a production environment, I gave up on Gluster and went with a Linux-HA/NFS cluster which just works. The problems I had with gluster were strange lock-ups, split brains, and too many instances where the whole cluster was off-line until I reloaded the data.<br>
<br>
So wiith the release of 3.3, I decided to give it another try. I created one relicated volume on my two NFS servers.<br>
<br>
I then mounted the volume on a client as follows:<br>
10.10.10.7:/pub2 /pub2 nfs rw,noacl,noatime,nodiratime,<u></u>soft,proto=tcp,vers=3,defaults 0 0<br>
<br>
I threw some data at it (find / -mount -print | cpio -pvdum /pub2/test)<br>
<br>
Within 10 seconds it locked up solid. No error messages on any of the servers, the client was unresponsive and load on the client was 15+. I restarted glusterd on both of my NFS servers, and the client remained locked. Finally I killed the cpio process on the client. When I started another cpio, it runs further than before, but now the logs on my NFS/Gluster server say:<br>
<br>
[2012-06-16 13:37:35.242754] I [afr-self-heal-common.c:1318:<u></u>afr_sh_missing_entries_lookup_<u></u>done] 0-pub2-replicate-0: No sources for dir of <gfid:4a787ad7-ab91-46ef-9b31-<u></u>715e49f5f818>/log/secure, in missing entry self-heal, continuing with the rest of the self-heals<br>
[2012-06-16 13:37:35.243315] I [afr-self-heal-common.c:994:<u></u>afr_sh_missing_entries_done] 0-pub2-replicate-0: split brain found, aborting selfheal of <gfid:4a787ad7-ab91-46ef-9b31-<u></u>715e49f5f818>/log/secure<br>
[2012-06-16 13:37:35.243350] E [afr-self-heal-common.c:2156:<u></u>afr_self_heal_completion_cbk] 0-pub2-replicate-0: background data gfid self-heal failed on <gfid:4a787ad7-ab91-46ef-9b31-<u></u>715e49f5f818>/log/secure<br>
<br>
<br>
This still seems to be an INCREDIBLY fragile system. Why would it lock solid while copying a large file? Why no errors in the logs?<br>
<br>
I am the only one seeing this kind of behavior?<span class="HOEnZb"><font color="#888888"><br>
<br>
sean<br>
<br>
<br>
<br>
<br>
<br>
-- <br>
Sean Fulton<br>
GCN Publishing, Inc.<br>
Internet Design, Development and Consulting For Today's Media Companies<br>
<a href="http://www.gcnpublishing.com" target="_blank">http://www.gcnpublishing.com</a><br>
<a href="tel:%28203%29%20665-6211%2C%20x203" value="+12036656211" target="_blank">(203) 665-6211, x203</a><br>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/<u></u>mailman/listinfo/gluster-users</a><br>
</font></span></blockquote></div><br></div>