when you mount xfs, also use the inode64 option. That will help with xfs performance.<div><br></div><div>My offhand guess is you are likely running into limited network bandwidth for the 2 bricks to sync. As the network gets flooded nfs response gets poor. Make sure you are getting full-duplex connections - or upgrade your network to 10G or (even better) Infiniband.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Nov 26, 2012 at 1:46 AM, ZHANG Cheng <span dir="ltr"><<a href="mailto:czhang.oss@gmail.com" target="_blank">czhang.oss@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Early this morning our 2 bricks replicated cluster had an outage. The<br>
disk space for one of the brick server (brick02) was used up. When we<br>
responded to the disk full alert, the issue already lasted for a few<br>
hours. We reclaimed some disk space, and reboot the brick02 server,<br>
expecting once it come back it will go self healing.<br>
<br>
It did go self healing, but just after couple minutes, access to<br>
gluster filesystem freeze. Tons of "nfs: server brick not responding,<br>
still trying" popped up in dmesg. The load average on app server went<br>
up to 200 something from usual 0.10. We had to shutdown brick02 server<br>
or stop gluster server process on it, to get the gluster cluster back<br>
working.<br>
<br>
How could we deal with this issue? Thanks in advance.<br>
<br>
Our gluster setup is followed the official doc.<br>
<br>
gluster> volume info<br>
<br>
Volume Name: staticvol<br>
Type: Replicate<br>
Volume ID: fdcbf635-5faf-45d6-ab4e-be97c74d7715<br>
Status: Started<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: brick01:/exports/static<br>
Brick2: brick02:/exports/static<br>
<br>
Underlying filesystem is xfs (on a lvm volume), as:<br>
/dev/mapper/vg_node-brick on /exports/static type xfs<br>
(rw,noatime,nodiratime,nobarrier,logbufs=8)<br>
<br>
The brick servers don't act as gluster client.<br>
<br>
Our app servers are the gluster client, mount via nfs.<br>
brick:/staticvol on /mnt/gfs-static type nfs<br>
(rw,noatime,nodiratime,vers=3,rsize=8192,wsize=8192,addr=10.10.10.51)<br>
<br>
brick is a DNS round-robin record for brick01 and brick02.<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div>