Can you give some more details about the event. Are the open fds files or directories? are they deleted? Are you accessing the filesystem via NFS or FUSE?<div><br></div><div>Thanks,</div><div>Avati<br><br><div class="gmail_quote">
On Tue, Aug 2, 2011 at 5:33 PM, Dave Bour <span dir="ltr"><<a href="mailto:dcbour@desktopsolutioncenter.ca">dcbour@desktopsolutioncenter.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I've got a 2 peer replicated setup with 2 bricks on each system.<br>
<br>
Starting sometime overnight, one system ground to a halt, to the point a<br>
local login was not possible.<br>
Restarted.<br>
A couple hours later I tried to log in via SSH, too many open files. A<br>
quick 'lsof" showed thousands of open files on glusterfs.<br>
<br>
I just restarted the server again...and appears to be doing the same<br>
thing. Addition of nearly 2500 files in 4 minutes, not 5 minutes after<br>
rebooting.<br>
<br>
for d in {1..1000};do a=`date`;b=`lsof|grep gluster|wc -l`;echo $a " "<br>
$b;sleep 60;done<br>
Tue Aug 2 07:56:37 EDT 2011 2924<br>
Tue Aug 2 07:57:37 EDT 2011 3568<br>
Tue Aug 2 07:58:38 EDT 2011 4154<br>
Tue Aug 2 07:59:38 EDT 2011 4718<br>
Tue Aug 2 08:00:38 EDT 2011 5376<br>
<br>
It's peer is more what's expected:<br>
for d in {1..1000};do a=`date`;b=`lsof|grep gluster|wc -l`;echo $a " "<br>
$b;sleep 60;done<br>
Tue Aug 2 07:57:18 EDT 2011 193<br>
Tue Aug 2 07:58:18 EDT 2011 192<br>
Tue Aug 2 07:59:18 EDT 2011 193<br>
Tue Aug 2 08:00:19 EDT 2011 193<br>
Tue Aug 2 08:01:19 EDT 2011 191<br>
<br>
Its running 3.2.2<br>
<br>
Any suggestions to what's happening?<br>
Where to start to troubleshoot.<br>
<br>
D.<br>
<br>
<br>
<br>
<br>
--<br>
Dave Bour<br>
Senior Consultant<br>
Desktop Solution Center<br>
905.381.0077 x501<br>
<a href="mailto:dcbour@desktopsolutioncenter.ca">dcbour@desktopsolutioncenter.ca</a><br>
<br>
Linked In: <a href="http://ca.linkedin.com/in/dcbour" target="_blank">http://ca.linkedin.com/in/dcbour</a><br>
Facebook: <a href="http://www.facebook.com/dcbour" target="_blank">http://www.facebook.com/dcbour</a><br>
<br>
<br>
Providing Complete IT Peace of Mind<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div>