<div dir="ltr">There was fdleak in 3.4.0, if you are running that you might consider upgrading to 3.4.1 or disable open-behind feature: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=991622">https://bugzilla.redhat.com/show_bug.cgi?id=991622</a><br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Oct 28, 2013 at 6:36 PM, Joel Stalder <span dir="ltr"><<a href="mailto:jstalder@panterranetworks.com" target="_blank">jstalder@panterranetworks.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-US"><div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Folks,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I’m using Gluster 3.4 on CentOS 6… very simple two-server, two-brick (replica 2) setup. The volume itself has many small files across a reasonably large directory tree, though I’m not sure if that plays a role. The FUSE client is being used. <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I can live with the performance limitations of small files with Gluster, but the problem I’m having is that file descriptor usage on the glusterfs servers just continues to grow… not sure when it might actually top off, if ever. No rebalance has been or is running. The application running on the client servers (two) are not leaving the files open.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I’ve tuned Linux behavior on the glusterfs servers, via /proc, to accept over 1 million per-process file descriptors, but that doesn’t seem to be enough. This volume hit the FD max some time ago and had to be recovered… I thought it was a fluke so started watching the open FD count and see that it’s growing again.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""># gluster volume top users open<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Brick: node-75:/storage/users<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Current open fds: 765651, Max open fds: 1048558, Max openfd time: 2013-10-02 22:26:18.327010<u></u><u></u></span></p><p class="MsoNormal">
<span style="font-size:9.0pt;font-family:"Courier New""><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Brick: node-76:/storage/users<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">Current open fds: 768936, Max open fds: 768938, Max openfd time: 2013-10-28 17:11:04.184964<u></u><u></u></span></p><p class="MsoNormal">
<u></u> <u></u></p><p class="MsoNormal">Clients:<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""># cat /proc/sys/fs/file-nr<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">5100 0 1572870<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New""># cat /proc/sys/fs/file-nr<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Courier New"">2550 0 1572870<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Looking for thoughts or suggestions here. Anyone else encountered this? Is the recommended solution to just define a ridiculously high per-process and global file descriptor max? <span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></p>
<span class="HOEnZb"><font color="#888888"><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">-Joel<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">
<u></u> <u></u></p></font></span></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>