<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Folks,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’m using Gluster 3.4 on CentOS 6… very simple two-server, two-brick (replica 2) setup. The volume itself has many small files across a reasonably large directory tree, though I’m not sure if that plays a role. The FUSE client is being used. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I can live with the performance limitations of small files with Gluster, but the problem I’m having is that file descriptor usage on the glusterfs servers just continues to grow… not sure when it might actually top off, if ever. No rebalance has been or is running. The application running on the client servers (two) are not leaving the files open.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’ve tuned Linux behavior on the glusterfs servers, via /proc, to accept over 1 million per-process file descriptors, but that doesn’t seem to be enough. This volume hit the FD max some time ago and had to be recovered… I thought it was a fluke so started watching the open FD count and see that it’s growing again.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'># gluster volume top users open<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'>Brick: node-75:/storage/users<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'>Current open fds: 765651, Max open fds: 1048558, Max openfd time: 2013-10-02 22:26:18.327010<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'>Brick: node-76:/storage/users<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'>Current open fds: 768936, Max open fds: 768938, Max openfd time: 2013-10-28 17:11:04.184964<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Clients:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'># cat /proc/sys/fs/file-nr<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'>5100 0 1572870<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'># cat /proc/sys/fs/file-nr<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Courier New"'>2550 0 1572870<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Looking for thoughts or suggestions here. Anyone else encountered this? Is the recommended solution to just define a ridiculously high per-process and global file descriptor max? <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>-Joel<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>