<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0cm;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
span.EmailStyle17
        {mso-style-type:personal;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
span.EmailStyle18
        {mso-style-type:personal-reply;
        font-family:"Calibri","sans-serif";
        color:#1F497D;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-size:10.0pt;}
@page WordSection1
        {size:612.0pt 792.0pt;
        margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
        {page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Some further information:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>When I run the command<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&#8220;gluster volume status audio detail&#8221; <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>I get the Zombie process created&#8230;. So it&#8217;s not the HERE document as I previously thought&#8230; it&#8217;s the command itself.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Does this happen with anyone else?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Steve<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Steve Thomas<br><b>Sent:</b> 24 March 2014 11:55<br><b>To:</b> Carlos Capriotti<br><b>Cc:</b> gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] Gluster 3.4.2 on Redhat 6.5<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p>&nbsp;</o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Hi Carlos,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Thanks for coming back to me&#8230; in response to your queries:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>PID is low, 1153 for glusterd with glusterfsd 1168 and 2 x glusterfs with 1318 and 1319 so I&#8217;d agree&#8230; it doesn&#8217;t seem that glusterd is crashing and being restarted.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>As of today, Monday morning top is reporting 1398 glusterd zombie processes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>I have this problem on all 4 of my gluster nodes and all four are being monitored by the attached nagios plugin.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>In terms of testing, I&#8217;ve prevented nagios from running the attached check script and restarted the glusterd process using <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>&#8220;service glusterd restart&#8221;. I&#8217;ve let it run for a few hours and haven&#8217;t yet seen any zombie processes created. This I think is good as, for whatever reason, it appears to point at the nagios check script being the problem.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>My next check was to run the nagios check once to see if it created a Zombie process&#8230; it did&#8230;. So I started looking at the script. I forced the script to exit after the first command &#8220;</span><span style='font-family:"Calibri","sans-serif"'>gluster volume heal audio info&#8221; </span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>and no Zombie process was created. This pointed me to the second which takes this form&#8230;. I&#8217;m no expert of HERE documents in shell but I think that it maybe causing the issue:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>while read -r line; do<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp; field=($(echo $line))<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp; case ${field[0]} in<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp; Brick) <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; brick=${field[@]:2}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ;;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp; Disk)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; key=${field[@]:0:3}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if [ &quot;${key}&quot; = &quot;Disk Space Free&quot; ]; then<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; freeunit=${field[@]:4}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; unit=${freeunit: -2}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; free=${freeunit%$unit}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if [ &quot;$unit&quot; != &quot;GB&quot; ]; then<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Exit UNKNOWN &quot;Unknown disk space size $freeunit\n&quot;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fi<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (( $(bc &lt;&lt;&lt; &quot;${free} &lt; ${freegb}&quot;) == 1 )); then<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; freegb=$free<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fi<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fi<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ;;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp; Online)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; online=${field[@]:2}<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if [ &quot;${online}&quot; = &quot;Y&quot; ]; then<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; let $((bricksfound++))<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; else<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; errors=(&quot;${errors[@]}&quot; &quot;$brick offline&quot;)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; fi<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ;;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>&nbsp;&nbsp;&nbsp;&nbsp; esac<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Courier New";color:#1F497D;mso-fareast-language:EN-US'>done &lt; &lt;( sudo gluster volume status ${VOLUME} detail)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Anyone spot why this would be an issue?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'>Steve<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Carlos Capriotti [<a href="mailto:capriotti.carlos@gmail.com">mailto:capriotti.carlos@gmail.com</a>] <br><b>Sent:</b> 22 March 2014 11:51<br><b>To:</b> Steve Thomas<br><b>Cc:</b> <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br><b>Subject:</b> Re: [Gluster-users] Gluster 3.4.2 on Redhat 6.5<o:p></o:p></span></p><p class=MsoNormal><o:p>&nbsp;</o:p></p><div><p class=MsoNormal>ok, let's see if we can gather more info.&nbsp;<o:p></o:p></p><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>I am not a specialist, but you know... another pair of eyes.<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>My system has a single glusterd process and it has a pretty low PID, meaning it has not crashed.<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>What is your PID for your glusterd ? how many zombie processes are there reported by top ?<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>I've been running my preliminary tests with gluster for a little over a month now and have never seen this. My platform is CentOS 6.5, so, I'd say it is pretty similar.<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>From my perspective, even making gluster sweat, running some intense rsync jobs in parallel, and seeing glusterd AND glusterfs take 120% of processing time on top (each on one core), they never crashed.<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>My zombie count, from top, &nbsp;is zero.<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>On the other hand, I had one of my nodes, the other day, crashing a process every time I started a high demanding task. Ends up I had (and still have) a hardware problem on one of the processor (or the main board; still undiagnosed).<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>Do you have this problem on one node only ?&nbsp;<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>Any chance you have something special compiled on your kernel ?<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>Any particularly memory-hungry tweak on your sysctl ?<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>Sounds like the system, not gluster.&nbsp;<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>KR,&nbsp;<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal>Carlos<o:p></o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div><div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p>&nbsp;</o:p></p><div><p class=MsoNormal>On Fri, Mar 21, 2014 at 10:29 PM, Steve Thomas &lt;<a href="mailto:sthomas@rpstechnologysolutions.co.uk" target="_blank">sthomas@rpstechnologysolutions.co.uk</a>&gt; wrote:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt'><div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Hi all&#8230;</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>&nbsp;</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Further investigation shows in excess of 500 glusterd zombie processes and continuing to climb on the box &#8230;</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>&nbsp;</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>Any suggestions? Am happy to provide logs etc to get to the bottom of this&#8230;.</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif";color:#1F497D'>&nbsp;</span><span style='font-family:"Calibri","sans-serif"'><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>_____________________________________________<br><b>From:</b> Steve Thomas <br><b>Sent:</b> 21 March 2014 13:21<br><b>To:</b> '<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>'<br><b>Subject:</b> Gluster 3.4.2 on Redhat 6.5<o:p></o:p></span></p></div><div><div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>Hi,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>I&#8217;m running Gluster 3.4.2 on Redhat 6.5 with 4 servers with a brick on each. This brick is mounted locally and used by apache to server audio files for an IVR system. Each of these audio files are typically around 80-100Kb.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>System appears to be working ok in terms of health and status via gluster CLI. <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>The system is monitored by nagios and there&#8217;s a check for zombie processes and the gluster status. It appears that over a 24 hour period the number of Zombie processes on the box has increased and is continually increasing. Investigating these are &#8220;glusterd&#8221; processes.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>I&#8217;m making an assumption but I&#8217;d suspect that the regular nagios checks are resulting in the increase in zombie processes as they are querying the glusterd process. The command that the nagios plugin is running is:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>#Check heal status<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>gluster volume heal audio info<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>#Check volume status<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>gluster volume status audio detail<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>Does anyone have any suggestions as to why glusterd is resulting in these zombie processes?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>Thanks for help in advance,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>Steve<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Calibri","sans-serif"'>&nbsp;<o:p></o:p></span></p></div></div></div></div><p class=MsoNormal><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p></blockquote></div><p class=MsoNormal><o:p>&nbsp;</o:p></p></div></div></body></html>