<div dir="ltr">ok, let's see if we can gather more info. <div><br></div><div>I am not a specialist, but you know... another pair of eyes.</div><div><br></div><div>My system has a single glusterd process and it has a pretty low PID, meaning it has not crashed.</div>
<div><br></div><div>What is your PID for your glusterd ? how many zombie processes are there reported by top ?</div><div><br></div><div>I've been running my preliminary tests with gluster for a little over a month now and have never seen this. My platform is CentOS 6.5, so, I'd say it is pretty similar.</div>
<div><br></div><div>From my perspective, even making gluster sweat, running some intense rsync jobs in parallel, and seeing glusterd AND glusterfs take 120% of processing time on top (each on one core), they never crashed.</div>
<div><br></div><div>My zombie count, from top, is zero.</div><div><br></div><div>On the other hand, I had one of my nodes, the other day, crashing a process every time I started a high demanding task. Ends up I had (and still have) a hardware problem on one of the processor (or the main board; still undiagnosed).</div>
<div><br></div><div>Do you have this problem on one node only ? </div><div><br></div><div>Any chance you have something special compiled on your kernel ?</div><div><br></div><div>Any particularly memory-hungry tweak on your sysctl ?</div>
<div><br></div><div>Sounds like the system, not gluster. </div><div><br></div><div>KR, </div><div><br></div><div>Carlos</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Mar 21, 2014 at 10:29 PM, Steve Thomas <span dir="ltr"><<a href="mailto:sthomas@rpstechnologysolutions.co.uk" target="_blank">sthomas@rpstechnologysolutions.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<font face="Calibri, sans-serif">
<div><font color="#1F497D">Hi all…</font></div>
<div><font color="#1F497D"> </font></div>
<div><font color="#1F497D">Further investigation shows in excess of 500 glusterd zombie processes and continuing to climb on the box …</font></div>
<div><font color="#1F497D"> </font></div>
<div><font color="#1F497D">Any suggestions? Am happy to provide logs etc to get to the bottom of this….</font></div>
<div><font face="Calibri, sans-serif" color="#1F497D"> </font></div>
<div><font face="Calibri, sans-serif">_____________________________________________<br>
<b>From:</b> Steve Thomas <br>
<b>Sent:</b> 21 March 2014 13:21<br>
<b>To:</b> '<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>'<br>
<b>Subject:</b> Gluster 3.4.2 on Redhat 6.5</font></div><div><div class="h5">
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">Hi,</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">I’m running Gluster 3.4.2 on Redhat 6.5 with 4 servers with a brick on each. This brick is mounted locally and used by apache to server audio files for an IVR system. Each of these audio files are typically around 80-100Kb.</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">System appears to be working ok in terms of health and status via gluster CLI. </font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">The system is monitored by nagios and there’s a check for zombie processes and the gluster status. It appears that over a 24 hour period the number of Zombie processes on the box has increased and is continually increasing.
Investigating these are “glusterd” processes.</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">I’m making an assumption but I’d suspect that the regular nagios checks are resulting in the increase in zombie processes as they are querying the glusterd process. The command that the nagios plugin is running is:</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">#Check heal status</font></div>
<div><font face="Calibri, sans-serif">gluster volume heal audio info</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">#Check volume status</font></div>
<div><font face="Calibri, sans-serif">gluster volume status audio detail</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">Does anyone have any suggestions as to why glusterd is resulting in these zombie processes?</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">Thanks for help in advance,</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif">Steve</font></div>
<div><font face="Calibri, sans-serif"> </font></div>
<div><font face="Calibri, sans-serif"> </font></div>
</div></div></font>
</div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>