<div dir="ltr"><div>Hi Vince,<br>It could be a behavioural change in heal process output capture with latest GlusterFS. If that is the case, we may tune the interval which nagios collect heal info output or some other settings to avoid continuous alerts. I am Ccing gluster nagios devs. <br><br></div>--Humble <br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>--Humble<br></div><br></div></div></div>
<br><div class="gmail_quote">On Wed, Nov 19, 2014 at 9:50 PM, Vince Loschiavo <span dir="ltr"><<a href="mailto:vloschiavo@gmail.com" target="_blank">vloschiavo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><br></div><div class="gmail_quote">Hello Gluster Community,</div><div class="gmail_quote"><br></div><div class="gmail_quote">I have been using the Nagios monitoring scripts, mentioned in the below thread, on 3.5.2 with great success. The most useful of these is the self heal. </div><div class="gmail_quote"><br></div><div class="gmail_quote">However, I've just upgraded to 3.6.1 on the lab and the self heal daemon has become quite aggressive. I continually get alerts/warnings on 3.6.1 that virt disk images need self heal, then they clear. This is not the case on 3.5.2. This </div><div class="gmail_quote"><br></div><div class="gmail_quote">Configuration:</div><div class="gmail_quote">2 node, 2 brick replicated volume with 2x1GB LAG network between the peers using this volume as a QEMU/KVM virt image store through the fuse mount on Centos 6.5.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Example:</div><div class="gmail_quote">on 3.5.2:</div><div class="gmail_quote"><i>gluster volume heal volumename info: </i>shows the bricks and number of entries to be healed: 0</div><div class="gmail_quote"><br></div><div class="gmail_quote">On v3.5.2 - During normal gluster operations, I can run this command over and over again, 2-4 times per second, and it will always show 0 entries to be healed. I've used this as an indicator that the bricks are synchronized. </div><div class="gmail_quote"><br></div><div class="gmail_quote">Last night, I upgraded to 3.6.1 in lab and I'm seeing different behavior.</div><div class="gmail_quote">Running <i>gluster volume heal volumename info</i>, during normal operations, will show a file out-of-sync, seemingly between every block written to disk then synced to the peer. I can run the command over and over again, 2-4 times per second, and it will almost always show something out of sync. The individual files change, meaning:</div><div class="gmail_quote"><br></div><div class="gmail_quote">Example:</div><div class="gmail_quote">1st Run: shows file1 out of sync</div><div class="gmail_quote">2nd run: shows file 2 and file 3 out of sync but file 1 is now in sync (not in the list)</div><div class="gmail_quote">3rd run: shows file 3 and file 4 out of sync but file 1 and 2 are in sync (not in the list).</div><div class="gmail_quote">...</div><div class="gmail_quote">nth run: shows 0 files out of sync</div><div class="gmail_quote">nth+1 run: shows file 3 and 12 out of sync. </div><div class="gmail_quote"><br></div><div class="gmail_quote">From looking at the virtual machines running off this gluster volume, it's obvious that gluster is working well. However, this obviously plays havoc with Nagios and alerts. Nagios will run the heal info and get different and non-useful results each time, and will send alerts.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Is this behavior change (3.5.2 vs 3.6.1) expected? Is there a way to tune the settings or change the monitoring method to get better results into Nagios.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Thank you,</div><div class="gmail_quote"><br>-- <br><div>-Vince Loschiavo</div></div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">On Wed, Nov 19, 2014 at 4:35 AM, Humble Devassy Chirammal <span dir="ltr"><<a href="mailto:humble.devassy@gmail.com" target="_blank">humble.devassy@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Hi Gopu,<br><br>Awesome !! <br><br>We can have a Gluster blog about this implementation. <br><br></div>--Humble<span><font color="#888888"><br><div><br><br></div></font></span></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div><div><div dir="ltr"><div>--Humble<br></div><br></div></div></div></font></span><div><div>
<br><div class="gmail_quote">On Wed, Nov 19, 2014 at 5:38 PM, Gopu Krishnan <span dir="ltr"><<a href="mailto:gopukrishnantec@gmail.com" target="_blank">gopukrishnantec@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Thanks for all your help... I was able to configure nagios using the glusterfs plugin. Following link shows how I configured it. Hope it helps someone else.:<br><br><a href="http://gopukrish.wordpress.com/2014/11/16/monitor-glusterfs-using-nagios-plugin/" target="_blank">http://gopukrish.wordpress.com/2014/11/16/monitor-glusterfs-using-nagios-plugin/</a><br></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 16, 2014 at 11:44 AM, Humble Devassy Chirammal <span dir="ltr"><<a href="mailto:humble.devassy@gmail.com" target="_blank">humble.devassy@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><div>Hi,<br><br>Please look at this thread <br><a href="http://gluster.org/pipermail/gluster-users.old/2014-June/017819.html" target="_blank">http://gluster.org/pipermail/gluster-users.old/2014-June/017819.html</a><br><br></div>Btw, if you are around, we have a talk on same topic in upcoming GlusterFS India meetup.<br><br>Details can be fetched from:<br> <a href="http://www.meetup.com/glusterfs-India/" target="_blank">http://www.meetup.com/glusterfs-India/</a><br><br></div>--Humble<br></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div>--Humble<br></div><br></div></div></div>
<br><div class="gmail_quote"><span>On Sun, Nov 16, 2014 at 11:23 AM, Gopu Krishnan <span dir="ltr"><<a href="mailto:gopukrishnantec@gmail.com" target="_blank">gopukrishnantec@gmail.com</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><div dir="ltr">How can we monitor the glusters and alert us if something happened wrong. I found some nagios plugins and didn't work until this time. I am still experimenting with those. Any suggestions would be much helpful</div>
<br></span>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br><br><br clear="all"><div><br></div>
</div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>