<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Ok, confirmed. Letting dbench work on different subfolders makes it work fine.<div>In terms of log messages:</div><div>I would love to monitor for problems with gluster. Failing self heals in the logs might not really be a problem per-se, any suggestions of what else I could parse the logs for?</div><div><br></div><div>Cheers,</div><div>Marc</div><div><br><div><div>On 19.02.2013, at 23:32, Anand Avati <<a href="mailto:anand.avati@gmail.com">anand.avati@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">They were just self-heal double-checks performed in the lookup path, as we perform the initial lookup unlocked for performance reasons. Those lines by themselves do not indicate anything harmful (yet). I admit the log messages could have been better though!<div>
<br></div><div>Avati<br><br><div class="gmail_quote">On Tue, Feb 19, 2013 at 2:20 PM, Marc Seeger <span dir="ltr"><<a href="mailto:marc.seeger@acquia.com" target="_blank">marc.seeger@acquia.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>I'll give it a try. Isn't it somewhat concerning that this would heal to missing gfids and failing self-heals though?</div>
<div><div class="h5"><br><div><div>On 19.02.2013, at 21:22, Anand Avati <<a href="mailto:anand.avati@gmail.com" target="_blank">anand.avati@gmail.com</a>> wrote:</div><br><blockquote type="cite">We run dbench in the pre-commit script for every patch. Looks like you are running dbench on the same directory from multiple clients. Is that even supposed to work? Did you instead lunch to run each dbench within their own subdirectory?<div>
<br></div><div>Avati<br><br><div class="gmail_quote">On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <span dir="ltr"><<a href="mailto:marc.seeger@acquia.com" target="_blank">marc.seeger@acquia.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>To test gluster's behavior under heavy load, I'm currently doing this on two machines sharing a common /mnt/gfs gluster mount:</div>
<div><br></div>ssh <a href="http://bal-6.example.com/" target="_blank">bal-6.example.com</a> apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs<div>ssh bal-7.<a href="http://example.com/" target="_blank">example.com</a> apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs</div>
<div><br></div><div><br></div><div>One of the processes usually dies pretty quickly like this:</div><div><br></div><div>[608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for handle 10003 (No such file or directory)</div>
<div>(610) ERROR: handle 10003 was not found,</div><div>Child failed with status 1</div><div><br></div><div><br></div><div>And the logs are full of things like this (ignore the initial timestamp, that's from our logging):</div>
<div><div><br></div><div>[2013-02-19 14:38:38.714493] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client5/~dmtmp/PM/MOVED.DOC, </div>
<div>[2013-02-19 14:38:38.724494] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client4/~dmtmp/PM/EVENTS.DOC, </div>
<div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client1/~dmtmp/PM/MOVED.DOC, </div>
<div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client0/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client4/~dmtmp/PM, [build-2 system.rb:340], I, </div>
<div>[2013-02-19T14:39:50.189970 #20802] INFO -- : </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div>
<div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div>
<div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.051890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients, </div>
<div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2, </div><div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3, </div>
<div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2, </div><div>[2013-02-19 14:38:36.081890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3, </div>
<div>[2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp, </div><div>[2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp, </div>
<div>[2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp, </div><div>[2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp, </div>
<div>[2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD, </div><div>
[2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.131890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/WORD, </div>
<div>[2013-02-19 14:38:36.141890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/CHAP10.DOC, </div>
<div>[2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/CHAP10.DOC, </div>
<div>[2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/BASEMACH.DOC, </div>
<div>[2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/BASEMACH.DOC, </div>
<div>[2013-02-19 14:38:36.171890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entr [build-2 system.rb:340], I, </div><div>[2013-02-19T14:39:50.189970 #20802] INFO -- : y missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, </div>
<div>[2013-02-19 14:38:36.181890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/FACTS.DOC, </div>
<div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL, </div><div>
[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/EXCEL, </div>
<div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client0/~dmtmp, </div>
<div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, </div>
<div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, </div>
<div>[2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/SALES.XLS, </div>
<div>[2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/SALES.XLS, </div>
<div>[2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT, </div><div>
[2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, </div>
<div>[2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.291890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, </div>
<div>[2013-02-19 14:38:36.311890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, </div>
<div>[2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, </div>
<div>[2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, </div>
<div>[2013-02-19 14:38:36.381890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, </div>
<div>[2013-02-19 14:38:36.391890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid sel [build-2 system.rb:340]</div></div><div><br></div><div>
<br>
</div><div><br></div><div>Any ideas? Can somebody confirm this happens for them too?</div><div><br></div><div>The setup is ubuntu lucid machines running 3.3.1 from this PPA: <a href="https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3" target="_blank">https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3</a></div>
<div><br></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div></div></blockquote></div><br></div>
</blockquote></div><br></div></body></html>