<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I'll give it a try. Isn't it somewhat concerning that this would heal to missing gfids and failing self-heals though?</div><br><div><div>On 19.02.2013, at 21:22, Anand Avati <<a href="mailto:anand.avati@gmail.com">anand.avati@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">We run dbench in the pre-commit script for every patch. Looks like you are running dbench on the same directory from multiple clients. Is that even supposed to work? Did you instead lunch to run each dbench within their own subdirectory?<div>
<br></div><div>Avati<br><br><div class="gmail_quote">On Tue, Feb 19, 2013 at 12:07 PM, Marc Seeger <span dir="ltr"><<a href="mailto:marc.seeger@acquia.com" target="_blank">marc.seeger@acquia.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>To test gluster's behavior under heavy load, I'm currently doing this on two machines sharing a common /mnt/gfs gluster mount:</div>
<div><br></div>ssh <a href="http://bal-6.example.com/" target="_blank">bal-6.example.com</a> apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs<div>ssh bal-7.<a href="http://example.com/" target="_blank">example.com</a> apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs</div>
<div><br></div><div><br></div><div>One of the processes usually dies pretty quickly like this:</div><div><br></div><div>[608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for handle 10003 (No such file or directory)</div>
<div>(610) ERROR: handle 10003 was not found,</div><div>Child failed with status 1</div><div><br></div><div><br></div><div>And the logs are full of things like this (ignore the initial timestamp, that's from our logging):</div>
<div><div><br></div><div>[2013-02-19 14:38:38.714493] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client5/~dmtmp/PM/MOVED.DOC, </div>
<div>[2013-02-19 14:38:38.724494] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client4/~dmtmp/PM/EVENTS.DOC, </div>
<div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client1/~dmtmp/PM/MOVED.DOC, </div>
<div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client0/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client4/~dmtmp/PM, [build-2 system.rb:340], I, </div>
<div>[2013-02-19T14:39:50.189970 #20802] INFO -- : </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div>
<div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div>
<div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.051890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients, </div>
<div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2, </div><div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3, </div>
<div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2, </div><div>[2013-02-19 14:38:36.081890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3, </div>
<div>[2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp, </div><div>[2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp, </div>
<div>[2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp, </div><div>[2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp, </div>
<div>[2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD, </div><div>
[2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.131890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/WORD, </div>
<div>[2013-02-19 14:38:36.141890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/CHAP10.DOC, </div>
<div>[2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/CHAP10.DOC, </div>
<div>[2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/BASEMACH.DOC, </div>
<div>[2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/BASEMACH.DOC, </div>
<div>[2013-02-19 14:38:36.171890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entr [build-2 system.rb:340], I, </div><div>[2013-02-19T14:39:50.189970 #20802] INFO -- : y missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, </div>
<div>[2013-02-19 14:38:36.181890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/FACTS.DOC, </div>
<div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL, </div><div>
[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/EXCEL, </div>
<div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client0/~dmtmp, </div>
<div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, </div>
<div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, </div>
<div>[2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/SALES.XLS, </div>
<div>[2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/SALES.XLS, </div>
<div>[2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT, </div><div>
[2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, </div>
<div>[2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.291890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, </div>
<div>[2013-02-19 14:38:36.311890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, </div>
<div>[2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, </div>
<div>[2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, </div>
<div>[2013-02-19 14:38:36.381890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, </div>
<div>[2013-02-19 14:38:36.391890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid sel [build-2 system.rb:340]</div></div><div><br></div><div><br>
</div><div><br></div><div>Any ideas? Can somebody confirm this happens for them too?</div><div><br></div><div>The setup is ubuntu lucid machines running 3.3.1 from this PPA: <a href="https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3" target="_blank">https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3</a></div>
<div><br></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></body></html>