<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>To test gluster's behavior under heavy load, I'm currently doing this on two machines sharing a common /mnt/gfs gluster mount:</div><div><br></div>ssh <a href="http://bal-6.example.com">bal-6.example.com</a> apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs<div>ssh bal-7.<a href="http://example.com">example.com</a> apt-get install dbench && dbench 6 -t 60 -D /mnt/gfs</div><div><br></div><div><br></div><div>One of the processes usually dies pretty quickly like this:</div><div><br></div><div>[608] open /mnt/gfs/clients/client5/~dmtmp/PWRPNT/PCBENCHM.PPT failed for handle 10003 (No such file or directory)</div><div>(610) ERROR: handle 10003 was not found,</div><div>Child failed with status 1</div><div><br></div><div><br></div><div>And the logs are full of things like this (ignore the initial timestamp, that's from our logging):</div><div><div><br></div><div>[2013-02-19 14:38:38.714493] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client5/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.724494] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client4/~dmtmp/PM/EVENTS.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client1/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background data missing-entry gfid self-heal failed on /clients/client0/~dmtmp/PM/MOVED.DOC, </div><div>[2013-02-19 14:38:38.734495] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client4/~dmtmp/PM, [build-2 system.rb:340], I, </div><div>[2013-02-19T14:39:50.189970 #20802] INFO -- : </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.041890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /, </div><div>[2013-02-19 14:38:36.051890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients, </div><div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2, </div><div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3, </div><div>[2013-02-19 14:38:36.071890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2, </div><div>[2013-02-19 14:38:36.081890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3, </div><div>[2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp, </div><div>[2013-02-19 14:38:36.091890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp, </div><div>[2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp, </div><div>[2013-02-19 14:38:36.101890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp, </div><div>[2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.111890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.131890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.141890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/WORD, </div><div>[2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/CHAP10.DOC, </div><div>[2013-02-19 14:38:36.151890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/CHAP10.DOC, </div><div>[2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/BASEMACH.DOC, </div><div>[2013-02-19 14:38:36.161890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/BASEMACH.DOC, </div><div>[2013-02-19 14:38:36.171890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entr [build-2 system.rb:340], I, </div><div>[2013-02-19T14:39:50.189970 #20802] INFO -- : y missing-entry gfid self-heal failed on /clients/client2/~dmtmp/WORD/FACTS.DOC, </div><div>[2013-02-19 14:38:36.181890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/WORD/FACTS.DOC, </div><div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.201890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/EXCEL, </div><div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client0/~dmtmp, </div><div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/PCMAGCD.XLS, </div><div>[2013-02-19 14:38:36.211890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/PCMAGCD.XLS, </div><div>[2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/EXCEL/SALES.XLS, </div><div>[2013-02-19 14:38:36.241890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/EXCEL/SALES.XLS, </div><div>[2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.271890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client2/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.281890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background entry self-heal failed on /clients/client3/~dmtmp/PWRPNT, </div><div>[2013-02-19 14:38:36.291890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PCBENCHM.PPT, </div><div>[2013-02-19 14:38:36.311890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/PCBENCHM.PPT, </div><div>[2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/ZD16.BMP, </div><div>[2013-02-19 14:38:36.351890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client3/~dmtmp/PWRPNT/ZD16.BMP, </div><div>[2013-02-19 14:38:36.381890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid self-heal failed on /clients/client2/~dmtmp/PWRPNT/PPTOOLS1.PPA, </div><div>[2013-02-19 14:38:36.391890] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-replicate0: background meta-data data entry missing-entry gfid sel [build-2 system.rb:340]</div></div><div><br></div><div><br></div><div><br></div><div>Any ideas? Can somebody confirm this happens for them too?</div><div><br></div><div>The setup is ubuntu lucid machines running 3.3.1 from this PPA: <a href="https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3">https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3</a></div><div><br></div></body></html>