<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi Amar.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Is there a projected release date for 3.1.5 and 3.2.1?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Burnash, James<br><b>Sent:</b> Friday, May 27, 2011 1:42 PM<br><b>To:</b> 'Amar Tumballi'<br><b>Cc:</b> gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] Files present on the backend but have become invisible from clients<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thank you Amar – this is much appreciated and gives me a better understanding of the meaning of some of these attributes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I would like to suggest that something at least on this level be added to the Gluster documentation for future use by others, as well as myself (I forget sometimes </span><span style='font-size:11.0pt;font-family:Wingdings;color:#1F497D'>J</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>So – as far as what I can do about these incorrect Replicate attributes – it appears that the answer is “nothing until the next release”? Or will triggering a self heal on those directories specifically clean things up?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks as always,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>James<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Amar Tumballi [mailto:amar@gluster.com] <br><b>Sent:</b> Friday, May 27, 2011 12:23 PM<br><b>To:</b> Burnash, James<br><b>Cc:</b> Mohit Anchlia; gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] Files present on the backend but have become invisible from clients<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>James,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>Replies inline.<o:p></o:p></p><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'>The directories are all still visible to the users, but scanning for attributes of 0sAAAAAAAAAAAAAAAA still yielded matches on the set of GlusterFS servers.<br><br><a href="http://pastebin.com/mxvFnFj4" target="_blank">http://pastebin.com/mxvFnFj4</a><br><br>I tried running this command, but as you can see it wasn't happy, even though the syntax was correct:<br><br>root@jc1letgfs17:~# gluster volume rebalance pfs-ro1 fix-layout start<br>Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status}<br><br>I suspect this is a bug because of the "-" in my volume name. I'll test and confirm and file when I get a chance.<o:p></o:p></p></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>This seems to be an bug with the 'fix-layout' CLI option itself (as i assume the version in 3.1.3, its fixed in 3.1.4+ or 3.2.0), please use just 'rebalance <VOLNAME> start'.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'>So I just did the standard rebalance command:<br> gluster volume rebalance pfs-ro1 start<br><br>and it trundled along for a while and then one time when checked it's status, it failed:<br> date; gluster volume rebalance pfs-ro1 status<br> Thu May 26 09:02:00 EDT 2011<br> rebalance failed<br><br>I re-ran it FOUR times getting a little farther with each attempt, and it eventually completed and then started doing the actual file migration part of the rebalance:<br> Thu May 26 12:22:25 EDT 2011<br> rebalance step 1: layout fix in progress: fixed layout 779<br> Thu May 26 12:23:25 EDT 2011<br> rebalance step 2: data migration in progress: rebalanced 71 files of size 136518704 (total files scanned 57702)<br><br>Now scanning for attributes of 0sAAAAAAAAAAAAAAAA yields less results, but some are still present:<o:p></o:p></p></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Now, doing a 'rebalance' is surely not the way to heal the 'replicate' related attributes. 'rebalance' is all about fixing the 'distribute' related 'layout's and rebalancing the data within the servers.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>It could have helped in resolving some of the attributes of 'replicate' as issuing a rebalance triggers a directory traversal on the volume (which is infact same as doing a 'ls -lR' or 'find' on volume). <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'><a href="http://pastebin.com/x4wYq8ic" target="_blank">http://pastebin.com/x4wYq8ic</a><br><br>As a possible sanity check, I did this command on my Read-Write GlusterFS storage servers (2 boxes, Distributed-Replicate), and got no "bad" attributes:<br> jc1ladmin1:~/projects/gluster loop_check ' getfattr -dm - /export/read-only/g*' jc1letgfs{13,16} | egrep "jc1letgfs|0sAAAAAAAAAAAAAAAA$|file:" | less<br> getfattr: /export/read-only/g*: No such file or directory<br> getfattr: /export/read-only/g*: No such file or directory<br> jc1letgfs13<br> jc1letgfs16<br><br>One difference in these two Storage server groups - the Read-Only group of 4 servers have their backend file systems formatted as XFS, while the Read-Write group of 2 are formatted with EXT4.<br><br>Suggestions, critiques, etc gratefully solicited.<o:p></o:p></p></blockquote><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Please, next time while looking at the GlusterFS attributes use '-e hex' for 'getfattr' command. Anyways, I think the issue here is mostly due to some sort of bug which resulted in writing attributes saying 'split-brain' happened, and if that is the attribute, 'replicate' module doesn't heal anything and leaves the file as is (without even fixing the attribute). <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>We are currently working on fixing these meta-data self-heal related issues right now and hope to fix many of them by 3.2.1 (and 3.1.5).<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Regards,<o:p></o:p></p></div><div><p class=MsoNormal>Amar<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><p class=MsoNormal><span style='color:#888888'>James Burnash<br>Unix Engineer.</span><o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div></blockquote></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p><p><span style='font-size:7.5pt;color:navy'>DISCLAIMER:<br>This e-mail, and any attachments thereto, is intended only for use by the addressee(s)named herein and<br>may contain legally privileged and/or confidential information. If you are not the intended recipient of this<br>e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachments</span><span style='color:navy'><br></span><span style='font-size:7.5pt;color:navy'>thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently<br>delete the original and any printout thereof.</span><span style='font-size:7.5pt'> <span style='color:navy'>E-mail transmission cannot be guaranteed to be secure or error-free.<br>The sender therefore does not accept liability for any errors or omissions in the contents of this message which<br>arise as a result of e-mail transmission.<br>NOTICE REGARDING PRIVACY AND CONFIDENTIALITY<br>Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications.</span></span><o:p></o:p></p><p><a href="http://www.knight.com/"><span style='font-size:7.5pt;color:navy'>http://www.knight.com</span></a><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p></div></body></html>