<html><head><base href="x-msg://110/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Jun 10, 2011, at 1:11 PM, Burnash, James wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hi Amar.<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Is there a projected release date for 3.1.5 and 3.2.1?<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div><div style="border-right-style: none; border-bottom-style: none; border-left-style: none; border-width: initial; border-color: initial; border-top-style: solid; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding-top: 3pt; padding-right: 0in; padding-bottom: 0in; padding-left: 0in; "><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:gluster-users-bounces@gluster.org" style="color: blue; text-decoration: underline; ">gluster-users-bounces@gluster.org</a><span class="Apple-converted-space"> </span>[mailto:gluster-users-bounces@gluster.org]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Burnash, James<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, May 27, 2011 1:42 PM<br><b>To:</b><span class="Apple-converted-space"> </span>'Amar Tumballi'<br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:gluster-users@gluster.org" style="color: blue; text-decoration: underline; ">gluster-users@gluster.org</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [Gluster-users] Files present on the backend but have become invisible from clients<o:p></o:p></span></div></div></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thank you Amar – this is much appreciated and gives me a better understanding of the meaning of some of these attributes.<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I would like to suggest that something at least on this level be added to the Gluster documentation for future use by others, as well as myself (I forget sometimes<span class="Apple-converted-space"> </span></span><span style="font-size: 11pt; font-family: Wingdings; color: rgb(31, 73, 125); ">J</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">)<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">So – as far as what I can do about these incorrect Replicate attributes – it appears that the answer is “nothing until the next release”? Or will triggering a self heal on those directories specifically clean things up?<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks as always,<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">James<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p> </o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span>Amar Tumballi [mailto:amar@gluster.com]<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, May 27, 2011 12:23 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Burnash, James<br><b>Cc:</b><span class="Apple-converted-space"> </span>Mohit Anchlia;<span class="Apple-converted-space"> </span><a href="mailto:gluster-users@gluster.org" style="color: blue; text-decoration: underline; ">gluster-users@gluster.org</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [Gluster-users] Files present on the backend but have become invisible from clients<o:p></o:p></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">James,<o:p></o:p></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><p class="MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 12pt; ">Replies inline.<o:p></o:p></p><div><blockquote style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-width: initial; border-color: initial; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-left-width: 1pt; padding-top: 0in; padding-right: 0in; padding-bottom: 0in; padding-left: 6pt; margin-left: 4.8pt; margin-top: 5pt; margin-right: 0in; margin-bottom: 5pt; "><p class="MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 12pt; ">The directories are all still visible to the users, but scanning for attributes of 0sAAAAAAAAAAAAAAAA still yielded matches on the set of GlusterFS servers.<br><br><a href="http://pastebin.com/mxvFnFj4" target="_blank" style="color: blue; text-decoration: underline; ">http://pastebin.com/mxvFnFj4</a><br><br>I tried running this command, but as you can see it wasn't happy, even though the syntax was correct:<br><br>root@jc1letgfs17:~# gluster volume rebalance pfs-ro1 fix-layout start<br>Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status}<br><br>I suspect this is a bug because of the "-" in my volume name. I'll test and confirm and file when I get a chance.<o:p></o:p></p></blockquote><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">This seems to be an bug with the 'fix-layout' CLI option itself (as i assume the version in 3.1.3, its fixed in 3.1.4+ or 3.2.0), please use just 'rebalance <VOLNAME> start'.<o:p></o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><blockquote style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-width: initial; border-color: initial; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-left-width: 1pt; padding-top: 0in; padding-right: 0in; padding-bottom: 0in; padding-left: 6pt; margin-left: 4.8pt; margin-top: 5pt; margin-right: 0in; margin-bottom: 5pt; "><p class="MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 12pt; ">So I just did the standard rebalance command:<br> gluster volume rebalance pfs-ro1 start<br><br>and it trundled along for a while and then one time when checked it's status, it failed:<br> date; gluster volume rebalance pfs-ro1 status<br> Thu May 26 09:02:00 EDT 2011<br> rebalance failed<br><br>I re-ran it FOUR times getting a little farther with each attempt, and it eventually completed and then started doing the actual file migration part of the rebalance:<br> Thu May 26 12:22:25 EDT 2011<br> rebalance step 1: layout fix in progress: fixed layout 779<br> Thu May 26 12:23:25 EDT 2011<br> rebalance step 2: data migration in progress: rebalanced 71 files of size 136518704 (total files scanned 57702)<br><br>Now scanning for attributes of 0sAAAAAAAAAAAAAAAA yields less results, but some are still present:<o:p></o:p></p></blockquote><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">Now, doing a 'rebalance' is surely not the way to heal the 'replicate' related attributes. 'rebalance' is all about fixing the 'distribute' related 'layout's and rebalancing the data within the servers.<o:p></o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">It could have helped in resolving some of the attributes of 'replicate' as issuing a rebalance triggers a directory traversal on the volume (which is infact same as doing a 'ls -lR' or 'find' on volume). <o:p></o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "> <o:p></o:p></div></div><blockquote style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-width: initial; border-color: initial; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-left-width: 1pt; padding-top: 0in; padding-right: 0in; padding-bottom: 0in; padding-left: 6pt; margin-left: 4.8pt; margin-top: 5pt; margin-right: 0in; margin-bottom: 5pt; "><p class="MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 12pt; "><a href="http://pastebin.com/x4wYq8ic" target="_blank" style="color: blue; text-decoration: underline; ">http://pastebin.com/x4wYq8ic</a><br><br>As a possible sanity check, I did this command on my Read-Write GlusterFS storage servers (2 boxes, Distributed-Replicate), and got no "bad" attributes:<br> jc1ladmin1:~/projects/gluster loop_check ' getfattr -dm - /export/read-only/g*' jc1letgfs{13,16} | egrep "jc1letgfs|0sAAAAAAAAAAAAAAAA$|file:" | less<br> getfattr: /export/read-only/g*: No such file or directory<br> getfattr: /export/read-only/g*: No such file or directory<br> jc1letgfs13<br> jc1letgfs16<br><br>One difference in these two Storage server groups - the Read-Only group of 4 servers have their backend file systems formatted as XFS, while the Read-Write group of 2 are formatted with EXT4.<br><br>Suggestions, critiques, etc gratefully solicited.<o:p></o:p></p></blockquote><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">Please, next time while looking at the GlusterFS attributes use '-e hex' for 'getfattr' command. Anyways, I think the issue here is mostly due to some sort of bug which resulted in writing attributes saying 'split-brain' happened, and if that is the attribute, 'replicate' module doesn't heal anything and leaves the file as is (without even fixing the attribute). <o:p></o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">We are currently working on fixing these meta-data self-heal related issues right now and hope to fix many of them by 3.2.1 (and 3.1.5).</div></div></div></div></div></div></span></blockquote><div><br></div><div><br></div>Is there a way to fix that attribute while the new version comes out?</div><div>unmounting and re-mounting doesn't seem to work, but other client nodes work fine.</div><div><div>[2011-06-10 18:28:08.966770] E [afr-common.c:110:afr_set_split_brain] gluster_replication-replicate-0: invalid argument: inode</div><div><br></div><div><div><b><font class="Apple-style-span" color="#235B21">$</font><font class="Apple-style-span" color="#BB251D"> gluster --version </font></b> ~</div><div>glusterfs 3.1.2 built on Jan 18 2011 11:19:54</div><div>Repository revision: v3.1.1-64-gf2a067c</div></div><div><br></div><blockquote type="cite"><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div><div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">Regards,<o:p></o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; ">Amar<o:p></o:p></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "> <o:p></o:p></div></div><blockquote style="border-top-style: none; border-right-style: none; border-bottom-style: none; border-width: initial; border-color: initial; border-left-style: solid; border-left-color: rgb(204, 204, 204); border-left-width: 1pt; padding-top: 0in; padding-right: 0in; padding-bottom: 0in; padding-left: 6pt; margin-left: 4.8pt; margin-top: 5pt; margin-right: 0in; margin-bottom: 5pt; position: static; z-index: auto; "><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><span style="color: rgb(136, 136, 136); ">James Burnash<br>Unix Engineer.</span><o:p></o:p></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "><o:p> </o:p></div></div></blockquote></div></div><p class="MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 12pt; "><o:p> </o:p></p><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 7.5pt; color: navy; ">DISCLAIMER:<br>This e-mail, and any attachments thereto, is intended only for use by the addressee(s)named herein and<br>may contain legally privileged and/or confidential information. If you are not the intended recipient of this<br>e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachments</span><span style="color: navy; "><br></span><span style="font-size: 7.5pt; color: navy; ">thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently<br>delete the original and any printout thereof.</span><span style="font-size: 7.5pt; "><span class="Apple-converted-space"> </span><span style="color: navy; ">E-mail transmission cannot be guaranteed to be secure or error-free.<br>The sender therefore does not accept liability for any errors or omissions in the contents of this message which<br>arise as a result of e-mail transmission.<br>NOTICE REGARDING PRIVACY AND CONFIDENTIALITY<br>Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications.</span></span><o:p></o:p></p><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><a href="http://www.knight.com/" style="color: blue; text-decoration: underline; "><span style="font-size: 7.5pt; color: navy; ">http://www.knight.com</span></a><o:p></o:p></p><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; margin-top: 0in; margin-bottom: 0.0001pt; "> <o:p></o:p></div></div>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" style="color: blue; text-decoration: underline; ">Gluster-users@gluster.org</a><br><a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" style="color: blue; text-decoration: underline; ">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br></div></blockquote></div><br></body></html>