James,<div><br></div><div>Replies inline.<br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
The directories are all still visible to the users, but scanning for attributes of 0sAAAAAAAAAAAAAAAA still yielded matches on the set of GlusterFS servers.<br>
<br>
<a href="http://pastebin.com/mxvFnFj4" target="_blank">http://pastebin.com/mxvFnFj4</a><br>
<br>
I tried running this command, but as you can see it wasn't happy, even though the syntax was correct:<br>
<br>
root@jc1letgfs17:~# gluster volume rebalance pfs-ro1 fix-layout start<br>
Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status}<br>
<br>
I suspect this is a bug because of the "-" in my volume name. I'll test and confirm and file when I get a chance.<br>
<br>
</blockquote><div><br></div><div>This seems to be an bug with the 'fix-layout' CLI option itself (as i assume the version in 3.1.3, its fixed in 3.1.4+ or 3.2.0), please use just 'rebalance <VOLNAME> start'.</div>
<div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">So I just did the standard rebalance command:<br>
gluster volume rebalance pfs-ro1 start<br>
<br>
and it trundled along for a while and then one time when checked it's status, it failed:<br>
date; gluster volume rebalance pfs-ro1 status<br>
Thu May 26 09:02:00 EDT 2011<br>
rebalance failed<br>
<br>
I re-ran it FOUR times getting a little farther with each attempt, and it eventually completed and then started doing the actual file migration part of the rebalance:<br>
Thu May 26 12:22:25 EDT 2011<br>
rebalance step 1: layout fix in progress: fixed layout 779<br>
Thu May 26 12:23:25 EDT 2011<br>
rebalance step 2: data migration in progress: rebalanced 71 files of size 136518704 (total files scanned 57702)<br>
<br>
Now scanning for attributes of 0sAAAAAAAAAAAAAAAA yields less results, but some are still present:<br>
<br>
<a href="http://pastebin.com/x4wYq8ic" target="_blank"></a></blockquote><div><br></div><div>Now, doing a 'rebalance' is surely not the way to heal the 'replicate' related attributes. 'rebalance' is all about fixing the 'distribute' related 'layout's and rebalancing the data within the servers.</div>
<div><br></div><div>It could have helped in resolving some of the attributes of 'replicate' as issuing a rebalance triggers a directory traversal on the volume (which is infact same as doing a 'ls -lR' or 'find' on volume). </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><a href="http://pastebin.com/x4wYq8ic" target="_blank">http://pastebin.com/x4wYq8ic</a><br>
<br>
As a possible sanity check, I did this command on my Read-Write GlusterFS storage servers (2 boxes, Distributed-Replicate), and got no "bad" attributes:<br>
jc1ladmin1:~/projects/gluster loop_check ' getfattr -dm - /export/read-only/g*' jc1letgfs{13,16} | egrep "jc1letgfs|0sAAAAAAAAAAAAAAAA$|file:" | less<br>
getfattr: /export/read-only/g*: No such file or directory<br>
getfattr: /export/read-only/g*: No such file or directory<br>
jc1letgfs13<br>
jc1letgfs16<br>
<br>
One difference in these two Storage server groups - the Read-Only group of 4 servers have their backend file systems formatted as XFS, while the Read-Write group of 2 are formatted with EXT4.<br>
<br>
Suggestions, critiques, etc gratefully solicited.<br>
<font color="#888888"><br></font></blockquote><div><br></div><div>Please, next time while looking at the GlusterFS attributes use '-e hex' for 'getfattr' command. Anyways, I think the issue here is mostly due to some sort of bug which resulted in writing attributes saying 'split-brain' happened, and if that is the attribute, 'replicate' module doesn't heal anything and leaves the file as is (without even fixing the attribute). </div>
<div><br></div><div>We are currently working on fixing these meta-data self-heal related issues right now and hope to fix many of them by 3.2.1 (and 3.1.5).</div><div><br></div><div>Regards,</div><div>Amar</div><div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><font color="#888888">
James Burnash<br>
Unix Engineer.<br>
</font><div class="im"><br></div></blockquote></div></div>