Is there any reason you didn't suggest upgrading to 3.2.6 instead?<br><br>
<div class="gmail_quote">On Fri, Oct 21, 2011 at 12:12 AM, Amar Tumballi <span dir="ltr"><<a href="mailto:amar@gluster.com">amar@gluster.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">Thanks for the logs. <br><br>This is due to another issue of 'gfid mismatches', which causes the some locking deadlock in replicate, which will make each request to fix-layout take 30mins (in your case i see frame timeout is set to very low value of 30, which is why you see it many times in 5mins), which explains the slowness of whole operation.<br>
<br>Please plan to upgrade to 3.2.4 version, which has most of the fixes related to gfid mismatch issues.<br><br>Regards,<br><font color="#888888">Amar</font>
<div>
<div></div>
<div class="h5"> </div></div></blockquote>
<div> </div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>
<div class="h5"><br>
<div class="gmail_quote">On Fri, Oct 21, 2011 at 12:23 PM, Changliang Chen <span dir="ltr"><<a href="mailto:hqucocl@gmail.com" target="_blank">hqucocl@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">Any help?<br><br>We notice that if the below errors appears,the rebalance fixed layout will become very slow,the number just increase about 4 per five minutes.<br>
<br>E [rpc-clnt.c:199:call_bail] dfs-client-0: bailing out frame type(GlusterFS 3.1) op(INODELK(29)) xid = 0x755696 sent = 2011-10-20 06:20:51.217782. timeout = 30<br><br> W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.<br>
<br>I [dht-common.c:369:dht_revalidate_cbk] dfs-dht: subvolume 19loudfs-replicate-2 returned -1 (Invalid argument)
<div>
<div></div>
<div><br><br>
<div class="gmail_quote">On Tue, Oct 18, 2011 at 5:45 PM, Changliang Chen <span dir="ltr"><<a href="mailto:hqucocl@gmail.com" target="_blank">hqucocl@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0pt 0pt 0pt 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>Thanks Amar,but it looks like that the v3.1.1 hasn't support the command <br></div>
<div>
<div><br>'gluster volume rebalance dfs migrate-data start' <br><br></div></div>
<div># gluster volume rebalance dfs migrate-data start<br>Usage: volume rebalance <VOLNAME> <start|stop|status><br>Rebalance of Volume dfs failed<br><br></div>
<div class="gmail_quote">
<div>On Tue, Oct 18, 2011 at 3:33 PM, Amar Tumballi <span dir="ltr"><<a href="mailto:amar@gluster.com" target="_blank">amar@gluster.com</a>></span> wrote:<br></div>
<div>
<div></div>
<div>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0pt 0pt 0pt 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">Hi Chen,<br><br>Can you restart the 'glusterd' and run 'gluster volume rebalance dfs migrate-data start' and check if your data migration happens? <br>
<br>Regards,<br>Amar<br><br>
<div class="gmail_quote">
<div>
<div></div>
<div>On Tue, Oct 18, 2011 at 12:54 PM, Changliang Chen <span dir="ltr"><<a href="mailto:hqucocl@gmail.com" target="_blank">hqucocl@gmail.com</a>></span> wrote:<br></div></div>
<blockquote style="BORDER-LEFT:rgb(204,204,204) 1px solid;MARGIN:0pt 0pt 0pt 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div>
<div></div>
<div>Hi guys,<br><br> we have a rebalance running on eight bricks since July and this is what the status looks like right now:<br><br>===Tue Oct 18 13:45:01 CST 2011 ====<br>rebalance step 1: layout fix in progress: fixed layout 223623<br>
<br>There are roughly 8T photos in the storage,so how long should this rebalance take?<br><br>What does the number (in this case) 22362 represent?<br><br>Our gluster infomation:<br>Repository revision: v3.1.1<br>Volume Name: dfs<br>
Type: Distributed-Replicate<br>Status: Started<br>Number of Bricks: 4 x 2 = 8<br>Transport-type: tcp<br>Bricks:<br>Brick1: 10.1.1.23:/data0<br>Brick2: 10.1.1.24:/data0<br>Brick3: 10.1.1.25:/data0<br>Brick4: 10.1.1.26:/data0<br>
Brick5: 10.1.1.27:/data0<br>Brick6: 10.1.1.28:/data0<br>Brick7: 10.1.1.64:/data0<br>Brick8: 10.1.1.65:/data0<br>Options Reconfigured:<br>cluster.min-free-disk: 10%<br>network.ping-timeout: 25<br>network.frame-timeout: 30<br>
performance.cache-max-file-size: 512KB<br>performance.cache-size: 3GB<br><font color="#888888"><br><br clear="all"><br>-- <br><br>Regards,<br><br>Cocl<br>OM manager<br>19lou Operation & Maintenance Dept<br></font><br>
</div></div>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br></blockquote></div></div></div>
<div>
<div></div>
<div><br><br clear="all"><br>-- <br><br>Regards,<br><br>Cocl<br>OM manager<br>19lou Operation & Maintenance Dept<br></div></div></blockquote></div><br><br clear="all"><br>-- <br><br>Regards,<br><br>Cocl<br>OM manager<br>
19lou Operation & Maintenance Dept<br></div></div><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br><br></blockquote></div><br></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br>