<div dir="ltr">Hi Viktor,<div><br></div><div>Thanks for the tips. I'm a bit confused, since the clients mount the share fine, and "gluster peer status" and "gluster volume status all detail" are happy.</div>
<div><br></div><div>What is the expected output of "rebalance status" for just a fix-layout run? I believe the last time I did that, the status was always 0s (which makes some sense, as files aren't moving) and the log was empty, but the operation seemed to complete successfully. Does a file rebalance first require a fix-layout operation internally, and is it possible that my volume is still in that phase? Or I making up an overly optimistic scenario?</div>
<div><br></div><div>Thanks,</div><div><br></div><div>Matt</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 27, 2014 at 8:33 PM, Viktor Villafuerte <span dir="ltr"><<a href="mailto:viktor.villafuerte@optusnet.com.au" target="_blank">viktor.villafuerte@optusnet.com.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Matt,<br>
<br>
if the 'status' says 0 for everything that's not good. Normally when I<br>
do rebalance the numbers should change (up). Also the rebalance log<br>
should show files being moved around.<br>
<br>
For the errors - my (limited) experience with Gluster is that the 'W'<br>
are normally harmless and they show up quite a bit. For the actuall<br>
error 'E' you could try to play with 'auth.allow' as suggested here<br>
<br>
<a href="http://gluster.org/pipermail/gluster-users/2011-November/009094.html" target="_blank">http://gluster.org/pipermail/gluster-users/2011-November/009094.html</a><br>
<br>
<br>
Normally when rebalancing I do count of files on the bricks and the<br>
Gluster mount to make sure they eventually add up. Also I grep and count<br>
'-T' and see how the count goes down and 'rw' count goes up.<br>
<span class="HOEnZb"><font color="#888888"><br>
v<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
On Thu 27 Feb 2014 00:57:28, Matt Edwards wrote:<br>
> Hopefully I'm not derailing this thread too far, but I have a related<br>
> rebalance progress/speed issue.<br>
><br>
> I have a rebalance process started that's been running for 3-4 days. Is<br>
> there a good way to see if it's running successfully, or might this be a<br>
> sign of some problem?<br>
><br>
> This is on a 4-node distribute setup with v3.4.2 and 45T of data.<br>
><br>
> The *-rebalance.log has been silent since some informational messages when<br>
> the rebalance started. There were a few initial warnings and errors that I<br>
> observed, though:<br>
><br>
><br>
> E [client-handshake.c:1397:client_setvolume_cbk] 0-cluster2-client-0:<br>
> SETVOLUME on remote-host failed: Authentication failed<br>
><br>
> W [client-handshake.c:1365:client_setvolume_cbk] 0-cluster2-client-4:<br>
> failed to set the volume (Permission denied)<br>
><br>
> W [client-handshake.c:1391:client_setvolume_cbk] 0-cluster2-client-4:<br>
> failed to get 'process-uuid' from reply dict<br>
><br>
> W [socket.c:514:__socket_rwv] 0-cluster2-client-3: readv failed (No data<br>
> available)<br>
><br>
><br>
> "gluster volume status" reports that the rebalance is in progress, the<br>
> process listed in vols/<volname>/rebalance/<hash>.pid is still running on<br>
> the server, but "gluster volume rebalance <volname> status" reports 0 for<br>
> everything (files scanned or rebalanced, failures, run time).<br>
><br>
> Thanks,<br>
><br>
> Matt<br>
><br>
><br>
> On Thu, Feb 27, 2014 at 12:39 AM, Shylesh Kumar <<a href="mailto:shmohan@redhat.com">shmohan@redhat.com</a>> wrote:<br>
><br>
> > Hi Viktor,<br>
> ><br>
> > Lots of optimizations and improvements went in for 3.4 so it should be<br>
> > faster than 3.2.<br>
> > Just to make sure what's happening could you please check rebalance logs<br>
> > which will be in<br>
> > /var/log/glusterfs/<volname>-rebalance.log and check is there any<br>
> > progress ?<br>
> ><br>
> > Thanks,<br>
> > Shylesh<br>
> ><br>
> ><br>
> > Viktor Villafuerte wrote:<br>
> ><br>
> >> Anybody can confirm/dispute that this is normal/abnormal?<br>
> >><br>
> >> v<br>
> >><br>
> >><br>
> >> On Tue 25 Feb 2014 15:21:40, Viktor Villafuerte wrote:<br>
> >><br>
> >>> Hi all,<br>
> >>><br>
> >>> I have distributed replicated set with 2 servers (replicas) and am<br>
> >>> trying to add another set of replicas: 1 x (1x1) => 2 x (1x1)<br>
> >>><br>
> >>> I have about 23G of data which I copy onto the first replica, check<br>
> >>> everything and then add the other set of replicas and eventually<br>
> >>> rebalance fix-layout, migrate-data.<br>
> >>><br>
> >>> Now on<br>
> >>><br>
> >>> Gluster v3.2.5 this took about 30 mins (to rebalance + migrate-data)<br>
> >>><br>
> >>> on<br>
> >>><br>
> >>> Gluster v3.4.2 this has been running for almost 4 hours and it's still<br>
> >>> not finished<br>
> >>><br>
> >>><br>
> >>> As I may have to do this in production, where the amount of data is<br>
> >>> significantly larger than 23G, I'm looking at about three weeks of wait<br>
> >>> to rebalance :)<br>
> >>><br>
> >>> Now my question is if this is as it's meant to be? I can see that v3.4.2<br>
> >>> gives me more info about the rebalance process etc, but that surely<br>
> >>> cannot justify the enormous time difference.<br>
> >>><br>
> >>> Is this normal/expected behaviour? If so I will have to stick with the<br>
> >>> v3.2.5 as it seems way quicker.<br>
> >>><br>
> >>> Please, let me know if there is any 'well known' option/way/secret to<br>
> >>> speed the rebalance up on v3.4.2.<br>
> >>><br>
> >>><br>
> >>> thanks<br>
> >>><br>
> >>><br>
> >>><br>
> >>> --<br>
> >>> Regards<br>
> >>><br>
> >>> Viktor Villafuerte<br>
> >>> Optus Internet Engineering<br>
> >>> t: 02 808-25265<br>
> >>> _______________________________________________<br>
> >>> Gluster-users mailing list<br>
> >>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >>> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
> >>><br>
> >><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
<br>
--<br>
Regards<br>
<br>
Viktor Villafuerte<br>
Optus Internet Engineering<br>
t: 02 808-25265<br>
</div></div></blockquote></div><br></div>