<div dir="ltr">Hopefully I'm not derailing this thread too far, but I have a related rebalance progress/speed issue.<div><br></div><div>I have a rebalance process started that's been running for 3-4 days. Is there a good way to see if it's running successfully, or might this be a sign of some problem?</div>
<div><br></div><div>This is on a 4-node distribute setup with v3.4.2 and 45T of data.</div><div><br></div><div>The *-rebalance.log has been silent since some informational messages when the rebalance started. There were a few initial warnings and errors that I observed, though:</div>
<div>
<p class=""><br></p><p class="">E [client-handshake.c:1397:client_setvolume_cbk] 0-cluster2-client-0: SETVOLUME on remote-host failed: Authentication failed</p><p class="">
</p><p class="">W [client-handshake.c:1365:client_setvolume_cbk] 0-cluster2-client-4: failed to set the volume (Permission denied)</p><p class="">
</p><p class="">W [client-handshake.c:1391:client_setvolume_cbk] 0-cluster2-client-4: failed to get 'process-uuid' from reply dict</p><p class="">
</p><p class="">W [socket.c:514:__socket_rwv] 0-cluster2-client-3: readv failed (No data available)</p><p class=""><br></p><p class="">"gluster volume status" reports that the rebalance is in progress, the process listed in vols/<volname>/rebalance/<hash>.pid is still running on the server, but "gluster volume rebalance <volname> status" reports 0 for everything (files scanned or rebalanced, failures, run time).</p>
<p class="">Thanks,</p><p class="">Matt</p></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 27, 2014 at 12:39 AM, Shylesh Kumar <span dir="ltr"><<a href="mailto:shmohan@redhat.com" target="_blank">shmohan@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Viktor,<br>
<br>
Lots of optimizations and improvements went in for 3.4 so it should be faster than 3.2.<br>
Just to make sure what's happening could you please check rebalance logs which will be in<br>
/var/log/glusterfs/<volname>-<u></u>rebalance.log and check is there any progress ?<br>
<br>
Thanks,<br>
Shylesh<div class="HOEnZb"><div class="h5"><br>
<br>
Viktor Villafuerte wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Anybody can confirm/dispute that this is normal/abnormal?<br>
<br>
v<br>
<br>
<br>
On Tue 25 Feb 2014 15:21:40, Viktor Villafuerte wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
I have distributed replicated set with 2 servers (replicas) and am<br>
trying to add another set of replicas: 1 x (1x1) => 2 x (1x1)<br>
<br>
I have about 23G of data which I copy onto the first replica, check<br>
everything and then add the other set of replicas and eventually<br>
rebalance fix-layout, migrate-data.<br>
<br>
Now on<br>
<br>
Gluster v3.2.5 this took about 30 mins (to rebalance + migrate-data)<br>
<br>
on<br>
<br>
Gluster v3.4.2 this has been running for almost 4 hours and it's still<br>
not finished<br>
<br>
<br>
As I may have to do this in production, where the amount of data is<br>
significantly larger than 23G, I'm looking at about three weeks of wait<br>
to rebalance :)<br>
<br>
Now my question is if this is as it's meant to be? I can see that v3.4.2<br>
gives me more info about the rebalance process etc, but that surely<br>
cannot justify the enormous time difference.<br>
<br>
Is this normal/expected behaviour? If so I will have to stick with the<br>
v3.2.5 as it seems way quicker.<br>
<br>
Please, let me know if there is any 'well known' option/way/secret to<br>
speed the rebalance up on v3.4.2.<br>
<br>
<br>
thanks<br>
<br>
<br>
<br>
-- <br>
Regards<br>
<br>
Viktor Villafuerte<br>
Optus Internet Engineering<br>
t: 02 808-25265<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
</blockquote></blockquote>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
</div></div></blockquote></div><br></div>