Hi Jochen,<br><br>Can you try running 'gluster volume rebalance <VOLNAME> fix-layout start' again?<br><br>Seems that some of the layout is not valid. Also, make sure all your bricks are up and running.<br><br>
Also, are you using ACLs? can the accessing of directories/files work as 'root' user?<br><br>Regards,<br>Amar<br><br><div class="gmail_quote">On Mon, Oct 17, 2011 at 6:29 PM, Jochen Lillich <span dir="ltr"><<a href="mailto:jochen@freistil.it">jochen@freistil.it</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi,<br>
<br>
when our two-node distributed backup storage cluster, running on<br>
GlusterFS 3.0.2 (Ubuntu Lucid packages), started running out of space,<br>
I added a third node and tried rebalancing them. The scale-n-defrag<br>
script didn't work, though. I always got a "stale NFS handle".<br>
<br>
So I decided to upgrade and installed the 3.2.4 packages from<br>
semiosis' Launchpad PPA, followed the upgrade guide for 3.0 -> 3.2 and<br>
then started "gluster volume rebalance". After two days, the three<br>
nodes are now about equally full, so the rebalancing worked.<br>
<br>
Unfortunately, it seems that it also broke the cluster filesystem. We<br>
get a lot of "permission denied" errors although file system<br>
permissions are correct. The log contains a lot of these entries:<br>
<br>
[2011-10-17 10:11:48.979801] I<br>
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote<br>
operation failed: Permission denied<br>
[2011-10-17 10:11:48.979892] I [dht-common.c:478:dht_revalidate_cbk]<br>
0-glfs-dht: subvolume glfs-client-2 for /bacula/mail02 returned -1<br>
(Permission denied)<br>
[2011-10-17 10:11:48.980216] I [dht-layout.c:192:dht_layout_search]<br>
0-glfs-dht: no subvolume for hash (value) = 1486987642<br>
[2011-10-17 10:11:48.981101] I<br>
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote<br>
operation failed: Permission denied<br>
[2011-10-17 10:11:48.981581] I [dht-layout.c:192:dht_layout_search]<br>
0-glfs-dht: no subvolume for hash (value) = 1486987642<br>
[2011-10-17 10:11:48.982468] I<br>
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote<br>
operation failed: Permission denied<br>
<br>
How can I repair this?<br>
<br>
Best regards,<br>
Jochen<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br>