<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On Wednesday 16 July 2014 10:18 AM,
David Raffelt wrote:<br>
</div>
<blockquote
cite="mid:CAOFq7B6Ur_ZeRuGh6JnS4ZHwu1FvcQXO9L2=606ucfT1VAHEwQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Raghavendra,
<div>No</div>
<div>Thanks </div>
<div>Dave</div>
</div>
<div class="gmail_extra"><br>
<br>
</div>
</blockquote>
<br>
As per the cmd_log_history file (a hidden file present in the log
directory which stores the CLI commands executed on that peer),
rebalance seems to be running (or was run).<br>
<br>
[2013-12-17 03:08:59.081232] : volume rebalance data start :
SUCCESS<br>
[2013-12-17 03:09:14.631826] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:09:22.761097] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:09:27.748014] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:09:28.839242] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:10:39.982747] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:14:30.919676] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:14:33.772300] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:29:14.467954] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:29:43.303852] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 03:30:04.309054] : volume rebalance data status :
SUCCESS<br>
[2013-12-17 04:35:45.631119] : volume rebalance data status :
SUCCESS<br>
<br>
<br>
I think this is what has happened. As part of rebalance layout
might have changed for some directories and distribute tries to
repair it by doing a self-heal when a lookup is performed on the
directory. Distribute performs self-heal as root. But when the
requests from that client comes to brick process, the requests from
root are changed by default to nfsnobody (uid: 65534) and that uid
does not have permissions to do some modifications (in this case
self-heal) on the directory which brick thinks is owned by root. So
self-heal does not happen properly and because of that some
operations performed (in this case rename of a file within that
directory)<br>
fails.<br>
<br>
Dave,<br>
Please let me know if I have missed anything. This is my observation
based on the log files.<br>
<br>
CCing Raghavendra G who might be able to clarify whether this is
what happened.<br>
<br>
Regards,<br>
Raghavendra Bhat<br>
<br>
<blockquote
cite="mid:CAOFq7B6Ur_ZeRuGh6JnS4ZHwu1FvcQXO9L2=606ucfT1VAHEwQ@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On 16 July 2014 14:47, Raghavendra Bhat
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:rabhat@redhat.com" target="_blank">rabhat@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5">
<div>On Tuesday 15 July 2014 01:57 PM, David Raffelt
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Raghavendra,
<div>Thanks for looking into this. Attached are
the log files from the 3 peers. The glusterfs
server is running on "Beauty". All 3 peers
mount the native gluster client on /home. Each
peer has a direct connection to each other,
addressable via the /etc/hosts file. </div>
<div><br>
</div>
<div>Note that I do not see any new output in the
log when this error occurs. Also note that I
tried to replicate this issue on Ubuntu 14.04
with a single brick and could not replicate it. </div>
<div><br>
</div>
<div>Below is some more output that might help.</div>
<div>Thanks!</div>
<div>Dave</div>
<div><br>
</div>
<div><br>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new, monospace"><b>dave@beauty:~$
glusterfs --version</b></font></div>
<div><font face="courier new, monospace">glusterfs
3.5git built on Jun 30 2014 15:58:19</font></div>
<div><font face="courier new, monospace">Repository
revision: git://<a moz-do-not-send="true"
href="http://git.gluster.com/glusterfs.git"
target="_blank">git.gluster.com/glusterfs.git</a></font></div>
<div><font face="courier new, monospace">Copyright
(c) 2006-2013 Red Hat, Inc. <<a
moz-do-not-send="true"
href="http://www.redhat.com/"
target="_blank">http://www.redhat.com/</a>></font></div>
<div><font face="courier new, monospace">GlusterFS
comes with ABSOLUTELY NO WARRANTY.</font></div>
<div><font face="courier new, monospace">It is
licensed to you under your choice of the GNU
Lesser</font></div>
<div><font face="courier new, monospace">General
Public License, version 3 or any later
version (LGPLv3</font></div>
<div><font face="courier new, monospace">or
later), or the GNU General Public License,
version 2 (GPLv2),</font></div>
<div><font face="courier new, monospace">in all
cases as published by the Free Software
Foundation.</font></div>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new, monospace"><b>dave@beauty:~$
uname -r</b></font></div>
<div><font face="courier new, monospace">3.15.4-1-ARCH</font></div>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new, monospace"><b>dave@beauty:~$
sudo gluster volume info </b></font></div>
<div><font face="courier new, monospace">Volume
Name: data</font></div>
<div><font face="courier new, monospace">Type:
Distribute</font></div>
<div><font face="courier new, monospace">Volume
ID: 1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc</font></div>
<div><font face="courier new, monospace">Status:
Started</font></div>
<div><font face="courier new, monospace">Number
of Bricks: 3</font></div>
<div><font face="courier new, monospace">Transport-type:
tcp</font></div>
<div><font face="courier new, monospace">Bricks:</font></div>
<div><font face="courier new, monospace">Brick1:
beauty:/export/beauty</font></div>
<div><font face="courier new, monospace">Brick2:
beast:/export/beast</font></div>
<div><font face="courier new, monospace">Brick3:
benji:/export/benji</font></div>
<div><font face="courier new, monospace">Options
Reconfigured:</font></div>
<div><font face="courier new, monospace">performance.cache-size:
32MB</font></div>
<div><font face="courier new, monospace">performance.write-behind-window-size:
1MB</font></div>
<div><font face="courier new, monospace">auth.allow:
172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2</font></div>
<div><font face="courier new, monospace">nfs.disable:
off</font></div>
<div><font face="courier new, monospace">diagnostics.brick-log-level:
ERROR</font></div>
<div><font face="courier new, monospace">diagnostics.client-log-level:
ERROR</font></div>
<div><font face="courier new, monospace">server.root-squash:
enable</font></div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
Hi Dave,<br>
<br>
Was rebalance running when you did above operations?<br>
<br>
<br>
Regards,<br>
Raghavendra Bhat
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 15 July 2014 15:29,
Raghavendra Bhat <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rabhat@redhat.com"
target="_blank">rabhat@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div>
<div>On Monday 14 July 2014 09:10 PM,
Pranith Kumar Karampuri wrote:<br>
</div>
<blockquote type="cite"> CCed
Raghavendra Bhat who may know about
the issue<br>
<br>
Pranith<br>
<div>On 07/14/2014 08:01 PM, Joe
Julian wrote:<br>
</div>
<blockquote type="cite"><a
moz-do-not-send="true"
href="https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS"
target="_blank">https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS</a><br>
<br>
Please file a bug report.<br>
<br>
<div class="gmail_quote">On July 14,
2014 12:38:11 AM PDT, David
Raffelt <a moz-do-not-send="true"
href="mailto:d.raffelt@brain.org.au" target="_blank"><d.raffelt@brain.org.au></a>
wrote:
<blockquote class="gmail_quote"
style="margin:0pt 0pt 0pt
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi All,
<div>After a recent update to
gluster 3.5 we are having
some issues renaming files
when root squashing is
enabled and the folder group
permissions are not set to
write. </div>
<div><br>
</div>
<div>For example if I create a
folder with the following
permissions</div>
<div><font face="courier new,
monospace">$ mkdir test</font></div>
<div><font face="courier new,
monospace">$ chmod g-w
test</font></div>
<div><font face="courier new,
monospace">$ ls -l </font></div>
<div>
<div><font face="courier
new, monospace">drwxr-xr-x
2 dave dave 22 Jul 14
17:16 test</font></div>
</div>
<div><br>
</div>
<div>When I create a file <i>within</i>
this folder, and try to
rename it I get a file
permissions error. </div>
<div><br>
</div>
<div><font face="courier new,
monospace">$ cd test</font></div>
<div><font face="courier new,
monospace">$ touch asdf</font></div>
<div><font face="courier new,
monospace">$ mv asdf asdf2</font></div>
<div>
<div><font face="courier
new, monospace">mv:
cannot move ‘asdf’ to
‘asdf2’: Permission
denied</font></div>
</div>
<div><br>
</div>
<div>A strace on the mv
command reveals the rename
system call fails with:</div>
<div>
<div><font face="courier
new, monospace">rename("asdf",
"asdf2") = -1 EACCES
(Permission denied)</font></div>
</div>
<div><br>
</div>
<div>However I can copy the
file and delete the old one
fine. </div>
<div><br>
</div>
<div>If I either disable
gluster root squashing, or
change the test_dir folder
group permission to write
then I can rename the file
without any problems.</div>
<div><br>
</div>
<div>System details are:<br>
</div>
<div>Arch linux</div>
<div>System umask is set to
002<br>
</div>
<div>Distributed volume, 3
peers, 1 brick per peer. </div>
<div><br>
</div>
<div>Any help is much
appreciated!</div>
<div>Dave</div>
<div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</blockquote>
<br>
</div>
</div>
Hi Dave,<br>
<br>
Can you please provide the brick and client
log files? Which client you were using? fuse
or nfs?<br>
<br>
Regards,<br>
Raghavendra Bhat
<div><br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0pt 0pt 0pt
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div> </div>
</div>
</div>
<pre><hr>
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android device with K-9
Mail. Please excuse my brevity. <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div><b><font color="#ff6600">David Raffelt
(PhD)</font></b></div>
<div><font color="#ff6600">Postdoctoral Fellow</font></div>
<div><br>
</div>
<div>The Florey Institute of Neuroscience and
Mental Health</div>
<div>Melbourne Brain Centre - Austin Campus</div>
<div>245 Burgundy Street</div>
<div>Heidelberg Vic 3084
<div>Ph: <a moz-do-not-send="true"
value="+61390357024">+61 3 9035 7024</a></div>
</div>
<div><a moz-do-not-send="true"
value="+61390357024">www.florey.edu.au</a></div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div><b><font color="#ff6600">David Raffelt (PhD)</font></b></div>
<div><font color="#ff6600">Postdoctoral Fellow</font></div>
<div><br>
</div>
<div>The Florey Institute of Neuroscience and Mental Health</div>
<div>Melbourne Brain Centre - Austin Campus</div>
<div>245 Burgundy Street</div>
<div>Heidelberg Vic 3084
<div>Ph: <a moz-do-not-send="true" value="+61390357024">+61
3 9035 7024</a></div>
</div>
<div><a moz-do-not-send="true" value="+61390357024">www.florey.edu.au</a></div>
</div>
</div>
</blockquote>
<br>
</body>
</html>