<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On Thursday 17 July 2014 03:46 PM,
David Raffelt wrote:<br>
</div>
<blockquote
cite="mid:CAOFq7B6eNuGgimebsCW47f9UhpgA7AE0jcyno09Lm6K=pxZcwg@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Raghavendra,
<div>I'm don't quite understand the issue. Yes, a rebalance was
was performed in December last year when I added a brick.
However, we have only just come across this (reproducible)
problem upon upgrading to 3.5. </div>
<div><br>
</div>
<div>Is there anything I can do to try and correct the issue?
Perhaps turn off root squashing while running "<span
style="color:rgb(0,0,0);font-family:arial,sans-serif;line-height:16.899999618530273px">gluster
volume rebalance </span><i
style="color:rgb(0,0,0);font-family:arial,sans-serif;line-height:16.899999618530273px">VOLNAME</i><span
style="color:rgb(0,0,0);font-family:arial,sans-serif;line-height:16.899999618530273px"> fix-layout
start"?</span></div>
<div><span
style="color:rgb(0,0,0);font-family:arial,sans-serif;line-height:16.899999618530273px"><br>
</span></div>
<div><span
style="color:rgb(0,0,0);font-family:arial,sans-serif;line-height:16.899999618530273px">Cheers,</span></div>
<div><span
style="color:rgb(0,0,0);font-family:arial,sans-serif;line-height:16.899999618530273px">Dave</span></div>
<div><br>
</div>
</div>
</blockquote>
<br>
Hi Dave,<br>
<br>
For now, you can turn off root-squashing. I am still trying to root
cause the issue. Will update ASAP with my finding.<br>
<br>
Regards,<br>
Raghavendra Bhat<br>
<br>
<blockquote
cite="mid:CAOFq7B6eNuGgimebsCW47f9UhpgA7AE0jcyno09Lm6K=pxZcwg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 17 July 2014 17:31, Raghavendra Bhat
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:rabhat@redhat.com" target="_blank">rabhat@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="">
<div>On Wednesday 16 July 2014 10:18 AM, David Raffelt
wrote:<br>
</div>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Raghavendra,
<div>No</div>
<div>Thanks </div>
<div>Dave</div>
</div>
<div class="gmail_extra"><br>
<br>
</div>
</blockquote>
<br>
As per the cmd_log_history file (a hidden file present in
the log directory which stores the CLI commands executed
on that peer), rebalance seems to be running (or was run).<br>
<br>
[2013-12-17 03:08:59.081232] : volume rebalance data
start : SUCCESS<br>
[2013-12-17 03:09:14.631826] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:09:22.761097] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:09:27.748014] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:09:28.839242] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:10:39.982747] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:14:30.919676] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:14:33.772300] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:29:14.467954] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:29:43.303852] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 03:30:04.309054] : volume rebalance data
status : SUCCESS<br>
[2013-12-17 04:35:45.631119] : volume rebalance data
status : SUCCESS<br>
<br>
<br>
I think this is what has happened. As part of rebalance
layout might have changed for some directories and
distribute tries to repair it by doing a self-heal when a
lookup is performed on the directory. Distribute performs
self-heal as root. But when the requests from that client
comes to brick process, the requests from root are changed
by default to nfsnobody (uid: 65534) and that uid does not
have permissions to do some modifications (in this case
self-heal) on the directory which brick thinks is owned by
root. So self-heal does not happen properly and because of
that some operations performed (in this case rename of a
file within that directory)<br>
fails.<br>
<br>
Dave,<br>
Please let me know if I have missed anything. This is my
observation based on the log files.<br>
<br>
CCing Raghavendra G who might be able to clarify whether
this is what happened.<br>
<br>
Regards,<br>
Raghavendra Bhat
<div>
<div class="h5"><br>
<br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On 16 July 2014 14:47,
Raghavendra Bhat <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rabhat@redhat.com"
target="_blank">rabhat@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div>
<div>On Tuesday 15 July 2014 01:57 PM,
David Raffelt wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Raghavendra,
<div>Thanks for looking into this.
Attached are the log files from
the 3 peers. The glusterfs server
is running on "Beauty". All 3
peers mount the native gluster
client on /home. Each peer has a
direct connection to each other,
addressable via the /etc/hosts
file. </div>
<div><br>
</div>
<div>Note that I do not see any new
output in the log when this error
occurs. Also note that I tried to
replicate this issue on Ubuntu
14.04 with a single brick and
could not replicate it. </div>
<div><br>
</div>
<div>Below is some more output that
might help.</div>
<div>Thanks!</div>
<div>Dave</div>
<div><br>
</div>
<div><br>
</div>
<div><font face="courier new,
monospace"><br>
</font></div>
<div>
<div><font face="courier new,
monospace"><b>dave@beauty:~$
glusterfs --version</b></font></div>
<div><font face="courier new,
monospace">glusterfs 3.5git
built on Jun 30 2014 15:58:19</font></div>
<div><font face="courier new,
monospace">Repository
revision: git://<a
moz-do-not-send="true"
href="http://git.gluster.com/glusterfs.git"
target="_blank">git.gluster.com/glusterfs.git</a></font></div>
<div><font face="courier new,
monospace">Copyright (c)
2006-2013 Red Hat, Inc. <<a
moz-do-not-send="true"
href="http://www.redhat.com/"
target="_blank">http://www.redhat.com/</a>></font></div>
<div><font face="courier new,
monospace">GlusterFS comes
with ABSOLUTELY NO WARRANTY.</font></div>
<div><font face="courier new,
monospace">It is licensed to
you under your choice of the
GNU Lesser</font></div>
<div><font face="courier new,
monospace">General Public
License, version 3 or any
later version (LGPLv3</font></div>
<div><font face="courier new,
monospace">or later), or the
GNU General Public License,
version 2 (GPLv2),</font></div>
<div><font face="courier new,
monospace">in all cases as
published by the Free Software
Foundation.</font></div>
</div>
<div><font face="courier new,
monospace"><br>
</font></div>
<div><font face="courier new,
monospace"><br>
</font></div>
<div>
<div><font face="courier new,
monospace"><b>dave@beauty:~$
uname -r</b></font></div>
<div><font face="courier new,
monospace">3.15.4-1-ARCH</font></div>
</div>
<div><font face="courier new,
monospace"><br>
</font></div>
<div><font face="courier new,
monospace"><br>
</font></div>
<div>
<div><font face="courier new,
monospace"><b>dave@beauty:~$
sudo gluster volume info </b></font></div>
<div><font face="courier new,
monospace">Volume Name: data</font></div>
<div><font face="courier new,
monospace">Type: Distribute</font></div>
<div><font face="courier new,
monospace">Volume ID:
1d5948c7-9b7a-40ca-8aa7-85c74bcef3bc</font></div>
<div><font face="courier new,
monospace">Status: Started</font></div>
<div><font face="courier new,
monospace">Number of Bricks: 3</font></div>
<div><font face="courier new,
monospace">Transport-type: tcp</font></div>
<div><font face="courier new,
monospace">Bricks:</font></div>
<div><font face="courier new,
monospace">Brick1:
beauty:/export/beauty</font></div>
<div><font face="courier new,
monospace">Brick2:
beast:/export/beast</font></div>
<div><font face="courier new,
monospace">Brick3:
benji:/export/benji</font></div>
<div><font face="courier new,
monospace">Options
Reconfigured:</font></div>
<div><font face="courier new,
monospace">performance.cache-size:
32MB</font></div>
<div><font face="courier new,
monospace">performance.write-behind-window-size:
1MB</font></div>
<div><font face="courier new,
monospace">auth.allow:
172.30.25.173,172.30.25.158,172.30.25.234,172.30.26.76,172.30.26.77,192.168.0.1,192.168.1.1,192.168.1.2,192.168.2.2,192.168.3.2,192.168.4.1,192.168.4.2,192.168.5.1,192.168.5.2</font></div>
<div><font face="courier new,
monospace">nfs.disable: off</font></div>
<div><font face="courier new,
monospace">diagnostics.brick-log-level:
ERROR</font></div>
<div><font face="courier new,
monospace">diagnostics.client-log-level:
ERROR</font></div>
<div><font face="courier new,
monospace">server.root-squash:
enable</font></div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
Hi Dave,<br>
<br>
Was rebalance running when you did above
operations?<br>
<br>
<br>
Regards,<br>
Raghavendra Bhat
<div>
<div><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 15 July
2014 15:29, Raghavendra Bhat <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:rabhat@redhat.com"
target="_blank">rabhat@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF">
<div>
<div>
<div>On Monday 14 July
2014 09:10 PM, Pranith
Kumar Karampuri wrote:<br>
</div>
<blockquote type="cite">
CCed Raghavendra Bhat
who may know about the
issue<br>
<br>
Pranith<br>
<div>On 07/14/2014 08:01
PM, Joe Julian wrote:<br>
</div>
<blockquote type="cite"><a
moz-do-not-send="true"
href="https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS"
target="_blank">https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS</a><br>
<br>
Please file a bug
report.<br>
<br>
<div
class="gmail_quote">On
July 14, 2014
12:38:11 AM PDT,
David Raffelt <a
moz-do-not-send="true"
href="mailto:d.raffelt@brain.org.au" target="_blank"><d.raffelt@brain.org.au></a>
wrote:
<blockquote
class="gmail_quote"
style="margin:0pt
0pt 0pt
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi
All,
<div>After a
recent update
to gluster 3.5
we are having
some issues
renaming files
when root
squashing is
enabled and
the folder
group
permissions
are not set to
write. </div>
<div><br>
</div>
<div>For example
if I create a
folder with
the following
permissions</div>
<div><font
face="courier
new,
monospace">$
mkdir test</font></div>
<div><font
face="courier
new,
monospace">$
chmod g-w test</font></div>
<div><font
face="courier
new,
monospace">$
ls -l </font></div>
<div>
<div><font
face="courier
new,
monospace">drwxr-xr-x
2 dave dave
22 Jul 14
17:16 test</font></div>
</div>
<div><br>
</div>
<div>When I
create a file
<i>within</i>
this folder,
and try to
rename it I
get a file
permissions
error. </div>
<div><br>
</div>
<div><font
face="courier
new,
monospace">$
cd test</font></div>
<div><font
face="courier
new,
monospace">$
touch asdf</font></div>
<div><font
face="courier
new,
monospace">$
mv asdf asdf2</font></div>
<div>
<div><font
face="courier
new,
monospace">mv:
cannot move
‘asdf’ to
‘asdf2’:
Permission
denied</font></div>
</div>
<div><br>
</div>
<div>A strace on
the mv command
reveals the
rename system
call fails
with:</div>
<div>
<div><font
face="courier
new,
monospace">rename("asdf",
"asdf2") = -1
EACCES
(Permission
denied)</font></div>
</div>
<div><br>
</div>
<div>However I
can copy the
file and
delete the old
one fine. </div>
<div><br>
</div>
<div>If I either
disable
gluster root
squashing, or
change the
test_dir
folder group
permission to
write then I
can rename the
file without
any problems.</div>
<div><br>
</div>
<div>System
details are:<br>
</div>
<div>Arch linux</div>
<div>System
umask is set
to 002<br>
</div>
<div>Distributed
volume, 3
peers, 1 brick
per peer. </div>
<div><br>
</div>
<div>Any help is
much
appreciated!</div>
<div>Dave</div>
<div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</blockquote>
<br>
</div>
</div>
Hi Dave,<br>
<br>
Can you please provide the
brick and client log files?
Which client you were using?
fuse or nfs?<br>
<br>
Regards,<br>
Raghavendra Bhat
<div><br>
<br>
<blockquote type="cite">
<blockquote type="cite">
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0pt
0pt 0pt
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>
<div> </div>
</div>
</div>
<pre><hr>
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
<br>
-- <br>
Sent from my Android
device with K-9 Mail.
Please excuse my
brevity. <br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div><b><font color="#ff6600">David
Raffelt (PhD)</font></b></div>
<div><font color="#ff6600">Postdoctoral
Fellow</font></div>
<div><br>
</div>
<div>The Florey Institute of
Neuroscience and Mental Health</div>
<div>Melbourne Brain Centre -
Austin Campus</div>
<div>245 Burgundy Street</div>
<div>Heidelberg Vic 3084
<div>Ph: <a
moz-do-not-send="true"
value="+61390357024">+61 3
9035 7024</a></div>
</div>
<div><a moz-do-not-send="true"
value="+61390357024">www.florey.edu.au</a></div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div><b><font color="#ff6600">David Raffelt
(PhD)</font></b></div>
<div><font color="#ff6600">Postdoctoral Fellow</font></div>
<div><br>
</div>
<div>The Florey Institute of Neuroscience and
Mental Health</div>
<div>Melbourne Brain Centre - Austin Campus</div>
<div>245 Burgundy Street</div>
<div>Heidelberg Vic 3084
<div>Ph: <a moz-do-not-send="true"
value="+61390357024">+61 3 9035 7024</a></div>
</div>
<div><a moz-do-not-send="true"
value="+61390357024">www.florey.edu.au</a></div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div><b><font color="#ff6600">David Raffelt (PhD)</font></b></div>
<div><font color="#ff6600">Postdoctoral Fellow</font></div>
<div><br>
</div>
<div>The Florey Institute of Neuroscience and Mental Health</div>
<div>Melbourne Brain Centre - Austin Campus</div>
<div>245 Burgundy Street</div>
<div>Heidelberg Vic 3084
<div>Ph: <a moz-do-not-send="true" value="+61390357024">+61
3 9035 7024</a></div>
</div>
<div><a moz-do-not-send="true" value="+61390357024">www.florey.edu.au</a></div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>