<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I've run across another problem - this one I'm pretty sure is a
problem with Gluster. I've opened
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=955753">https://bugzilla.redhat.com/show_bug.cgi?id=955753</a>.<br>
<br>
I'm using Oracle DNFS still and it's erroring out on some of its
logfiles:<br>
<tt>ARC3: Error 19508 Closing archive log file
'/db/flash_recovery_area/ALTUS/archivelog/2013_04_22/o1_mf_1_1093__1366653401581181_.arc'</tt><br>
<br>
Gluster is reporting:<br>
<tt>[2013-04-22 13:57:22.073354] W
[client3_1-fops.c:707:client3_1_truncate_cbk] 0-gv0-client-9:
remote operation failed: Pe</tt><tt>rmission denied</tt><tt><br>
</tt><tt>[2013-04-22 13:57:22.073496] W
[client3_1-fops.c:707:client3_1_truncate_cbk] 0-gv0-client-8:
remote operation failed: Pe</tt><tt>rmission denied</tt><tt><br>
</tt><tt>[2013-04-22 13:57:22.073805] W
[nfs3.c:889:nfs3svc_truncate_cbk] 0-nfs: 8b534455:
/fleming1/db0/ALTUS_flash/archivelog/2013_04_22/.o1_mf_1_1093__1366653401581181_.arc
=> -1 (Permission denied)</tt><tt><br>
</tt><tt>[2013-04-22 13:57:22.082594] E
[nfs3.c:3408:nfs3_remove_resume] 0-nfs-nfsv3: Unable to resolve
FH: (192.168.10.3:46391) gv0 :
82c4c5ec-f3ad-4074-ac66-c5a455146d71</tt><tt><br>
</tt><br>
Immediately prior to this, that file has attributes:<br>
Regular File mode:0640 uid:500 gid:1000, size: 476959744<br>
<br>
The actual NFS RPC causing this error is [1]. Briefly:<br>
<tt>Remote Procedure Call, Type:Call XID:0x8b534455</tt><tt><br>
</tt><tt>Network File System, SETATTR Call FH:0x5c191ad8</tt><tt><br>
</tt><tt> new_attributes</tt><tt><br>
</tt><tt> mode: value follows</tt><tt><br>
</tt><tt><tt> set_it: value follows (1)</tt><tt><br>
</tt> Mode: 0440, S_IRUSR, S_IRGRP</tt><tt><br>
</tt><tt> size: value follows</tt><tt><br>
</tt><tt> set_it: value follows (1)</tt><tt><br>
</tt><tt> size: 476959744</tt><tt><br>
</tt><br>
In other words, a "truncate" and "chmod 440" in the same call.<br>
<br>
Gluster is replying with [2]:<br>
<tt>Remote Procedure Call, Type:Reply XID:0x8b534455</tt><tt><br>
</tt><tt>Network File System, SETATTR Reply Error:NFS3ERR_ACCES</tt><tt><br>
</tt><tt> Status: NFS3ERR_ACCES (13)</tt><tt><br>
</tt><br>
What's happening is that gluster is processing the mode change
before the truncate, causing the truncate to fail.<br>
<br>
Incidentally, this also causes gluster to think that these files
need healing:<br>
Gathering Heal info on volume gv0 has been successful<br>
…<br>
Brick fearless1:/export/bricks/500117310007a7ec/glusterdata<br>
/fleming1/db0/ALTUS_flash/archivelog/2013_04_22/.o1_mf_1_1093__1366653401581181_.arc<br>
…<br>
Brick fearless2:/export/bricks/500117310007a74c/glusterdata<br>
/fleming1/db0/ALTUS_flash/archivelog/2013_04_22/.o1_mf_1_1093__1366653401581181_.arc<br>
<br>
So, arguably gluster should be doing the truncate before the chmod.
Perhaps the Most Correct thing is to always chmod last if removing
permissions. That's a longer discussion :p<br>
<br>
[1] Full RPC Call<br>
<tt>Remote Procedure Call, Type:Call XID:0x8b534455</tt><tt><br>
</tt><tt> Fragment header: Last fragment, 172 bytes</tt><tt><br>
</tt><tt> 1... .... .... .... .... .... .... .... = Last
Fragment: Yes</tt><tt><br>
</tt><tt> .000 0000 0000 0000 0000 0000 1010 1100 = Fragment
Length: 172</tt><tt><br>
</tt><tt> XID: 0x8b534455 (2337490005)</tt><tt><br>
</tt><tt> Message Type: Call (0)</tt><tt><br>
</tt><tt> RPC Version: 2</tt><tt><br>
</tt><tt> Program: NFS (100003)</tt><tt><br>
</tt><tt> Program Version: 3</tt><tt><br>
</tt><tt> Procedure: SETATTR (2)</tt><tt><br>
</tt><tt> [The reply to this request is in frame 293325]</tt><tt><br>
</tt><tt> Credentials</tt><tt><br>
</tt><tt> Flavor: AUTH_UNIX (1)</tt><tt><br>
</tt><tt> Length: 52</tt><tt><br>
</tt><tt> Stamp: 0xabcdefab</tt><tt><br>
</tt><tt> Machine Name: fleming1.netdirect.ca</tt><tt><br>
</tt><tt> length: 21</tt><tt><br>
</tt><tt> contents: fleming1.netdirect.ca</tt><tt><br>
</tt><tt> fill bytes: opaque data</tt><tt><br>
</tt><tt> UID: 500</tt><tt><br>
</tt><tt> GID: 1000</tt><tt><br>
</tt><tt> Auxiliary GIDs</tt><tt><br>
</tt><tt> GID: 1000</tt><tt><br>
</tt><tt> GID: 1030</tt><tt><br>
</tt><tt> Verifier</tt><tt><br>
</tt><tt> Flavor: AUTH_NULL (0)</tt><tt><br>
</tt><tt> Length: 0</tt><tt><br>
</tt><tt>Network File System, SETATTR Call FH:0x5c191ad8</tt><tt><br>
</tt><tt> [Program Version: 3]</tt><tt><br>
</tt><tt> [V3 Procedure: SETATTR (2)]</tt><tt><br>
</tt><tt> object</tt><tt><br>
</tt><tt> length: 36</tt><tt><br>
</tt><tt> [hash (CRC-32): 0x5c191ad8]</tt><tt><br>
</tt><tt> [Name: .o1_mf_1_1093__1366653401581181_.arc]</tt><tt><br>
</tt><tt> [Full Name:
192.168.10.1:/gv0/fleming1/db0/ALTUS_flash/archivelog/2013_04_22/.o1_mf_1_1093__1366653401581181_.arc]</tt><tt><br>
</tt><tt> decode type as: unknown</tt><tt><br>
</tt><tt> filehandle:
3a4f474c20117b487f884f169490a0349afacf71e16a95fc...</tt><tt><br>
</tt><tt> new_attributes</tt><tt><br>
</tt><tt> mode: value follows</tt><tt><br>
</tt><tt> set_it: value follows (1)</tt><tt><br>
</tt><tt> Mode: 0440, S_IRUSR, S_IRGRP</tt><tt><br>
</tt><tt> .... .... .... .... .... 0... .... .... =
S_ISUID: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .0.. .... .... =
S_ISGID: No</tt><tt><br>
</tt><tt> .... .... .... .... .... ..0. .... .... =
S_ISVTX: No</tt><tt><br>
</tt><tt> .... .... .... .... .... ...1 .... .... =
S_IRUSR: Yes</tt><tt><br>
</tt><tt> .... .... .... .... .... .... 0... .... =
S_IWUSR: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .... .0.. .... =
S_IXUSR: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .... ..1. .... =
S_IRGRP: Yes</tt><tt><br>
</tt><tt> .... .... .... .... .... .... ...0 .... =
S_IWGRP: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .... .... 0... =
S_IXGRP: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .... .... .0.. =
S_IROTH: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .... .... ..0. =
S_IWOTH: No</tt><tt><br>
</tt><tt> .... .... .... .... .... .... .... ...0 =
S_IXOTH: No</tt><tt><br>
</tt><tt> uid: no value</tt><tt><br>
</tt><tt> set_it: no value (0)</tt><tt><br>
</tt><tt> gid: no value</tt><tt><br>
</tt><tt> set_it: no value (0)</tt><tt><br>
</tt><tt> size: value follows</tt><tt><br>
</tt><tt> set_it: value follows (1)</tt><tt><br>
</tt><tt> size: 476959744</tt><tt><br>
</tt><tt> atime: don't change</tt><tt><br>
</tt><tt> set_it: don't change (0)</tt><tt><br>
</tt><tt> mtime: don't change</tt><tt><br>
</tt><tt> set_it: don't change (0)</tt><tt><br>
</tt><tt> guard: no value</tt><tt><br>
</tt><tt> check: no value (0)</tt><tt><br>
</tt><br>
[2] Full Reply<br>
<tt>Ethernet II, Src: Ibm_36:f7:d0 (5c:f3:fc:36:f7:d0), Dst:
IntelCor_38:e7:58 (00:1e:67:38:e7:58)</tt><tt><br>
</tt><tt>Internet Protocol Version 4, Src: 192.168.10.1
(192.168.10.1), Dst: 192.168.10.3 (192.168.10.3)</tt><tt><br>
</tt><tt>Transmission Control Protocol, Src Port: 38467 (38467), Dst
Port: 46391 (46391), Seq: 1230671698, Ack: 2230824272, Len: 40</tt><tt><br>
</tt><tt>Remote Procedure Call, Type:Reply XID:0x8b534455</tt><tt><br>
</tt><tt> Fragment header: Last fragment, 36 bytes</tt><tt><br>
</tt><tt> 1... .... .... .... .... .... .... .... = Last
Fragment: Yes</tt><tt><br>
</tt><tt> .000 0000 0000 0000 0000 0000 0010 0100 = Fragment
Length: 36</tt><tt><br>
</tt><tt> XID: 0x8b534455 (2337490005)</tt><tt><br>
</tt><tt> Message Type: Reply (1)</tt><tt><br>
</tt><tt> [Program: NFS (100003)]</tt><tt><br>
</tt><tt> [Program Version: 3]</tt><tt><br>
</tt><tt> [Procedure: SETATTR (2)]</tt><tt><br>
</tt><tt> Reply State: accepted (0)</tt><tt><br>
</tt><tt> [This is a reply to a request in frame 293324]</tt><tt><br>
</tt><tt> [Time from request: 0.001547000 seconds]</tt><tt><br>
</tt><tt> Verifier</tt><tt><br>
</tt><tt> Flavor: AUTH_NULL (0)</tt><tt><br>
</tt><tt> Length: 0</tt><tt><br>
</tt><tt> Accept State: RPC executed successfully (0)</tt><tt><br>
</tt><tt>Network File System, SETATTR Reply Error:NFS3ERR_ACCES</tt><tt><br>
</tt><tt> [Program Version: 3]</tt><tt><br>
</tt><tt> [V3 Procedure: SETATTR (2)]</tt><tt><br>
</tt><tt> Status: NFS3ERR_ACCES (13)</tt><tt><br>
</tt><tt> obj_wcc</tt><tt><br>
</tt><tt> before</tt><tt><br>
</tt><tt> attributes_follow: no value (0)</tt><tt><br>
</tt><tt> after</tt><tt><br>
</tt><tt> attributes_follow: no value (0)</tt><br>
<br>
<pre class="moz-signature" cols="72">--
Michael Brown | `One of the main causes of the fall of
Systems Consultant | the Roman Empire was that, lacking zero,
Net Direct Inc. | they had no way to indicate successful
☎: +1 519 883 1172 x5106 | termination of their C programs.' - Firth
</pre>
</body>
</html>