Tomaoki, excellent debugging! Please add yourself to CC - <a href="https://bugzilla.redhat.com/show_bug.cgi?id=835336">https://bugzilla.redhat.com/show_bug.cgi?id=835336</a><div><br></div><div>Avati<br><br><div class="gmail_quote">
On Sun, Jun 24, 2012 at 10:55 PM, Tomoaki Sato <span dir="ltr"><<a href="mailto:tsato@valinux.co.jp" target="_blank">tsato@valinux.co.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Avati,<br>
<br>
Are these intended ?:<br>
- hashcount value of 'bar'(0) is not same as 'foo/..'(2) and,<br>
- hashcount value of 'foo'(1) is not same as 'foo/../foo'(3).<div class="im"><br>
<br>
# tshark -i 1 -R nfs<br>
Running as user "root" and group "root". This could be dangerous.<br>
Capturing on eth0<br></div>
2.386732 192.168.1.23 -> 192.168.1.132 NFS V3 GETATTR Call, FH:0x43976ad5<br>
2.387772 192.168.1.132 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 7) Directory mode:0755 uid:0 gid:0<br>
3.666252 192.168.1.23 -> 192.168.1.132 NFS V3 GETATTR Call, FH:0x43976ad5<br>
3.667112 192.168.1.132 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 17) Directory mode:0755 uid:0 gid:0<br>
3.667260 192.168.1.23 -> 192.168.1.132 NFS V3 LOOKUP Call, DH:0x43976ad5/foo /* bar/foo */<br>
3.668321 192.168.1.132 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 19), FH:0x3f9fd887<br>
11.386638 192.168.1.23 -> 192.168.1.132 NFS V3 GETATTR Call, FH:0x43976ad5<br>
11.387664 192.168.1.132 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 52) Directory mode:0755 uid:0 gid:0<br>
20.386438 192.168.1.23 -> 192.168.1.132 NFS V3 GETATTR Call, FH:0x43976ad5<br>
20.387436 192.168.1.132 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 95) Directory mode:0755 uid:0 gid:0<br>
29.382531 192.168.1.23 -> 192.168.1.132 NFS V3 GETATTR Call, FH:0x43976ad5<br>
29.383796 192.168.1.132 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 126) Directory mode:0755 uid:0 gid:0<br>
33.666658 192.168.1.23 -> 192.168.1.132 NFS V3 LOOKUP Call, DH:0x3f9fd887/.. /* foo/.. */<br>
33.668097 192.168.1.132 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 144), FH:0x42966b36<br>
33.668310 192.168.1.23 -> 192.168.1.132 NFS V3 READDIRPLUS Call, FH:0x42966b36<br>
33.669996 192.168.1.132 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 146) .. foo .<br>
33.670188 192.168.1.23 -> 192.168.1.132 NFS V3 LOOKUP Call, DH:0x42966b36/.. /* bar/.. */<br>
33.671279 192.168.1.132 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 148), FH:0xbc1b2900<br>
33.671425 192.168.1.23 -> 192.168.1.132 NFS V3 LOOKUP Call, DH:0x42966b36/foo /* bar/foo */<br>
33.672421 192.168.1.132 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 150), FH:0x3e9ed964<br>
20 packets captured<br>
<br>
# egrep "nfs3_log_fh_entry_call|nfs3_<u></u>log_newfh_res" /var/log/glusterfs/nfs.log | tail -8<br>
[2012-06-25 14:28:40.090333] D [nfs3-helpers.c:1645:nfs3_log_<u></u>fh_entry_call] 0-nfs-nfsv3: XID: 3d78d872, LOOKUP: args: FH: hashcount 0, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 00000000-0000-0000-0000-<u></u>000000000001, name: foo /* bar/foo */<br>
[2012-06-25 14:28:40.091108] D [nfs3-helpers.c:3462:nfs3_log_<u></u>newfh_res] 0-nfs-nfsv3: XID: 3d78d872, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 1, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 7c4b5a51-0108-4ac9-8fd2-<u></u>4b843dcb2715<br>
[2012-06-25 14:29:10.089791] D [nfs3-helpers.c:1645:nfs3_log_<u></u>fh_entry_call] 0-nfs-nfsv3: XID: 3d78d879, LOOKUP: args: FH: hashcount 1, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 7c4b5a51-0108-4ac9-8fd2-<u></u>4b843dcb2715, name: .. /* foo/.. */<br>
[2012-06-25 14:29:10.090872] D [nfs3-helpers.c:3462:nfs3_log_<u></u>newfh_res] 0-nfs-nfsv3: XID: 3d78d879, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 2, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 00000000-0000-0000-0000-<u></u>000000000001<br>
[2012-06-25 14:29:10.093266] D [nfs3-helpers.c:1645:nfs3_log_<u></u>fh_entry_call] 0-nfs-nfsv3: XID: 3d78d87b, LOOKUP: args: FH: hashcount 2, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 00000000-0000-0000-0000-<u></u>000000000001, name: .. /* bar/.. */<br>
[2012-06-25 14:29:10.094056] D [nfs3-helpers.c:3462:nfs3_log_<u></u>newfh_res] 0-nfs-nfsv3: XID: 3d78d87b, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 3, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 6edd430d-bc57-470e-8e98-<u></u>eacfe1a91040<br>
[2012-06-25 14:29:10.094498] D [nfs3-helpers.c:1645:nfs3_log_<u></u>fh_entry_call] 0-nfs-nfsv3: XID: 3d78d87c, LOOKUP: args: FH: hashcount 2, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 00000000-0000-0000-0000-<u></u>000000000001, name: foo /* bar/foo */<br>
[2012-06-25 14:29:10.095198] D [nfs3-helpers.c:3462:nfs3_log_<u></u>newfh_res] 0-nfs-nfsv3: XID: 3d78d87c, LOOKUP: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 3, exportid b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b, gfid 7c4b5a51-0108-4ac9-8fd2-<u></u>4b843dcb2715<br>
<br>
Regards,<br>
<br>
Tomo<br>
<br>
Anand Avati wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
Tomoaki, this is very useful. I will look deeper soon.<br>
Thanks!<br>
<br>
Avati<br>
<br></div><div><div class="h5">
On Thu, Jun 21, 2012 at 9:21 PM, Tomoaki Sato <<a href="mailto:tsato@valinux.co.jp" target="_blank">tsato@valinux.co.jp</a> <mailto:<a href="mailto:tsato@valinux.co.jp" target="_blank">tsato@valinux.co.jp</a>>> wrote:<br>
<br>
Avati,<br>
<br>
tshark says ...<br>
FH values that the linux kernel NFS server returns stays constant for every LOOKUP 'foo' but,<br>
FH values that the GlusterFS(NFS) returns are non-constant.<br>
<br>
operaions at the ESXi host:<br>
<br>
~ # ./getcwd /vmfs/volumes/94925201-<u></u>78f190e0/foo<br>
========= sleep 30 ================<br>
/vmfs/volumes/94925201-<u></u>78f190e0/foo<br>
~ #<br>
<br>
tshark's output at the linux kernel NFS server:<br>
<br>
# tshark -i 2 -R nfs<br>
Running as user "root" and group "root". This could be dangerous.<br>
Capturing on br0<br>
/* chdir */<br>
2.056680 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff<br>
2.056990 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 13) Directory mode:0755 uid:0 gid:0<br>
9.848666 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff<br>
9.848767 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 60) Directory mode:0755 uid:0 gid:0<br>
9.848966 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo<br>
9.849049 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 62), FH:0xdb05b90a <=====<br>
20.055508 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff<br>
20.055702 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 103) Directory mode:0755 uid:0 gid:0<br>
29.054939 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff<br>
29.055180 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 132) Directory mode:0755 uid:0 gid:0<br>
38.054338 192.168.1.23 -> 192.168.1.254 NFS V3 GETATTR Call, FH:0x1ffd38ff<br>
38.054583 192.168.1.254 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 151) Directory mode:0755 uid:0 gid:0<br>
/* getcwd */<br>
39.849107 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0xdb05b90a/..<br>
39.849449 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 170), FH:0x1ffd38ff<br>
39.849676 192.168.1.23 -> 192.168.1.254 NFS V3 READDIRPLUS Call, FH:0x1ffd38ff<br>
39.849833 192.168.1.254 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 172) . .. foo<br>
39.850071 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo<br>
39.850149 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 174), FH:0xdb05b90a<br>
39.850746 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0xdb05b90a/..<br>
39.850814 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 176), FH:0x1ffd38ff<br>
39.851014 192.168.1.23 -> 192.168.1.254 NFS V3 READDIRPLUS Call, FH:0x1ffd38ff<br>
39.851095 192.168.1.254 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 178) . .. foo<br>
39.851329 192.168.1.23 -> 192.168.1.254 NFS V3 LOOKUP Call, DH:0x1ffd38ff/foo<br>
39.851438 192.168.1.254 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 180), FH:0xdb05b90a <=====<br>
<br>
operations at the ESXi host:<br>
<br>
~ # ./getcwd /vmfs/volumes/ef172a87-<u></u>e5ae817f/foo<br>
========= sleep 30 ================<br>
getcwd: No such file or directory<br>
~ #<br>
<br>
tshark's output at the GlusterFS(NFS) server:<br>
<br>
# tshark -i 1 -R nfs<br>
Running as user "root" and group "root". This could be dangerous.<br>
Capturing on eth0<br>
/* chdir */<br>
1.228396 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5<br>
1.229406 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 6) Directory mode:0755 uid:0 gid:0<br>
4.445894 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5<br>
4.446916 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 16) Directory mode:0755 uid:0 gid:0<br>
4.447099 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x43976ad5/foo<br>
4.448147 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 18), FH:0x3f9fd887 <=====<br>
10.228438 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5<br>
10.229432 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 31) Directory mode:0755 uid:0 gid:0<br>
19.228321 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5<br>
19.229309 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 47) Directory mode:0755 uid:0 gid:0<br>
28.228139 192.168.1.23 -> 192.168.1.136 NFS V3 GETATTR Call, FH:0x43976ad5<br>
28.229112 192.168.1.136 -> 192.168.1.23 NFS V3 GETATTR Reply (Call In 70) Directory mode:0755 uid:0 gid:0<br>
/* getcwd */<br>
34.448796 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x3f9fd887/..<br>
34.450119 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 81), FH:0x42966b36<br>
34.450343 192.168.1.23 -> 192.168.1.136 NFS V3 READDIRPLUS Call, FH:0x42966b36<br>
34.452105 192.168.1.136 -> 192.168.1.23 NFS V3 READDIRPLUS Reply (Call In 83) .. foo .<br>
34.452311 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x42966b36/..<br>
34.453464 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 85), FH:0xbc1b2900<br>
34.453648 192.168.1.23 -> 192.168.1.136 NFS V3 LOOKUP Call, DH:0x42966b36/foo<br>
34.454677 192.168.1.136 -> 192.168.1.23 NFS V3 LOOKUP Reply (Call In 87), FH:0x3e9ed964 <======<br>
<br>
Regards,<br>
<br>
Tomo<br>
<br>
(2012年06月20日 16:28), Tomoaki Sato wrote:<br>
> Avati,<br>
><br>
> I've tried following:<br>
> 1) 'esxcfg-nas -d gluster_nfs' at the ESXi host.<br>
> 2) 'volume set bar nfs.enable-ino32 on' at the 192.168.1.136 host.<br>
> 3) 'volume stop bar' and 'volume start bar' at the 192.168.1.136 host.<br>
> 4) 'esxcfg-nas -a -o 192.168.1.136 -s /bar gluster_nfs' at the ESXi host.<br>
><br>
> on the ESXi host:<br>
><br>
> ~ # uname -m<br>
> x86_64<br>
> ~ # mkdir /vmfs/volumes/ef172a87-<u></u>e5ae817f/after-enable-ino32-on<br>
> ~ # ls -liR /vmfs/volumes/ef172a87-<u></u>e5ae817f<br>
> /vmfs/volumes/ef172a87-<u></u>e5ae817f:<br>
> -2118204814 drwxr-xr-x 1 root root 4096 Jun 20 07:13 after-enable-ino32-on<br>
> 1205893126 drwxr-xr-x 1 root root 4096 Jun 20 07:08 baz<br>
> -1291907235 drwx------ 1 root root 16384 Jun 6 23:41 lost+found<br>
><br>
> /vmfs/volumes/ef172a87-<u></u>e5ae817f/after-enable-ino32-<u></u>on:<br>
><br>
> /vmfs/volumes/ef172a87-<u></u>e5ae817f/baz:<br>
> -1374929331 drwxr-xr-x 1 root root 4096 Jun 19 06:41 foo<br>
><br>
> /vmfs/volumes/ef172a87-<u></u>e5ae817f/baz/foo:<br>
><br>
> /vmfs/volumes/ef172a87-<u></u>e5ae817f/lost+found:<br>
> ~ # ./getcwd /vmfs/volumes/ef172a87-<u></u>e5ae817f/after-enable-ino32-on<br>
> getcwd: No such file or directory<br>
> ~ #<br>
><br>
> on the 192.168.1.136 host:<br>
><br>
> # gluster volume info bar<br>
><br>
> Volume Name: bar<br>
> Type: Distribute<br>
> Volume ID: b2d75589-8370-4528-ab4e-<u></u>b543b3abdc3b<br>
> Status: Started<br>
> Number of Bricks: 1<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: bar-1-private:/mnt/brick<br>
> Options Reconfigured:<br>
> diagnostics.brick-log-level: TRACE<br>
> diagnostics.client-log-level: TRACE<br>
> nfs.enable-ino32: on<br>
><br>
> please fine attached nfs.log5.<br>
><br>
> Regards,<br>
><br>
> Tomo<br>
><br>
> (2012/06/20 16:11), Anand Avati wrote:<br>
>> -1374929331 drwxr-xr-x 1 root root 4096 Jun 19 06:41 foo<br>
>><br>
>> ...<br>
>><br>
>> 2920037965 drwxr-xr-x 2 root root 4096 Jun 19 15:41 foo<br>
>><br>
>><br>
>> Ouch!<br>
>><br>
>> -1374929331 == (int32_t) 2920037965<br>
>><br>
>> 'uname -m' from the ESXi host please! Is it a 32bit OS? Can you try 'gluster volume set bar nfs.enable-ino32 on' and retry?<br>
>><br>
>> Avati<br>
><br>
<br>
<br>
</div></div></blockquote>
<br>
<br>
</blockquote></div><br></div>