<div dir="ltr">Wannes:<div><br></div><div>There is a current issue with EXT4, the Linux Kernel and Gluster. It is a long story, but, in the end there is a bug there that won't be addressed any time soon.</div><div><br>
</div><div>If at all possible, like gluster's/ RH's documentation recommends, use XFS and make sure you prepare the partition with -i size=512.</div><div><br></div><div>At least then you are fully compliant.</div>
<div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 1, 2014 at 4:21 PM, VAN CAUSBROECK Wannes <span dir="ltr"><<a href="mailto:Wannes.VANCAUSBROECK@onprvp.fgov.be" target="_blank">Wannes.VANCAUSBROECK@onprvp.fgov.be</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="FR-BE" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="color:#1f497d">Hello Joe,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d">Could it be that gluster has issues with the 64 bit ext4 filesystem? As a test I’m moving one of the volumes to an xfs volume… I hope I’ll be able to answer that question myself soon.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Joe Julian
[mailto:<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>] <br>
<b>Sent:</b> dinsdag 1 april 2014 16:14<br>
<b>To:</b> VAN CAUSBROECK Wannes; '<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>'</span></p><div class=""><br>
<b>Subject:</b> Re: [Gluster-users] nfs acces denied<u></u><u></u></div><p></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">[2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in
/2011/201107/20110716/55. holes=1 overlaps=0</span></p><div><div class="h5"><br>
<br>
This tells us that there is a chunk of hash map missing. My guess would be that the server you're nfs mounting from cannot connect to one of the bricks.
<u></u><u></u></div></div><p></p><div><div class="h5">
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">On March 31, 2014 6:58:53 AM PDT, VAN CAUSBROECK Wannes <<a href="mailto:Wannes.VANCAUSBROECK@onprvp.fgov.be" target="_blank">Wannes.VANCAUSBROECK@onprvp.fgov.be</a>> wrote:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Hello all,<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I’ve already tried to post this, but i’m unsure it arrived to the mailing list.<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I have some issues regarding my nfs mounts. My setup is as follows:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Rhel 6.4, gluster 3.4.2-1 running on a vm (4 cores, 8GB ram) attached to a san. I have one disk on which are all the bricks (formatted ext4 in 64 bit mode) of 25TB.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">On the gluster side of things, everything works without issues. The trouble starts when I mount a volume as an nfs mount.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Lots of volumes work without issues, but others behave strangely. The volumes that act weird generally contain many files (can be accidental?).<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">The volumes in question mount without issues, but when I try to go into any subdirectory sometimes it works, sometimes I get errors.<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">On windows with nfs client: access denied<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">In nfslog:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.771241] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>. holes=1 overlaps=0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.771348] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:c8d94120-6851-46ea-9f28-c629a44b1015>: Invalid argument<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.771380] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (<a href="http://192.168.148.46:984" target="_blank">192.168.148.46:984</a>) caviar_data11 : c8d94120-6851-46ea-9f28-c629a44b1015<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.771819] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28530, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.798967] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>. holes=1 overlaps=0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.799039] E [nfs3-helpers.c:3595:nfs3_fh_resolve_inode_lookup_cbk] 0-nfs-nfsv3: Lookup failed: <gfid:14972193-1039-4d7a-aed5-0d7e7eccf57b>: Invalid argument<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.799056] E [nfs3.c:1380:nfs3_lookup_resume] 0-nfs-nfsv3: Unable to resolve FH: (<a href="http://192.168.148.46:984" target="_blank">192.168.148.46:984</a>) caviar_data11 : 14972193-1039-4d7a-aed5-0d7e7eccf57b<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 13:57:58.799088] W [nfs3-helpers.c:3380:nfs3_log_common_res] 0-nfs-nfsv3: XID: 1ec28531, LOOKUP: NFS: 22(Invalid argument for operation), POSIX: 14(Bad address)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">….<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">On linux:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">ls: /media/2011/201105/20110530/37: No such file or directory<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">total 332<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">…<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 32<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 34<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 35<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 36<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">drwxrwsr-x 2 nfsnobody 1003 4096 Jun 6 2011 37<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">…<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[root@lpr-nas01 brick-xiv2]# ll /media/2011/201105/20110530/37<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">ls: /media/2011/201105/20110530/37/NN.0000073824357.00001.tif: No such file or directory<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">ls: /media/2011/201105/20110530/37/NN.0000073824357.00003.tif: No such file or directory<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">total 54<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">-rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00001.tif<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">-rwxrwxr-x 1 nfsnobody 1003 35312 Jun 6 2011 NN.0000073824357.00002.tif<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">-rwxrwxr-x 0 nfsnobody 1003 9340 Jun 6 2011 NN.0000073824357.00003.tif<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">I see in the nfslog:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">…<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.941083] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/55. holes=1 overlaps=0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.958078] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/30. holes=1 overlaps=0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.959980] I [dht-layout.c:638:dht_layout_normalize] 0-caviar_data11-dht: found anomalies in /2011/201107/20110716/90. holes=1 overlaps=0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.961094] E [dht-helper.c:429:dht_subvol_get_hashed] (-->/usr/lib64/glusterfs/3.4.2/xlator/debug/io-stats.so(io_stats_lookup+0x157) [0x7fd6a61282e7] (-->/usr/lib64/libglusterfs.so.0(default_lookup+0x6d)
[0x3dfe01c03d] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup+0xa7e) [0x7fd6a656af2e]))) 0-caviar_data11-dht: invalid argument: loc->parent<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.961283] W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-caviar_data11-client-0: remote operation failed: Invalid argument. Path: <gfid:00000000-0000-0000-0000-000000000000> (00000000-0000-0000-0000-000000000000)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.961319] E [acl3.c:334:acl3_getacl_resume] 0-nfs-ACL: Unable to resolve FH: (<a href="http://192.168.151.21:740" target="_blank">192.168.151.21:740</a>) caviar_data11 : 00000000-0000-0000-0000-000000000000<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">[2014-03-31 12:44:18.961338] E [acl3.c:342:acl3_getacl_resume] 0-nfs-ACL: unable to open_and_resume<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">…<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">The weirdest thing is it changes from time to time which files and directories work and which don’t
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Any ideas?<u></u><u></u></span></p>
<p><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks!<u></u><u></u></span></p>
<pre style="text-align:center"><hr size="2" width="100%" align="center"></pre>
<pre><br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></pre>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.<u></u><u></u></span></p>
</div></div></div>
</div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>