<html><head></head><body>What's the backend filesystem?<br>
Were there any brick errors, probably around 2014-03-31 22:44:04 (half an hour before the frame timeout)? <br>
<br><br><div class="gmail_quote">On April 9, 2014 7:10:58 AM PDT, James Cuff <james_cuff@harvard.edu> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="k9mail">Hi team,<br /><br />I hate "me too" emails sometimes not at all constructive, but I feel I<br />really ought chip in from real world systems we use in anger and at<br />massive scale here.<br /><br />So we also use NFS to "mask" this and other performance issues. The<br />cluster.readdir-optimize gave us similar results unfortunately.<br /><br />We reported our other challenge back last summer but we stalled on this:<br /><br /><a href="http://www.gluster.org/pipermail/gluster-users/2013-June/036252.html">http://www.gluster.org/pipermail/gluster-users/2013-June/036252.html</a><br /><br />We also unfortunately now see a new NFS phenotype that I've pasted<br />below which is again is causing real heartburn.<br /><br />Small files, always difficult for any FS, might be worth doing some<br />regression testing with small file directory scenarios in test - it's<br />an easy reproducer on even moderately sized gluster clusters. Hope<br />some good progress can be
made, and I understand it's a tough one to<br />track down performance hangs and issues. I just wanted to say that we<br />really do see them, and have tried many things to avoid them.<br /><br />Here's the note from my team:<br /><br />We were hitting 30 minute timeouts on getxattr/system.posix_acl_access<br />calls on directories in a NFS v3 mount (w/ acl option) of a 10-node<br />40-brick gluster 3.4.0 volume. Strace shows where the client hangs:<br /><br />$ strace -tt -T getfacl d6h_take1<br />...<br />18:43:57.929225 lstat("d6h_take1", {st_mode=S_IFDIR|0755,<br />st_size=7024, ...}) = 0 <0.257107><br />18:43:58.186461 getxattr("d6h_take1", "system.posix_acl_access",<br />0x7fffdf2b9f50, 132) = -1 ENODATA (No data available) <1806.296893><br />19:14:04.483556 stat("d6h_take1", {st_mode=S_IFDIR|0755, st_size=7024,<br />...}) = 0 <0.642362><br />19:14:05.126025 getxattr("d6h_take1", "system.posix_acl_default",<br />0x7fffdf2b9f50, 132) = -1 ENODATA (No data
available) <0.000024><br />19:14:05.126114 stat("d6h_take1", {st_mode=S_IFDIR|0755, st_size=7024,<br />...}) = 0 <0.000010><br />...<br /><br />Load on the servers was moderate. While the above was hanging,<br />getfacl worked nearly instantaneously on that directory on all bricks.<br /> When it finally hit the 30 minute timeout, gluster logged it in<br />nfs.log:<br /><br />[2014-03-31 23:14:04.481154] E [rpc-clnt.c:207:call_bail]<br />0-holyscratch-client-36: bailing out frame type(GlusterFS 3.3)<br />op(GETXATTR(18)) xid = 0x8168809x sent = 2014-03-31 22:43:58.442411.<br />timeout = 1800<br />[2014-03-31 23:14:04.481233] W<br />[client-rpc-fops.c:1112:client3_3_getxattr_cbk]<br />0-holyscratch-client-36: remote operation failed: Transport endpoint<br />is not connected. Path: <gfid:b116fb01-b13d-448a-90d0-a8693a98698b><br />(b116fb01-b13d-448a-90d0-a8693a98698b). Key: (null)<br /><br />Other than that, we didn't see anything directly related in the nfs or<br
/>brick logs or anything out of sorts with the gluster services. A<br />couple other errors raise eyebrows, but these are different<br />directories (neighbors of the example above) and at different times:<br /><br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:30:47.794454]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:31:47.794447]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:33:47.802135]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:34:47.802182]<br />I
[dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:36:47.764329]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:37:47.773164]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:39:47.774285]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:40:47.780338]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht:
found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br />holyscratch07: /var/log/glusterfs/nfs.log:[2014-03-31 19:42:47.730345]<br />I [dht-layout.c:630:dht_layout_normalize] 0-holyscratch-dht: found<br />anomalies in /ramanathan_lab/dhuh/d9_take2_BGI/Diffreg. holes=1<br />overlaps=0<br /><br />holyscratch08: /var/log/glusterfs/bricks/holyscratch08_03-brick.log:[2014-03-31<br />00:57:51.973565] E [posix-helpers.c:696:posix_handle_pair]<br />0-holyscratch-posix:<br />/holyscratch08_03/brick/ramanathan_lab/dhuh/d9_take2_BGI/cuffdiffRN.txt:<br />key:system.posix_acl_access error:Invalid argument<br />holyscratch08: /var/log/glusterfs/bricks/holyscratch08_03-brick.log:[2014-03-31<br />01:18:12.345818] E [posix-helpers.c:696:posix_handle_pair]<br />0-holyscratch-posix:<br />/holyscratch08_03/brick/ramanathan_lab/dhuh/d9_take2_BGI/cuffdiffRN.txt:<br />key:system.posix_acl_access error:Invalid argument<br />holyscratch05:
/var/log/glusterfs/bricks/holyscratch05_04-brick.log:[2014-03-31<br />21:16:37.057674] E [posix-helpers.c:696:posix_handle_pair]<br />0-holyscratch-posix:<br />/holyscratch05_04/brick/ramanathan_lab/dhuh/d9_take2_BGI/Diffreg/cuffdiffRN.txt:<br />key:system.posix_acl_access error:Invalid argument<br /><br />--<br />dr. james cuff, assistant dean for research computing, harvard<br />university | division of science | thirty eight oxford street,<br />cambridge. ma. 02138 | +1 617 384 7647 | <a href="http://rc.fas.harvard.edu">http://rc.fas.harvard.edu</a><br /><br /><br />On Wed, Apr 9, 2014 at 9:52 AM, <james.bellinger@icecube.wisc.edu> wrote:<br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"> I am seeing something perhaps similar. 3.4.2-1, 2 servers, each with 1<br /> brick, replicated. A du of a local (ZFS) directory tree of 297834 files<br /> and 525GB takes about 17 minutes. A du of the gluster copy
is still not<br /> finished after 22 hours. Network activity has been about 5-6KB/sec until<br /> (I gather) du hit a directory with 22450 files, when activity jumped to<br /> 300KB/sec (200 packets/sec) for about 15-20 minutes. If I assume that the<br /> spike came from scanning the two largest directories, that looks like<br /> about 8K of traffic per file, and about 5 packets.<br /><br /> A 3.3.2 gluster installation that we are trying to retire is not afflicted<br /> this way.<br /><br /> James Bellinger<br /><br /><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;"><br /> Am I the only person using Gluster suffering from very slow directory<br /> access? It's so seriously bad that it almost makes Gluster unusable.<br /><br /> Using NFS instead of the Fuse client masks the problem as long as the<br /> directories are cached but it's still hellishly slow when you first<br /> access them.<br /><br /> Has there
been any progress at all fixing this bug?<br /><br /> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1067256">https://bugzilla.redhat.com/show_bug.cgi?id=1067256</a><br /><br /> Cheers,<br /><br /><hr /><br /> Gluster-users mailing list<br /> Gluster-users@gluster.org<br /> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote><br /><br /><br /><hr /><br /> Gluster-users mailing list<br /> Gluster-users@gluster.org<br /> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br /></blockquote><hr /><br />Gluster-users mailing list<br />Gluster-users@gluster.org<br /><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br /></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>