<div dir="ltr"><div class="gmail_extra">Hi Ira, Vivek,</div><div class="gmail_extra"> </div><div class="gmail_extra">Thanks, I will open a BZ sometime today.</div><div class="gmail_extra"> </div><div class="gmail_extra">Ira, comments and answers to your questions<br>
interstitially below.<br><br>Note that everything is done on the same box, so the<br>networking is all virtual, through the 'lo' device</div><div class="gmail_extra"> </div><div class="gmail_quote">On Tue, Feb 25, 2014 at 10:19 PM, Ira Cooper <span dir="ltr"><<a href="mailto:ira@redhat.com" target="_blank">ira@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">Hello Jeff,<br>
<br>
First of all, thank you for your work here. I appreciate anyone disecting a performance issue. I do a few thoughts, and asks of you, if you don't mind.<br>
<div><br>
<br>
><br>
> I have a problem with very slow Windows Explorer browsing<br>
> when there are a large number of directories/files.<br>
> In this case, the top level folder has almost 6000 directories,<br>
> admittedly large, but it works almost instantaneously when a<br>
> Windows Server share was being used.<br>
> Migrating to a Samba/GlusterFS share, there is almost a 20<br>
> second delay while the explorer window populates the list.<br>
> This leaves a bad impression on the storage performance. The<br>
> systems are otherwise idle.<br>
> To isolate the cause, I've eliminated everything, from<br>
> networking, Windows, and have narrowed in on GlusterFS<br>
> being the sole cause of most of the directory lag.<br>
> I was optimistic on using the GlusterFS VFS libgfapi instead<br>
> of FUSE with Samba, and it does help performance<br>
> dramatically in some cases, but it does not help (and<br>
> sometimes hurts) when compared to the CIFS FUSE mount<br>
> for directory listings.<br>
<br>
</div>This should be investigated further.<br>
<div><br>
><br>
> NFS for directory listings, and small I/O's seems to be<br>
> better, but I cannot use NFS, as I need to use CIFS for<br>
> Windows clients, need ACL's, Active Directory, etc.<br>
<br>
</div>Understood.<br>
<div><br>
<br>
><br>
> Directory listing of 6000 empty directories ('stat-prefetch'<br>
> is on):<br>
><br>
> Directory listing the ext4 mount directly is almost<br>
> instantaneous of course.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/<br>
> >/dev/null<br>
> real 0m41.235s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/<br>
> >/dev/null<br>
> real 0m0.110s<br>
> # time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/<br>
> >/dev/null<br>
> real 0m0.109s<br>
<br>
</div>The cache population time matches what I'd expect, for hitting this data cold.<br>
<div><br>
> Directory listing the NFS mount is also very fast.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
> real 0m44.352s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
> real 0m0.471s<br>
> # time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
> real 0m0.114s<br>
<br>
</div>Note that the last measurement is within a small amount of the ext4 times. That looks "right" to me.<br>
<br>
Now, I'd be interested in what happens if you wait ~30m and try it again with the caches warm. (ls -l the local directory on the brick, to warm the cache.)<br></blockquote><div>Not sure if I did what you were asking here:</div>
<div>I repeated the test with a 30 minute wait, and did not see<br>the initial long cache populate time reoccur, even after<br>waiting more than an hour. I assume you wanted to see this<br>for the NFS mount to the GlusterFS volume, correct?</div>
<div> </div><div># sync;sync; echo '3' > /proc/sys/vm/drop_caches<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>real 0m43.903s<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
real 0m0.407s<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>real 0m0.289s<br># date<br>Wed Feb 26 07:17:53 PST 2014<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null^C<br>
# date<br>Wed Feb 26 07:52:16 PST 2014<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br># date<br>Wed Feb 26 09:10:28 PST 2014<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
real 0m1.018s<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>real 0m0.116s</div><div> </div><div>A similar experiment I did was to drop_caches, warm the FS cache up by</div><div>an "ls -l" on the brick, then time the warmed "ls -l" on the NFS mount:</div>
<div> </div><div> # date<br>Wed Feb 26 10:29:15 PST 2014<br># sync;sync; echo '3' > /proc/sys/vm/drop_caches<br># time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>real 0m41.899s<br>
# time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>real 0m2.400s<br># time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>real 0m0.115s<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<br>
That should expire the NFS cache, and give an idea of how fast things are with the protocol overhead pulled out.<br>
<br>
My guess: ~4s. See below for why.<br>
<div><br>
> Directory listing the CIFS FUSE mount is so slow, almost 16<br>
> seconds!<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /mnt/nas-cbs-0005-cifs/manydirs/ >/dev/null<br>
> real 0m56.573s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /mnt/nas-cbs-0005-cifs/manydirs/ >/dev/null<br>
> real 0m16.101s<br>
> # time ls -l /mnt/nas-cbs-0005-cifs/manydirs/ >/dev/null<br>
> real 0m15.986s<br>
<br>
> Directory listing the CIFS VFS libgfapi mount is about twice<br>
> as fast as FUSE, but still slow at 8 seconds.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /mnt/nas-cbs-0005-cifs-vfs/cifs_share/manydirs/ >/dev/null<br>
> real 0m48.839s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /mnt/nas-cbs-0005-cifs-vfs/cifs_share/manydirs/ >/dev/null<br>
> real 0m8.197s<br>
> # time ls -l /mnt/nas-cbs-0005-cifs-vfs/cifs_share/manydirs/ >/dev/null<br>
> real 0m8.450s<br>
<br>
</div>Looking at the numbers, it looks like the network is being consulted, and the data pulled back across.<br>
<br>
So let's do some quick math:<br>
<br>
FUSE:<br>
<br>
56s for the initial read.<br>
16s for each read there after.<br>
---<br>
40s of cache population time.<br>
<br>
VFS Module:<br>
<br>
48s for the initial read.<br>
8s for each read there after.<br>
---<br>
40s of cache population time.<br>
<br>
The fact that the cache population time drops out as a constant, tells me that in fact, it is likely re-reading the data over the network, and not caching.<br>
<br>
That should be controllable via mount parameters in mount.cifs. Now, that doesn't mean that Samba taking 8s to do the actual work, and NFS taking in my guesstimate, 4s. Is actually good. But it certainly puts the performance in another light.<br>
<div><div class="h5"><br>
> ####################<br>
><br>
> Retesting directory list with Gluster default settings,<br>
> including 'stat-prefetch' off, due to:<br>
><br>
> Bug 1004327 - New files are not inheriting ACL from parent directory<br>
> unless "stat-prefetch" is off for the respective gluster<br>
> volume<br>
> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1004327" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1004327</a><br>
><br>
> # gluster volume info nas-cbs-0005<br>
><br>
> Volume Name: nas-cbs-0005<br>
> Type: Distribute<br>
> Volume ID: 5068e9a5-d60f-439c-b319-befbf9a73a50<br>
> Status: Started<br>
> Number of Bricks: 1<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: 192.168.5.181:/exports/nas-segment-0004/nas-cbs-0005<br>
> Options Reconfigured:<br>
> performance.stat-prefetch: off<br>
> server.allow-insecure: on<br>
> nfs.rpc-auth-allow: *<br>
> nfs.disable: off<br>
> nfs.addr-namelookup: off<br>
><br>
> Directory listing of 6000 empty directories ('stat-prefetch'<br>
> is off):<br>
><br>
> Accessing the ext4 mount directly is almost instantaneous of<br>
> course.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/<br>
> >/dev/null<br>
> real 0m39.483s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/<br>
> >/dev/null<br>
> real 0m0.136s<br>
> # time ls -l /exports/nas-segment-0004/nas-cbs-0005/cifs_share/manydirs/<br>
> >/dev/null<br>
> real 0m0.109s<br>
><br>
> Accessing the NFS mount is also very fast.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
> real 0m43.819s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
> real 0m0.342s<br>
> # time ls -l /mnt/nas-cbs-0005/cifs_share/manydirs/ >/dev/null<br>
> real 0m0.200s<br>
><br>
> Accessing the CIFS FUSE mount is slow, almost 14 seconds!<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /mnt/nas-cbs-0005-cifs/manydirs/ >/dev/null<br>
> real 0m55.759s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /mnt/nas-cbs-0005-cifs/manydirs/ >/dev/null<br>
> real 0m13.458s<br>
> # time ls -l /mnt/nas-cbs-0005-cifs/manydirs/ >/dev/null<br>
> real 0m13.665s<br>
><br>
> Accessing the CIFS VFS libgfapi mount is now about twice as<br>
> slow as FUSE, at almost 26 seconds due to 'stat-prefetch'<br>
> being off!<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # time ls -l /mnt/nas-cbs-0005-cifs-vfs/cifs_share/manydirs/ >/dev/null<br>
> real 1m2.821s (Throw away first time for ext4 FS cache population?)<br>
> # time ls -l /mnt/nas-cbs-0005-cifs-vfs/cifs_share/manydirs/ >/dev/null<br>
> real 0m25.563s<br>
> # time ls -l /mnt/nas-cbs-0005-cifs-vfs/cifs_share/manydirs/ >/dev/null<br>
> real 0m26.949s<br>
<br>
</div></div>This data is all showing the same behaviors I described above. I expect the NFS client to win.<br>
<br>
The likely reason for the difference in the performance of FUSE vs. libgfapi here is: Caching.<br>
<br>
> ####################<br>
<div><div class="h5"><br>
> 4KB Writes:<br>
><br>
> NFS very small block writes are very slow at about 4 MB/sec.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync if=/dev/zero<br>
> of=/mnt/nas-cbs-0005/cifs_share/testfile count=20k<br>
> time to transfer data was 20.450521 secs, 4.10 MB/sec<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync if=/dev/zero<br>
> of=/mnt/nas-cbs-0005/cifs_share/testfile count=20k<br>
> time to transfer data was 19.669923 secs, 4.26 MB/sec<br>
><br>
> CIFS FUSE very small block writes are faster, at about 11<br>
> MB/sec.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync if=/dev/zero<br>
> of=/mnt/nas-cbs-0005-cifs/testfile count=20k<br>
> time to transfer data was 7.247578 secs, 11.57 MB/sec<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync if=/dev/zero<br>
> of=/mnt/nas-cbs-0005-cifs/testfile count=20k<br>
> time to transfer data was 7.422002 secs, 11.30 MB/sec<br>
><br>
> CIFS VFS libgfapi very small block writes are twice as fast<br>
> as CIFS FUSE, at about 22 MB/sec.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync if=/dev/zero<br>
> of=/mnt/nas-cbs-0005-cifs-vfs/cifs_share/testfile count=20k<br>
> time to transfer data was 3.766179 secs, 22.27 MB/sec<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync if=/dev/zero<br>
> of=/mnt/nas-cbs-0005-cifs-vfs/cifs_share/testfile count=20k<br>
> time to transfer data was 3.761176 secs, 22.30 MB/sec<br>
<br>
</div></div>I'm betting if you look at the back-end during the NFS transfer, it is at 100%.<br>
<br>
For Samba, it is likely SMB and a chattier protocol holding it back.<br>
<div><div class="h5"><br>
> 4KB Reads:<br>
><br>
> NFS very small block reads are very fast at about 346<br>
> MB/sec.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync of=/dev/null<br>
> if=/mnt/nas-cbs-0005/cifs_share/testfile count=20k<br>
> time to transfer data was 0.244960 secs, 342.45 MB/sec<br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync of=/dev/null<br>
> if=/mnt/nas-cbs-0005/cifs_share/testfile count=20k<br>
> time to transfer data was 0.240472 secs, 348.84 MB/sec<br>
><br>
> CIFS FUSE very small block reads are less than half as fast<br>
> as NFS, at about 143 MB/sec.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync of=/dev/null<br>
> if=/mnt/nas-cbs-0005-cifs/testfile count=20k<br>
> time to transfer data was 0.606534 secs, 138.30 MB/sec<br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync of=/dev/null<br>
> if=/mnt/nas-cbs-0005-cifs/testfile count=20k<br>
> time to transfer data was 0.576185 secs, 145.59 MB/sec<br>
><br>
> CIFS VFS libgfapi very small block reads a slight bit slower<br>
> than CIFS FUSE, at about 137 MB/sec.<br>
><br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync of=/dev/null<br>
> if=/mnt/nas-cbs-0005-cifs-vfs/cifs_share/testfile count=20k<br>
> time to transfer data was 0.611328 secs, 137.22 MB/sec<br>
> # sync;sync; echo '3' > /proc/sys/vm/drop_caches<br>
> # sgp_dd time=1 thr=4 bs=4k bpt=1 iflag=dsync oflag=dsync of=/dev/null<br>
> if=/mnt/nas-cbs-0005-cifs-vfs/cifs_share/testfile count=20k<br>
> time to transfer data was 0.615834 secs, 136.22 MB/sec<br>
<br>
<br>
</div></div>I'm not totally shocked by the bad performance here of Samba.<br>
<br>
It matches what I've seen in other scenarios.<br>
<br>
There may be things that can be done to speed things up. But I think the issue is probably Samba+CIFS client. (And I truly expect it is Samba, from my experience with Windows.)<br>
<br>
I'd encourage you to also do a raw test of Samba+CIFS client without Gluster in the mix. It would help confirm some of these thoughts.<br></blockquote><div> </div><div>The first test I did, which unfortunately didn't get<br>
reported was to make sure that this was not just a<br>Samba/CIFS issue. To do this, I make a CIFS mount directly<br>to the storage brick/segment, bypassing GlusterFS, and<br>mounted it.<br></div><div>Note that there is neither the long cache population time on<br>
the first run, nor the very long delays, although there is a<br>consistent 1.8 second delay which must be attributed to<br>Samba/CIFS itself:</div><div> </div><div>[nas-cbs-0005-seg]<br> path = /exports/nas-segment-0004/nas-cbs-0005<br>
admin users = "localadmin"<br> valid users = "localadmin"<br> invalid users =<br> read list =<br> write list = "localadmin"<br> guest ok = yes<br> read only = no<br> hide unreadable = yes<br>
hide dot files = yes<br> available = yes</div><div><br># mount |grep seg<br>//<a href="http://10.10.200.181/nas-cbs-0005-seg">10.10.200.181/nas-cbs-0005-seg</a> on /mnt/nas-cbs-0005-seg type cifs (rw,username=localadmin,password=localadmin)</div>
<div># sync;sync; echo '3' > /proc/sys/vm/drop_caches<br># time ls -l /mnt/nas-cbs-0005-seg/cifs_share/manyfiles/ >/dev/null<br>real 0m1.745s<br># time ls -l /mnt/nas-cbs-0005-seg/cifs_share/manyfiles/ >/dev/null<br>
real 0m1.819s<br># time ls -l /mnt/nas-cbs-0005-seg/cifs_share/manyfiles/ >/dev/null<br>real 0m1.781s </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<br>
I'd also encourage you to open a gluster BZ, so we don't lose track of this data. Alas, I know we can't act on it immediately.<br>
<br>
Also any replication and setup scripts you are using, would be GREAT to put in that BZ. It will really help us reproduce the issue.<br>
<br>
Thanks,<br>
<br>
-Ira / ira@(<a href="http://redhat.com" target="_blank">redhat.com</a>|<a href="http://samba.org" target="_blank">samba.org</a>)<br>
</blockquote></div><div class="gmail_extra"><br><br clear="all"><br>-- <br></div><div class="gmail_extra" dir="ltr">~ Jeff Byers ~</div><div class="gmail_extra">
</div></div>