Joseph,<br><br>Thank you for your response. Yours, combined with Whit's, led me to come up with a pretty solid repro case, and a pinpointing of what I think is going on.<br><br>I tried your additional SMB configuration settings, and was hopeful, but it didn't alleviate the issue. But it was helpful, your interpretation of the logs. It makes sense now that Samba was pounding on GlusterFS, doing it's string of getdents operations.<br>
<br>I also took your advice last night on stat-cache (I assume that was on the Gluster side, which I enabled), and wasn't sure where fast lookups was. That didn't seem to make a noticeable difference either.<br><br>
I think the lockups are happening as a result of being crippled by GlusterFS's relatively slow directory listing (5x-10x slower generating a dir listing than a raw SMB share), combined with FUSE's blocking readdir(). I'm not positive on that last point since there was only one mention of that on the internet. Am praying that somebody will see this and say, "oh yeah, well sure, just change this one thing in FUSE and you're good to go!" Somehow I don't think that's going to happen. :)<br>
<br>Ken<br><br><div class="gmail_quote">On Sun, Jul 17, 2011 at 10:35 PM, Joe Landman <span dir="ltr"><<a href="mailto:landman@scalableinformatics.com">landman@scalableinformatics.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On 07/17/2011 11:19 PM, Ken Randall wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Joe,<br>
<br>
Thank you for your response. After seeing what you wrote, I bumped up<br>
the performance.cache-size up to 4096MB, the max allowed, and ran into<br>
the same wall.<br>
</blockquote>
<br></div>
Hmmm ...<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I wouldn't think that any SMB caching would help in this case, since the<br>
same Samba server on top of the raw Gluster data wasn't exhibiting any<br>
trouble, or am I deceived?<br>
</blockquote>
<br></div>
Samba could cache better so it didn't have to hit Gluster so hard.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I haven't used strace before, but I ran it on the glusterfs process, and<br>
saw a lot of:<br>
epoll_wait(3, {{EPOLLIN, {u32=9, u64=9}}}, 257, 4294967295) = 1<br>
readv(9, [{"\200\0\16,", 4}], 1) = 4<br>
readv(9, [{"\0\n;\227\0\0\0\1", 8}], 1) = 8<br>
readv(9,<br>
[{"\0\0\0\0\0\0\0\0\0\0\0\0\0\<u></u>0\0\0\0\0\0\31\0\0\0\0\0\0\0\<u></u>1\0\0\0\0"..., 3620}],<br>
1) = 1436<br>
readv(9, 0xa90b1b8, 1) = -1 EAGAIN (Resource<br>
temporarily unavailable)<br>
</blockquote>
<br></div>
Interesting ... I am not sure why its reporting an EAGAIN for readv, other than it can't fill the vector from the read.<div><div></div><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
And when I ran it on smbd, I saw a constant stream of this kind of activity:<br>
getdents(29, /* 25 entries */, 32768) = 840<br>
getdents(29, /* 25 entries */, 32768) = 856<br>
getdents(29, /* 25 entries */, 32768) = 848<br>
getdents(29, /* 24 entries */, 32768) = 856<br>
getdents(29, /* 25 entries */, 32768) = 864<br>
getdents(29, /* 24 entries */, 32768) = 832<br>
getdents(29, /* 25 entries */, 32768) = 832<br>
getdents(29, /* 24 entries */, 32768) = 856<br>
getdents(29, /* 25 entries */, 32768) = 840<br>
getdents(29, /* 24 entries */, 32768) = 832<br>
getdents(29, /* 25 entries */, 32768) = 784<br>
getdents(29, /* 25 entries */, 32768) = 824<br>
getdents(29, /* 25 entries */, 32768) = 808<br>
getdents(29, /* 25 entries */, 32768) = 840<br>
getdents(29, /* 25 entries */, 32768) = 864<br>
getdents(29, /* 25 entries */, 32768) = 872<br>
getdents(29, /* 25 entries */, 32768) = 832<br>
getdents(29, /* 24 entries */, 32768) = 832<br>
getdents(29, /* 25 entries */, 32768) = 840<br>
getdents(29, /* 25 entries */, 32768) = 824<br>
getdents(29, /* 25 entries */, 32768) = 824<br>
getdents(29, /* 24 entries */, 32768) = 864<br>
getdents(29, /* 25 entries */, 32768) = 848<br>
getdents(29, /* 24 entries */, 32768) = 840<br>
</blockquote>
<br></div></div>
Get directory entries. This is the stuff that NTFS is caching for its web server, and it appears Samba is not.<br>
<br>
Try<br>
<br>
aio read size = 32768<br>
csc policy = documents<br>
dfree cache time = 60<br>
directory name cache size = 100000<br>
fake oplocks = yes<br>
getwd cache = yes<br>
level2 oplocks = yes<br>
max stat cache size = 16384<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That chunk would get repeated over and over and over again as fast as<br>
the screen could go, with the occasional (every 5-10 seconds or so),<br>
would you see anything that you'd normally expect to see, such as:<br>
close(29) = 0<br>
stat("Storage/01", 0x7fff07dae870) = -1 ENOENT (No such file or directory)<br>
write(23,<br>
"\0\0\0#\377SMB24\0\0\300\<u></u>210A\310\0\0\0\0\0\0\0\0\0\0\<u></u>0\0\1\0d\233"...,<br>
39) = 39<br>
select(38, [5 20 23 27 30 31 35 36 37], [], NULL, {60, 0}) = 1 (in [23],<br>
left {60, 0})<br>
read(23, "\0\0\0x", 4) = 4<br>
read(23,<br>
"\377SMB2\0\0\0\0\30\7\310\0\<u></u>0\0\0\0\0\0\0\0\0\0\0\1\0\<u></u>250P\273\0[8"...,<br>
120) = 120<br>
stat("Storage", {st_mode=S_IFDIR|0755, st_size=1581056, ...}) = 0<br>
stat("Storage/011235", 0x7fff07dad470) = -1 ENOENT (No such file or<br>
directory)<br>
stat("Storage/011235", 0x7fff07dad470) = -1 ENOENT (No such file or<br>
directory)<br>
open("Storage", O_RDONLY|O_NONBLOCK|O_<u></u>DIRECTORY) = 29<br>
fcntl(29, F_SETFD, FD_CLOEXEC) = 0<br>
<br>
(The no such file or directory part is expected since some of the image<br>
references don't exist.)<br>
<br>
</blockquote>
<br></div>
Ok. It looks like Samba is pounding on GlusterFS metadata (getdents). GlusterFS doesn't really do a great job in this case ... you have to give it help and cache pretty aggressively here. Samba can do this caching to some extent. You might want to enable stat-cache and fast lookups. These have been problematic for us in the past though, and I'd recommend caution.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Ken<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/<u></u>mailman/listinfo/gluster-users</a><br>
</blockquote>
<br>
<br>
-- <br>
Joseph Landman, Ph.D<br>
Founder and CEO<br>
Scalable Informatics, Inc.<br>
email: <a href="mailto:landman@scalableinformatics.com" target="_blank">landman@scalableinformatics.<u></u>com</a><br>
web : <a href="http://scalableinformatics.com" target="_blank">http://scalableinformatics.com</a><br>
<a href="http://scalableinformatics.com/sicluster" target="_blank">http://scalableinformatics.<u></u>com/sicluster</a><br>
phone: <a href="tel:%2B1%20734%20786%208423%20x121" value="+17347868423" target="_blank">+1 734 786 8423 x121</a><br>
fax : <a href="tel:%2B1%20866%20888%203112" value="+18668883112" target="_blank">+1 866 888 3112</a><br>
cell : <a href="tel:%2B1%20734%20612%204615" value="+17346124615" target="_blank">+1 734 612 4615</a><br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/<u></u>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br>