<div dir="ltr">Thanks Dan, very helpful indeed</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Apr 28, 2014 at 7:44 PM, Dan Mons <span dir="ltr"><<a href="mailto:dmons@cuttingedge.com.au" target="_blank">dmons@cuttingedge.com.au</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 28 April 2014 16:11, Andrew Hatfield<br>
<div class=""><<a href="mailto:andrew.hatfield@cynosureservices.com">andrew.hatfield@cynosureservices.com</a>> wrote:<br>
> Hey Dan,<br>
><br>
> Did you ever test SMB with the streams xattr vfs object?<br>
<br>
</div>Yes. :)<br>
<div class=""><br>
> Did it work? Was performance ok?<br>
<br>
</div>It functions, but forces us to compromise. It was the only way to get<br>
GlusterFS working via SMB on MacOSX.<br>
<br>
For anyone who hasn't discovered this: MacOSX's Finder looks for a<br>
"._filename" resource fork file for every "filename" data file it<br>
finds. GlusterFS (and any clustered file system, to be fair) is quite<br>
poor at negative lookups (i.e.: when it can't find a file on a<br>
particular brick, it has to run around asking every brick in the<br>
cluster if they have the file), and so a folder with ~1000 files in it<br>
(quite common for a VFX studio, across dozens of shots inside dozens<br>
of projects) can take several minutes to populate when trying to view<br>
it in OSX's Finder.<br>
<br>
streams xattr means that the resource fork information can be held in<br>
the xattr component of the underlying file system (for us that's XFS<br>
on our CentOS6 bricks), and completely removes the need for the<br>
negative lookup, as that resource fork data lives with the file. (I<br>
for one find the whole resource fork concept totally outdated and<br>
silly, but Apple seem keen to keep it around).<br>
<br>
Downsides:<br>
<br>
1) SMB is slower than either NFS or FUSE+GlusterFS. vfs_glusterfs<br>
helps, but still doesn't compete in real world usage.<br>
<br>
2) MacOSX does not allow system-wide mounting of SMB shares, ala NFS3.<br>
Our studio relies heavily on this as machines need network file<br>
systems mounted up so that multiple users can hit them at once. We<br>
run a large number of machines in what's called a "render farm" which<br>
are batch-processing multiple jobs each in parallel. Our standard spec<br>
render node currently is a dual Xeon (8 cores per proc, 2 procs, per<br>
node, with HT = 32 threads / logical CPUs) and 128GB RAM. These<br>
sometimes do one very large job, and sometimes many smaller jobs in<br>
parallel. Jobs run as the UID of the person who submitted them, so<br>
NFS3 style network mounting is mandatory for this to work. There is<br>
no such thing as "one user on one machine" in our world.<br>
<br>
MacOSX can't do this via SMB. In 10.8 and 10.9, whichever user<br>
initiates the SMB connection "owns" the mount (regardless of the<br>
credentials they use at mount time), and any other UID is locked out<br>
of that share.<br>
<br>
Upsides:<br>
<br>
1) At least it works at all, unlike NFS which Apple broke in Finder<br>
back in 10.5 and have refused to even acknowledge let alone fix.<br>
Honestly, how does an OS based on UNIX not do NFS? 10.9 now<br>
represents 5 complete production versions of their OS in a row with<br>
broken NFS. Ludicrous.<br>
<br>
2) AFP has the same UID-clobbering mount issues as SMB, but SMB is<br>
faster than AFP (thanks to streams xattr) and we can cluster SMB more<br>
easily.<br>
Long story short, SMB is a stop-gap. Ideally either Apple fix NFS<br>
(I'm not holding my breath, as it's clear Apple's priority is selling<br>
iPhones and media, and not actually offering a usable operating<br>
system), or the Gluster community add the ever moving target of MacOSX<br>
support to their FUSE+GlusterFS client, which is now moving along<br>
nicely as per this thread.<br>
<br>
So my eternal thanks to everyone contributing to this OSX porting<br>
effort (and of course everyone who's ever contributed a line of code<br>
to GlusterFS in general). You are all wonderful human beings. :)<br>
<span class="HOEnZb"><font color="#888888"><br>
-Dan<br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br></div>