<div dir="ltr">Off and on for the past 8 months I've been working on setting up a rather sizable Gluster configuration on top of ZFS. I wanted to report back my experiences.<div><br></div><div>First, I'm running Ubuntu 12.04 with <a href="http://zfsonlinux.org">zfsonlinux.org</a> packages. Gluster is installed from the semiosis PPA.</div>
<div><br></div><div>My test config is as follows:</div><div><br></div><div>Two brick Gluster in mirroring (replica 2)</div><div>Each brick has 8 TB of hard drives configured with ZFS using RAID-Z.</div><div>Each brick has a 256 GB SSD drive to use as cache for ZFS.</div>
<div>A third server for geo-replication slave also with RAID-Z storage.</div><div>Private Gigabit network segment for the gluster servers (and the application servers that talk to them)</div><div>Geo-rep will be placed three states away in a colo center, but is currently on the same private segment.</div>
<div><br></div><div>Data:</div><div>About 6 TB of test data, the final deploy will be much larger. Most files are around 40MB in size. Files are stored in directories going 4 deep, no more than 255 files in a single directory.</div>
<div><br></div><div>All data is already compressed and encrypted with AES-256 prior to storing on the filesystem, so the comrpession feature of ZFS isn't useful to me.</div><div><br></div><div>The default configs for ZFS caused this to be a miserable failure. Configuring Gluster geo-replication over SSH was also a difficult task, because the documentation is wrong on many counts when compared to the version in the PPA.</div>
<div><br></div><div>With a default config, directory listings (remember, less than 255 files and folders in each directory) would take about 15 seconds to complete over the Gluster share.</div><div><br></div><div>By simply tweaking the following ZFS configs:</div>
<div><div>zfs set atime=off [volname]</div><div>zfs set xattr=sa [volname]</div><div>zfs set exec=off [volname]</div><div>zfs set sync=disabled [volname]</div></div><div><br></div><div>I was able to get near instantaneous directory listings and dramatically improved read performance, although I haven't bothered getting hard numbers on that.</div>
<div><br></div><div>I just wanted to pass this along. Here's my blog posts about ZFS and Gluster, which pretty much says the same thing I just did but offers more about installing ZFS:</div><div><br></div><div><a href="http://www.ossramblings.com/zfs-filesystem-on-linux">http://www.ossramblings.com/zfs-filesystem-on-linux</a><br>
</div><div><a href="http://www.ossramblings.com/gluster-geo-replication-with-zfs">http://www.ossramblings.com/gluster-geo-replication-with-zfs</a><br></div><div><br></div><div>-Tony</div></div>