Hi,<br><br>We've been using GlusterFS to manage shared files across a number of hosts in the past few months and have ran into a few problems -- basically one every month, roughly. The problems are occasionally extremely difficult to track down to GlusterFS, as they often masquerade as something else in the application log files that we have. The problems have been one instance of split-brain and then a number of instances of "stuck" files (i.e. any stat calls would block for an hour and then timeout with an error) as well as a couple instances of "ghost" files (remove the file, but GlusterFS continues to show it for a little while until the cache times out).<br>
<br>We do <i>not</i> place a large amount of load on GlusterFS, and don't have any significant performance issues to deal with. With that in mind, the core question of this e-mail is: "How can I modify our configuration to be the absolute <i>most</i> stable (problem free) that it can be, even if it means sacrificing performance?" In sum, I don't have any particular performance concerns at this moment, but the GlusterFS bugs that we encounter are quite problematic -- so I'm willing to entertain any suggested stability improvement, even if it has a negative impact on performance (I suspect that the answer here is just "turn off all performance-enhancing gluster caching", but I wanted to validate that is actually true before going so far). Thus please suggest anything that could be done to improve the stability of our setup -- as an aside, I think that this would be an advantageous thing to add to the FAQ. Right now the FAQ contains information for <i>performance</i> tuning, but not for <i>stability</i> tuning.<br>
<br>Thanks for any help that you can give/suggestions that you can make.<br><br>Here are the details of our environment:<br><br>OS: RHEL5<br>GlusterFS Version: 3.1.5<br>Mount method: glusterfsd/FUSE<br clear="all">GlusterFS Servers: web01, web02<br>
GlusterFS Clients: web01, web02, dj01, dj02<br><br><span style="font-family: courier new,monospace;">$ sudo gluster volume info</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Volume Name: shared-application-data</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Type: Replicate</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Status: Started</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Number of Bricks: 2</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Transport-type: tcp</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Bricks:</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Brick1: web01:/var/glusterfs/bricks/shared</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">Brick2: web02:/var/glusterfs/bricks/shared</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">Options Reconfigured:</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">network.ping-timeout: 5</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">nfs.disable: on</span><br><br>Configuration File Contents:<br><b>/etc/glusterd/vols/shared-application-data/shared-application-data-fuse.vol</b><br><span style="font-family: courier new,monospace;">volume shared-application-data-client-0</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type protocol/client</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> option remote-host web01</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> option remote-subvolume /var/glusterfs/bricks/shared</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> option transport-type tcp</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> option ping-timeout 5</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;">
<br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-client-1</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> type protocol/client</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> option remote-host web02</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> option remote-subvolume /var/glusterfs/bricks/shared</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> option transport-type tcp</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> option ping-timeout 5</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-replicate-0</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type cluster/replicate</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-client-0 shared-application-data-client-1</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-write-behind</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type performance/write-behind</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-replicate-0</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-read-ahead</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type performance/read-ahead</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-write-behind</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-io-cache</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type performance/io-cache</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-read-ahead</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-quick-read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type performance/quick-read</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-io-cache</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data-stat-prefetch</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type performance/stat-prefetch</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-quick-read</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">volume shared-application-data</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type debug/io-stats</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> subvolumes shared-application-data-stat-prefetch</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">end-volume</span><br><br><b>/etc/glusterfs/glusterd.vol</b><br><span style="font-family: courier new,monospace;">volume management</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> type mgmt/glusterd</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> option working-directory /etc/glusterd</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> option transport-type socket,rdma</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;"> option transport.socket.keepalive-time 10</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"> option transport.socket.keepalive-interval 2</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">end-volume</span><br>
<br>-- <br><div>Remi Broemeling</div><div>System Administrator</div><span style="font-family:arial, sans-serif;font-size:13px;border-collapse:collapse"><div><font face="arial, helvetica, sans-serif">Clio - Practice Management Simplified</font></div>
<div><font face="arial, helvetica, sans-serif">1-888-858-2546 x(2^5) | <a href="mailto:remi@goclio.com" target="_blank">remi@goclio.com</a></font></div><div><font face="arial, helvetica, sans-serif"><a href="http://www.goclio.com/" target="_blank">www.goclio.com</a> | <a href="http://www.goclio.com/blog" target="_blank">blog</a> | <a href="http://www.twitter.com/goclio" target="_blank">twitter</a> | <a href="http://www.facebook.com/goclio" target="_blank">facebook</a><br>
</font><span style="border-collapse:separate;color:rgb(255, 255, 255);font-family:'Times New Roman';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><span style="font-family:verdana, arial, helvetica, sans-serif;font-size:small"><pre style="margin-top:10px;margin-right:10px;margin-bottom:10px;margin-left:10px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">
<code style="font-style:normal;font-weight:normal"> ____
_⌠ oo ⌡_
(_ _)
| |
⌡_⌡⌡_⌡</code></pre></span></span><br></div></span><br>