<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;"><span lang="en-US">
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">GlusterFS Alert -<br>
<br>
Problem: GFID Mismatch<br>
<br>
Severity: 7 (out of 10) - Loss of service but ultimately no loss of data<br>
<br>
PREVENTION: To *prevent* the issue, please install GlusterFS 3.2.2. If you're using 3.1.x, upgrade to 3.1.5.
<br>
Download 3.2.2 here: </span></font><a href="http://download.gluster.com/pub/gluster/glusterfs/LATEST/" target="_blank">http://download.gluster.com/pub/gluster/glusterfs/LATEST/</a>
<br>
<font size="2"><span style="font-family: Calibri;">Download 3.1.5 here:</span></font>
<a href="http://download.gluster.com/pub/gluster/glusterfs/3.1/LATEST/" target="_blank">
http://download.gluster.com/pub/gluster/glusterfs/3.1/LATEST/</a> <br>
<font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"><br>
FIX: </span></font><span lang="en-US">
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">To check for mismatched GFIDs, please review your client logs and grep for the words:</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">“gfid different” or “gfid differs”</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"> </span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">If you see either of these conditions, simply upgrading will not fix the problem. You will need to use our tools here:
</span></font><a href="https://github.com/vikasgorur/gfid" target="_blank">https://github.com/vikasgorur/gfid</a>
</div>
</span><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">See details below for instructions. Upgrading will not fix the issue if you've already experienced GFID mismatches.<br>
<br>
</span></font><span lang="en-US">
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"><br>
DETAILS:<br>
Over the last 3 weeks we have seen a growing number of GlusterFS implementations experiencing an issue where mismatched GFIDs are appearing within the filesystem.
<br>
<br>
Each file/directory on a Gluster volume has a unique 128-bit number associated with it called the GFID. This is true regardless of Gluster configuration (distribute or distribute/replicate). One inode, one GFID. The GFID is stored on the backend as the value
of the extended attribute "trusted.gfid". Under normal circumstances, the value of this attribute is the same on all the backend bricks. However, certain conditions can cause the value on one or more of the bricks to differ from that on the other bricks. This
causes the GlusterFS client to become confused and throw errors. This applies to both the 3.1.4 and 3.2.1 versions of the filesystem, and previous versions in those series. This can happen with the Native GlusterFS, NFS, or CIFS.</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"> </span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">PREVENTION:<br>
</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">To prevent this issue from occurring, please upgrade immediately to 3.1.5, or 3.2.2. This will not correct the issue should it already be present in your cluster.</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"><br>
FIX:<br>
***IMPORTANT***<br>
</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">To check for mismatched GFIDs, please review your client logs and grep for the words:</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">“gfid different” or “gfid differs”</span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"> </span></font></div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">If you see either of these conditions, simply upgrading will not fix the problem. You will need to download tools here:
</span></font><a href="https://github.com/vikasgorur/gfid" target="_blank">https://github.com/vikasgorur/gfid</a>
<br>
</div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;"><br>
Follow the instructions in the README:<br>
</span></font><a href="https://github.com/vikasgorur/gfid/blob/master/README" target="_blank">https://github.com/vikasgorur/gfid/blob/master/README</a>
<br>
<br>
Here's the quick-start version:<br>
<br>
<pre><div class="line" id="LC22">1. The first step is to construct the master list of all files:</div><div class="line" id="LC23"><br></div><div class="line" id="LC24"># cd /export/brick1</div><div class="line" id="LC25"># find . > brick1.txt</div><div class="line" id="LC26">... (do for all bricks)<br></div></pre>
# cat brick1.txt brick2.txt... | sort -u > master_list.txt<br>
<br>
2. Then we need to get the gfid's of all the inodes from these bricks:
<pre><div class="line" id="LC39"><pre><div class="line" id="LC40"># cd /export/brick1</div><div class="line" id="LC41"># gfid-list /path/to/master_list.txt > brick1.gfid<br>... (do for all bricks)<br><br><pre><div class="line" id="LC52">3. Identify the mismatched inodes:</div><div class="line" id="LC53"><br></div><div class="line" id="LC76"># gfid-mismatch brick1.gfid brick2.gfid brick3.gfid brick4.gfid<br><br>4. Delete the gfid's now by doing:<span style="font-family: Tahoma;"><br><br></span># gluster volume stop <affected volume><span style="font-family: Tahoma;"><br></span># gfid-mismatch brick1.gfid brick2.gfid brick3.gfid brick4.gfid | cut -f1 -d: > mismatched.txt<span style="font-family: Tahoma;"><br></span># cd /export/brick1<br># gfid-delete /path/to/mismatched.txt</div></pre></div></pre></div></pre>
<div class="line" id="LC37">
<div class="line" id="LC77">
<pre>Repeat for the other bricks.<br><br>5. Check logs<span style="font-family: Tahoma;"><br><br>'</span>gfid-delete' will produce a log<span style="font-family: Tahoma;"> </span>with one entry for each file, which is either:<span style="font-family: Tahoma;"><br><br></span>usr/bin/factor: removed OK<span style="font-family: Tahoma;"><br></span> OR<span style="font-family: Tahoma;"><br></span>usr/bin/vim: No such file or directory<span style="font-family: Tahoma;"><br><br></span>IMPORTANT NOTE: The deletion of gfid's must be done ONLY ON A STOPPED<span style="font-family: Tahoma;"> </span>VOLUME. <br>Deleting the gfid's on a running volume with mounted clients<span style="font-family: Tahoma;"> </span>will cause more <br>problems instead of solving them.</pre>
</div>
</div>
<br>
</div>
<div style="margin: 0pt;"><font face="Calibri,sans-serif" size="2"><span style="font-size: 11pt;">Please feel free to contact me directly with any questions.<br>
<br>
</span></font></div>
</span></div>
</span></div>
</body>
</html>