<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hi all,<br>
<br>
I had an instance of heal-failed today on a 3x2 replicated volume with 17TB on ubuntu 12.04 xfs bricks running gluster 3.5.2<br>
<br>
Initially:<br>
<br>
on the brick log:<br>
<br>
<pre>Warnings in <a href="http://xymonprod001.sl1.shopzilla.sea/xymon-cgi/svcstatus.sh?CLIENT=glusterprod001.shopzilla.laxhq&SECTION=msgs:/var/log/glusterfs/glustershd.log">/var/log/glusterfs/glustershd.log</a>
<pre>.2014-09-25 15:56:10.200387] E [afr-self-heal-common.c:1615:afr_sh_common_lookup_cbk] 0-sas03-replicate-0: Conflicting entries for /RdB2C_20140917.dat
.2014-09-25 15:56:10.653858] E [afr-self-heal-common.c:1615:afr_sh_common_lookup_cbk] 0-sas03-replicate-0: Conflicting entries for /RdB2C_20140917.dat
</pre></pre>
<br>
which staying a file having conflich. NO split-brain detected but a heal-failed:<br>
<pre class="textmate-source"><pre class="sunburst"><span class="meta meta_paragraph meta_paragraph_text">root@glusterprod001:~# gluster volume heal sas03 info heal-failed
Gathering list of heal failed entries on volume sas03 has been successful
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod001.shopzilla.laxhq:/brick03/gfs
Number of entries: 2
at path on brick
-----------------------------------
2014-09-25 15:56:10 /HypDataSata03/data/RdctB2C
2014-09-25 16:06:09 /HypDataSata03/data/RdctB2C
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod002.shopzilla.laxhq:/brick03/gfs
Number of entries: 1
at path on brick
-----------------------------------
2014-09-25 15:58:37 /HypDataSata03//data//RdctB2C
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod003.shopzilla.laxhq:/brick03/gfs
Number of entries: 0
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod004.shopzilla.laxhq:/brick03/gfs
Number of entries: 0
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod005.shopzilla.laxhq:/brick03/gfs
Number of entries: 0
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod006.shopzilla.laxhq:/brick03/gfs
Number of entries: 0</span></pre></pre>
Noticed it stays the directory heal-failed instead of the file.<br>
<br>
Gluster clients sees the error on the file with an invalid file while doing ls against it.<br>
<br>
Then:<br>
<br>
I tried to restart glusterfs-server on both prod001 and prod002 as that's how I used to resolve the heal-failed.<br>
<br>
and it became like this:<br>
<br>
<pre class="textmate-source"><pre class="sunburst"><span class="meta meta_paragraph meta_paragraph_text">root@glusterprod001:~# gluster volume heal sas03 info heal-failed
Gathering list of heal failed entries on volume sas03 has been successful
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod001.shopzilla.laxhq:/brick03/gfs
Number of entries: 2
at path on brick
-----------------------------------
2014-09-25 16:17:51 <gfid:9ec801a3-53d4-4d98-b950-14211920694e>
2014-09-25 16:17:53 <gfid:9ec801a3-53d4-4d98-b950-14211920694e>
</span>
<span class="meta meta_paragraph meta_paragraph_text">Brick glusterprod002.shopzilla.laxhq:/brick03/gfs
Number of entries: 3
at path on brick
-----------------------------------
2014-09-25 16:15:43 <gfid:9ec801a3-53d4-4d98-b950-14211920694e>
2014-09-25 16:15:44 <gfid:9ec801a3-53d4-4d98-b950-14211920694e>
2014-09-25 16:17:51 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br><br>Seems like the folder turns into gfid <br><br>And then:<br></span></pre></pre>
I identified the file on the brick and removed the invalid copy then issue a volume heal<br>
<br>
# gluster volume heal sas03<br>
<br>
This fixed the client access to the file<br>
<br>
but info heal-failed got this:<br>
root@glusterprod001:/# gluster volume heal sas03 info heal-failed<br>
Gathering list of heal failed entries on volume sas03 has been successful<br>
<br>
Brick glusterprod001.shopzilla.laxhq:/brick03/gfs<br>
Number of entries: 3<br>
at path on brick<br>
-----------------------------------<br>
2014-09-25 16:17:51 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
2014-09-25 16:17:53 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
2014-09-25 16:27:53 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
<br>
Brick glusterprod002.shopzilla.laxhq:/brick03/gfs<br>
Number of entries: 5<br>
at path on brick<br>
-----------------------------------<br>
2014-09-25 16:15:43 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
2014-09-25 16:15:44 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
2014-09-25 16:17:51 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
2014-09-25 16:25:44 <gfid:9ec801a3-53d4-4d98-b950-14211920694e><br>
2014-09-25 16:34:11 /HypDataSata03/data/RdctB2C<br>
<br>
Brick glusterprod003.shopzilla.laxhq:/brick03/gfs<br>
Number of entries: 0<br>
<br>
Brick glusterprod004.shopzilla.laxhq:/brick03/gfs<br>
Number of entries: 0<br>
<br>
Brick glusterprod005.shopzilla.laxhq:/brick03/gfs<br>
Number of entries: 0<br>
<br>
Brick glusterprod006.shopzilla.laxhq:/brick03/gfs<br>
Number of entries: 0<br>
<br>
which has all the gfid and the directory showed up on the heal-failed<br>
<br>
Finally:<br>
<br>
I restarted glusterfs-server on both prod001 and prod002 and that cleared the heal-failed entries<br>
<br>
Should there be a better way to resolve the heal-failed and file conflict?<br>
<br>
Thanks<br>
Peter<br>
<br>
</div>
</body>
</html>