<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Dear Ravi,</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">You hit the nail right on the head. After successful MD5sum’s I reseted the the afr extended attributes of the bricks as you proposed and self heal daemon now runs without complains.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Thanks,</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Mark</div> <br><p style="color:#A0A0A8;">On 25 november 2013 at 12:22:27, Ravishankar N (<a href="mailto://ravishankar@redhat.com">ravishankar@redhat.com</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div bgcolor="#FFFFFF" text="#000000"><div>
<title></title>
<div class="moz-cite-prefix">On 11/25/2013 01:47 AM, Mark Ruys
wrote:<br></div>
<blockquote cite="mid:etPan.52925eec.515f007c.154@Rajka.local" type="cite">
<div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">
So I decided to bite the bullet and upgraded from 3.3 to 3.4.
Somehow this was a painful proces for me (the glusterfs daemon
refused to start), so I decided to configure our Gluster pool from
scratch. Everything seems to work nicely, except for the self-heal
daemon. In the logs, I get every 10 minutes the following
line:</div>
<div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">
<br></div>
<div id="bloop_customfont" style="color: rgb(0, 0, 0); margin: 0px;">
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">[2013-11-24 19:50:34.495204] E
[afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
0-GLUSTER-SHARE-replicate-0: Unable to self-heal contents of
'<gfid:00000000-0000-0000-0000-000000000001>' (possible
split-brain). Please delete the file from all but the preferred
subvolume.- Pending matrix: [ [ 0 2 ] [ 2 0 ]
]</span></font></p>
</div>
<div id="bloop_sign_1385322684854820096">
<div style="orphans: 2; widows: 2; line-height: normal; font-size: 11px; font-family: Helvetica;">
<br></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial">I’ve removed and recreated
the .glusterfs/00/00/00000000-0000-0000-0000-000000000001, but
that doesn’t seem to make a difference. </font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial"><br></font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial">How to fix the self-heal daemon?</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial"><br></font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial">Mark</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial"><br></font></div>
<div style="orphans: auto; widows: auto;">
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;"># find . -name
00000000-0000-0000-0000-000000000001 -ls</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">1447202 0 ---------- 2 root
root 0 Nov
23 22:35
./export-share-1/.glusterfs/indices/xattrop/00000000-0000-0000-0000-000000000001</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">1319116 0 lrwxrwxrwx 1 root
root 8 Nov
23 22:35
./export-share-1/.glusterfs/00/00/00000000-0000-0000-0000-000000000001
-> ../../..</span></font></p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;">
<br></p>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial">Brick 1:</font></div>
<div style="orphans: auto; widows: auto;">
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;"># getfattr -m . -d -e hex
export-share-1</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;"># file: export-share-1</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-0=0x000000000000000000000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-1=0x000000000000000200000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.gfid=0x00000000000000000000000000000001</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.quota.dirty=0x3000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.quota.size=0x0000000000000000</span></font></p>
<p style="margin: 0px;"></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.volume-id=0xe6eb05aabe3b456cbf3027275faa529c</span></font></p>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial"><br></font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial">Brick 2:</font></div>
<div style="orphans: auto; widows: auto;">
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;"># getfattr -m . -d -e hex
export-share-2</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;"># file: export-share-2</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-0=0x000000000000000200000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-1=0x000000000000000000000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.gfid=0x00000000000000000000000000000001</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.quota.dirty=0x3000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.quota.size=0x0000000000000000</span></font></p>
<p style="margin: 0px;"></p>
<p style="margin: 0px;"><font face="Courier New"><span style="font-size: 11px;">trusted.glusterfs.volume-id=0xe6eb05aabe3b456cbf3027275faa529c</span></font></p>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial"><br></font></div>
</div>
</blockquote>
From the afr extended attributes, it seems you have hit a
metadata-split-brain of the top level (brick) directory (having
gfid 01). If you are you able to perform I/O on all files
from the mount point without error (EIO) and the file
contents are identical on both the bricks (check with md5sum), you
could safely clear the afr extended attributes of the
bricks:<br>
<br>
setfattr -n trusted.afr.<font face="Courier New"><span style="font-size: 11px;">GLUSTER-SHARE-client-0 </span></font> -v
0x000000000000000000000000 /<font face="Courier New"><span style="font-size: 11px;">export-share-1</span></font><br>
setfattr -n trusted.afr.<font face="Courier New"><span style="font-size: 11px;">GLUSTER-SHARE-client-1 </span></font> -v
0x000000000000000000000000 /<font face="Courier New"><span style="font-size: 11px;">export-share-1</span></font><br>
<br>
setfattr -n trusted.afr.<font face="Courier New"><span style="font-size: 11px;">GLUSTER-SHARE-client-0 </span></font> -v
0x000000000000000000000000 /<font face="Courier New"><span style="font-size: 11px;">export-share-2</span></font><br>
setfattr -n trusted.afr.<font face="Courier New"><span style="font-size: 11px;">GLUSTER-SHARE-client-1 </span></font> -v
0x000000000000000000000000 /<font face="Courier New"><span style="font-size: 11px;">export-share-2</span></font><br>
<br>
Thanks,<br>
Ravi<br>
<br>
<blockquote cite="mid:etPan.52925eec.515f007c.154@Rajka.local" type="cite">
<div id="bloop_sign_1385322684854820096">
<div style="orphans: auto; widows: auto;"><font face="Helvetica, Arial"><br></font></div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a>
</pre></blockquote>
<br>
</div></div></span></blockquote> <div id="bloop_sign_1385381127593515008"><div style="background-color: rgb(255, 255, 255); orphans: 2; widows: 2; line-height: normal; font-size: 11px; font-family: Helvetica;"><span class="Apple-style-span" style="font-family: monospace;">---</span></div><div style="background-color: rgb(255, 255, 255); font-family: Helvetica; font-size: 11px; line-height: normal; orphans: 2; widows: 2;"><span style="font-family: monospace;">Oudenhof 4c, 4191NW Geldermalsen, The Netherlands</span></div><div style="background-color: rgb(255, 255, 255); font-family: Helvetica; font-size: 11px; line-height: normal; orphans: 2; widows: 2;"><span class="Apple-style-span" style="font-family: monospace;">Web site and travel directions: </span><span class="Apple-style-span" style="font-family: monospace;"><a href="http://peercode.nl/">www.peercode.nl</a><br></span><span class="Apple-style-span" style="font-family: monospace;">Phone +31.88.0084124 :: Mobile +31.6.51298623</span></div></div></body></html>