<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 11/25/2013 01:47 AM, Mark Ruys
wrote:<br>
</div>
<blockquote cite="mid:etPan.52925eec.515f007c.154@Rajka.local"
type="cite">
<div id="bloop_customfont"
style="font-family:Helvetica,Arial;font-size:13px; color:
rgba(0,0,0,1.0); margin: 0px; line-height: auto;">So I decided
to bite the bullet and upgraded from 3.3 to 3.4. Somehow this
was a painful proces for me (the glusterfs daemon refused to
start), so I decided to configure our Gluster pool from scratch.
Everything seems to work nicely, except for the self-heal
daemon. In the logs, I get every 10 minutes the following line:</div>
<div id="bloop_customfont"
style="font-family:Helvetica,Arial;font-size:13px; color:
rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br>
</div>
<div id="bloop_customfont" style="color: rgb(0, 0, 0); margin:
0px;">
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">[2013-11-24 19:50:34.495204] E
[afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
0-GLUSTER-SHARE-replicate-0: Unable to self-heal contents
of '<gfid:00000000-0000-0000-0000-000000000001>'
(possible split-brain). Please delete the file from all
but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [
2 0 ] ]</span></font></p>
</div>
<div id="bloop_sign_1385322684854820096"><span
style="font-family:helvetica,arial;font-size:13px"></span>
<div style="orphans: 2; widows: 2; line-height: normal;
font-size: 11px; font-family: Helvetica;"><br>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial">I’ve removed and recreated
the .glusterfs/00/00/00000000-0000-0000-0000-000000000001,
but that doesn’t seem to make a difference. </font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial"><br>
</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial">How to fix the self-heal daemon?</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial"><br>
</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial">Mark</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial"><br>
</font></div>
<div style="orphans: auto; widows: auto;">
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;"># find . -name
00000000-0000-0000-0000-000000000001 -ls</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">1447202 0 ---------- 2
root root 0 Nov 23 22:35
./export-share-1/.glusterfs/indices/xattrop/00000000-0000-0000-0000-000000000001</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">1319116 0 lrwxrwxrwx 1
root root 8 Nov 23 22:35
./export-share-1/.glusterfs/00/00/00000000-0000-0000-0000-000000000001
-> ../../..</span></font></p>
<p style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</p>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial">Brick 1:</font></div>
<div style="orphans: auto; widows: auto;">
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;"># getfattr -m . -d -e hex
export-share-1</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;"># file: export-share-1</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-0=0x000000000000000000000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-1=0x000000000000000200000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.gfid=0x00000000000000000000000000000001</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.quota.dirty=0x3000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.quota.size=0x0000000000000000</span></font></p>
<p style="margin: 0px;">
</p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.volume-id=0xe6eb05aabe3b456cbf3027275faa529c</span></font></p>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial"><br>
</font></div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial">Brick 2:</font></div>
<div style="orphans: auto; widows: auto;">
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;"># getfattr -m . -d -e hex
export-share-2</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;"># file: export-share-2</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-0=0x000000000000000200000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.afr.GLUSTER-SHARE-client-1=0x000000000000000000000000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.gfid=0x00000000000000000000000000000001</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.dht=0x000000010000000000000000ffffffff</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.quota.dirty=0x3000</span></font></p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.quota.size=0x0000000000000000</span></font></p>
<p style="margin: 0px;">
</p>
<p style="margin: 0px;"><font face="Courier New"><span
style="font-size: 11px;">trusted.glusterfs.volume-id=0xe6eb05aabe3b456cbf3027275faa529c</span></font></p>
</div>
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial"><br>
</font></div>
</div>
</blockquote>
From the afr extended attributes, it seems you have hit a
metadata-split-brain of the top level (brick) directory (having gfid
01). If you are you able to perform I/O on all files from the mount
point without error (EIO) and the file contents are identical on
both the bricks (check with md5sum), you could safely clear the afr
extended attributes of the bricks:<br>
<br>
setfattr -n trusted.afr.<font face="Courier New"><span
style="font-size: 11px;">GLUSTER-SHARE-client-0 </span></font>
-v 0x000000000000000000000000 /<font face="Courier New"><span
style="font-size: 11px;">export-share-1</span></font><br>
setfattr -n trusted.afr.<font face="Courier New"><span
style="font-size: 11px;">GLUSTER-SHARE-client-1 </span></font>
-v 0x000000000000000000000000 /<font face="Courier New"><span
style="font-size: 11px;">export-share-1</span></font><br>
<br>
setfattr -n trusted.afr.<font face="Courier New"><span
style="font-size: 11px;">GLUSTER-SHARE-client-0 </span></font>
-v 0x000000000000000000000000 /<font face="Courier New"><span
style="font-size: 11px;">export-share-2</span></font><br>
setfattr -n trusted.afr.<font face="Courier New"><span
style="font-size: 11px;">GLUSTER-SHARE-client-1 </span></font>
-v 0x000000000000000000000000 /<font face="Courier New"><span
style="font-size: 11px;">export-share-2</span></font><br>
<br>
Thanks,<br>
Ravi<br>
<br>
<blockquote cite="mid:etPan.52925eec.515f007c.154@Rajka.local"
type="cite">
<div id="bloop_sign_1385322684854820096">
<div style="orphans: auto; widows: auto;"><font face="Helvetica,
Arial"><br>
</font></div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>