<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Found the BZ <a href="https://bugzilla.redhat.com/show_bug.cgi?id=960190" style="font-family:arial" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=960190</a> - so I restarted one of the volumes and it seems to have restarted the all daemons again.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Self heal started again, but I seem to have split-brain issues everywhere. There's over 100 different entries on each node, what's the best way to restore this now? Short of having to manually go through and delete 200+ files. It looks like a full split brain as the file sizes on the different nodes are out of balance by about 100GB or so.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Any suggestions would be much appreciated!</div><div class="gmail_default" style="font-family:tahoma,sans-serif">
<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Cheers.</div>
<div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 24, 2013 at 10:32 PM, Andrew Lau <span dir="ltr"><<a href="mailto:andrew@andrewklau.com" target="_blank">andrew@andrewklau.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div style="font-family:tahoma,sans-serif">
Hi,</div><div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif">
Right now, I have a 2x1 replica. Ever since I had to reinstall one of the gluster servers, there's been issues with split-brain. The self-heal daemon doesn't seem to be running on either of the nodes.</div><div style="font-family:tahoma,sans-serif">
<br></div><div style="font-family:tahoma,sans-serif">To reinstall the gluster server (the original brick data was intact but the OS had to be reinstalled)</div><div style="font-family:tahoma,sans-serif">
- Reinstalled gluster</div><div style="font-family:tahoma,sans-serif">- Copied over the old uuid from backup</div><div style="font-family:tahoma,sans-serif">- gluster peer probe</div>
<div><span style="font-family:tahoma,sans-serif">- </span><font face="tahoma, sans-serif">gluster volume sync $othernode all</font></div><div><font face="tahoma, sans-serif">- mount -t glusterfs localhost:STORAGE /mnt</font></div>
<div><font face="tahoma, sans-serif">- find /mnt -noleaf -print0 | xargs --null stat >/dev/null 2>/var/log/glusterfs/mnt-selfheal.log</font></div><div style="font-family:tahoma,sans-serif">
<br></div><div style="font-family:tahoma,sans-serif">I let it resync and it was working fine, atleast so I thought. I just came back a few days later to see there's a miss match in the brick volumes. One is 50GB ahead of the other.</div>
<div style="font-family:tahoma,sans-serif"><br></div><div style="font-family:tahoma,sans-serif"># gluster volume heal STORAGE info</div><div><font face="tahoma, sans-serif">Status: self-heal-daemon is not running on 966456a1-b8a6-4ca8-9da7-d0eb96997cbe</font><br>
</div><div><font face="tahoma, sans-serif"><br></font></div><div><font face="tahoma, sans-serif">/var/log/gluster/glustershd.log doesn't seem to have any recent logs, only those from when the two original gluster servers were running.</font></div>
<div><font face="tahoma, sans-serif"><br></font></div><div><font face="tahoma, sans-serif"># gluster volume status</font></div><div><font face="tahoma, sans-serif"><br>
</font></div><div><font face="tahoma, sans-serif"><div>Self-heal Daemon on localhost<span style="white-space:pre-wrap"> </span>N/A<span style="white-space:pre-wrap"> </span>N<span style="white-space:pre-wrap"> </span>N/A</div>
<div><br></div><div>Any suggestions would be much appreciated!</div><div><br></div><div>Cheers</div><span><font color="#888888"><div>Andrew.</div></font></span></font></div>
</div>
</blockquote></div><br></div></div>