Sure,<br><br>These files are just a sampling -- a lot of other files are showing the same &quot;split-brain&quot; behaviour.<br><br><span style="font-family: courier new,monospace;">[14:42:45][root@web01:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809223185/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809223185/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAABQAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:45:15][root@web02:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809223185/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809223185/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAACOwAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:42:53][root@web01:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809223185/event.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809223185/event.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAADgAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:45:24][root@web02:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809223185/event.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809223185/event.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAGXQAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:43:02][root@web01:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809223635/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809223635/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAACgAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:45:28][root@web02:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809223635/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809223635/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAELQAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:43:39][root@web01:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809224061/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809224061/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAACQAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:45:32][root@web02:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809224061/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809224061/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAD+AAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:43:42][root@web01:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809224321/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809224321/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAACAAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:45:37][root@web02:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809224321/contact.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809224321/contact.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAERAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:43:45][root@web01:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809215319/event.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809215319/event.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAAAAAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAABwAAAAAAAAAA</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">[14:45:45][root@web02:/var/glusterfs/bricks/shared]# getfattr -d -m &quot;trusted.afr*&quot; agc/production/log/809215319/event.log</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;"># file: agc/production/log/809215319/event.log</span><br style="font-family: courier new,monospace;"><span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-0=0sAAAC/QAAAAAAAAAA</span><br style="font-family: courier new,monospace;">
<span style="font-family: courier new,monospace;">trusted.afr.shared-application-data-client-1=0sAAAAAAAAAAAAAAAA</span><br><br><div class="gmail_quote">On Wed, May 18, 2011 at 01:31, Pranith Kumar. Karampuri <span dir="ltr">&lt;<a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">hi Remi,<br>
      It seems the split-brain is detected on following files:<br>
/agc/production/log/809223185/contact.log<br>
/agc/production/log/809223185/event.log<br>
/agc/production/log/809223635/contact.log<br>
/agc/production/log/809224061/contact.log<br>
/agc/production/log/809224321/contact.log<br>
/agc/production/log/809215319/event.log<br>
<br>
Could you give the output of the following command for each file above on both the bricks in the replica pair.<br>
<br>
getxattr -d -m &quot;trusted.afr*&quot; &lt;filepath&gt;<br>
<br>
Thanks<br>
<div class="im">Pranith<br>
<br>
----- Original Message -----<br>
From: &quot;Remi Broemeling&quot; &lt;<a href="mailto:remi@goclio.com">remi@goclio.com</a>&gt;<br>
To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
</div><div><div></div><div class="h5">Sent: Tuesday, May 17, 2011 9:02:44 PM<br>
Subject: Re: [Gluster-users] Rebuild Distributed/Replicated Setup<br>
<br>
<br>
Hi Pranith. Sure, here is a pastebin sampling of logs from one of the hosts: <a href="http://pastebin.com/1U1ziwjC" target="_blank">http://pastebin.com/1U1ziwjC</a><br>
<br>
<br>
On Mon, May 16, 2011 at 20:48, Pranith Kumar. Karampuri &lt; <a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a> &gt; wrote:<br>
<br>
<br>
hi Remi,<br>
Would it be possible to post the logs on the client, so that we can find what issue you are running into.<br>
<br>
Pranith<br>
<br>
<br>
<br>
----- Original Message -----<br>
From: &quot;Remi Broemeling&quot; &lt; <a href="mailto:remi@goclio.com">remi@goclio.com</a> &gt;<br>
To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
Sent: Monday, May 16, 2011 10:47:33 PM<br>
Subject: [Gluster-users] Rebuild Distributed/Replicated Setup<br>
<br>
<br>
Hi,<br>
<br>
I&#39;ve got a distributed/replicated GlusterFS v3.1.2 (installed via RPM) setup across two servers (web01 and web02) with the following vol config:<br>
<br>
volume shared-application-data-client-0<br>
type protocol/client<br>
option remote-host web01<br>
option remote-subvolume /var/glusterfs/bricks/shared<br>
option transport-type tcp<br>
option ping-timeout 5<br>
end-volume<br>
<br>
volume shared-application-data-client-1<br>
type protocol/client<br>
option remote-host web02<br>
option remote-subvolume /var/glusterfs/bricks/shared<br>
option transport-type tcp<br>
option ping-timeout 5<br>
end-volume<br>
<br>
volume shared-application-data-replicate-0<br>
type cluster/replicate<br>
subvolumes shared-application-data-client-0 shared-application-data-client-1<br>
end-volume<br>
<br>
volume shared-application-data-write-behind<br>
type performance/write-behind<br>
subvolumes shared-application-data-replicate-0<br>
end-volume<br>
<br>
volume shared-application-data-read-ahead<br>
type performance/read-ahead<br>
subvolumes shared-application-data-write-behind<br>
end-volume<br>
<br>
volume shared-application-data-io-cache<br>
type performance/io-cache<br>
subvolumes shared-application-data-read-ahead<br>
end-volume<br>
<br>
volume shared-application-data-quick-read<br>
type performance/quick-read<br>
subvolumes shared-application-data-io-cache<br>
end-volume<br>
<br>
volume shared-application-data-stat-prefetch<br>
type performance/stat-prefetch<br>
subvolumes shared-application-data-quick-read<br>
end-volume<br>
<br>
volume shared-application-data<br>
type debug/io-stats<br>
subvolumes shared-application-data-stat-prefetch<br>
end-volume<br>
<br>
In total, four servers mount this via GlusterFS FUSE. For whatever reason (I&#39;m really not sure why), the GlusterFS filesystem has run into a bit of split-brain nightmare (although to my knowledge an actual split brain situation has never occurred in this environment), and I have been getting solidly corrupted issues across the filesystem as well as complaints that the filesystem cannot be self-healed.<br>

<br>
What I would like to do is completely empty one of the two servers (here I am trying to empty server web01), making the other one (in this case web02) the authoritative source for the data; and then have web01 completely rebuild it&#39;s mirror directly from web02.<br>

<br>
What&#39;s the easiest/safest way to do this? Is there a command that I can run that will force web01 to re-initialize it&#39;s mirror directly from web02 (and thus completely eradicate all of the split-brain errors and data inconsistencies)?<br>

<br>
Thanks!<br clear="all"></div></div></blockquote></div><br>-- <br><div>Remi Broemeling</div><div>System Administrator</div><span style="font-family:arial, sans-serif;font-size:13px;border-collapse:collapse"><div><font face="arial, helvetica, sans-serif">Clio - Practice Management Simplified</font></div>
<div><font face="arial, helvetica, sans-serif">1-888-858-2546 x(2^5) | <a href="mailto:remi@goclio.com" target="_blank">remi@goclio.com</a></font></div><div><font face="arial, helvetica, sans-serif"><a href="http://www.goclio.com/" target="_blank">www.goclio.com</a> | <a href="http://www.goclio.com/blog" target="_blank">blog</a> | <a href="http://www.twitter.com/goclio" target="_blank">twitter</a> | <a href="http://www.facebook.com/goclio" target="_blank">facebook</a></font></div>
</span><br>