<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Just FYI, I have errors similar to Remi, on 3.2.0 for certain dirs., same type of dist/repl setup.. Input/Output errors, unable to self-heal, split-brain. Except mine say it’s a permissions problem, not size differences, even though the permissions on both backends for the files/dirs show the same.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Only way I fixed mine was removing the offending directory and recreating it.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-Tony<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>---------------------------<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>Manager, IT Operations<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>Format Dynamics, Inc.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>P: 303-228-7327<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'>F: 303-228-7305<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'><a href="mailto:abiacco@formatdynamics.com">abiacco@formatdynamics.com</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New";color:#1F497D'><a href="http://www.formatdynamics.com/">http://www.formatdynamics.com</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Remi Broemeling<br><b>Sent:</b> Tuesday, May 17, 2011 9:33 AM<br><b>To:</b> gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] Rebuild Distributed/Replicated Setup<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>Hi Pranith. Sure, here is a pastebin sampling of logs from one of the hosts: <a href="http://pastebin.com/1U1ziwjC">http://pastebin.com/1U1ziwjC</a><o:p></o:p></p><div><p class=MsoNormal>On Mon, May 16, 2011 at 20:48, Pranith Kumar. Karampuri <<a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal>hi Remi,<br> Would it be possible to post the logs on the client, so that we can find what issue you are running into.<br><br>Pranith<o:p></o:p></p><div><div><p class=MsoNormal style='margin-bottom:12.0pt'>----- Original Message -----<br>From: "Remi Broemeling" <<a href="mailto:remi@goclio.com">remi@goclio.com</a>><br>To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>Sent: Monday, May 16, 2011 10:47:33 PM<br>Subject: [Gluster-users] Rebuild Distributed/Replicated Setup<br><br><br>Hi,<br><br>I've got a distributed/replicated GlusterFS v3.1.2 (installed via RPM) setup across two servers (web01 and web02) with the following vol config:<br><br>volume shared-application-data-client-0<br>type protocol/client<br>option remote-host web01<br>option remote-subvolume /var/glusterfs/bricks/shared<br>option transport-type tcp<br>option ping-timeout 5<br>end-volume<br><br>volume shared-application-data-client-1<br>type protocol/client<br>option remote-host web02<br>option remote-subvolume /var/glusterfs/bricks/shared<br>option transport-type tcp<br>option ping-timeout 5<br>end-volume<br><br>volume shared-application-data-replicate-0<br>type cluster/replicate<br>subvolumes shared-application-data-client-0 shared-application-data-client-1<br>end-volume<br><br>volume shared-application-data-write-behind<br>type performance/write-behind<br>subvolumes shared-application-data-replicate-0<br>end-volume<br><br>volume shared-application-data-read-ahead<br>type performance/read-ahead<br>subvolumes shared-application-data-write-behind<br>end-volume<br><br>volume shared-application-data-io-cache<br>type performance/io-cache<br>subvolumes shared-application-data-read-ahead<br>end-volume<br><br>volume shared-application-data-quick-read<br>type performance/quick-read<br>subvolumes shared-application-data-io-cache<br>end-volume<br><br>volume shared-application-data-stat-prefetch<br>type performance/stat-prefetch<br>subvolumes shared-application-data-quick-read<br>end-volume<br><br>volume shared-application-data<br>type debug/io-stats<br>subvolumes shared-application-data-stat-prefetch<br>end-volume<br><br>In total, four servers mount this via GlusterFS FUSE. For whatever reason (I'm really not sure why), the GlusterFS filesystem has run into a bit of split-brain nightmare (although to my knowledge an actual split brain situation has never occurred in this environment), and I have been getting solidly corrupted issues across the filesystem as well as complaints that the filesystem cannot be self-healed.<br><br>What I would like to do is completely empty one of the two servers (here I am trying to empty server web01), making the other one (in this case web02) the authoritative source for the data; and then have web01 completely rebuild it's mirror directly from web02.<br><br>What's the easiest/safest way to do this? Is there a command that I can run that will force web01 to re-initialize it's mirror directly from web02 (and thus completely eradicate all of the split-brain errors and data inconsistencies)?<br><br>Thanks!<br>--<br><br>Remi Broemeling<br>System Administrator<br>Clio - Practice Management Simplified<br><a href="tel:1-888-858-2546">1-888-858-2546</a> x(2^5) | <a href="mailto:remi@goclio.com">remi@goclio.com</a><br><a href="http://www.goclio.com" target="_blank">www.goclio.com</a> | blog | twitter | facebook<o:p></o:p></p></div></div><p class=MsoNormal>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><o:p></o:p></p></div><p class=MsoNormal><br><br clear=all><br>-- <o:p></o:p></p><div><p class=MsoNormal>Remi Broemeling<o:p></o:p></p></div><div><p class=MsoNormal>System Administrator<o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Clio - Practice Management Simplified<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>1-888-858-2546 x(2^5) | <a href="mailto:remi@goclio.com" target="_blank">remi@goclio.com</a><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><a href="http://www.goclio.com/" target="_blank">www.goclio.com</a> | <a href="http://www.goclio.com/blog" target="_blank">blog</a> | <a href="http://www.twitter.com/goclio" target="_blank">twitter</a> | <a href="http://www.facebook.com/goclio" target="_blank">facebook</a><o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>