<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Sprechblasentext Zchn";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.E-MailFormatvorlage17
{mso-style-type:personal-compose;
font-family:"Arial","sans-serif";
font-variant:normal !important;
color:black;
text-transform:none;
font-weight:normal;
font-style:normal;
text-decoration:none none;
vertical-align:baseline;}
span.SprechblasentextZchn
{mso-style-name:"Sprechblasentext Zchn";
mso-style-priority:99;
mso-style-link:Sprechblasentext;
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:289089337;
mso-list-type:hybrid;
mso-list-template-ids:-1248556190 864189428 67567619 67567621 67567617 67567619 67567621 67567617 67567619 67567621;}
@list l0:level1
{mso-level-start-at:5;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Arial","sans-serif";
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Hi!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>I just gave GlusterFS a try and experienced two problems. First some background:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>I want to set up a file server with synchronous replication between branch offices, similar to Windows DFS-Replication. The goal is _not_ high-availability or cluster-scaleout, but just having all files locally available at each branch office.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>To test GlusterFS, I installed two virtual machines in different locations, Ubuntu 12.04, with the GlusterFS 3.3 packages from the PPA.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>Both machines shall be server and and client, and export the GlusterFS volume via samba.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>I set up a file system in replica mode according to the quick start guide (except that I used ext4 instead of xfs for the brick, I had bad experiences with xfs)<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>I mounted the filesystem on both machines as localhost:/gv0, and shared the mount via samba.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>At first it seemed to work fine (Copying files from/to the share, files appear instantly on the other host), until I did some robustness tests:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>I severed the connection between the two hosts to provoke a split-brain scenario, just to see what happens. I expected both hosts to work, but on one of them the GlusterFS volume froze. After restarting the glusterfs-server service, it came back.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>Then I intentionally created a conflicting file on each host.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>After reconnecting the host, I got “Input/Output error” on both the conflicting file and the volume root inode. I found this <a href="http://blog.oneiroi.co.uk/linux/gluster-resolving-a-split-brain-in-a-replicated-setup/">http://blog.oneiroi.co.uk/linux/gluster-resolving-a-split-brain-in-a-replicated-setup/</a> which fixed it for me, but having to manually fix the filesystem whenever a branch office link goes down does not feel very trustworthy. Is there some auto-conflict-resolving feature (last one wins, or renaming conflicting files)?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>Then I took a look at the performance, and copied an ISO image (~ 700MB) to the filesystem. Worked fine, until I tried to md5sum it from both hosts. While the one node took a few seconds (what I expected), the other one took several minutes. Then I found out that it read the file over the WAN link from the distant host instead from itself. It should have had time enough (one hour) to replicate the file across both hosts…<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>(By the way, I also wanted to try geo-replication (which might suffice for my needs with a tight-enough schedule), but I was not able to create a volume with only one brick…<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>So I wonder: What did I do wrong?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>Thanks<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'>Martin<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#404040;mso-fareast-language:DE'><o:p> </o:p></span></p></div></body></html>