<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>I have four servers, absolutely identical, connected to the same switches. One interface is on a 100Mb switch, the other is on a 1Gb switch. I access the nodes via the 100Mb port, gluster is configured on the 1Gb port. The nodes are all loaded with Scientific Linux 6.3, Virtualization Host, with glusterfs-3.2.7 from EPEL. The nodes are 2-socket quad-core AMD (so 8 cores total) servers with 6x 300GB internal drives. I’m using LVM on top of h/w RAID0, and have a 1.5TB xfs brick on each node. I have libvirtd running, but no VMs created yet.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I initially configured each pair of servers as a separate cluster with a 1x2 replicated volume. Mount the volumes as glusterfs from localhost, and dd tests gives me ~90MB/s.. pretty decent for 1Gb network (max 125MB/s). So, tear that all down, and join all four nodes together, and create a 2x2 distributed-replicated volume. Now is where it gets interesting. First node dd test is consistent. Second node dd test is half-speed. Third node dd test is back to full speed. Fourth node dd test is back to half-speed. So when I look in the bricks directly, I see that the nodes that were slower had their file in a brick that was not part of the replica they were hosting.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>For example..<o:p></o:p></p><p class=MsoNormal>gluster volume create vol1 replica 2 transport tcp server1:/brick1 server2:/brick2 server3:/brick3 server4:/brick4<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>server1:/brick1 and server2:/brick2 are the first replica pair<o:p></o:p></p><p class=MsoNormal>server3:/brick3 and server4:/brick4 are the second replica pair<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>server1.. file1 goes into brick1/brick2 - fast<o:p></o:p></p><p class=MsoNormal>server2.. file2 goes into brick3/brick4 - slow<o:p></o:p></p><p class=MsoNormal>server3.. file3 goes into brick3/brick4 - fast<o:p></o:p></p><p class=MsoNormal>server4.. file4 goes into brick1/brick2 - slow<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So I delete that volume, and create another..<o:p></o:p></p><p class=MsoNormal>gluster volume create vol2 replica 2 transport tcp server2:/brick2 server3:/brick3 server4:/brick4 server1:/brick1<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>server2:/brick2 and server3:/brick3 are the first replica pair<o:p></o:p></p><p class=MsoNormal>server4:/brick4 and server1:/brick1 are the second replica pair<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>server2.. file2 goes into brick2/brick3 - fast<o:p></o:p></p><p class=MsoNormal>server3.. file3 goes into brick2/brick3 - fast<o:p></o:p></p><p class=MsoNormal>server4.. file4 goes into brick4/brick1 - fast<o:p></o:p></p><p class=MsoNormal>server1.. file1 goes into brick4/brick1 - fast<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>So now I’m like seriously WTF. So I remove all output files, and try four consecutive tests from the same node with output file1, file2, file3, file4. And sure enough two of them are fast and two are slow, and the fast ones are placed in “its” replica pair and the slow ones are in the other. And I notice that every time I delete them, the files get created in the same replica pair each time, no matter what order I create them. I’ve tried this with nfs mounts also (instead of glusterfs), and the results are the same.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Has anyone seen this behavior before? Is this a known issue or mis-configuration?<o:p></o:p></p></div></body></html>