<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Jaden,<br>
<br>
May I ask some more info about your setup?<br>
<br>
Are you using NFS client or gluster FUSE client?<br>
<br>
If you are using NFS Client write data goes to one of nodes of
replica pair and that node sends write replica data to the other
node. If you are using one switch for client and server
connections and one 1GbE port on each device, data received in the
first node is re-sended to the other node simultaneously and, in
theory, you may reach speeds closer to 100MBps.<br>
<br>
In case of gluster FUSE Client, write data goes simultaneously to
both server nodes using half bandwidth for each of the client's
1GbE port because replica is done by client side, that results on
a writing speed around 50MBps(<60MBps).<br>
<br>
I hope this helps.<br>
<br>
El 03/09/14 a les 07:02, Jaden Liang ha escrit:<br>
</div>
<blockquote
cite="mid:CAJ6qgSfCqsapeDwKuAUCAJnm_eE3w_04NTweSxZzN0=Dq5Gg0Q@mail.gmail.com"
type="cite">Hi all,
<div><br>
</div>
<div>We did some more tests and analysis yesterday. It looks like
50MB/s is the top theoretical speed in replica 1 volume over
1Gbps network. GlusterFS write 128KB data once a block, then
wait for return. The 128KB data would cost about 1ms in 1Gbps
network. And in the server-side, it took about 800us to 1000us
to write 128KB to the HDD and return. Plus some other 100us to
200us time elapsed. GlusterFS would take about 2ms-2.2ms to
finish a 128KB block data writing, which is about 50MB/s.</div>
<div><br>
</div>
<div>The question is that why don't glusterfs use pipeline
writting or reading to speed up this chatty process?<br>
<br>
On Tuesday, September 2, 2014, Jaden Liang <<a
moz-do-not-send="true" href="mailto:jaden1q84@gmail.com">jaden1q84@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,
gluster-devel and gluster-users team,
<div><br>
</div>
<div>We are running a performance test in a replica 1 volume
and find out the single file sequence writing performance
only get about 50MB/s in a 1Gbps Ethernet. However, if we
test multiple files sequence writing, the writing
performance can go up to 120MB/s which is the top speed of
network. </div>
<div><br>
</div>
<div>We also tried to use the stat xlator to find out where is
the bottleneck of single file write performance. Here is the
stat data:</div>
<div><br>
</div>
<div>Client-side:</div>
<div>......</div>
<div>
<div>
vs_vol_rep1-client-8.latency.WRITE=total:21834371.000000us,
mean:2665.328491us, count:8192, max:4063475, min:1849</div>
</div>
<div>......</div>
<div><br>
</div>
<div>Server-side:</div>
<div>......</div>
<div>
<div>/data/sdb1/brick1.latency.WRITE=total:6156857.000000us,
mean:751.569458us, count:8192, max:230864, min:611</div>
</div>
<div>......</div>
<div><br>
</div>
<div>Note that the test is write a 1GB single file
sequentially to a replica 1 volume through 1Gbps Ethernet
network. </div>
<div><br>
</div>
<div>On the client-side, we can see there are 8192 write
requests totally. Every request will write 128KB data. Total
eclipsed time is 21834371us, about 21 seconds. The mean time
of request is 2665us, about 2.6ms which means it could only
serves about 380 requests in 1 seconds. Plus there are other
time consuming like statfs, lookup, but those are not major
reasons.</div>
<div><br>
</div>
<div>On the server-side, the mean time of request is 751us
include write data to HDD disk. So we think that is not the
major reason.</div>
<div><br>
</div>
<div>And we also modify some codes to do the statistic of
system epoll elapsed time. It only took about 20us from
enqueue data to finish sent-out.</div>
<div><br>
</div>
<div>Now we are heading to the rpc mechanism in glusterfs.
Still, we think this issue maybe encountered in
gluster-devel or gluster-users teams. Therefor, any
suggestions would be grateful. Or have anyone know such
issue?</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Jaden Liang</div>
<div>9/2/2014</div>
<br>
<br>
-- <br>
<div dir="ltr">
<div><font face="'courier new', monospace">Best regards,</font></div>
<font face="'courier new', monospace">Jaden Liang</font><br>
</div>
<br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
</blockquote>
<br>
</body>
</html>