Hi Ramon,<div><br></div><div>Sorry for replying so late.</div><div><br></div><div>Our disks are Enterprese SATA disk spining at 7200rpm, so it is not the disk issue.</div><div>In these days, we traced the code into the fuse kernel module and found out that is the kernel fuse module</div><div>problem.</div><div><br></div><div>Here is some code fragment in fs/fuse/file.c of linux-3.4.87:</div><div><br></div><div>[any kind of fuse write]->fuse_send_write()->fuse_request_send()->queue_request()</div><div><br></div><div><div>static size_t fuse_send_write(struct fuse_req *req, struct file *file,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span> loff_t pos, size_t count, fl_owner_t owner)</div><div>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>.....</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>fuse_request_send(fc, req);</div></div><div><br></div><div><div>void fuse_request_send(struct fuse_conn *fc, struct fuse_req *req)</div><div>{</div><div> ......</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>queue_request(fc, req);</div><div> ......</div><div><span class="Apple-tab-span" style="white-space:pre"> </span><b>request_wait_answer(fc, req);</b></div><div> ......</div><div>}</div></div><div><br></div><div>The write operation will go into fuse_send_write() then call fuse_request_send(). In fuse_request_send(), we</div><div>can see after enqueuing the request, will get in the request_wait_answer() which will wait the write operation </div><div>finisded. This will obviously stop the pipeline and fall into stop-and-wait flow. That is why we can only get about </div><div>50MB/s in sequence writing one single file(according to the calculation in my former mail).</div><div><br></div><div>Now we are trying to use the libgfapi to check the speed.</div><div><br></div><div>Thanks for the advises.</div><div><br></div><div><br>On Friday, September 5, 2014, Ramon Selga <<a href="mailto:ramon.selga@gmail.com">ramon.selga@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Jaden,<br>
<br>
Sorry, from your subject I misunderstood your setup.<br>
<br>
In a pure distributed volume, each file goes to a brick and only
that brick. That unique brick is computed with the elastic hash
algorithm. <br>
<br>
If you get near wire speed, 1Gbps or 120MBps, when writing several
files at once but only roughly half speed writing only one file,
maybe each brick limits write speed: one "green" SATA disk running
at 5400rpm reaches 75MBps maximum writing big files sequentially
(Enterprise SATA disk spinning at 7200rpm reaches around 115MBps).<br>
<br>
Can you, please, explain which type of bricks do you have on each
server node?<br>
<br>
I'll try to emulate your setup and test it.<br>
<br>
Thank you! <br>
<br>
<br>
El 04/09/14 a les 03:20, Jaden Liang ha escrit:<br>
</div>
<blockquote type="cite">Hi Ramon,
<div><br>
</div>
<div>I am running on gluster FUSE client.</div>
<div><br>
</div>
<div>I maybe not stat clearly my testing environment. Let me
explain. The volume is configured on 2 servers. There is no
replication at all, just distributed volume. So I don't think it
is the replicated data issue. Actually, we can reach 100MB/s
when writing mutiple files at the same time.</div>
<div><br>
</div>
<div>Here is the volume info:</div>
<div><br>
</div>
<div># gluster volume info vs_vol_rep1</div>
<div> </div>
<div>Volume Name: vs_vol_rep1</div>
<div>Type: Distribute</div>
<div>Volume ID: cd137b57-e98a-4755-939a-7fc578f2a8c0</div>
<div>Status: Started</div>
<div>Number of Bricks: 10</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1:
host-001e67a3486c:/sf/data/vs/local/dfb0edaa-cfcb-4536-b5cb-a89aabaf8b4d/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick2:
host-001e67a3486c:/sf/data/vs/local/ac752388-1c2d-43a2-9396-7bedaf9abce2/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick3:
host-001e67a3486c:/sf/data/vs/local/6ef6c20e-ed59-4f3c-a354-a47caf11bbb0/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick4:
host-001e67a3486c:/sf/data/vs/local/4fa375da-265f-4436-8385-6af949581e16/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick5:
host-001e67a3486c:/sf/data/vs/local/184f174a-c5ee-45e8-8cbc-20ae518ad7b1/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick6:
host-001e67a3486c:/sf/data/vs/local/0a20eb9a-bba4-4cfd-be8f-542eac7a1f98/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick7:
host-001e67a3486c:/sf/data/vs/local/03648144-fec1-4471-9aa7-45fc2123867a/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick8:
host-001e67a349d4:/sf/data/vs/local/e7de2d40-6ebd-4867-b2a6-c19c669ecc83/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick9:
host-001e67a349d4:/sf/data/vs/local/896da577-cd03-42a0-8f5c-469759dd7f7b/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Brick10:
host-001e67a349d4:/sf/data/vs/local/6f274934-7e8b-4145-9f3a-bab549e2a95d/49ea070f-1480-4838-8182-95d1a6f17d81</div>
<div>Options Reconfigured:</div>
<div>diagnostics.latency-measurement: on</div>
<div>nfs.disable: on </div>
<div><br>
On Wednesday, September 3, 2014, Ramon Selga <<a href="javascript:_e(%7B%7D,'cvml','ramon.selga@gmail.com');" target="_blank">ramon.selga@gmail.com</a>>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi Jaden,<br>
<br>
May I ask some more info about your setup?<br>
<br>
Are you using NFS client or gluster FUSE client?<br>
<br>
If you are using NFS Client write data goes to one of
nodes of replica pair and that node sends write replica
data to the other node. If you are using one switch for
client and server connections and one 1GbE port on each
device, data received in the first node is re-sended to
the other node simultaneously and, in theory, you may
reach speeds closer to 100MBps.<br>
<br>
In case of gluster FUSE Client, write data goes
simultaneously to both server nodes using half bandwidth
for each of the client's 1GbE port because replica is done
by client side, that results on a writing speed around
50MBps(<60MBps).<br>
<br>
I hope this helps.<br>
<br>
El 03/09/14 a les 07:02, Jaden Liang ha escrit:<br>
</div>
<blockquote type="cite">Hi all,
<div><br>
</div>
<div>We did some more tests and analysis yesterday. It
looks like 50MB/s is the top theoretical speed in
replica 1 volume over 1Gbps network. GlusterFS write
128KB data once a block, then wait for return. The 128KB
data would cost about 1ms in 1Gbps network. And in the
server-side, it took about 800us to 1000us to write
128KB to the HDD and return. Plus some other 100us to
200us time elapsed. GlusterFS would take about 2ms-2.2ms
to finish a 128KB block data writing, which is about
50MB/s.</div>
<div><br>
</div>
<div>The question is that why don't glusterfs use pipeline
writting or reading to speed up this chatty process?<br>
<br>
On Tuesday, September 2, 2014, Jaden Liang <<a>jaden1q84@gmail.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,
gluster-devel and gluster-users team,
<div><br>
</div>
<div>We are running a performance test in a replica 1
volume and find out the single file sequence writing
performance only get about 50MB/s in a 1Gbps
Ethernet. However, if we test multiple
files sequence writing, the writing performance can
go up to 120MB/s which is the top speed of network. </div>
<div><br>
</div>
<div>We also tried to use the stat xlator to find out
where is the bottleneck of single file write
performance. Here is the stat data:</div>
<div><br>
</div>
<div>Client-side:</div>
<div>......</div>
<div>
<div>
vs_vol_rep1-client-8.latency.WRITE=total:21834371.000000us,
mean:2665.328491us, count:8192, max:4063475,
min:1849</div>
</div>
<div>......</div>
<div><br>
</div>
<div>Server-side:</div>
<div>......</div>
<div>
<div>/data/sdb1/brick1.latency.WRITE=total:6156857.000000us,
mean:751.569458us, count:8192, max:230864, min:611</div>
</div>
<div>......</div>
<div><br>
</div>
<div>Note that the test is write a 1GB single file
sequentially to a replica 1 volume through 1Gbps
Ethernet network. </div>
<div><br>
</div>
<div>On the client-side, we can see there are 8192
write requests totally. Every request will write
128KB data. Total eclipsed time is 21834371us, about
21 seconds. The mean time of request is 2665us,
about 2.6ms which means it could only serves about
380 requests in 1 seconds. Plus there are other time
consuming like statfs, lookup, but those are not
major reasons.</div>
<div><br>
</div>
<div>On the server-side, the mean time of request is
751us include write data to HDD disk. So we think
that is not the major reason.</div>
<div><br>
</div>
<div>And we also modify some codes to do the statistic
of system epoll elapsed time. It only took about
20us from enqueue data to finish sent-out.</div>
<div><br>
</div>
<div>Now we are heading to the rpc mechanism in
glusterfs. Still, we think this issue maybe
encountered in gluster-devel or gluster-users teams.
Therefor, any suggestions would be grateful. Or have
anyone know such issue?</div>
<div><br>
</div>
<div>Best regards,</div>
<div>Jaden Liang</div>
<div>9/2/2014</div>
<br>
<br>
-- <br>
<div dir="ltr">
<div><font face="'courier new', monospace">Best
regards,</font></div>
<font face="'courier new', monospace">Jaden Liang</font><br>
</div>
<br>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-devel mailing list
<a>Gluster-devel@gluster.org</a>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote></div>