<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-2022-jp"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Anand,<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I use 2 gluster server , this is my volume info:</div><div><div>Volume Name: soul</div><div>Type: Distribute</div><div>Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6</div><div>Status: Started</div><div>Number of Bricks: 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.101.133:/dcsdata/d0</div><div>Brick2: 192.168.101.134:/dcsdata/d0</div></div><div><br></div><div>each brick use a raid 5 logic disk with 8*2TSATA hdd.</div><div><br></div><div>smb.conf:</div><div><div>[gvol]</div><div> comment = For samba export of volume test</div><div> vfs objects = glusterfs</div><div> glusterfs:volfile_server = localhost</div><div> glusterfs:volume = soul</div><div> path = /</div><div> read only = no</div><div> guest ok = yes</div></div><div><br></div><div>this my testparm result:</div><div><div>[global]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>workgroup = MYGROUP</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>server string = DCS Samba Server</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>log file = /var/log/samba/log.vfs</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>max log size = 500000</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>max xmit = 262144</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>stat cache = No</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>kernel oplocks = No</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>idmap config * : backend = tdb</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>aio read size = 262144</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>aio write size = 262144</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>aio write behind = true</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>cups options = raw</div></div><div><br></div><div>in client mount the smb share with cifs to dir /mnt/vfs, </div><div>then use iozone executed in the cifs mount dir "/mnt/vfs":</div><div>$ ./iozone -s 10G -r 128k -i0 -i1 -t 4</div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>File size set to 10485760 KB</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Record Size 128 KB</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Output is in Kbytes/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Time Resolution = 0.000001 seconds.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Processor cache size set to 1024 Kbytes.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Processor cache line size set to 32 bytes.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>File stride size set to 17 * record size.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Throughput test with 4 processes</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Each process writes a 10485760 Kbyte file in 128 Kbyte records</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 initial writers <span class="Apple-tab-span" style="white-space:pre"> </span>= 534315.84 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 initial writers <span class="Apple-tab-span" style="white-space:pre"> </span>= 519428.83 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 133154.69 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 134341.05 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 133578.96 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 10391296.00 KB</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 rewriters <span class="Apple-tab-span" style="white-space:pre"> </span>= 536634.88 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 rewriters <span class="Apple-tab-span" style="white-space:pre"> </span>= 522618.54 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 133408.80 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 134721.36 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 134158.72 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 10384384.00 KB</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 readers <span class="Apple-tab-span" style="white-space:pre"> </span>= 77403.54 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 readers <span class="Apple-tab-span" style="white-space:pre"> </span>= 77402.86 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 19349.42 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 19353.42 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 19350.88 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 10483712.00 KB</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 re-readers <span class="Apple-tab-span" style="white-space:pre"> </span>= 77424.40 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 re-readers <span class="Apple-tab-span" style="white-space:pre"> </span>= 77423.89 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 19354.75 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 19358.50 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 19356.10 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 10483840.00 KB</div></div><div><div><div><br></div><div>then the use the same command test in the dir mounted with glister fuse:</div><div><div><span class="Apple-tab-span" style="white-space:pre"> </span>File size set to 10485760 KB</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Record Size 128 KB</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Output is in Kbytes/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Time Resolution = 0.000001 seconds.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Processor cache size set to 1024 Kbytes.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Processor cache line size set to 32 bytes.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>File stride size set to 17 * record size.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Throughput test with 4 processes</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Each process writes a 10485760 Kbyte file in 128 Kbyte records</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 initial writers <span class="Apple-tab-span" style="white-space:pre"> </span>= 887534.72 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 initial writers <span class="Apple-tab-span" style="white-space:pre"> </span>= 848830.39 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 220140.91 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 223690.45 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 221883.68 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 10319360.00 KB</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 rewriters <span class="Apple-tab-span" style="white-space:pre"> </span>= 892774.92 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 rewriters <span class="Apple-tab-span" style="white-space:pre"> </span>= 871186.83 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 222326.44 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 223970.17 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 223193.73 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 10431360.00 KB</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Children see throughput for 4 readers <span class="Apple-tab-span" style="white-space:pre"> </span>= 605889.12 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Parent sees throughput for 4 readers <span class="Apple-tab-span" style="white-space:pre"> </span>= 601767.96 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 143133.14 KB/sec </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Max throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 159550.88 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Avg throughput per process <span class="Apple-tab-span" style="white-space:pre"> </span>= 151472.28 KB/sec</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Min xfer <span class="Apple-tab-span" style="white-space:pre"> </span>= 9406848.00 KB</div></div><div><br></div><div>it shows much higher perf.</div><div><br></div><div>any places i did wrong?</div><div><br></div><div><br></div><div>thank you</div><div>-Kane</div><div><br></div><div>$B:_(B 2013-9-18$B!$2<8a(B1:19$B!$(BAnand Avati <<a href="mailto:avati@gluster.org">avati@gluster.org</a>> $B<LF;!'(B</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">How are you testing this? What tool are you using?<div><br></div><div>Avati</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 17, 2013 at 9:02 PM, kane <span dir="ltr"><<a href="mailto:stef_9k@163.com" target="_blank">stef_9k@163.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Vijay<br>
<br>
I used the code in <a href="https://github.com/gluster/glusterfs.git" target="_blank">https://github.com/gluster/glusterfs.git</a> with the lasted commit:<br>
commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>
Author: Anand Avati <<a href="mailto:avati@redhat.com">avati@redhat.com</a>><br>
Date: Tue Sep 17 16:45:03 2013 -0700<br>
<br>
Revert "cluster/distribute: Rebalance should also verify free inodes"<br>
<br>
This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>
<br>
Realized soon after merging, $B!D(B.<br>
<br>
which include the patch you mentioned last time improve read perf, written by Anand.<br>
<br>
but the read perf was still slow:<br>
write: 500MB/s<br>
read: 77MB/s<br>
<br>
while via fuse :<br>
write 800MB/s<br>
read 600MB/s<br>
<br>
any advises?<br>
<br>
<br>
Thank you.<br>
-Kane<br>
<br>
$B:_(B 2013-9-13$B!$2<8a(B10:37$B!$(Bkane <<a href="mailto:stef_9k@163.com">stef_9k@163.com</a>> $B<LF;!'(B<br>
<div class="HOEnZb"><div class="h5"><br>
> Hi Vijay$B!$(B<br>
><br>
> thank you for post this message, i will try it soon<br>
><br>
> -kane<br>
><br>
><br>
><br>
> $B:_(B 2013-9-13$B!$2<8a(B9:21$B!$(BVijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> $B<LF;!'(B<br>
><br>
>> On 09/13/2013 06:10 PM, kane wrote:<br>
>>> Hi<br>
>>><br>
>>> We use gluster samba vfs test io,but the read performance via vfs is<br>
>>> half of write perfomance,<br>
>>> but via fuse the read and write performance is almost the same.<br>
>>><br>
>>> this is our smb.conf:<br>
>>> [global]<br>
>>> workgroup = MYGROUP<br>
>>> server string = DCS Samba Server<br>
>>> log file = /var/log/samba/log.vfs<br>
>>> max log size = 500000<br>
>>> # use sendfile = true<br>
>>> aio read size = 262144<br>
>>> aio write size = 262144<br>
>>> aio write behind = true<br>
>>> min receivefile size = 262144<br>
>>> write cache size = 268435456<br>
>>> security = user<br>
>>> passdb backend = tdbsam<br>
>>> load printers = yes<br>
>>> cups options = raw<br>
>>> read raw = yes<br>
>>> write raw = yes<br>
>>> max xmit = 262144<br>
>>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144<br>
>>> SO_SNDBUF=262144<br>
>>> kernel oplocks = no<br>
>>> stat cache = no<br>
>>><br>
>>> any advises helpful?<br>
>>><br>
>><br>
>> This patch has shown improvement in read performance with libgfapi:<br>
>><br>
>> <a href="http://review.gluster.org/#/c/5897/" target="_blank">http://review.gluster.org/#/c/5897/</a><br>
>><br>
>> Would it be possible for you to try this patch and check if it improves performance in your case?<br>
>><br>
>> -Vijay<br>
>><br>
><br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></div></div></blockquote></div><br></div>
</blockquote></div><br></div></body></html>