<div dir="ltr">Can you get the volume profile dumps for both the runs and compare them?<div><br></div><div>Avati</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Sep 17, 2013 at 10:46 PM, kane <span dir="ltr"><<a href="mailto:stef_9k@163.com" target="_blank">stef_9k@163.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have already used "kernel oplocks = no" in the smb.conf, next is my original smb.conf file global settings:<br>
<div class="im">[global]<br>
workgroup = MYGROUP<br>
server string = DCS Samba Server<br>
log file = /var/log/samba/log.vfs<br>
max log size = 500000<br>
</div><div class="im"> aio read size = 262144<br>
aio write size = 262144<br>
aio write behind = true<br>
</div><div class="im"> security = user<br>
passdb backend = tdbsam<br>
load printers = yes<br>
cups options = raw<br>
read raw = yes<br>
write raw = yes<br>
max xmit = 262144<br>
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144<br>
</div># max protocol = SMB2<br>
<div class="im"> kernel oplocks = no<br>
stat cache = no<br>
<br>
</div>thank you<br>
<span class="HOEnZb"><font color="#888888">-Kane<br>
$B:_(B 2013-9-18$B!$2<8a(B1:38$B!$(BAnand Avati <<a href="mailto:avati@redhat.com">avati@redhat.com</a>> $B<LF;!'(B<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
> On 9/17/13 10:34 PM, kane wrote:<br>
>> Hi Anand,<br>
>><br>
>> I use 2 gluster server , this is my volume info:<br>
>> Volume Name: soul<br>
>> Type: Distribute<br>
>> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6<br>
>> Status: Started<br>
>> Number of Bricks: 2<br>
>> Transport-type: tcp<br>
>> Bricks:<br>
>> Brick1: 192.168.101.133:/dcsdata/d0<br>
>> Brick2: 192.168.101.134:/dcsdata/d0<br>
>><br>
>> each brick use a raid 5 logic disk with 8*2TSATA hdd.<br>
>><br>
>> smb.conf:<br>
>> [gvol]<br>
>> comment = For samba export of volume test<br>
>> vfs objects = glusterfs<br>
>> glusterfs:volfile_server = localhost<br>
>> glusterfs:volume = soul<br>
>> path = /<br>
>> read only = no<br>
>> guest ok = yes<br>
>><br>
>> this my testparm result:<br>
>> [global]<br>
>> workgroup = MYGROUP<br>
>> server string = DCS Samba Server<br>
>> log file = /var/log/samba/log.vfs<br>
>> max log size = 500000<br>
>> max xmit = 262144<br>
>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144<br>
>> SO_SNDBUF=262144<br>
>> stat cache = No<br>
>> kernel oplocks = No<br>
>> idmap config * : backend = tdb<br>
>> aio read size = 262144<br>
>> aio write size = 262144<br>
>> aio write behind = true<br>
>> cups options = raw<br>
>><br>
>> in client mount the smb share with cifs to dir /mnt/vfs,<br>
>> then use iozone executed in the cifs mount dir "/mnt/vfs":<br>
>> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>
>> File size set to 10485760 KB<br>
>> Record Size 128 KB<br>
>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>
>> Output is in Kbytes/sec<br>
>> Time Resolution = 0.000001 seconds.<br>
>> Processor cache size set to 1024 Kbytes.<br>
>> Processor cache line size set to 32 bytes.<br>
>> File stride size set to 17 * record size.<br>
>> Throughput test with 4 processes<br>
>> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>
>><br>
>> Children see throughput for 4 initial writers = 534315.84 KB/sec<br>
>> Parent sees throughput for 4 initial writers = 519428.83 KB/sec<br>
>> Min throughput per process = 133154.69 KB/sec<br>
>> Max throughput per process = 134341.05 KB/sec<br>
>> Avg throughput per process = 133578.96 KB/sec<br>
>> Min xfer = 10391296.00 KB<br>
>><br>
>> Children see throughput for 4 rewriters = 536634.88 KB/sec<br>
>> Parent sees throughput for 4 rewriters = 522618.54 KB/sec<br>
>> Min throughput per process = 133408.80 KB/sec<br>
>> Max throughput per process = 134721.36 KB/sec<br>
>> Avg throughput per process = 134158.72 KB/sec<br>
>> Min xfer = 10384384.00 KB<br>
>><br>
>> Children see throughput for 4 readers = 77403.54 KB/sec<br>
>> Parent sees throughput for 4 readers = 77402.86 KB/sec<br>
>> Min throughput per process = 19349.42 KB/sec<br>
>> Max throughput per process = 19353.42 KB/sec<br>
>> Avg throughput per process = 19350.88 KB/sec<br>
>> Min xfer = 10483712.00 KB<br>
>><br>
>> Children see throughput for 4 re-readers = 77424.40 KB/sec<br>
>> Parent sees throughput for 4 re-readers = 77423.89 KB/sec<br>
>> Min throughput per process = 19354.75 KB/sec<br>
>> Max throughput per process = 19358.50 KB/sec<br>
>> Avg throughput per process = 19356.10 KB/sec<br>
>> Min xfer = 10483840.00 KB<br>
>><br>
>> then the use the same command test in the dir mounted with glister fuse:<br>
>> File size set to 10485760 KB<br>
>> Record Size 128 KB<br>
>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>
>> Output is in Kbytes/sec<br>
>> Time Resolution = 0.000001 seconds.<br>
>> Processor cache size set to 1024 Kbytes.<br>
>> Processor cache line size set to 32 bytes.<br>
>> File stride size set to 17 * record size.<br>
>> Throughput test with 4 processes<br>
>> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>
>><br>
>> Children see throughput for 4 initial writers = 887534.72 KB/sec<br>
>> Parent sees throughput for 4 initial writers = 848830.39 KB/sec<br>
>> Min throughput per process = 220140.91 KB/sec<br>
>> Max throughput per process = 223690.45 KB/sec<br>
>> Avg throughput per process = 221883.68 KB/sec<br>
>> Min xfer = 10319360.00 KB<br>
>><br>
>> Children see throughput for 4 rewriters = 892774.92 KB/sec<br>
>> Parent sees throughput for 4 rewriters = 871186.83 KB/sec<br>
>> Min throughput per process = 222326.44 KB/sec<br>
>> Max throughput per process = 223970.17 KB/sec<br>
>> Avg throughput per process = 223193.73 KB/sec<br>
>> Min xfer = 10431360.00 KB<br>
>><br>
>> Children see throughput for 4 readers = 605889.12 KB/sec<br>
>> Parent sees throughput for 4 readers = 601767.96 KB/sec<br>
>> Min throughput per process = 143133.14 KB/sec<br>
>> Max throughput per process = 159550.88 KB/sec<br>
>> Avg throughput per process = 151472.28 KB/sec<br>
>> Min xfer = 9406848.00 KB<br>
>><br>
>> it shows much higher perf.<br>
>><br>
>> any places i did wrong?<br>
>><br>
>><br>
>> thank you<br>
>> -Kane<br>
>><br>
>> $B:_(B 2013-9-18$B!$2<8a(B1:19$B!$(BAnand Avati <<a href="mailto:avati@gluster.org">avati@gluster.org</a><br>
>> <mailto:<a href="mailto:avati@gluster.org">avati@gluster.org</a>>> $B<LF;!'(B<br>
>><br>
>>> How are you testing this? What tool are you using?<br>
>>><br>
>>> Avati<br>
>>><br>
>>><br>
>>> On Tue, Sep 17, 2013 at 9:02 PM, kane <<a href="mailto:stef_9k@163.com">stef_9k@163.com</a><br>
>>> <mailto:<a href="mailto:stef_9k@163.com">stef_9k@163.com</a>>> wrote:<br>
>>><br>
>>> Hi Vijay<br>
>>><br>
>>> I used the code in<br>
>>> <a href="https://github.com/gluster/glusterfs.git" target="_blank">https://github.com/gluster/glusterfs.git</a> with the lasted commit:<br>
>>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>
>>> Author: Anand Avati <<a href="mailto:avati@redhat.com">avati@redhat.com</a> <mailto:<a href="mailto:avati@redhat.com">avati@redhat.com</a>>><br>
>>> Date: Tue Sep 17 16:45:03 2013 -0700<br>
>>><br>
>>> Revert "cluster/distribute: Rebalance should also verify free<br>
>>> inodes"<br>
>>><br>
>>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>
>>><br>
>>> Realized soon after merging, ….<br>
>>><br>
>>> which include the patch you mentioned last time improve read perf,<br>
>>> written by Anand.<br>
>>><br>
>>> but the read perf was still slow:<br>
>>> write: 500MB/s<br>
>>> read: 77MB/s<br>
>>><br>
>>> while via fuse :<br>
>>> write 800MB/s<br>
>>> read 600MB/s<br>
>>><br>
>>> any advises?<br>
>>><br>
>>><br>
>>> Thank you.<br>
>>> -Kane<br>
>>><br>
>>> $B:_(B 2013-9-13$B!$2<8a(B10:37$B!$(Bkane <<a href="mailto:stef_9k@163.com">stef_9k@163.com</a><br>
>>> <mailto:<a href="mailto:stef_9k@163.com">stef_9k@163.com</a>>> $B<LF;!'(B<br>
>>><br>
>>>> Hi Vijay$B!$(B<br>
>>>><br>
>>>> thank you for post this message, i will try it soon<br>
>>>><br>
>>>> -kane<br>
>>>><br>
>>>><br>
>>>><br>
>>>> $B:_(B 2013-9-13$B!$2<8a(B9:21$B!$(BVijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a><br>
>>> <mailto:<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>>> $B<LF;!'(B<br>
>>>><br>
>>>>> On 09/13/2013 06:10 PM, kane wrote:<br>
>>>>>> Hi<br>
>>>>>><br>
>>>>>> We use gluster samba vfs test io,but the read performance via<br>
>>> vfs is<br>
>>>>>> half of write perfomance,<br>
>>>>>> but via fuse the read and write performance is almost the same.<br>
>>>>>><br>
>>>>>> this is our smb.conf:<br>
>>>>>> [global]<br>
>>>>>> workgroup = MYGROUP<br>
>>>>>> server string = DCS Samba Server<br>
>>>>>> log file = /var/log/samba/log.vfs<br>
>>>>>> max log size = 500000<br>
>>>>>> # use sendfile = true<br>
>>>>>> aio read size = 262144<br>
>>>>>> aio write size = 262144<br>
>>>>>> aio write behind = true<br>
>>>>>> min receivefile size = 262144<br>
>>>>>> write cache size = 268435456<br>
>>>>>> security = user<br>
>>>>>> passdb backend = tdbsam<br>
>>>>>> load printers = yes<br>
>>>>>> cups options = raw<br>
>>>>>> read raw = yes<br>
>>>>>> write raw = yes<br>
>>>>>> max xmit = 262144<br>
>>>>>> socket options = TCP_NODELAY IPTOS_LOWDELAY<br>
>>> SO_RCVBUF=262144<br>
>>>>>> SO_SNDBUF=262144<br>
>>>>>> kernel oplocks = no<br>
>>>>>> stat cache = no<br>
>>>>>><br>
>>>>>> any advises helpful?<br>
>>>>>><br>
>>>>><br>
>>>>> This patch has shown improvement in read performance with libgfapi:<br>
>>>>><br>
>>>>> <a href="http://review.gluster.org/#/c/5897/" target="_blank">http://review.gluster.org/#/c/5897/</a><br>
>>>>><br>
>>>>> Would it be possible for you to try this patch and check if it<br>
>>> improves performance in your case?<br>
>>>>><br>
>>>>> -Vijay<br>
>>>>><br>
>>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Gluster-users mailing list<br>
>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>><br>
>>> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
>>><br>
>>><br>
>><br>
><br>
> Please add 'kernel oplocks = no' in the [gvol] section and try again.<br>
><br>
> Avati<br>
><br>
<br>
<br>
</div></div></blockquote></div><br></div>