Can you try "gluster volume set <volname> performance.read-ahead-page-count 64" or some value higher or lower?<div><br></div><div>Avati<br><br><div class="gmail_quote">On Thu, Feb 28, 2013 at 7:15 PM, Thomas Wakefield <span dir="ltr"><<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Good point, forgot to set a blcoksize, here are the redone dd tests:<div><div><br></div>
<div><div>[root@cpu_crew1 ~]# dd if=/shared/working/benchmark/test.cpucrew1 of=/dev/null bs=128k</div><div>40000+0 records in</div><div>40000+0 records out</div><div>5242880000 bytes (5.2 GB) copied, 65.4928 seconds, 80.1 MB/s</div>
<div>[root@cpu_crew1 ~]# dd if=/shared/working/benchmark/test.cpucrew1 of=/dev/null bs=1M</div><div>5000+0 records in</div><div>5000+0 records out</div><div>5242880000 bytes (5.2 GB) copied, 49.0907 seconds, 107 MB/s</div>
<div>[root@cpu_crew1 ~]# dd if=/shared/working/benchmark/test.cpucrew1 of=/dev/null bs=4M</div><div>1250+0 records in</div><div>1250+0 records out</div><div>5242880000 bytes (5.2 GB) copied, 44.5724 seconds, 118 MB/s</div>
<div><br></div><div>Still not impressive.</div><div><br></div><div>-Tom</div><div><br></div><div><br></div><div><div>On Feb 28, 2013, at 8:42 PM, Jeff Anderson-Lee <<a href="mailto:jonah@eecs.berkeley.edu" target="_blank">jonah@eecs.berkeley.edu</a>> wrote:</div>
<br><blockquote type="cite">
<div text="#000000" bgcolor="#FFFFFF">
<div>Thomas,<br>
<br>
You have not specified a block size, so you are doing a huge
number of small(ish) reads with associated round trips. What
happens with dd bs=128k ..?<br>
<br>
Jeff Anderson-Lee<br>
<br>
On 2/28/2013 5:30 PM, Thomas Wakefield wrote:<br>
</div>
<blockquote type="cite">
Did a fresh dd test just to confirm, same results:
<div><br>
</div>
<div>
<div>[root@cpu_crew1 benchmark]# dd if=/dev/zero
of=/shared/working/benchmark/test.cpucrew1 bs=512k count=10000</div>
<div>10000+0 records in</div>
<div>10000+0 records out</div>
<div>5242880000 bytes (5.2 GB) copied, 7.43695 seconds, 705 MB/s</div>
<div>[root@cpu_crew1 benchmark]# dd
if=/shared/working/benchmark/test.cpucrew1 of=/dev/null</div>
<div>552126+0 records in</div>
<div>552125+0 records out</div>
<div>282688000 bytes (283 MB) copied, 37.8514 seconds, 7.5 MB/s</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>On Feb 28, 2013, at 8:14 PM, Bryan Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">How are you doing the reading? Is this still
an iozone benchmark?
<div><br>
</div>
<div>if you simply dd if=/glustermount/bigfile
of=/dev/null, is the speed better?</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Feb 28, 2013 at 5:05 PM,
Thomas Wakefield <span dir="ltr"><<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I get great speed
locally, it's only when I add gluster in that it
slows down. I get 2GB/s locally to the exact same
brick. It's gluster that is having the read issue
(80MB/s). But Gluster can write just fine, 800MB/s.
<div>
<br>
</div>
<div>The blockdev idea is a good one, and I have
already done it. Thanks though.</div>
<div><br>
</div>
<div>-Tom</div>
<div><br>
</div>
<div>
<div>
<div>On Feb 28, 2013, at 7:53 PM, Ling Ho <<a href="mailto:ling@slac.stanford.edu" target="_blank">ling@slac.stanford.edu</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> Tom,<br>
<br>
What type of disks do you have? If they are
raid 5 or 6, have you try setting the
read-ahead size to 8192 or 16384 (blockdev
--setra 8192 /dev/<sd?> ?<br>
<br>
...<br>
ling<br>
<br>
On 02/28/2013 04:23 PM, Thomas Wakefield
wrote:
<blockquote type="cite"> Did anyone else
have any ideas on performance tuning for
reads?
<div><br>
<div>
<div>On Feb 27, 2013, at 9:29 PM,
Thomas Wakefield <<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">Bryan-
<div><br>
</div>
<div>Yes I can write at
700-800MBytes/sec, but i can
only read at <b>70-80
MBytes/sec</b>. I would be
very happy if I could get it to
read at the same speed it can
write at. And the 70-80 is
sequential, not random for
reads, same exact test commands
on the disk server are in the
2+GB/s range, so I know the disk
server can do it.</div>
<div><br>
</div>
<div>-Tom</div>
<div><br>
</div>
<div><br>
<div>
<div>On Feb 27, 2013, at 7:41
PM, Bryan Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Are your
figures 700-800M<b>Byte</b>/sec?
Because that is probably
as fast as your 10G nic
cards are able to do. You
can test that by trying to
push a large amount of
data over nc or ftp.
<div><br>
</div>
<div> Might want to try
Infiniband. 40G cards
are pretty routine.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Feb 27, 2013 at
3:45 PM, Thomas
Wakefield <span dir="ltr"><<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I
also get the same
performance running
iozone for large
file sizes, iozone
-u 1 -r 512k -s 2G
-I -F.
<div> <br>
</div>
<div>Large file IO
is what I need the
system to do. I
am just shocked at
the huge
difference between
local IO and
gluster client IO.
I know there
should be some
difference, but
10x is
unacceptable.</div>
<div><br>
</div>
<div>-Tom</div>
<div>
<div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On Feb
27, 2013, at
5:31 PM, Bryan
Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Every
time you
open/close a
file or a
directory you
will have to
wait for locks
which take
time. This is
totally
expected.
<div><br>
</div>
<div>Why don't
you share what
you want to
do? iozone
benchmarks
look like crap
but serving
qcow2 files to
qemu works
fantastic for
me. What are
you doing?
Make a
benchmark that
does that. If
you are going
to have many
files with a
wide variety of
sizes
glusterfs/fuse
might not be
what you are
looking for.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Wed, Feb 27,
2013 at 12:56
PM, Thomas
Wakefield <span dir="ltr"><<a href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I
have tested
everything,
small and
large files.
I have used
file sizes
ranging from
128k up to
multiple GB
files. All
the reads are
bad.
<div> <br>
</div>
<div><br>
<div><br>
</div>
<div>Here is a
fairly
exhaustive
iozone auto
test:</div>
<div><br>
</div>
<div>
<div>
random
random
bkwd record
stride
</div>
<div>
KB reclen
write
rewrite
read
reread
read write
read
rewrite
read fwrite
frewrite
fread freread</div>
<div>
64 4
40222
63492 26868
30060
1620 71037
1572
70570
31294
77096
72475 14736
13928</div>
<div>
64 8
99207
116366
13591
13513 3214
97690
<a href="tel:3155%20%C2%A0%20109978" value="+13155109978" target="_blank">3155 109978</a>
28920
152018
158480 18936
17625</div>
<div>
64 16
230257
253766
25156
28713 10867
223732
8873 244297
54796
303383
312204 15062
13545</div>
<div>
64 32
255943
234481
5735102
7100397
11897 318502
13681
347801
24214
695778
528618 25838
28094</div>
<div>
64 64
214096
681644
6421025
7100397
27453 292156
28117
621657
27338
376062
512471 28569
32534</div>
<div>
128 4
74329 <a href="tel:75468%20%C2%A0%20%C2%A026428" value="+17546826428" target="_blank">75468 26428</a> 41089 1131 72857 1118
66976
1597 73778
78343
13351 13026</div>
<div>
128 8
100862 <a href="tel:135170%20%C2%A0%20%C2%A024966" value="+13517024966" target="_blank">135170
24966</a>
16734 2617
118966
2560 120406
39156
125121
146613 16177
16180</div>
<div>
128 16
115114
253983
28212
17854 5307
246180
5431 229843
47335
255920
271173 27256
24445</div>
<div>
128 32
256042
391360
39848
64258 11329
290230
9905 429563
38176
490380
463696 20917
19219</div>
<div>
128 64
248573
592699
4557257
6812590
19583 452366
29263
603357
42967
814915
692017 76327
37604</div>
<div>
128 128
921183
526444
5603747
5379161
45614 390222
65441
826202
41384
662962
1040839
78526 39023</div>
<div>
256 4
76212 <a href="tel:77337%20%C2%A0%20%C2%A040295" value="+17733740295" target="_blank">77337 40295</a> 32125 1289 71866 1261
64645
1436 57309
53048
23073 29550</div>
<div>
256 8
126922 <a href="tel:141976%20%C2%A0%20%C2%A026237" value="+14197626237" target="_blank">141976
26237</a>
25130 2566
128058
<a href="tel:2565%20%C2%A0%20138981" value="+12565138981" target="_blank">2565 138981</a>
2985
125060
133603 22840
24955</div>
<div>
256 16
242883
263636
41850
24371 4902
250009
5290 248792
89353
243821
247303 26965
26199</div>
<div>
256 32
409074
439732
40101
39335 11953
436870
11209 430218
83743
409542
479390 30821
27750</div>
<div>
256 64
259935
571502
64840 <a href="tel:71847%20%C2%A0%2022537" value="+17184722537" target="_blank">71847
22537</a>
617161
23383 392047
91852
672010
802614 41673
53111</div>
<div>
256 128
847597
812329
185517 <a href="tel:83198%20%C2%A0%2049383" value="+18319849383" target="_blank">83198
49383</a>
708831
44668 794889
74267
1180188
1662639
54303 41018</div>
<div>
256 256
481324
709299
5217259
5320671
44668 719277
40954
808050
41302
790209
771473 62224
35754</div>
<div>
512 4
77667
75226 35102
29696
1337 66262
1451
67680
1413 69265
69142
42084 27897</div>
<div>
512 8
134311 <a href="tel:144341%20%C2%A0%20%C2%A030144" value="+14434130144" target="_blank">144341
30144</a>
24646 2102
134143
2209 134699
2296
108110
128616 25104
29123</div>
<div>
512 16
200085
248787
30235
25697 4196
247240
<a href="tel:4179%20%C2%A0%20256116" value="+14179256116" target="_blank">4179 256116</a>
4768
250003
226436 32351
28455</div>
<div>
512 32
330341
439805
26440
39284 8744
457611
<a href="tel:8006%20%C2%A0%20424168" value="+18006424168" target="_blank">8006 424168</a>
125953
425935
448813 27660
26951</div>
<div>
512 64
483906
733729
48747
41121 16032
555938
17424 587256
187343
366977
735740 41700
41548</div>
<div>
512 128
836636
907717
69359 <a href="tel:94921%20%C2%A0%2042443" value="+19492142443" target="_blank">94921
42443</a>
761031
36828 964378
123165
651383
695697 58368
44459</div>
<div>
512 256
520879
860437
145534
135523 40267
847532
31585 663252
69696
1270846
1492545
48822 48092</div>
<div>
512 512
782951
973118
3099691
2942541
42328 871966
46218
911184
49791
953248
1036527
52723 48347</div>
<div>
1024 4
76218
69362 36431
28711
1137 66171
1174
68938
1125 70566
70845
34942 28914</div>
<div>
1024 8
126045 <a href="tel:140524%20%C2%A0%20%C2%A037836" value="+14052437836" target="_blank">140524
37836</a>
15664 2698
126000
2557 125566
2567
110858
127255 26764
27945</div>
<div>
1024 16
243398
261429
40238
23263 3987
246400
3882 260746
4093
236652
236874 31429
25076</div>
<div>
1024 32
383109
422076
41731
41605 8277
473441
7775 415261
8588
394765
407306 40089
28537</div>
<div>
1024 64
590145
619156
39623
53267 15051
722717
14624 753000
257294
597784
620946 38619
44073</div>
<div>
1024 128
1077836 <a href="tel:1124099%20%C2%A0%20%C2%A056192" value="+12409956192" target="_blank">1124099
56192</a>
64916 36851
1102176
37198 1082454
281548
829175
792604 47975
51913</div>
<div>
1024 256
941918
1074331
72783 <a href="tel:81450%20%C2%A0%2026778" value="+18145026778" target="_blank">81450
26778</a>
1099636
32395 1060013
183218
1024121
995171 44371
45448</div>
<div>
1024 512
697483
1130312
100324
114682 48215
1041758
41480 1058967
90156
994020
1563622
56328 46370</div>
<div>
1024 1024
931702
1087111
4609294
4199201
44191 949834
45594
970656
56674
933525
1075676
44876 46115</div>
<div>
2048 4
71438
67066 58319
38913
1147 44147
1043
42916
967 66416
67205
45953 96750</div>
<div>
2048 8
141926
134567
61101
55445 2596
77528
2564 80402
4258
124211
120747 53888
100337</div>
<div>
2048 16
254344
255585
71550
74500 5410
139365
5201 141484
5171
205521
213113 67048
57304</div>
<div>
2048 32
397833
411261
56676 <a href="tel:80027%20%C2%A0%2010440" value="+18002710440" target="_blank">80027
10440</a>
260034
10126 230238
10814
391665
383379 79333
60877</div>
<div>
2048 64
595167
687205
64262
<a href="tel:87327%20%C2%A0%2020772" value="+18732720772" target="_blank">87327 20772</a>
456430
19960 477064
23190
540220
563096 86812
92565</div>
<div>
2048 128
833585
933403
121926
118621 37700
690020
37575 733254
567449
712337
734006 92011
104934</div>
<div>
2048 256
799003
949499
143688
<a href="tel:125659%20%C2%A0%2040871" value="+12565940871" target="_blank">125659 40871</a>
892757
37977 880494
458281
836263
901375 131332
110237</div>
<div>
2048 512
979936
1040724
120896
138013 54381
859783
48721 780491
279203
1068824
1087085
97886 98078</div>
<div>
2048 1024
901754
987938
53352
<a href="tel:53043%20%C2%A0%2072727" value="+15304372727" target="_blank">53043 72727</a>
1054522
68269 992275
181253
1309480
1524983
121600
95585</div>
<div>
2048 2048
831890
1021540
4257067
3302797
75672 984203
80181
826209
94278
966920
1027159
111832
105921</div>
<div>
4096 4
66195
67316 62171
74785
1328 28963
1329
26397
1223 71470
69317
55903 84915</div>
<div>
4096 8
122221
120057
90537
60958 2598
47312
2468 59783
2640
128674
127872 41285
40422</div>
<div>
4096 16
238321
239251
29336
32121 4153
89262
3986 96930
4608
229970
237108 55039
56983</div>
<div>
4096 32
417110
421356
30974
50000 8382
156676
7886 153841
7900
359585
367288 26611
25952</div>
<div>
4096 64
648008
668066
32193
29389 14830
273265
14822 282211
19653
581898
620798 51281
50218</div>
<div>
4096 128
779422
848564
55594
<a href="tel:60253%20%C2%A0%2037108" value="+16025337108" target="_blank">60253 37108</a>
451296
35908 491361
37567
738163
728059 67681
66440</div>
<div>
4096 256
865623
886986
71368
<a href="tel:63947%20%C2%A0%2044255" value="+16394744255" target="_blank">63947 44255</a>
645961
42689 719491
736707
819696
837641 57059
60347</div>
<div>
4096 512
852099
889650
68870
73891 31185
845224
30259 830153
392334
910442
961983 60083
55558</div>
<div>
4096 1024
710357
867810
29377
29522 49954
846640
43665 926298
213677
986226
1115445
55130 59205</div>
<div>
4096 2048
826479
908420
43191
42075 59684
904022
58601 855664
115105
1418322
1524415
60548 66066</div>
<div>
4096 4096
793351
855111
3232454
3673419
66018 861413
48833
847852
45914
852268
842075 42980
48374</div>
<div>
8192 4
67340
69421 42198
31740
994 23251
1166 16813
837
73827
73126 25169
29610</div>
<div>
8192 8
137150
125622
29131
36439 2051
44342
1988 48930
2315
134183
135367 31080
33573</div>
<div>
8192 16
237366
220826
24810
26584 3576
88004
3769 78717
4289
233751
235355 23302
28742</div>
<div>
8192 32
457447
454404
31594
27750 8141
142022
7846 143984
9322
353147
396188 34203
33265</div>
<div>
8192 64
670645
655259
28630
23255 16669
237476
16965 244968
15607
590365
575320 49998
43305</div>
<div>
8192 128
658676
760982
44197
47802 28693
379523
26614 378328
27184
720997
702038 51707
49733</div>
<div>
8192 256
643370
698683
56233
<a href="tel:63165%20%C2%A0%2028846" value="+16316528846" target="_blank">63165 28846</a>
543952
27745 576739
44014
701007
725534 59611
58985</div>
<div>
8192 512
696884
776793
67258
52705 18711
698854
21004 694124
621695
784812
773331 43101
47659</div>
<div>
8192 1024
729664
810451
15470
15875 31318
801490
38123 812944
301222
804323
832765 54308
53376</div>
<div>
8192 2048
749217
68757 21914
22667
48971 783309
48132
782738
172848
907408
929324 51156
50565</div>
<div>
8192 4096
707677
763960
32063
<a href="tel:31928%20%C2%A0%2047809" value="+13192847809" target="_blank">31928 47809</a>
751692
49560 786339
93445
1046761
1297876
48037 51680</div>
<div>
8192 8192
623817
746288
2815955
3137358
48722 741633
35428
753787
49626
803683
823800 48977
52895</div>
<div>
16384 4
72372
73651 34471
30788
960 23610
903 22316
891
71445
71138 56451
55129</div>
<div>
16384 8
137920
141704
50830
33857 1935
41934
2275 35588
3608
130757
137801 51621
48525</div>
<div>
16384 16
245369
242460
41808
29770 3605
75682
4355 75315
4767
241100
239693 53263
30785</div>
<div>
16384 32
448877
433956
31846
35010 7973
118181
8819 112703
8177
381734
391651 57749
63417</div>
<div>
16384 64
710831
700712
66792
68864 20176
209806
<a href="tel:19034%20%C2%A0%20207852" value="+19034207852" target="_blank">19034 207852</a>
21255
589503
601379 104567
105162</div>
<div>
16384 128
836901
860867
104226
100373 40899
358865
40946 360562
39415
675968
691538 96086
105695</div>
<div>
16384 256
798081
828146
107103
120433 39084
595325
39050 593110
56925
763466
797859 109645
113414</div>
<div>
16384 512
810851
843931
113564
106202 35111
714831
46244 745947
53636
802902
760172 110492
100879</div>
<div>
16384 1024
726399
820219
22106
22987 53087
749053
54781 777705
1075341
772686
809723 100349
96619</div>
<div>
16384 2048
807772
856458
23920
23617 66320
829576
72105 740848
656379
864539
835446 93499
101714</div>
<div>
16384 4096
797470
840596
27270
28132 88784
834938
83428 836809
342898
953274
969699 96499
100657</div>
<div>
16384 8192
760063
812942
41022
40014 102020
812135
104063
820346
229870
1252409
1398306
100959
103793</div>
<div>
16384 16384
743107
825054
3074035
3551062
97740 826175
103597
803926
94681
820474
829106 97168
92018</div>
<div>
32768 64
713528
714585
71310
74296 21135
194177
21551 202998
21110
571030
595232 98598
103373</div>
<div>
32768 128
766698
820615
103028
102269 39013
328409
40325 346403
41084
684604
702513 103887
100689</div>
<div>
32768 256
766591
819138
107557
113878 42088
574878
41778 557989
55045
749922
754434 100964
104866</div>
<div>
32768 512
774365
846915
104255
106094 35280
728957
25829 734316
26990
750614
800955 55013
63750</div>
<div>
32768 1024
759113
806241
22033
22622 37322
761870
45003 766950
54519
755666
741339 70018
66255</div>
<div>
32768 2048
797566
860344
22893
23859 49973
778872
53509 779539
796674
817078
797585 56750
57047</div>
<div>
32768 4096
796693
798985
24267
25222 54410
781810
53053 802845
421421
832858
855811 56748
60507</div>
<div>
32768 8192
741543
795144
29254
29150 58678
830011
55545 833914
221323
914182
952916 56120
60014</div>
<div>
32768 16384
754641
819797
44183
45718 67293
785047
65078 807273
132136
1240414
1267713
56585 56167</div>
<div>
65536 64
663333
656588
32211
31925 17324
202043
15477 215981
15496
592126
597188 86431
93969</div>
<div>
65536 128
716218
791603
92035
95614 40330
330127
39295 346542
39469
697385
714453 105094
98327</div>
<div>
65536 256
769283
798701
110285
110085 38404
535730
39939 585023
52402
741206
749899 98671
100106</div>
<div>
65536 512
728151
814516
112550
108873 32284
690164
38911 708244
50047
778669
765303 98333
92728</div>
<div>
65536 1024
773504
827108
21542
20741 45901
775473
57282 807232
69359
770793
766736 101465
100905</div>
<div>
65536 2048
767661
853446
21077
21260 59825
762711
71985 802488
87290
772402
775685 96832
103189</div>
<div>
65536 4096
767669
828165
20681
21480 79798
829757
91755 838367
1379411
791248
783023 106433
112769</div>
<div>
65536 8192
739291
784946
23100
22454 85945
785134
87081 828489
642088
816781
798275 93307
95318</div>
<div>
65536 16384
736401
791669
26345
26384 93082
799356
91419 751766
385721
911919
884688 109668
96374</div>
<div>
131072 64
658361
661211
79790
81016 19460
182084
19325 195428
19553
551809
554426 96581
81979</div>
<div>
131072 128
759806
838994
51063
56976 33624
326699
36329 360593
35679
710602
716538 81725
51819</div>
<div>
131072 256
787375
843763
54664
65031 33847
557817
35093 581552
45163
750234
758452 82704
83472</div>
<div>
131072 512
777460
818049
87831
87110 27994
705621
31860 723071
38483
779489
793068 88281
86068</div>
<div>
131072 1024
737158
782780
19005
18897 36614
735240
38900 812074
52149
746598
762694 74376
82534</div>
<div>
131072 2048
777386
817330
24169
24693 55171
769662
56082 788848
55937
757716
763147 73508
82351</div>
<div>
131072 4096
751757
792201
25283
25173 48455
813945
56557 837418
59578
749106
757939 67846
70964</div>
<div>
131072 8192
743004
796673
20369
20327 88990
780925
95956 777211
951528
762755
739851 70850
69943</div>
<div>
131072 16384
766023
823611
22333
22142 108337
796852
110848
777704
718954
809861
802822 115896
82903</div>
<div>
262144 64
655769
695392
45973
45668 19083
209417
18768 207272
19133
601329
599267 107705
87261</div>
<div>
262144 128
750052
798999
72711
74713 37329
339480
36854 365587
37187
710898
714372 108400
78806</div>
<div>
262144 256
788355
815575
80897
75141 37080
549762
37521 574393
51731
780429
751131 97395
73842</div>
<div>
262144 512
714994
736649
80224
80195 32986
708196
36048 628508
44043
767699
781863 123267
83558</div>
<div>
262144 1024
793541
800826
20200
20428 45599
752027
48232 798229
57011
715643
763691 76741
73211</div>
<div>
262144 2048
699395
782935
20227
20519 65785
779219
80486 803403
93381
650054
712171 115611
108457</div>
<div>
262144 4096
629531
766261
20544
21563 83107
819917
93008 844103
90625
741879
754931 94114
94475</div>
<div>
262144 8192
731695
804112
21158
21589 82987
787869
84858 799483
95127
731822
737397 95950
95160</div>
<div>
262144 16384
703376
802552
21940
22371 87748
777024
90349 779626
1018679
765278
619969 96051
92734</div>
<div>
524288 64
649937
710342
73778
70893 19734
207179
19367 199709
19493
585910
581340 91902
91443</div>
<div>
524288 128
739270
805153
91507
90395 38547
319049
37988 358516
38123
712602
711814 92463
90851</div>
<div>
524288 256
751279
802496
89481
91022 38869
544097
38244 569142
56130
756367
752466 92677
92875</div>
<div>
524288 512
758393
819788
58085
73437 29030
700120
28767 739236
33925
770177
795879 58963
72689</div>
<div>
524288 1024
778385
818368
18822
18639 38774
798827
50898 799492
55679
801849
800917 92778
82686</div>
<div>
524288 2048
780594
834612
19936
19994 63147
769555
63007 802512
66322
772900
771314 86628
93485</div>
<div>
524288 4096
769806
812570
20165
20137 79896
783615
71217 80835
68550
80166
73733 122559
116844</div>
<div>
524288 8192
797504
887823
123824
124513 94520
844880
86441 710319
65077
636713
819868 87765
80121</div>
<div>
524288 16384
788468
858216
92580
117354 109583
850179
63966 856965
121151
789252
805040 123810
119088</div>
<div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div><br>
</div>
<div><br>
<div>
<div>On Feb
27, 2013, at
2:46 PM, Bryan
Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">How
are you doing
the read/write
tests on the
fuse/glusterfs
mountpoint?
Many small
files will be
slow because
all the time
is spent
coordinating
locks.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Wed, Feb
27, 2013 at
9:31 AM,
Thomas
Wakefield <span dir="ltr"><<a href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Help
please-
<div><br>
</div>
<div><br>
</div>
<div>I am
running 3.3.1
on Centos
using a 10GB
network. I
get reasonable
write speeds,
although I
think they
could be
faster. But
my read speeds
are REALLY
slow.</div>
<div><br>
</div>
<div>Executive
summary:</div>
<div><br>
</div>
<div>On
gluster
client-</div>
<div>Writes
average about
700-800MB/s</div>
<div>Reads
average about
70-80MB/s</div>
<div><br>
</div>
<div>On
server-</div>
<div>Writes
average about
1-1.5GB/s</div>
<div>Reads
average about
2-3GB/s</div>
<div><br>
</div>
<div>Any
thoughts?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Here are
some
additional
details:</div>
<div><br>
</div>
<div>Nothing
interesting in
any of the log
files,
everything is
very quite.</div>
<div>All
servers had no
other load,
and all
clients are
performing the
same way.</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>Volume
Name: shared</div>
<div>Type:
Distribute</div>
<div>Volume
ID:
de11cc19-0085-41c3-881e-995cca244620</div>
<div>Status:
Started</div>
<div>Number of
Bricks: 26</div>
<div>Transport-type:
tcp</div>
<div>Bricks:</div>
<div>Brick1:
fs-disk2:/storage/disk2a</div>
<div>Brick2:
fs-disk2:/storage/disk2b</div>
<div>Brick3:
fs-disk2:/storage/disk2d</div>
<div>Brick4:
fs-disk2:/storage/disk2e</div>
<div>Brick5:
fs-disk2:/storage/disk2f</div>
<div>Brick6:
fs-disk2:/storage/disk2g</div>
<div>Brick7:
fs-disk2:/storage/disk2h</div>
<div>Brick8:
fs-disk2:/storage/disk2i</div>
<div> Brick9:
fs-disk2:/storage/disk2j</div>
<div>Brick10:
fs-disk2:/storage/disk2k</div>
<div>Brick11:
fs-disk2:/storage/disk2l</div>
<div>Brick12:
fs-disk2:/storage/disk2m</div>
<div>Brick13:
fs-disk2:/storage/disk2n</div>
<div>Brick14:
fs-disk2:/storage/disk2o</div>
<div> Brick15:
fs-disk2:/storage/disk2p</div>
<div>Brick16:
fs-disk2:/storage/disk2q</div>
<div>Brick17:
fs-disk2:/storage/disk2r</div>
<div>Brick18:
fs-disk2:/storage/disk2s</div>
<div>Brick19:
fs-disk2:/storage/disk2t</div>
<div> Brick20:
fs-disk2:/storage/disk2u</div>
<div>Brick21:
fs-disk2:/storage/disk2v</div>
<div>Brick22:
fs-disk2:/storage/disk2w</div>
<div>Brick23:
fs-disk2:/storage/disk2x</div>
<div>Brick24:
fs-disk3:/storage/disk3a</div>
<div>Brick25:
fs-disk3:/storage/disk3b</div>
<div> Brick26:
fs-disk3:/storage/disk3c</div>
<div>Options
Reconfigured:</div>
<div>performance.write-behind:
on</div>
<div>performance.read-ahead:
on</div>
<div>performance.io-cache:
on</div>
<div>performance.stat-prefetch:
on</div>
<div>performance.quick-read:
on</div>
<div>cluster.min-free-disk:
500GB</div>
<div>nfs.disable:
off</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>sysctl.conf
settings for
10GBe</div>
<div>
<div>#
increase TCP
max buffer
size settable
using
setsockopt()</div>
<div>net.core.rmem_max
= 67108864 </div>
<div>net.core.wmem_max
= 67108864 </div>
<div>#
increase Linux
autotuning TCP
buffer limit</div>
<div>net.ipv4.tcp_rmem
= 4096 87380
67108864</div>
<div>net.ipv4.tcp_wmem
= 4096 65536
67108864</div>
<div>#
increase the
length of the
processor
input queue</div>
<div>net.core.netdev_max_backlog
= 250000</div>
<div>#
recommended
default
congestion
control is
htcp </div>
<div>net.ipv4.tcp_congestion_control=htcp</div>
<div> #
recommended
for hosts with
jumbo frames
enabled</div>
<div>net.ipv4.tcp_mtu_probing=1</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<br>
<div> <span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px">
<div> <span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px">
<div style="word-wrap:break-word">
<div>Thomas W.<br>
Sr. Systems
Administrator
COLA/IGES<br>
<a href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a><br>
</div>
</div>
</span></span></div>
Affiliate <span style="font-size:12px">Computer Scientist GMU</span></span> </div>
<br>
</div>
<br>
_______________________________________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</div>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote></div><br></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>