<div dir="ltr">How are you doing the reading? Is this still an iozone benchmark?<div><br></div><div style>if you simply dd if=/glustermount/bigfile of=/dev/null, is the speed better?</div></div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Thu, Feb 28, 2013 at 5:05 PM, Thomas Wakefield <span dir="ltr"><<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I get great speed locally, it's only when I add gluster in that it slows down. I get 2GB/s locally to the exact same brick. It's gluster that is having the read issue (80MB/s). But Gluster can write just fine, 800MB/s.<div>
<br></div><div>The blockdev idea is a good one, and I have already done it. Thanks though.</div><div><br></div><div>-Tom</div><div><br></div><div><div><div>On Feb 28, 2013, at 7:53 PM, Ling Ho <<a href="mailto:ling@slac.stanford.edu" target="_blank">ling@slac.stanford.edu</a>> wrote:</div>
<br><blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
Tom,<br>
<br>
What type of disks do you have? If they are raid 5 or 6, have you
try setting the read-ahead size to 8192 or 16384 (blockdev --setra
8192 /dev/<sd?> ?<br>
<br>
...<br>
ling<br>
<br>
On 02/28/2013 04:23 PM, Thomas Wakefield wrote:
<blockquote type="cite">
Did anyone else have any ideas on performance tuning for reads?
<div><br>
<div>
<div>On Feb 27, 2013, at 9:29 PM, Thomas Wakefield <<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div style="word-wrap:break-word">Bryan-
<div><br>
</div>
<div>Yes I can write at 700-800MBytes/sec, but i can only
read at <b>70-80 MBytes/sec</b>. I would be very happy
if I could get it to read at the same speed it can write
at. And the 70-80 is sequential, not random for reads,
same exact test commands on the disk server are in the
2+GB/s range, so I know the disk server can do it.</div>
<div><br>
</div>
<div>-Tom</div>
<div><br>
</div>
<div><br>
<div>
<div>On Feb 27, 2013, at 7:41 PM, Bryan Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Are your figures 700-800M<b>Byte</b>/sec?
Because that is probably as fast as your 10G nic
cards are able to do. You can test that by trying
to push a large amount of data over nc or ftp.
<div><br>
</div>
<div>
Might want to try Infiniband. 40G cards are
pretty routine.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Feb 27, 2013 at
3:45 PM, Thomas Wakefield <span dir="ltr"><<a href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I also get
the same performance running iozone for
large file sizes, iozone -u 1 -r 512k -s 2G
-I -F.
<div>
<br>
</div>
<div>Large file IO is what I need the system
to do. I am just shocked at the huge
difference between local IO and gluster
client IO. I know there should be some
difference, but 10x is unacceptable.</div>
<div><br>
</div>
<div>-Tom</div>
<div>
<div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On Feb 27, 2013, at 5:31 PM,
Bryan Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Every time you
open/close a file or a directory
you will have to wait for locks
which take time. This is totally
expected.
<div><br>
</div>
<div>Why don't you share what
you want to do? iozone
benchmarks look like crap but
serving qcow2 files to qemu
works fantastic for me. What
are you doing? Make a
benchmark that does that. If
you are going to have many
files with a wide variety of
sizes glusterfs/fuse might not
be what you are looking for.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed,
Feb 27, 2013 at 12:56 PM,
Thomas Wakefield <span dir="ltr"><<a href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">I
have tested everything,
small and large files. I
have used file sizes
ranging from 128k up to
multiple GB files. All
the reads are bad.
<div>
<br>
</div>
<div><br>
<div><br>
</div>
<div>Here is a fairly
exhaustive iozone auto
test:</div>
<div><br>
</div>
<div>
<div>
random random
bkwd record
stride
</div>
<div> KB
reclen write
rewrite read
reread read
write read
rewrite read
fwrite frewrite
fread freread</div>
<div> 64
4 40222
63492 26868
30060 1620
71037 1572
70570 31294
77096 72475
14736 13928</div>
<div> 64
8 99207
116366 13591
13513 3214
97690 3155
109978 28920
152018 158480
18936 17625</div>
<div> 64
16 230257
253766 25156
28713 10867
223732 8873
244297 54796
303383 312204
15062 13545</div>
<div> 64
32 255943
234481 5735102
7100397 11897
318502 13681
347801 24214
695778 528618
25838 28094</div>
<div> 64
64 214096
681644 6421025
7100397 27453
292156 28117
621657 27338
376062 512471
28569 32534</div>
<div> 128
4 74329
<a href="tel:75468%20%C2%A0%20%C2%A026428" value="+17546826428" target="_blank">75468 26428</a>
41089 1131
72857 1118
66976 1597
73778 78343
13351 13026</div>
<div> 128
8 100862
<a href="tel:135170%20%C2%A0%20%C2%A024966" value="+13517024966" target="_blank">135170 24966</a>
16734 2617
118966 2560
120406 39156
125121 146613
16177 16180</div>
<div> 128
16 115114
253983 28212
17854 5307
246180 5431
229843 47335
255920 271173
27256 24445</div>
<div> 128
32 256042
391360 39848
64258 11329
290230 9905
429563 38176
490380 463696
20917 19219</div>
<div> 128
64 248573
592699 4557257
6812590 19583
452366 29263
603357 42967
814915 692017
76327 37604</div>
<div> 128
128 921183
526444 5603747
5379161 45614
390222 65441
826202 41384
662962 1040839
78526 39023</div>
<div> 256
4 76212
<a href="tel:77337%20%C2%A0%20%C2%A040295" value="+17733740295" target="_blank">77337 40295</a>
32125 1289
71866 1261
64645 1436
57309 53048
23073 29550</div>
<div> 256
8 126922
<a href="tel:141976%20%C2%A0%20%C2%A026237" value="+14197626237" target="_blank">141976 26237</a>
25130 2566
128058 2565
138981 2985
125060 133603
22840 24955</div>
<div> 256
16 242883
263636 41850
24371 4902
250009 5290
248792 89353
243821 247303
26965 26199</div>
<div> 256
32 409074
439732 40101
39335 11953
436870 11209
430218 83743
409542 479390
30821 27750</div>
<div> 256
64 259935
571502 64840
<a href="tel:71847%20%C2%A0%2022537" value="+17184722537" target="_blank">71847 22537</a>
617161 23383
392047 91852
672010 802614
41673 53111</div>
<div> 256
128 847597
812329 185517
<a href="tel:83198%20%C2%A0%2049383" value="+18319849383" target="_blank">83198 49383</a>
708831 44668
794889 74267
1180188 1662639
54303 41018</div>
<div> 256
256 481324
709299 5217259
5320671 44668
719277 40954
808050 41302
790209 771473
62224 35754</div>
<div> 512
4 77667
75226 35102
29696 1337
66262 1451
67680 1413
69265 69142
42084 27897</div>
<div> 512
8 134311
<a href="tel:144341%20%C2%A0%20%C2%A030144" value="+14434130144" target="_blank">144341 30144</a>
24646 2102
134143 2209
134699 2296
108110 128616
25104 29123</div>
<div> 512
16 200085
248787 30235
25697 4196
247240 4179
256116 4768
250003 226436
32351 28455</div>
<div> 512
32 330341
439805 26440
39284 8744
457611 8006
424168 125953
425935 448813
27660 26951</div>
<div> 512
64 483906
733729 48747
41121 16032
555938 17424
587256 187343
366977 735740
41700 41548</div>
<div> 512
128 836636
907717 69359
<a href="tel:94921%20%C2%A0%2042443" value="+19492142443" target="_blank">94921 42443</a>
761031 36828
964378 123165
651383 695697
58368 44459</div>
<div> 512
256 520879
860437 145534
135523 40267
847532 31585
663252 69696
1270846 1492545
48822 48092</div>
<div> 512
512 782951
973118 3099691
2942541 42328
871966 46218
911184 49791
953248 1036527
52723 48347</div>
<div> 1024
4 76218
69362 36431
28711 1137
66171 1174
68938 1125
70566 70845
34942 28914</div>
<div> 1024
8 126045
<a href="tel:140524%20%C2%A0%20%C2%A037836" value="+14052437836" target="_blank">140524 37836</a>
15664 2698
126000 2557
125566 2567
110858 127255
26764 27945</div>
<div> 1024
16 243398
261429 40238
23263 3987
246400 3882
260746 4093
236652 236874
31429 25076</div>
<div> 1024
32 383109
422076 41731
41605 8277
473441 7775
415261 8588
394765 407306
40089 28537</div>
<div> 1024
64 590145
619156 39623
53267 15051
722717 14624
753000 257294
597784 620946
38619 44073</div>
<div> 1024
128 1077836
<a href="tel:1124099%20%C2%A0%20%C2%A056192" value="+12409956192" target="_blank">1124099 56192</a>
64916 36851
1102176 37198
1082454 281548
829175 792604
47975 51913</div>
<div> 1024
256 941918
1074331 72783
<a href="tel:81450%20%C2%A0%2026778" value="+18145026778" target="_blank">81450 26778</a>
1099636 32395
1060013 183218
1024121 995171
44371 45448</div>
<div> 1024
512 697483
1130312 100324
114682 48215
1041758 41480
1058967 90156
994020 1563622
56328 46370</div>
<div> 1024
1024 931702
1087111 4609294
4199201 44191
949834 45594
970656 56674
933525 1075676
44876 46115</div>
<div> 2048
4 71438
67066 58319
38913 1147
44147 1043
42916 967
66416 67205
45953 96750</div>
<div> 2048
8 141926
134567 61101
55445 2596
77528 2564
80402 4258
124211 120747
53888 100337</div>
<div> 2048
16 254344
255585 71550
74500 5410
139365 5201
141484 5171
205521 213113
67048 57304</div>
<div> 2048
32 397833
411261 56676
<a href="tel:80027%20%C2%A0%2010440" value="+18002710440" target="_blank">80027 10440</a>
260034 10126
230238 10814
391665 383379
79333 60877</div>
<div> 2048
64 595167
687205 64262
87327 20772
456430 19960
477064 23190
540220 563096
86812 92565</div>
<div> 2048
128 833585
933403 121926
118621 37700
690020 37575
733254 567449
712337 734006
92011 104934</div>
<div> 2048
256 799003
949499 143688
125659 40871
892757 37977
880494 458281
836263 901375
131332 110237</div>
<div> 2048
512 979936
1040724 120896
138013 54381
859783 48721
780491 279203
1068824 1087085
97886 98078</div>
<div> 2048
1024 901754
987938 53352
53043 72727
1054522 68269
992275 181253
1309480 1524983
121600 95585</div>
<div> 2048
2048 831890
1021540 4257067
3302797 75672
984203 80181
826209 94278
966920 1027159
111832 105921</div>
<div> 4096
4 66195
67316 62171
74785 1328
28963 1329
26397 1223
71470 69317
55903 84915</div>
<div> 4096
8 122221
120057 90537
60958 2598
47312 2468
59783 2640
128674 127872
41285 40422</div>
<div> 4096
16 238321
239251 29336
32121 4153
89262 3986
96930 4608
229970 237108
55039 56983</div>
<div> 4096
32 417110
421356 30974
50000 8382
156676 7886
153841 7900
359585 367288
26611 25952</div>
<div> 4096
64 648008
668066 32193
29389 14830
273265 14822
282211 19653
581898 620798
51281 50218</div>
<div> 4096
128 779422
848564 55594
60253 37108
451296 35908
491361 37567
738163 728059
67681 66440</div>
<div> 4096
256 865623
886986 71368
63947 44255
645961 42689
719491 736707
819696 837641
57059 60347</div>
<div> 4096
512 852099
889650 68870
73891 31185
845224 30259
830153 392334
910442 961983
60083 55558</div>
<div> 4096
1024 710357
867810 29377
29522 49954
846640 43665
926298 213677
986226 1115445
55130 59205</div>
<div> 4096
2048 826479
908420 43191
42075 59684
904022 58601
855664 115105
1418322 1524415
60548 66066</div>
<div> 4096
4096 793351
855111 3232454
3673419 66018
861413 48833
847852 45914
852268 842075
42980 48374</div>
<div> 8192
4 67340
69421 42198
31740 994
23251 1166
16813 837
73827 73126
25169 29610</div>
<div> 8192
8 137150
125622 29131
36439 2051
44342 1988
48930 2315
134183 135367
31080 33573</div>
<div> 8192
16 237366
220826 24810
26584 3576
88004 3769
78717 4289
233751 235355
23302 28742</div>
<div> 8192
32 457447
454404 31594
27750 8141
142022 7846
143984 9322
353147 396188
34203 33265</div>
<div> 8192
64 670645
655259 28630
23255 16669
237476 16965
244968 15607
590365 575320
49998 43305</div>
<div> 8192
128 658676
760982 44197
47802 28693
379523 26614
378328 27184
720997 702038
51707 49733</div>
<div> 8192
256 643370
698683 56233
63165 28846
543952 27745
576739 44014
701007 725534
59611 58985</div>
<div> 8192
512 696884
776793 67258
52705 18711
698854 21004
694124 621695
784812 773331
43101 47659</div>
<div> 8192
1024 729664
810451 15470
15875 31318
801490 38123
812944 301222
804323 832765
54308 53376</div>
<div> 8192
2048 749217
68757 21914
22667 48971
783309 48132
782738 172848
907408 929324
51156 50565</div>
<div> 8192
4096 707677
763960 32063
31928 47809
751692 49560
786339 93445
1046761 1297876
48037 51680</div>
<div> 8192
8192 623817
746288 2815955
3137358 48722
741633 35428
753787 49626
803683 823800
48977 52895</div>
<div> 16384
4 72372
73651 34471
30788 960
23610 903
22316 891
71445 71138
56451 55129</div>
<div> 16384
8 137920
141704 50830
33857 1935
41934 2275
35588 3608
130757 137801
51621 48525</div>
<div> 16384
16 245369
242460 41808
29770 3605
75682 4355
75315 4767
241100 239693
53263 30785</div>
<div> 16384
32 448877
433956 31846
35010 7973
118181 8819
112703 8177
381734 391651
57749 63417</div>
<div> 16384
64 710831
700712 66792
68864 20176
209806 19034
207852 21255
589503 601379
104567 105162</div>
<div> 16384
128 836901
860867 104226
100373 40899
358865 40946
360562 39415
675968 691538
96086 105695</div>
<div> 16384
256 798081
828146 107103
120433 39084
595325 39050
593110 56925
763466 797859
109645 113414</div>
<div> 16384
512 810851
843931 113564
106202 35111
714831 46244
745947 53636
802902 760172
110492 100879</div>
<div> 16384
1024 726399
820219 22106
22987 53087
749053 54781
777705 1075341
772686 809723
100349 96619</div>
<div> 16384
2048 807772
856458 23920
23617 66320
829576 72105
740848 656379
864539 835446
93499 101714</div>
<div> 16384
4096 797470
840596 27270
28132 88784
834938 83428
836809 342898
953274 969699
96499 100657</div>
<div> 16384
8192 760063
812942 41022
40014 102020
812135 104063
820346 229870
1252409 1398306
100959 103793</div>
<div> 16384
16384 743107
825054 3074035
3551062 97740
826175 103597
803926 94681
820474 829106
97168 92018</div>
<div> 32768
64 713528
714585 71310
74296 21135
194177 21551
202998 21110
571030 595232
98598 103373</div>
<div> 32768
128 766698
820615 103028
102269 39013
328409 40325
346403 41084
684604 702513
103887 100689</div>
<div> 32768
256 766591
819138 107557
113878 42088
574878 41778
557989 55045
749922 754434
100964 104866</div>
<div> 32768
512 774365
846915 104255
106094 35280
728957 25829
734316 26990
750614 800955
55013 63750</div>
<div> 32768
1024 759113
806241 22033
22622 37322
761870 45003
766950 54519
755666 741339
70018 66255</div>
<div> 32768
2048 797566
860344 22893
23859 49973
778872 53509
779539 796674
817078 797585
56750 57047</div>
<div> 32768
4096 796693
798985 24267
25222 54410
781810 53053
802845 421421
832858 855811
56748 60507</div>
<div> 32768
8192 741543
795144 29254
29150 58678
830011 55545
833914 221323
914182 952916
56120 60014</div>
<div> 32768
16384 754641
819797 44183
45718 67293
785047 65078
807273 132136
1240414 1267713
56585 56167</div>
<div> 65536
64 663333
656588 32211
31925 17324
202043 15477
215981 15496
592126 597188
86431 93969</div>
<div> 65536
128 716218
791603 92035
95614 40330
330127 39295
346542 39469
697385 714453
105094 98327</div>
<div> 65536
256 769283
798701 110285
110085 38404
535730 39939
585023 52402
741206 749899
98671 100106</div>
<div> 65536
512 728151
814516 112550
108873 32284
690164 38911
708244 50047
778669 765303
98333 92728</div>
<div> 65536
1024 773504
827108 21542
20741 45901
775473 57282
807232 69359
770793 766736
101465 100905</div>
<div> 65536
2048 767661
853446 21077
21260 59825
762711 71985
802488 87290
772402 775685
96832 103189</div>
<div> 65536
4096 767669
828165 20681
21480 79798
829757 91755
838367 1379411
791248 783023
106433 112769</div>
<div> 65536
8192 739291
784946 23100
22454 85945
785134 87081
828489 642088
816781 798275
93307 95318</div>
<div> 65536
16384 736401
791669 26345
26384 93082
799356 91419
751766 385721
911919 884688
109668 96374</div>
<div> 131072
64 658361
661211 79790
81016 19460
182084 19325
195428 19553
551809 554426
96581 81979</div>
<div> 131072
128 759806
838994 51063
56976 33624
326699 36329
360593 35679
710602 716538
81725 51819</div>
<div> 131072
256 787375
843763 54664
65031 33847
557817 35093
581552 45163
750234 758452
82704 83472</div>
<div> 131072
512 777460
818049 87831
87110 27994
705621 31860
723071 38483
779489 793068
88281 86068</div>
<div> 131072
1024 737158
782780 19005
18897 36614
735240 38900
812074 52149
746598 762694
74376 82534</div>
<div> 131072
2048 777386
817330 24169
24693 55171
769662 56082
788848 55937
757716 763147
73508 82351</div>
<div> 131072
4096 751757
792201 25283
25173 48455
813945 56557
837418 59578
749106 757939
67846 70964</div>
<div> 131072
8192 743004
796673 20369
20327 88990
780925 95956
777211 951528
762755 739851
70850 69943</div>
<div> 131072
16384 766023
823611 22333
22142 108337
796852 110848
777704 718954
809861 802822
115896 82903</div>
<div> 262144
64 655769
695392 45973
45668 19083
209417 18768
207272 19133
601329 599267
107705 87261</div>
<div> 262144
128 750052
798999 72711
74713 37329
339480 36854
365587 37187
710898 714372
108400 78806</div>
<div> 262144
256 788355
815575 80897
75141 37080
549762 37521
574393 51731
780429 751131
97395 73842</div>
<div> 262144
512 714994
736649 80224
80195 32986
708196 36048
628508 44043
767699 781863
123267 83558</div>
<div> 262144
1024 793541
800826 20200
20428 45599
752027 48232
798229 57011
715643 763691
76741 73211</div>
<div> 262144
2048 699395
782935 20227
20519 65785
779219 80486
803403 93381
650054 712171
115611 108457</div>
<div> 262144
4096 629531
766261 20544
21563 83107
819917 93008
844103 90625
741879 754931
94114 94475</div>
<div> 262144
8192 731695
804112 21158
21589 82987
787869 84858
799483 95127
731822 737397
95950 95160</div>
<div> 262144
16384 703376
802552 21940
22371 87748
777024 90349
779626 1018679
765278 619969
96051 92734</div>
<div> 524288
64 649937
710342 73778
70893 19734
207179 19367
199709 19493
585910 581340
91902 91443</div>
<div> 524288
128 739270
805153 91507
90395 38547
319049 37988
358516 38123
712602 711814
92463 90851</div>
<div> 524288
256 751279
802496 89481
91022 38869
544097 38244
569142 56130
756367 752466
92677 92875</div>
<div> 524288
512 758393
819788 58085
73437 29030
700120 28767
739236 33925
770177 795879
58963 72689</div>
<div> 524288
1024 778385
818368 18822
18639 38774
798827 50898
799492 55679
801849 800917
92778 82686</div>
<div> 524288
2048 780594
834612 19936
19994 63147
769555 63007
802512 66322
772900 771314
86628 93485</div>
<div> 524288
4096 769806
812570 20165
20137 79896
783615 71217
80835 68550
80166 73733
122559 116844</div>
<div> 524288
8192 797504
887823 123824
124513 94520
844880 86441
710319 65077
636713 819868
87765 80121</div>
<div> 524288
16384 788468
858216 92580
117354 109583
850179 63966
856965 121151
789252 805040
123810 119088</div>
<div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div><br>
</div>
<div><br>
<div>
<div>On Feb
27, 2013, at
2:46 PM, Bryan
Whitehead <<a href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">How
are you doing
the read/write
tests on the
fuse/glusterfs
mountpoint?
Many small
files will be
slow because
all the time
is spent
coordinating
locks.</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On Wed, Feb
27, 2013 at
9:31 AM,
Thomas
Wakefield <span dir="ltr"><<a href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Help
please-
<div><br>
</div>
<div><br>
</div>
<div>I am
running 3.3.1
on Centos
using a 10GB
network. I
get reasonable
write speeds,
although I
think they
could be
faster. But
my read speeds
are REALLY
slow.</div>
<div><br>
</div>
<div>Executive
summary:</div>
<div><br>
</div>
<div>On
gluster
client-</div>
<div>Writes
average about
700-800MB/s</div>
<div>Reads
average about
70-80MB/s</div>
<div><br>
</div>
<div>On
server-</div>
<div>Writes
average about
1-1.5GB/s</div>
<div>Reads
average about
2-3GB/s</div>
<div><br>
</div>
<div>Any
thoughts?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Here are
some
additional
details:</div>
<div><br>
</div>
<div>Nothing
interesting in
any of the log
files,
everything is
very quite.</div>
<div>All
servers had no
other load,
and all
clients are
performing the
same way.</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>Volume
Name: shared</div>
<div>Type:
Distribute</div>
<div>Volume
ID:
de11cc19-0085-41c3-881e-995cca244620</div>
<div>Status:
Started</div>
<div>Number of
Bricks: 26</div>
<div>Transport-type:
tcp</div>
<div>Bricks:</div>
<div>Brick1:
fs-disk2:/storage/disk2a</div>
<div>Brick2:
fs-disk2:/storage/disk2b</div>
<div>Brick3:
fs-disk2:/storage/disk2d</div>
<div>Brick4:
fs-disk2:/storage/disk2e</div>
<div>Brick5:
fs-disk2:/storage/disk2f</div>
<div>Brick6:
fs-disk2:/storage/disk2g</div>
<div>Brick7:
fs-disk2:/storage/disk2h</div>
<div>Brick8:
fs-disk2:/storage/disk2i</div>
<div>
Brick9:
fs-disk2:/storage/disk2j</div>
<div>Brick10:
fs-disk2:/storage/disk2k</div>
<div>Brick11:
fs-disk2:/storage/disk2l</div>
<div>Brick12:
fs-disk2:/storage/disk2m</div>
<div>Brick13:
fs-disk2:/storage/disk2n</div>
<div>Brick14:
fs-disk2:/storage/disk2o</div>
<div>
Brick15:
fs-disk2:/storage/disk2p</div>
<div>Brick16:
fs-disk2:/storage/disk2q</div>
<div>Brick17:
fs-disk2:/storage/disk2r</div>
<div>Brick18:
fs-disk2:/storage/disk2s</div>
<div>Brick19:
fs-disk2:/storage/disk2t</div>
<div>
Brick20:
fs-disk2:/storage/disk2u</div>
<div>Brick21:
fs-disk2:/storage/disk2v</div>
<div>Brick22:
fs-disk2:/storage/disk2w</div>
<div>Brick23:
fs-disk2:/storage/disk2x</div>
<div>Brick24:
fs-disk3:/storage/disk3a</div>
<div>Brick25:
fs-disk3:/storage/disk3b</div>
<div>
Brick26:
fs-disk3:/storage/disk3c</div>
<div>Options
Reconfigured:</div>
<div>performance.write-behind:
on</div>
<div>performance.read-ahead:
on</div>
<div>performance.io-cache:
on</div>
<div>performance.stat-prefetch:
on</div>
<div>performance.quick-read:
on</div>
<div>cluster.min-free-disk:
500GB</div>
<div>nfs.disable:
off</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>sysctl.conf
settings for
10GBe</div>
<div>
<div>#
increase TCP
max buffer
size settable
using
setsockopt()</div>
<div>net.core.rmem_max
= 67108864 </div>
<div>net.core.wmem_max
= 67108864 </div>
<div>#
increase Linux
autotuning TCP
buffer limit</div>
<div>net.ipv4.tcp_rmem
= 4096 87380
67108864</div>
<div>net.ipv4.tcp_wmem
= 4096 65536
67108864</div>
<div>#
increase the
length of the
processor
input queue</div>
<div>net.core.netdev_max_backlog
= 250000</div>
<div>#
recommended
default
congestion
control is
htcp </div>
<div>net.ipv4.tcp_congestion_control=htcp</div>
<div>
# recommended
for hosts with
jumbo frames
enabled</div>
<div>net.ipv4.tcp_mtu_probing=1</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<br>
<br>
<div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px">
<div>
<span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px">
<div style="word-wrap:break-word">
<div>Thomas W.<br>
Sr. Systems
Administrator
COLA/IGES<br>
<a href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a><br>
</div>
</div>
</span></span></div>
Affiliate <span style="font-size:12px">Computer Scientist GMU</span></span>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div><br></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>