<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I get great speed locally, it's only when I add gluster in that it slows down. &nbsp;I get 2GB/s locally to the exact same brick. &nbsp;It's gluster that is having the read issue (80MB/s). &nbsp;But Gluster can write just fine, 800MB/s.<div><br></div><div>The blockdev idea is a good one, and I have already done it. &nbsp;Thanks though.</div><div><br></div><div>-Tom</div><div><br></div><div><div><div>On Feb 28, 2013, at 7:53 PM, Ling Ho &lt;<a href="mailto:ling@slac.stanford.edu">ling@slac.stanford.edu</a>&gt; wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
  
    <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
  
  <div bgcolor="#FFFFFF" text="#000000">
    Tom,<br>
    <br>
    What type of disks do you have? If they are raid 5 or 6, have you
    try setting the read-ahead size to 8192 or 16384 (blockdev --setra
    8192 /dev/&lt;sd?&gt;&nbsp;&nbsp; ?<br>
    <br>
    ...<br>
    ling<br>
    <br>
    On 02/28/2013 04:23 PM, Thomas Wakefield wrote:
    <blockquote cite="mid:FFFBE352-E1C4-4FC2-AA5A-6703EC580F9E@iges.org" type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=ISO-8859-1">
      Did anyone else have any ideas on performance tuning for reads?
      <div><br>
        <div>
          <div>On Feb 27, 2013, at 9:29 PM, Thomas Wakefield &lt;<a moz-do-not-send="true" href="mailto:twake@iges.org">twake@iges.org</a>&gt;
            wrote:</div>
          <br class="Apple-interchange-newline">
          <blockquote type="cite">
            <meta http-equiv="Content-Type" content="text/html;
              charset=ISO-8859-1">
            <div style="word-wrap: break-word; -webkit-nbsp-mode: space;
              -webkit-line-break: after-white-space; ">Bryan-
              <div><br>
              </div>
              <div>Yes I can write at 700-800MBytes/sec, but i can only
                read at <b>70-80 MBytes/sec</b>. &nbsp;I would be very happy
                if I could get it to read at the same speed it can write
                at. &nbsp;And the 70-80 is sequential, not random for reads,
                same exact test commands on the disk server are in the
                2+GB/s range, so I know the disk server can do it.</div>
              <div><br>
              </div>
              <div>-Tom</div>
              <div><br>
              </div>
              <div><br>
                <div>
                  <div>On Feb 27, 2013, at 7:41 PM, Bryan Whitehead &lt;<a moz-do-not-send="true" href="mailto:driver@megahappy.net">driver@megahappy.net</a>&gt;
                    wrote:</div>
                  <br class="Apple-interchange-newline">
                  <blockquote type="cite">
                    <div dir="ltr">Are your figures 700-800M<b>Byte</b>/sec?
                      Because that is probably as fast as your 10G nic
                      cards are able to do. You can test that by trying
                      to push a large amount of data over nc or ftp.
                      <div><br>
                      </div>
                      <div style="">
                        Might want to try Infiniband. 40G cards are
                        pretty routine.</div>
                    </div>
                    <div class="gmail_extra"><br>
                      <br>
                      <div class="gmail_quote">On Wed, Feb 27, 2013 at
                        3:45 PM, Thomas Wakefield <span dir="ltr">&lt;<a moz-do-not-send="true" href="mailto:twake@iges.org" target="_blank">twake@iges.org</a>&gt;</span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0
                          0 0 .8ex;border-left:1px #ccc
                          solid;padding-left:1ex">
                          <div style="word-wrap:break-word">I also get
                            the same performance running iozone for
                            large file sizes,&nbsp;iozone -u 1 -r 512k -s 2G
                            -I -F.
                            <div>
                              <br>
                            </div>
                            <div>Large file IO is what I need the system
                              to do. &nbsp;I am just shocked at the huge
                              difference between local IO and gluster
                              client IO. &nbsp;I know there should be some
                              difference, but 10x is unacceptable.</div>
                            <div><br>
                            </div>
                            <div>-Tom</div>
                            <div>
                              <div class="h5">
                                <div><br>
                                </div>
                                <div><br>
                                </div>
                                <div><br>
                                  <div>
                                    <div>On Feb 27, 2013, at 5:31 PM,
                                      Bryan Whitehead &lt;<a moz-do-not-send="true" href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>&gt;
                                      wrote:</div>
                                    <br>
                                    <blockquote type="cite">
                                      <div dir="ltr">Every time you
                                        open/close a file or a directory
                                        you will have to wait for locks
                                        which take time. This is totally
                                        expected.
                                        <div><br>
                                        </div>
                                        <div>Why don't you share what
                                          you want to do? iozone
                                          benchmarks look like crap but
                                          serving qcow2 files to qemu
                                          works fantastic for me. What
                                          are you doing? Make a
                                          benchmark that does that. If
                                          you are going to have many
                                          files with a wide&nbsp;variety&nbsp;of
                                          sizes glusterfs/fuse might not
                                          be what you are looking for.</div>
                                      </div>
                                      <div class="gmail_extra"><br>
                                        <br>
                                        <div class="gmail_quote">On Wed,
                                          Feb 27, 2013 at 12:56 PM,
                                          Thomas Wakefield <span dir="ltr">&lt;<a moz-do-not-send="true" href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a>&gt;</span>
                                          wrote:<br>
                                          <blockquote class="gmail_quote" style="margin:0 0 0
                                            .8ex;border-left:1px #ccc
                                            solid;padding-left:1ex">
                                            <div style="word-wrap:break-word">I
                                              have tested everything,
                                              small and large files. &nbsp;I
                                              have used file sizes
                                              ranging from 128k up to
                                              multiple GB files. &nbsp;All
                                              the reads are bad.
                                              <div>
                                                <br>
                                              </div>
                                              <div><br>
                                                <div><br>
                                                </div>
                                                <div>Here is a fairly
                                                  exhaustive iozone auto
                                                  test:</div>
                                                <div><br>
                                                </div>
                                                <div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; random &nbsp;random &nbsp;
                                                    &nbsp;bkwd &nbsp; record &nbsp;
                                                    stride &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                                                    &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; KB
                                                    &nbsp;reclen &nbsp; write
                                                    rewrite &nbsp; &nbsp;read &nbsp;
                                                    &nbsp;reread &nbsp; &nbsp;read &nbsp;
                                                    write &nbsp; &nbsp;read
                                                    &nbsp;rewrite &nbsp; &nbsp; read &nbsp;
                                                    fwrite frewrite &nbsp;
                                                    fread &nbsp;freread</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 64
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 40222 &nbsp;
                                                    63492 &nbsp; &nbsp;26868 &nbsp;
                                                    &nbsp;30060 &nbsp; &nbsp;1620 &nbsp;
                                                    71037 &nbsp; &nbsp;1572 &nbsp;
                                                    &nbsp;70570 &nbsp; &nbsp;31294 &nbsp;
                                                    &nbsp;77096 &nbsp; &nbsp;72475 &nbsp;
                                                    14736 &nbsp; &nbsp;13928</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 64
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp; 99207
                                                    &nbsp;116366 &nbsp; &nbsp;13591 &nbsp;
                                                    &nbsp;13513 &nbsp; &nbsp;3214 &nbsp;
                                                    97690 &nbsp; &nbsp;3155 &nbsp;
                                                    109978 &nbsp; &nbsp;28920 &nbsp;
                                                    152018 &nbsp; 158480 &nbsp;
                                                    18936 &nbsp; &nbsp;17625</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 64
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;230257
                                                    &nbsp;253766 &nbsp; &nbsp;25156 &nbsp;
                                                    &nbsp;28713 &nbsp; 10867
                                                    &nbsp;223732 &nbsp; &nbsp;8873 &nbsp;
                                                    244297 &nbsp; &nbsp;54796 &nbsp;
                                                    303383 &nbsp; 312204 &nbsp;
                                                    15062 &nbsp; &nbsp;13545</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 64
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;255943
                                                    &nbsp;234481 &nbsp;5735102
                                                    &nbsp;7100397 &nbsp; 11897
                                                    &nbsp;318502 &nbsp; 13681 &nbsp;
                                                    347801 &nbsp; &nbsp;24214 &nbsp;
                                                    695778 &nbsp; 528618 &nbsp;
                                                    25838 &nbsp; &nbsp;28094</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 64
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;214096
                                                    &nbsp;681644 &nbsp;6421025
                                                    &nbsp;7100397 &nbsp; 27453
                                                    &nbsp;292156 &nbsp; 28117 &nbsp;
                                                    621657 &nbsp; &nbsp;27338 &nbsp;
                                                    376062 &nbsp; 512471 &nbsp;
                                                    28569 &nbsp; &nbsp;32534</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 74329 &nbsp;
                                                    75468 &nbsp; &nbsp;26428 &nbsp;
                                                    &nbsp;41089 &nbsp; &nbsp;1131 &nbsp;
                                                    72857 &nbsp; &nbsp;1118 &nbsp;
                                                    &nbsp;66976 &nbsp; &nbsp; 1597 &nbsp;
                                                    &nbsp;73778 &nbsp; &nbsp;78343 &nbsp;
                                                    13351 &nbsp; &nbsp;13026</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;100862
                                                    &nbsp;135170 &nbsp; &nbsp;24966 &nbsp;
                                                    &nbsp;16734 &nbsp; &nbsp;2617
                                                    &nbsp;118966 &nbsp; &nbsp;2560 &nbsp;
                                                    120406 &nbsp; &nbsp;39156 &nbsp;
                                                    125121 &nbsp; 146613 &nbsp;
                                                    16177 &nbsp; &nbsp;16180</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;115114
                                                    &nbsp;253983 &nbsp; &nbsp;28212 &nbsp;
                                                    &nbsp;17854 &nbsp; &nbsp;5307
                                                    &nbsp;246180 &nbsp; &nbsp;5431 &nbsp;
                                                    229843 &nbsp; &nbsp;47335 &nbsp;
                                                    255920 &nbsp; 271173 &nbsp;
                                                    27256 &nbsp; &nbsp;24445</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;256042
                                                    &nbsp;391360 &nbsp; &nbsp;39848 &nbsp;
                                                    &nbsp;64258 &nbsp; 11329
                                                    &nbsp;290230 &nbsp; &nbsp;9905 &nbsp;
                                                    429563 &nbsp; &nbsp;38176 &nbsp;
                                                    490380 &nbsp; 463696 &nbsp;
                                                    20917 &nbsp; &nbsp;19219</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;248573
                                                    &nbsp;592699 &nbsp;4557257
                                                    &nbsp;6812590 &nbsp; 19583
                                                    &nbsp;452366 &nbsp; 29263 &nbsp;
                                                    603357 &nbsp; &nbsp;42967 &nbsp;
                                                    814915 &nbsp; 692017 &nbsp;
                                                    76327 &nbsp; &nbsp;37604</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;128
                                                    &nbsp; &nbsp; 128 &nbsp;921183
                                                    &nbsp;526444 &nbsp;5603747
                                                    &nbsp;5379161 &nbsp; 45614
                                                    &nbsp;390222 &nbsp; 65441 &nbsp;
                                                    826202 &nbsp; &nbsp;41384 &nbsp;
                                                    662962 &nbsp;1040839 &nbsp;
                                                    78526 &nbsp; &nbsp;39023</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 76212 &nbsp;
                                                    77337 &nbsp; &nbsp;40295 &nbsp;
                                                    &nbsp;32125 &nbsp; &nbsp;1289 &nbsp;
                                                    71866 &nbsp; &nbsp;1261 &nbsp;
                                                    &nbsp;64645 &nbsp; &nbsp; 1436 &nbsp;
                                                    &nbsp;57309 &nbsp; &nbsp;53048 &nbsp;
                                                    23073 &nbsp; &nbsp;29550</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;126922
                                                    &nbsp;141976 &nbsp; &nbsp;26237 &nbsp;
                                                    &nbsp;25130 &nbsp; &nbsp;2566
                                                    &nbsp;128058 &nbsp; &nbsp;2565 &nbsp;
                                                    138981 &nbsp; &nbsp; 2985 &nbsp;
                                                    125060 &nbsp; 133603 &nbsp;
                                                    22840 &nbsp; &nbsp;24955</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;242883
                                                    &nbsp;263636 &nbsp; &nbsp;41850 &nbsp;
                                                    &nbsp;24371 &nbsp; &nbsp;4902
                                                    &nbsp;250009 &nbsp; &nbsp;5290 &nbsp;
                                                    248792 &nbsp; &nbsp;89353 &nbsp;
                                                    243821 &nbsp; 247303 &nbsp;
                                                    26965 &nbsp; &nbsp;26199</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;409074
                                                    &nbsp;439732 &nbsp; &nbsp;40101 &nbsp;
                                                    &nbsp;39335 &nbsp; 11953
                                                    &nbsp;436870 &nbsp; 11209 &nbsp;
                                                    430218 &nbsp; &nbsp;83743 &nbsp;
                                                    409542 &nbsp; 479390 &nbsp;
                                                    30821 &nbsp; &nbsp;27750</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;259935
                                                    &nbsp;571502 &nbsp; &nbsp;64840 &nbsp;
                                                    &nbsp;71847 &nbsp; 22537
                                                    &nbsp;617161 &nbsp; 23383 &nbsp;
                                                    392047 &nbsp; &nbsp;91852 &nbsp;
                                                    672010 &nbsp; 802614 &nbsp;
                                                    41673 &nbsp; &nbsp;53111</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; 128 &nbsp;847597
                                                    &nbsp;812329 &nbsp; 185517 &nbsp;
                                                    &nbsp;83198 &nbsp; 49383
                                                    &nbsp;708831 &nbsp; 44668 &nbsp;
                                                    794889 &nbsp; &nbsp;74267
                                                    &nbsp;1180188 &nbsp;1662639 &nbsp;
                                                    54303 &nbsp; &nbsp;41018</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;256
                                                    &nbsp; &nbsp; 256 &nbsp;481324
                                                    &nbsp;709299 &nbsp;5217259
                                                    &nbsp;5320671 &nbsp; 44668
                                                    &nbsp;719277 &nbsp; 40954 &nbsp;
                                                    808050 &nbsp; &nbsp;41302 &nbsp;
                                                    790209 &nbsp; 771473 &nbsp;
                                                    62224 &nbsp; &nbsp;35754</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 77667 &nbsp;
                                                    75226 &nbsp; &nbsp;35102 &nbsp;
                                                    &nbsp;29696 &nbsp; &nbsp;1337 &nbsp;
                                                    66262 &nbsp; &nbsp;1451 &nbsp;
                                                    &nbsp;67680 &nbsp; &nbsp; 1413 &nbsp;
                                                    &nbsp;69265 &nbsp; &nbsp;69142 &nbsp;
                                                    42084 &nbsp; &nbsp;27897</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;134311
                                                    &nbsp;144341 &nbsp; &nbsp;30144 &nbsp;
                                                    &nbsp;24646 &nbsp; &nbsp;2102
                                                    &nbsp;134143 &nbsp; &nbsp;2209 &nbsp;
                                                    134699 &nbsp; &nbsp; 2296 &nbsp;
                                                    108110 &nbsp; 128616 &nbsp;
                                                    25104 &nbsp; &nbsp;29123</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;200085
                                                    &nbsp;248787 &nbsp; &nbsp;30235 &nbsp;
                                                    &nbsp;25697 &nbsp; &nbsp;4196
                                                    &nbsp;247240 &nbsp; &nbsp;4179 &nbsp;
                                                    256116 &nbsp; &nbsp; 4768 &nbsp;
                                                    250003 &nbsp; 226436 &nbsp;
                                                    32351 &nbsp; &nbsp;28455</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;330341
                                                    &nbsp;439805 &nbsp; &nbsp;26440 &nbsp;
                                                    &nbsp;39284 &nbsp; &nbsp;8744
                                                    &nbsp;457611 &nbsp; &nbsp;8006 &nbsp;
                                                    424168 &nbsp; 125953 &nbsp;
                                                    425935 &nbsp; 448813 &nbsp;
                                                    27660 &nbsp; &nbsp;26951</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;483906
                                                    &nbsp;733729 &nbsp; &nbsp;48747 &nbsp;
                                                    &nbsp;41121 &nbsp; 16032
                                                    &nbsp;555938 &nbsp; 17424 &nbsp;
                                                    587256 &nbsp; 187343 &nbsp;
                                                    366977 &nbsp; 735740 &nbsp;
                                                    41700 &nbsp; &nbsp;41548</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; 128 &nbsp;836636
                                                    &nbsp;907717 &nbsp; &nbsp;69359 &nbsp;
                                                    &nbsp;94921 &nbsp; 42443
                                                    &nbsp;761031 &nbsp; 36828 &nbsp;
                                                    964378 &nbsp; 123165 &nbsp;
                                                    651383 &nbsp; 695697 &nbsp;
                                                    58368 &nbsp; &nbsp;44459</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; 256 &nbsp;520879
                                                    &nbsp;860437 &nbsp; 145534 &nbsp;
                                                    135523 &nbsp; 40267
                                                    &nbsp;847532 &nbsp; 31585 &nbsp;
                                                    663252 &nbsp; &nbsp;69696
                                                    &nbsp;1270846 &nbsp;1492545 &nbsp;
                                                    48822 &nbsp; &nbsp;48092</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;512
                                                    &nbsp; &nbsp; 512 &nbsp;782951
                                                    &nbsp;973118 &nbsp;3099691
                                                    &nbsp;2942541 &nbsp; 42328
                                                    &nbsp;871966 &nbsp; 46218 &nbsp;
                                                    911184 &nbsp; &nbsp;49791 &nbsp;
                                                    953248 &nbsp;1036527 &nbsp;
                                                    52723 &nbsp; &nbsp;48347</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 76218 &nbsp;
                                                    69362 &nbsp; &nbsp;36431 &nbsp;
                                                    &nbsp;28711 &nbsp; &nbsp;1137 &nbsp;
                                                    66171 &nbsp; &nbsp;1174 &nbsp;
                                                    &nbsp;68938 &nbsp; &nbsp; 1125 &nbsp;
                                                    &nbsp;70566 &nbsp; &nbsp;70845 &nbsp;
                                                    34942 &nbsp; &nbsp;28914</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;126045
                                                    &nbsp;140524 &nbsp; &nbsp;37836 &nbsp;
                                                    &nbsp;15664 &nbsp; &nbsp;2698
                                                    &nbsp;126000 &nbsp; &nbsp;2557 &nbsp;
                                                    125566 &nbsp; &nbsp; 2567 &nbsp;
                                                    110858 &nbsp; 127255 &nbsp;
                                                    26764 &nbsp; &nbsp;27945</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;243398
                                                    &nbsp;261429 &nbsp; &nbsp;40238 &nbsp;
                                                    &nbsp;23263 &nbsp; &nbsp;3987
                                                    &nbsp;246400 &nbsp; &nbsp;3882 &nbsp;
                                                    260746 &nbsp; &nbsp; 4093 &nbsp;
                                                    236652 &nbsp; 236874 &nbsp;
                                                    31429 &nbsp; &nbsp;25076</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;383109
                                                    &nbsp;422076 &nbsp; &nbsp;41731 &nbsp;
                                                    &nbsp;41605 &nbsp; &nbsp;8277
                                                    &nbsp;473441 &nbsp; &nbsp;7775 &nbsp;
                                                    415261 &nbsp; &nbsp; 8588 &nbsp;
                                                    394765 &nbsp; 407306 &nbsp;
                                                    40089 &nbsp; &nbsp;28537</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;590145
                                                    &nbsp;619156 &nbsp; &nbsp;39623 &nbsp;
                                                    &nbsp;53267 &nbsp; 15051
                                                    &nbsp;722717 &nbsp; 14624 &nbsp;
                                                    753000 &nbsp; 257294 &nbsp;
                                                    597784 &nbsp; 620946 &nbsp;
                                                    38619 &nbsp; &nbsp;44073</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; 128 1077836
                                                    1124099 &nbsp; &nbsp;56192 &nbsp;
                                                    &nbsp;64916 &nbsp; 36851
                                                    1102176 &nbsp; 37198
                                                    &nbsp;1082454 &nbsp; 281548 &nbsp;
                                                    829175 &nbsp; 792604 &nbsp;
                                                    47975 &nbsp; &nbsp;51913</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; 256 &nbsp;941918
                                                    1074331 &nbsp; &nbsp;72783 &nbsp;
                                                    &nbsp;81450 &nbsp; 26778
                                                    1099636 &nbsp; 32395
                                                    &nbsp;1060013 &nbsp; 183218
                                                    &nbsp;1024121 &nbsp; 995171 &nbsp;
                                                    44371 &nbsp; &nbsp;45448</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp; 512 &nbsp;697483
                                                    1130312 &nbsp; 100324 &nbsp;
                                                    114682 &nbsp; 48215
                                                    1041758 &nbsp; 41480
                                                    &nbsp;1058967 &nbsp; &nbsp;90156 &nbsp;
                                                    994020 &nbsp;1563622 &nbsp;
                                                    56328 &nbsp; &nbsp;46370</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 1024
                                                    &nbsp; &nbsp;1024 &nbsp;931702
                                                    1087111 &nbsp;4609294
                                                    &nbsp;4199201 &nbsp; 44191
                                                    &nbsp;949834 &nbsp; 45594 &nbsp;
                                                    970656 &nbsp; &nbsp;56674 &nbsp;
                                                    933525 &nbsp;1075676 &nbsp;
                                                    44876 &nbsp; &nbsp;46115</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 71438 &nbsp;
                                                    67066 &nbsp; &nbsp;58319 &nbsp;
                                                    &nbsp;38913 &nbsp; &nbsp;1147 &nbsp;
                                                    44147 &nbsp; &nbsp;1043 &nbsp;
                                                    &nbsp;42916 &nbsp; &nbsp; &nbsp;967 &nbsp;
                                                    &nbsp;66416 &nbsp; &nbsp;67205 &nbsp;
                                                    45953 &nbsp; &nbsp;96750</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;141926
                                                    &nbsp;134567 &nbsp; &nbsp;61101 &nbsp;
                                                    &nbsp;55445 &nbsp; &nbsp;2596 &nbsp;
                                                    77528 &nbsp; &nbsp;2564 &nbsp;
                                                    &nbsp;80402 &nbsp; &nbsp; 4258 &nbsp;
                                                    124211 &nbsp; 120747 &nbsp;
                                                    53888 &nbsp; 100337</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;254344
                                                    &nbsp;255585 &nbsp; &nbsp;71550 &nbsp;
                                                    &nbsp;74500 &nbsp; &nbsp;5410
                                                    &nbsp;139365 &nbsp; &nbsp;5201 &nbsp;
                                                    141484 &nbsp; &nbsp; 5171 &nbsp;
                                                    205521 &nbsp; 213113 &nbsp;
                                                    67048 &nbsp; &nbsp;57304</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;397833
                                                    &nbsp;411261 &nbsp; &nbsp;56676 &nbsp;
                                                    &nbsp;80027 &nbsp; 10440
                                                    &nbsp;260034 &nbsp; 10126 &nbsp;
                                                    230238 &nbsp; &nbsp;10814 &nbsp;
                                                    391665 &nbsp; 383379 &nbsp;
                                                    79333 &nbsp; &nbsp;60877</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;595167
                                                    &nbsp;687205 &nbsp; &nbsp;64262 &nbsp;
                                                    &nbsp;87327 &nbsp; 20772
                                                    &nbsp;456430 &nbsp; 19960 &nbsp;
                                                    477064 &nbsp; &nbsp;23190 &nbsp;
                                                    540220 &nbsp; 563096 &nbsp;
                                                    86812 &nbsp; &nbsp;92565</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; 128 &nbsp;833585
                                                    &nbsp;933403 &nbsp; 121926 &nbsp;
                                                    118621 &nbsp; 37700
                                                    &nbsp;690020 &nbsp; 37575 &nbsp;
                                                    733254 &nbsp; 567449 &nbsp;
                                                    712337 &nbsp; 734006 &nbsp;
                                                    92011 &nbsp; 104934</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; 256 &nbsp;799003
                                                    &nbsp;949499 &nbsp; 143688 &nbsp;
                                                    125659 &nbsp; 40871
                                                    &nbsp;892757 &nbsp; 37977 &nbsp;
                                                    880494 &nbsp; 458281 &nbsp;
                                                    836263 &nbsp; 901375
                                                    &nbsp;131332 &nbsp; 110237</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp; 512 &nbsp;979936
                                                    1040724 &nbsp; 120896 &nbsp;
                                                    138013 &nbsp; 54381
                                                    &nbsp;859783 &nbsp; 48721 &nbsp;
                                                    780491 &nbsp; 279203
                                                    &nbsp;1068824 &nbsp;1087085 &nbsp;
                                                    97886 &nbsp; &nbsp;98078</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp;1024 &nbsp;901754
                                                    &nbsp;987938 &nbsp; &nbsp;53352 &nbsp;
                                                    &nbsp;53043 &nbsp; 72727
                                                    1054522 &nbsp; 68269 &nbsp;
                                                    992275 &nbsp; 181253
                                                    &nbsp;1309480 &nbsp;1524983
                                                    &nbsp;121600 &nbsp; &nbsp;95585</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 2048
                                                    &nbsp; &nbsp;2048 &nbsp;831890
                                                    1021540 &nbsp;4257067
                                                    &nbsp;3302797 &nbsp; 75672
                                                    &nbsp;984203 &nbsp; 80181 &nbsp;
                                                    826209 &nbsp; &nbsp;94278 &nbsp;
                                                    966920 &nbsp;1027159
                                                    &nbsp;111832 &nbsp; 105921</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 66195 &nbsp;
                                                    67316 &nbsp; &nbsp;62171 &nbsp;
                                                    &nbsp;74785 &nbsp; &nbsp;1328 &nbsp;
                                                    28963 &nbsp; &nbsp;1329 &nbsp;
                                                    &nbsp;26397 &nbsp; &nbsp; 1223 &nbsp;
                                                    &nbsp;71470 &nbsp; &nbsp;69317 &nbsp;
                                                    55903 &nbsp; &nbsp;84915</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;122221
                                                    &nbsp;120057 &nbsp; &nbsp;90537 &nbsp;
                                                    &nbsp;60958 &nbsp; &nbsp;2598 &nbsp;
                                                    47312 &nbsp; &nbsp;2468 &nbsp;
                                                    &nbsp;59783 &nbsp; &nbsp; 2640 &nbsp;
                                                    128674 &nbsp; 127872 &nbsp;
                                                    41285 &nbsp; &nbsp;40422</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;238321
                                                    &nbsp;239251 &nbsp; &nbsp;29336 &nbsp;
                                                    &nbsp;32121 &nbsp; &nbsp;4153 &nbsp;
                                                    89262 &nbsp; &nbsp;3986 &nbsp;
                                                    &nbsp;96930 &nbsp; &nbsp; 4608 &nbsp;
                                                    229970 &nbsp; 237108 &nbsp;
                                                    55039 &nbsp; &nbsp;56983</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;417110
                                                    &nbsp;421356 &nbsp; &nbsp;30974 &nbsp;
                                                    &nbsp;50000 &nbsp; &nbsp;8382
                                                    &nbsp;156676 &nbsp; &nbsp;7886 &nbsp;
                                                    153841 &nbsp; &nbsp; 7900 &nbsp;
                                                    359585 &nbsp; 367288 &nbsp;
                                                    26611 &nbsp; &nbsp;25952</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;648008
                                                    &nbsp;668066 &nbsp; &nbsp;32193 &nbsp;
                                                    &nbsp;29389 &nbsp; 14830
                                                    &nbsp;273265 &nbsp; 14822 &nbsp;
                                                    282211 &nbsp; &nbsp;19653 &nbsp;
                                                    581898 &nbsp; 620798 &nbsp;
                                                    51281 &nbsp; &nbsp;50218</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; 128 &nbsp;779422
                                                    &nbsp;848564 &nbsp; &nbsp;55594 &nbsp;
                                                    &nbsp;60253 &nbsp; 37108
                                                    &nbsp;451296 &nbsp; 35908 &nbsp;
                                                    491361 &nbsp; &nbsp;37567 &nbsp;
                                                    738163 &nbsp; 728059 &nbsp;
                                                    67681 &nbsp; &nbsp;66440</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; 256 &nbsp;865623
                                                    &nbsp;886986 &nbsp; &nbsp;71368 &nbsp;
                                                    &nbsp;63947 &nbsp; 44255
                                                    &nbsp;645961 &nbsp; 42689 &nbsp;
                                                    719491 &nbsp; 736707 &nbsp;
                                                    819696 &nbsp; 837641 &nbsp;
                                                    57059 &nbsp; &nbsp;60347</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp; 512 &nbsp;852099
                                                    &nbsp;889650 &nbsp; &nbsp;68870 &nbsp;
                                                    &nbsp;73891 &nbsp; 31185
                                                    &nbsp;845224 &nbsp; 30259 &nbsp;
                                                    830153 &nbsp; 392334 &nbsp;
                                                    910442 &nbsp; 961983 &nbsp;
                                                    60083 &nbsp; &nbsp;55558</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp;1024 &nbsp;710357
                                                    &nbsp;867810 &nbsp; &nbsp;29377 &nbsp;
                                                    &nbsp;29522 &nbsp; 49954
                                                    &nbsp;846640 &nbsp; 43665 &nbsp;
                                                    926298 &nbsp; 213677 &nbsp;
                                                    986226 &nbsp;1115445 &nbsp;
                                                    55130 &nbsp; &nbsp;59205</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp;2048 &nbsp;826479
                                                    &nbsp;908420 &nbsp; &nbsp;43191 &nbsp;
                                                    &nbsp;42075 &nbsp; 59684
                                                    &nbsp;904022 &nbsp; 58601 &nbsp;
                                                    855664 &nbsp; 115105
                                                    &nbsp;1418322 &nbsp;1524415 &nbsp;
                                                    60548 &nbsp; &nbsp;66066</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 4096
                                                    &nbsp; &nbsp;4096 &nbsp;793351
                                                    &nbsp;855111 &nbsp;3232454
                                                    &nbsp;3673419 &nbsp; 66018
                                                    &nbsp;861413 &nbsp; 48833 &nbsp;
                                                    847852 &nbsp; &nbsp;45914 &nbsp;
                                                    852268 &nbsp; 842075 &nbsp;
                                                    42980 &nbsp; &nbsp;48374</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 67340 &nbsp;
                                                    69421 &nbsp; &nbsp;42198 &nbsp;
                                                    &nbsp;31740 &nbsp; &nbsp; 994 &nbsp;
                                                    23251 &nbsp; &nbsp;1166 &nbsp;
                                                    &nbsp;16813 &nbsp; &nbsp; &nbsp;837 &nbsp;
                                                    &nbsp;73827 &nbsp; &nbsp;73126 &nbsp;
                                                    25169 &nbsp; &nbsp;29610</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;137150
                                                    &nbsp;125622 &nbsp; &nbsp;29131 &nbsp;
                                                    &nbsp;36439 &nbsp; &nbsp;2051 &nbsp;
                                                    44342 &nbsp; &nbsp;1988 &nbsp;
                                                    &nbsp;48930 &nbsp; &nbsp; 2315 &nbsp;
                                                    134183 &nbsp; 135367 &nbsp;
                                                    31080 &nbsp; &nbsp;33573</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;237366
                                                    &nbsp;220826 &nbsp; &nbsp;24810 &nbsp;
                                                    &nbsp;26584 &nbsp; &nbsp;3576 &nbsp;
                                                    88004 &nbsp; &nbsp;3769 &nbsp;
                                                    &nbsp;78717 &nbsp; &nbsp; 4289 &nbsp;
                                                    233751 &nbsp; 235355 &nbsp;
                                                    23302 &nbsp; &nbsp;28742</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;457447
                                                    &nbsp;454404 &nbsp; &nbsp;31594 &nbsp;
                                                    &nbsp;27750 &nbsp; &nbsp;8141
                                                    &nbsp;142022 &nbsp; &nbsp;7846 &nbsp;
                                                    143984 &nbsp; &nbsp; 9322 &nbsp;
                                                    353147 &nbsp; 396188 &nbsp;
                                                    34203 &nbsp; &nbsp;33265</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;670645
                                                    &nbsp;655259 &nbsp; &nbsp;28630 &nbsp;
                                                    &nbsp;23255 &nbsp; 16669
                                                    &nbsp;237476 &nbsp; 16965 &nbsp;
                                                    244968 &nbsp; &nbsp;15607 &nbsp;
                                                    590365 &nbsp; 575320 &nbsp;
                                                    49998 &nbsp; &nbsp;43305</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; 128 &nbsp;658676
                                                    &nbsp;760982 &nbsp; &nbsp;44197 &nbsp;
                                                    &nbsp;47802 &nbsp; 28693
                                                    &nbsp;379523 &nbsp; 26614 &nbsp;
                                                    378328 &nbsp; &nbsp;27184 &nbsp;
                                                    720997 &nbsp; 702038 &nbsp;
                                                    51707 &nbsp; &nbsp;49733</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; 256 &nbsp;643370
                                                    &nbsp;698683 &nbsp; &nbsp;56233 &nbsp;
                                                    &nbsp;63165 &nbsp; 28846
                                                    &nbsp;543952 &nbsp; 27745 &nbsp;
                                                    576739 &nbsp; &nbsp;44014 &nbsp;
                                                    701007 &nbsp; 725534 &nbsp;
                                                    59611 &nbsp; &nbsp;58985</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp; 512 &nbsp;696884
                                                    &nbsp;776793 &nbsp; &nbsp;67258 &nbsp;
                                                    &nbsp;52705 &nbsp; 18711
                                                    &nbsp;698854 &nbsp; 21004 &nbsp;
                                                    694124 &nbsp; 621695 &nbsp;
                                                    784812 &nbsp; 773331 &nbsp;
                                                    43101 &nbsp; &nbsp;47659</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp;1024 &nbsp;729664
                                                    &nbsp;810451&nbsp; &nbsp;15470 &nbsp;
                                                    &nbsp;15875 &nbsp; 31318
                                                    &nbsp;801490 &nbsp; 38123 &nbsp;
                                                    812944 &nbsp; 301222 &nbsp;
                                                    804323 &nbsp; 832765 &nbsp;
                                                    54308 &nbsp; &nbsp;53376</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp;2048 &nbsp;749217 &nbsp;
                                                    68757 &nbsp; &nbsp;21914 &nbsp;
                                                    &nbsp;22667 &nbsp; 48971
                                                    &nbsp;783309 &nbsp; 48132 &nbsp;
                                                    782738 &nbsp; 172848 &nbsp;
                                                    907408 &nbsp; 929324 &nbsp;
                                                    51156 &nbsp; &nbsp;50565</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp;4096 &nbsp;707677
                                                    &nbsp;763960 &nbsp; &nbsp;32063 &nbsp;
                                                    &nbsp;31928 &nbsp; 47809
                                                    &nbsp;751692 &nbsp; 49560 &nbsp;
                                                    786339 &nbsp; &nbsp;93445
                                                    &nbsp;1046761 &nbsp;1297876 &nbsp;
                                                    48037 &nbsp; &nbsp;51680</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8192
                                                    &nbsp; &nbsp;8192 &nbsp;623817
                                                    &nbsp;746288 &nbsp;2815955
                                                    &nbsp;3137358 &nbsp; 48722
                                                    &nbsp;741633 &nbsp; 35428 &nbsp;
                                                    753787 &nbsp; &nbsp;49626 &nbsp;
                                                    803683 &nbsp; 823800 &nbsp;
                                                    48977 &nbsp; &nbsp;52895</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; &nbsp; 4 &nbsp; 72372 &nbsp;
                                                    73651 &nbsp; &nbsp;34471 &nbsp;
                                                    &nbsp;30788 &nbsp; &nbsp; 960 &nbsp;
                                                    23610 &nbsp; &nbsp; 903 &nbsp;
                                                    &nbsp;22316 &nbsp; &nbsp; &nbsp;891 &nbsp;
                                                    &nbsp;71445 &nbsp; &nbsp;71138 &nbsp;
                                                    56451 &nbsp; &nbsp;55129</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; &nbsp; 8 &nbsp;137920
                                                    &nbsp;141704 &nbsp; &nbsp;50830 &nbsp;
                                                    &nbsp;33857 &nbsp; &nbsp;1935 &nbsp;
                                                    41934 &nbsp; &nbsp;2275 &nbsp;
                                                    &nbsp;35588 &nbsp; &nbsp; 3608 &nbsp;
                                                    130757 &nbsp; 137801 &nbsp;
                                                    51621 &nbsp; &nbsp;48525</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; &nbsp;16 &nbsp;245369
                                                    &nbsp;242460 &nbsp; &nbsp;41808 &nbsp;
                                                    &nbsp;29770 &nbsp; &nbsp;3605 &nbsp;
                                                    75682 &nbsp; &nbsp;4355 &nbsp;
                                                    &nbsp;75315 &nbsp; &nbsp; 4767 &nbsp;
                                                    241100 &nbsp; 239693 &nbsp;
                                                    53263 &nbsp; &nbsp;30785</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; &nbsp;32 &nbsp;448877
                                                    &nbsp;433956 &nbsp; &nbsp;31846 &nbsp;
                                                    &nbsp;35010 &nbsp; &nbsp;7973
                                                    &nbsp;118181 &nbsp; &nbsp;8819 &nbsp;
                                                    112703 &nbsp; &nbsp; 8177 &nbsp;
                                                    381734 &nbsp; 391651 &nbsp;
                                                    57749 &nbsp; &nbsp;63417</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;710831
                                                    &nbsp;700712 &nbsp; &nbsp;66792 &nbsp;
                                                    &nbsp;68864 &nbsp; 20176
                                                    &nbsp;209806 &nbsp; 19034 &nbsp;
                                                    207852 &nbsp; &nbsp;21255 &nbsp;
                                                    589503 &nbsp; 601379
                                                    &nbsp;104567 &nbsp; 105162</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; 128 &nbsp;836901
                                                    &nbsp;860867 &nbsp; 104226 &nbsp;
                                                    100373 &nbsp; 40899
                                                    &nbsp;358865 &nbsp; 40946 &nbsp;
                                                    360562 &nbsp; &nbsp;39415 &nbsp;
                                                    675968 &nbsp; 691538 &nbsp;
                                                    96086 &nbsp; 105695</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; 256 &nbsp;798081
                                                    &nbsp;828146 &nbsp; 107103 &nbsp;
                                                    120433 &nbsp; 39084
                                                    &nbsp;595325 &nbsp; 39050 &nbsp;
                                                    593110 &nbsp; &nbsp;56925 &nbsp;
                                                    763466 &nbsp; 797859
                                                    &nbsp;109645 &nbsp; 113414</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp; 512 &nbsp;810851
                                                    &nbsp;843931 &nbsp; 113564 &nbsp;
                                                    106202 &nbsp; 35111
                                                    &nbsp;714831 &nbsp; 46244 &nbsp;
                                                    745947 &nbsp; &nbsp;53636 &nbsp;
                                                    802902 &nbsp; 760172
                                                    &nbsp;110492 &nbsp; 100879</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp;1024 &nbsp;726399
                                                    &nbsp;820219 &nbsp; &nbsp;22106 &nbsp;
                                                    &nbsp;22987 &nbsp; 53087
                                                    &nbsp;749053 &nbsp; 54781 &nbsp;
                                                    777705 &nbsp;1075341 &nbsp;
                                                    772686 &nbsp; 809723
                                                    &nbsp;100349 &nbsp; &nbsp;96619</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp;2048 &nbsp;807772
                                                    &nbsp;856458 &nbsp; &nbsp;23920 &nbsp;
                                                    &nbsp;23617 &nbsp; 66320
                                                    &nbsp;829576 &nbsp; 72105 &nbsp;
                                                    740848 &nbsp; 656379 &nbsp;
                                                    864539 &nbsp; 835446 &nbsp;
                                                    93499 &nbsp; 101714</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp;4096 &nbsp;797470
                                                    &nbsp;840596 &nbsp; &nbsp;27270 &nbsp;
                                                    &nbsp;28132 &nbsp; 88784
                                                    &nbsp;834938 &nbsp; 83428 &nbsp;
                                                    836809 &nbsp; 342898 &nbsp;
                                                    953274 &nbsp; 969699 &nbsp;
                                                    96499 &nbsp; 100657</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; &nbsp;8192 &nbsp;760063
                                                    &nbsp;812942 &nbsp; &nbsp;41022 &nbsp;
                                                    &nbsp;40014 &nbsp;102020
                                                    &nbsp;812135 &nbsp;104063 &nbsp;
                                                    820346 &nbsp; 229870
                                                    &nbsp;1252409 &nbsp;1398306
                                                    &nbsp;100959 &nbsp; 103793</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;16384
                                                    &nbsp; 16384 &nbsp;743107
                                                    &nbsp;825054 &nbsp;3074035
                                                    &nbsp;3551062 &nbsp; 97740
                                                    &nbsp;826175 &nbsp;103597 &nbsp;
                                                    803926 &nbsp; &nbsp;94681 &nbsp;
                                                    820474 &nbsp; 829106 &nbsp;
                                                    97168 &nbsp; &nbsp;92018</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;713528
                                                    &nbsp;714585 &nbsp; &nbsp;71310 &nbsp;
                                                    &nbsp;74296 &nbsp; 21135
                                                    &nbsp;194177 &nbsp; 21551 &nbsp;
                                                    202998 &nbsp; &nbsp;21110 &nbsp;
                                                    571030 &nbsp; 595232 &nbsp;
                                                    98598 &nbsp; 103373</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp; 128 &nbsp;766698
                                                    &nbsp;820615 &nbsp; 103028 &nbsp;
                                                    102269 &nbsp; 39013
                                                    &nbsp;328409 &nbsp; 40325 &nbsp;
                                                    346403 &nbsp; &nbsp;41084 &nbsp;
                                                    684604 &nbsp; 702513
                                                    &nbsp;103887 &nbsp; 100689</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp; 256 &nbsp;766591
                                                    &nbsp;819138 &nbsp; 107557 &nbsp;
                                                    113878 &nbsp; 42088
                                                    &nbsp;574878 &nbsp; 41778 &nbsp;
                                                    557989 &nbsp; &nbsp;55045 &nbsp;
                                                    749922 &nbsp; 754434
                                                    &nbsp;100964 &nbsp; 104866</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp; 512 &nbsp;774365
                                                    &nbsp;846915 &nbsp; 104255 &nbsp;
                                                    106094 &nbsp; 35280
                                                    &nbsp;728957 &nbsp; 25829 &nbsp;
                                                    734316 &nbsp; &nbsp;26990 &nbsp;
                                                    750614 &nbsp; 800955 &nbsp;
                                                    55013 &nbsp; &nbsp;63750</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp;1024 &nbsp;759113
                                                    &nbsp;806241 &nbsp; &nbsp;22033 &nbsp;
                                                    &nbsp;22622 &nbsp; 37322
                                                    &nbsp;761870 &nbsp; 45003 &nbsp;
                                                    766950 &nbsp; &nbsp;54519 &nbsp;
                                                    755666 &nbsp; 741339 &nbsp;
                                                    70018 &nbsp; &nbsp;66255</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp;2048 &nbsp;797566
                                                    &nbsp;860344 &nbsp; &nbsp;22893 &nbsp;
                                                    &nbsp;23859 &nbsp; 49973
                                                    &nbsp;778872 &nbsp; 53509 &nbsp;
                                                    779539 &nbsp; 796674 &nbsp;
                                                    817078 &nbsp; 797585 &nbsp;
                                                    56750 &nbsp; &nbsp;57047</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp;4096 &nbsp;796693
                                                    &nbsp;798985 &nbsp; &nbsp;24267 &nbsp;
                                                    &nbsp;25222 &nbsp; 54410
                                                    &nbsp;781810 &nbsp; 53053 &nbsp;
                                                    802845 &nbsp; 421421 &nbsp;
                                                    832858 &nbsp; 855811 &nbsp;
                                                    56748 &nbsp; &nbsp;60507</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; &nbsp;8192 &nbsp;741543
                                                    &nbsp;795144 &nbsp; &nbsp;29254 &nbsp;
                                                    &nbsp;29150 &nbsp; 58678
                                                    &nbsp;830011 &nbsp; 55545 &nbsp;
                                                    833914 &nbsp; 221323 &nbsp;
                                                    914182 &nbsp; 952916 &nbsp;
                                                    56120 &nbsp; &nbsp;60014</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32768
                                                    &nbsp; 16384 &nbsp;754641
                                                    &nbsp;819797 &nbsp; &nbsp;44183 &nbsp;
                                                    &nbsp;45718 &nbsp; 67293
                                                    &nbsp;785047 &nbsp; 65078 &nbsp;
                                                    807273 &nbsp; 132136
                                                    &nbsp;1240414 &nbsp;1267713 &nbsp;
                                                    56585 &nbsp; &nbsp;56167</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;663333
                                                    &nbsp;656588 &nbsp; &nbsp;32211 &nbsp;
                                                    &nbsp;31925 &nbsp; 17324
                                                    &nbsp;202043 &nbsp; 15477 &nbsp;
                                                    215981 &nbsp; &nbsp;15496 &nbsp;
                                                    592126 &nbsp; 597188 &nbsp;
                                                    86431 &nbsp; &nbsp;93969</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp; 128 &nbsp;716218
                                                    &nbsp;791603 &nbsp; &nbsp;92035 &nbsp;
                                                    &nbsp;95614 &nbsp; 40330
                                                    &nbsp;330127 &nbsp; 39295 &nbsp;
                                                    346542 &nbsp; &nbsp;39469 &nbsp;
                                                    697385 &nbsp; 714453
                                                    &nbsp;105094 &nbsp; &nbsp;98327</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp; 256 &nbsp;769283
                                                    &nbsp;798701 &nbsp; 110285 &nbsp;
                                                    110085 &nbsp; 38404
                                                    &nbsp;535730 &nbsp; 39939 &nbsp;
                                                    585023 &nbsp; &nbsp;52402 &nbsp;
                                                    741206 &nbsp; 749899 &nbsp;
                                                    98671 &nbsp; 100106</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp; 512 &nbsp;728151
                                                    &nbsp;814516 &nbsp; 112550 &nbsp;
                                                    108873 &nbsp; 32284
                                                    &nbsp;690164 &nbsp; 38911 &nbsp;
                                                    708244 &nbsp; &nbsp;50047 &nbsp;
                                                    778669 &nbsp; 765303 &nbsp;
                                                    98333 &nbsp; &nbsp;92728</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp;1024 &nbsp;773504
                                                    &nbsp;827108 &nbsp; &nbsp;21542 &nbsp;
                                                    &nbsp;20741 &nbsp; 45901
                                                    &nbsp;775473 &nbsp; 57282 &nbsp;
                                                    807232 &nbsp; &nbsp;69359 &nbsp;
                                                    770793 &nbsp; 766736
                                                    &nbsp;101465 &nbsp; 100905</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp;2048 &nbsp;767661
                                                    &nbsp;853446 &nbsp; &nbsp;21077 &nbsp;
                                                    &nbsp;21260 &nbsp; 59825
                                                    &nbsp;762711 &nbsp; 71985 &nbsp;
                                                    802488 &nbsp; &nbsp;87290 &nbsp;
                                                    772402 &nbsp; 775685 &nbsp;
                                                    96832 &nbsp; 103189</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp;4096 &nbsp;767669
                                                    &nbsp;828165 &nbsp; &nbsp;20681 &nbsp;
                                                    &nbsp;21480 &nbsp; 79798
                                                    &nbsp;829757 &nbsp; 91755 &nbsp;
                                                    838367 &nbsp;1379411 &nbsp;
                                                    791248 &nbsp; 783023
                                                    &nbsp;106433 &nbsp; 112769</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; &nbsp;8192 &nbsp;739291
                                                    &nbsp;784946 &nbsp; &nbsp;23100 &nbsp;
                                                    &nbsp;22454 &nbsp; 85945
                                                    &nbsp;785134 &nbsp; 87081 &nbsp;
                                                    828489 &nbsp; 642088 &nbsp;
                                                    816781 &nbsp; 798275 &nbsp;
                                                    93307 &nbsp; &nbsp;95318</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;65536
                                                    &nbsp; 16384 &nbsp;736401
                                                    &nbsp;791669 &nbsp; &nbsp;26345 &nbsp;
                                                    &nbsp;26384 &nbsp; 93082
                                                    &nbsp;799356 &nbsp; 91419 &nbsp;
                                                    751766 &nbsp; 385721 &nbsp;
                                                    911919 &nbsp; 884688
                                                    &nbsp;109668 &nbsp; &nbsp;96374</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;658361
                                                    &nbsp;661211 &nbsp; &nbsp;79790 &nbsp;
                                                    &nbsp;81016 &nbsp; 19460
                                                    &nbsp;182084 &nbsp; 19325 &nbsp;
                                                    195428 &nbsp; &nbsp;19553 &nbsp;
                                                    551809 &nbsp; 554426 &nbsp;
                                                    96581 &nbsp; &nbsp;81979</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp; 128 &nbsp;759806
                                                    &nbsp;838994 &nbsp; &nbsp;51063 &nbsp;
                                                    &nbsp;56976 &nbsp; 33624
                                                    &nbsp;326699 &nbsp; 36329 &nbsp;
                                                    360593 &nbsp; &nbsp;35679 &nbsp;
                                                    710602 &nbsp; 716538 &nbsp;
                                                    81725 &nbsp; &nbsp;51819</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp; 256 &nbsp;787375
                                                    &nbsp;843763 &nbsp; &nbsp;54664 &nbsp;
                                                    &nbsp;65031 &nbsp; 33847
                                                    &nbsp;557817 &nbsp; 35093 &nbsp;
                                                    581552 &nbsp; &nbsp;45163 &nbsp;
                                                    750234 &nbsp; 758452 &nbsp;
                                                    82704 &nbsp; &nbsp;83472</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp; 512 &nbsp;777460
                                                    &nbsp;818049 &nbsp; &nbsp;87831 &nbsp;
                                                    &nbsp;87110 &nbsp; 27994
                                                    &nbsp;705621 &nbsp; 31860 &nbsp;
                                                    723071 &nbsp; &nbsp;38483 &nbsp;
                                                    779489 &nbsp; 793068 &nbsp;
                                                    88281 &nbsp; &nbsp;86068</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp;1024 &nbsp;737158
                                                    &nbsp;782780 &nbsp; &nbsp;19005 &nbsp;
                                                    &nbsp;18897 &nbsp; 36614
                                                    &nbsp;735240 &nbsp; 38900 &nbsp;
                                                    812074 &nbsp; &nbsp;52149 &nbsp;
                                                    746598 &nbsp; 762694 &nbsp;
                                                    74376 &nbsp; &nbsp;82534</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp;2048 &nbsp;777386
                                                    &nbsp;817330 &nbsp; &nbsp;24169 &nbsp;
                                                    &nbsp;24693 &nbsp; 55171
                                                    &nbsp;769662 &nbsp; 56082 &nbsp;
                                                    788848 &nbsp; &nbsp;55937 &nbsp;
                                                    757716 &nbsp; 763147 &nbsp;
                                                    73508 &nbsp; &nbsp;82351</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp;4096 &nbsp;751757
                                                    &nbsp;792201 &nbsp; &nbsp;25283 &nbsp;
                                                    &nbsp;25173 &nbsp; 48455
                                                    &nbsp;813945 &nbsp; 56557 &nbsp;
                                                    837418 &nbsp; &nbsp;59578 &nbsp;
                                                    749106 &nbsp; 757939 &nbsp;
                                                    67846 &nbsp; &nbsp;70964</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; &nbsp;8192 &nbsp;743004
                                                    &nbsp;796673 &nbsp; &nbsp;20369 &nbsp;
                                                    &nbsp;20327 &nbsp; 88990
                                                    &nbsp;780925 &nbsp; 95956 &nbsp;
                                                    777211 &nbsp; 951528 &nbsp;
                                                    762755 &nbsp; 739851 &nbsp;
                                                    70850 &nbsp; &nbsp;69943</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 131072
                                                    &nbsp; 16384 &nbsp;766023
                                                    &nbsp;823611 &nbsp; &nbsp;22333 &nbsp;
                                                    &nbsp;22142 &nbsp;108337
                                                    &nbsp;796852 &nbsp;110848 &nbsp;
                                                    777704 &nbsp; 718954 &nbsp;
                                                    809861 &nbsp; 802822
                                                    &nbsp;115896 &nbsp; &nbsp;82903</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;655769
                                                    &nbsp;695392 &nbsp; &nbsp;45973 &nbsp;
                                                    &nbsp;45668 &nbsp; 19083
                                                    &nbsp;209417 &nbsp; 18768 &nbsp;
                                                    207272 &nbsp; &nbsp;19133 &nbsp;
                                                    601329 &nbsp; 599267
                                                    &nbsp;107705 &nbsp; &nbsp;87261</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp; 128 &nbsp;750052
                                                    &nbsp;798999 &nbsp; &nbsp;72711 &nbsp;
                                                    &nbsp;74713 &nbsp; 37329
                                                    &nbsp;339480 &nbsp; 36854 &nbsp;
                                                    365587 &nbsp; &nbsp;37187 &nbsp;
                                                    710898 &nbsp; 714372
                                                    &nbsp;108400 &nbsp; &nbsp;78806</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp; 256 &nbsp;788355
                                                    &nbsp;815575 &nbsp; &nbsp;80897 &nbsp;
                                                    &nbsp;75141 &nbsp; 37080
                                                    &nbsp;549762 &nbsp; 37521 &nbsp;
                                                    574393 &nbsp; &nbsp;51731 &nbsp;
                                                    780429 &nbsp; 751131 &nbsp;
                                                    97395 &nbsp; &nbsp;73842</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp; 512 &nbsp;714994
                                                    &nbsp;736649 &nbsp; &nbsp;80224 &nbsp;
                                                    &nbsp;80195 &nbsp; 32986
                                                    &nbsp;708196 &nbsp; 36048 &nbsp;
                                                    628508 &nbsp; &nbsp;44043 &nbsp;
                                                    767699 &nbsp; 781863
                                                    &nbsp;123267 &nbsp; &nbsp;83558</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp;1024 &nbsp;793541
                                                    &nbsp;800826 &nbsp; &nbsp;20200 &nbsp;
                                                    &nbsp;20428 &nbsp; 45599
                                                    &nbsp;752027 &nbsp; 48232 &nbsp;
                                                    798229 &nbsp; &nbsp;57011 &nbsp;
                                                    715643 &nbsp; 763691 &nbsp;
                                                    76741 &nbsp; &nbsp;73211</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp;2048 &nbsp;699395
                                                    &nbsp;782935 &nbsp; &nbsp;20227 &nbsp;
                                                    &nbsp;20519 &nbsp; 65785
                                                    &nbsp;779219 &nbsp; 80486 &nbsp;
                                                    803403 &nbsp; &nbsp;93381 &nbsp;
                                                    650054 &nbsp; 712171
                                                    &nbsp;115611 &nbsp; 108457</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp;4096 &nbsp;629531
                                                    &nbsp;766261 &nbsp; &nbsp;20544 &nbsp;
                                                    &nbsp;21563 &nbsp; 83107
                                                    &nbsp;819917 &nbsp; 93008 &nbsp;
                                                    844103 &nbsp; &nbsp;90625 &nbsp;
                                                    741879 &nbsp; 754931 &nbsp;
                                                    94114 &nbsp; &nbsp;94475</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; &nbsp;8192 &nbsp;731695
                                                    &nbsp;804112 &nbsp; &nbsp;21158 &nbsp;
                                                    &nbsp;21589 &nbsp; 82987
                                                    &nbsp;787869 &nbsp; 84858 &nbsp;
                                                    799483 &nbsp; &nbsp;95127 &nbsp;
                                                    731822 &nbsp; 737397 &nbsp;
                                                    95950 &nbsp; &nbsp;95160</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 262144
                                                    &nbsp; 16384 &nbsp;703376
                                                    &nbsp;802552 &nbsp; &nbsp;21940 &nbsp;
                                                    &nbsp;22371 &nbsp; 87748
                                                    &nbsp;777024 &nbsp; 90349 &nbsp;
                                                    779626 &nbsp;1018679 &nbsp;
                                                    765278 &nbsp; 619969 &nbsp;
                                                    96051 &nbsp; &nbsp;92734</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp; &nbsp;64 &nbsp;649937
                                                    &nbsp;710342 &nbsp; &nbsp;73778 &nbsp;
                                                    &nbsp;70893 &nbsp; 19734
                                                    &nbsp;207179 &nbsp; 19367 &nbsp;
                                                    199709 &nbsp; &nbsp;19493 &nbsp;
                                                    585910 &nbsp; 581340 &nbsp;
                                                    91902 &nbsp; &nbsp;91443</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp; 128 &nbsp;739270
                                                    &nbsp;805153 &nbsp; &nbsp;91507 &nbsp;
                                                    &nbsp;90395 &nbsp; 38547
                                                    &nbsp;319049 &nbsp; 37988 &nbsp;
                                                    358516 &nbsp; &nbsp;38123 &nbsp;
                                                    712602 &nbsp; 711814 &nbsp;
                                                    92463 &nbsp; &nbsp;90851</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp; 256 &nbsp;751279
                                                    &nbsp;802496 &nbsp; &nbsp;89481 &nbsp;
                                                    &nbsp;91022 &nbsp; 38869
                                                    &nbsp;544097 &nbsp; 38244 &nbsp;
                                                    569142 &nbsp; &nbsp;56130 &nbsp;
                                                    756367 &nbsp; 752466 &nbsp;
                                                    92677 &nbsp; &nbsp;92875</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp; 512 &nbsp;758393
                                                    &nbsp;819788 &nbsp; &nbsp;58085 &nbsp;
                                                    &nbsp;73437 &nbsp; 29030
                                                    &nbsp;700120 &nbsp; 28767 &nbsp;
                                                    739236 &nbsp; &nbsp;33925 &nbsp;
                                                    770177 &nbsp; 795879 &nbsp;
                                                    58963 &nbsp; &nbsp;72689</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp;1024 &nbsp;778385
                                                    &nbsp;818368 &nbsp; &nbsp;18822 &nbsp;
                                                    &nbsp;18639 &nbsp; 38774
                                                    &nbsp;798827 &nbsp; 50898 &nbsp;
                                                    799492 &nbsp; &nbsp;55679 &nbsp;
                                                    801849 &nbsp; 800917 &nbsp;
                                                    92778 &nbsp; &nbsp;82686</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp;2048 &nbsp;780594
                                                    &nbsp;834612 &nbsp; &nbsp;19936 &nbsp;
                                                    &nbsp;19994 &nbsp; 63147
                                                    &nbsp;769555 &nbsp; 63007 &nbsp;
                                                    802512 &nbsp; &nbsp;66322 &nbsp;
                                                    772900 &nbsp; 771314 &nbsp;
                                                    86628 &nbsp; &nbsp;93485</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp;4096 &nbsp;769806
                                                    &nbsp;812570 &nbsp; &nbsp;20165 &nbsp;
                                                    &nbsp;20137 &nbsp; 79896
                                                    &nbsp;783615 &nbsp; 71217 &nbsp;
                                                    &nbsp;80835 &nbsp; &nbsp;68550 &nbsp;
                                                    &nbsp;80166 &nbsp; &nbsp;73733
                                                    &nbsp;122559 &nbsp; 116844</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; &nbsp;8192 &nbsp;797504
                                                    &nbsp;887823 &nbsp; 123824 &nbsp;
                                                    124513 &nbsp; 94520
                                                    &nbsp;844880 &nbsp; 86441 &nbsp;
                                                    710319 &nbsp; &nbsp;65077 &nbsp;
                                                    636713 &nbsp; 819868 &nbsp;
                                                    87765 &nbsp; &nbsp;80121</div>
                                                  <div>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 524288
                                                    &nbsp; 16384 &nbsp;788468
                                                    &nbsp;858216 &nbsp; &nbsp;92580 &nbsp;
                                                    117354 &nbsp;109583
                                                    &nbsp;850179 &nbsp; 63966 &nbsp;
                                                    856965 &nbsp; 121151 &nbsp;
                                                    789252 &nbsp; 805040
                                                    &nbsp;123810 &nbsp; 119088</div>
                                                  <div>
                                                    <div><br>
                                                    </div>
                                                    <div><br>
                                                    </div>
                                                    <div><br>
                                                      <div><br>
                                                      </div>
                                                      <div><br>
                                                        <div>
                                                          <div>On Feb
                                                          27, 2013, at
                                                          2:46 PM, Bryan
                                                          Whitehead &lt;<a moz-do-not-send="true" href="mailto:driver@megahappy.net" target="_blank">driver@megahappy.net</a>&gt;
                                                          wrote:</div>
                                                          <br>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">How
                                                          are you doing
                                                          the read/write
                                                          tests on the
                                                          fuse/glusterfs
                                                          mountpoint?
                                                          Many small
                                                          files will be
                                                          slow because
                                                          all the time
                                                          is spent
                                                          coordinating
                                                          locks.</div>
                                                          <div class="gmail_extra"><br>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          On Wed, Feb
                                                          27, 2013 at
                                                          9:31 AM,
                                                          Thomas
                                                          Wakefield <span dir="ltr">&lt;<a moz-do-not-send="true" href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0
                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div style="word-wrap:break-word">Help
                                                          please-
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>I am
                                                          running 3.3.1
                                                          on Centos
                                                          using a 10GB
                                                          network. &nbsp;I
                                                          get reasonable
                                                          write speeds,
                                                          although I
                                                          think they
                                                          could be
                                                          faster. &nbsp;But
                                                          my read speeds
                                                          are REALLY
                                                          slow.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Executive
                                                          summary:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>On
                                                          gluster
                                                          client-</div>
                                                          <div>Writes
                                                          average about
                                                          700-800MB/s</div>
                                                          <div>Reads
                                                          average about
                                                          70-80MB/s</div>
                                                          <div><br>
                                                          </div>
                                                          <div>On
                                                          server-</div>
                                                          <div>Writes
                                                          average about
                                                          1-1.5GB/s</div>
                                                          <div>Reads
                                                          average about
                                                          2-3GB/s</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Any
                                                          thoughts?</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Here are
                                                          some
                                                          additional
                                                          details:</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Nothing
                                                          interesting in
                                                          any of the log
                                                          files,
                                                          everything is
                                                          very quite.</div>
                                                          <div>All
                                                          servers had no
                                                          other load,
                                                          and all
                                                          clients are
                                                          performing the
                                                          same way.</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>Volume
                                                          Name: shared</div>
                                                          <div>Type:
                                                          Distribute</div>
                                                          <div>Volume
                                                          ID:
                                                          de11cc19-0085-41c3-881e-995cca244620</div>
                                                          <div>Status:
                                                          Started</div>
                                                          <div>Number of
                                                          Bricks: 26</div>
                                                          <div>Transport-type:
                                                          tcp</div>
                                                          <div>Bricks:</div>
                                                          <div>Brick1:
                                                          fs-disk2:/storage/disk2a</div>
                                                          <div>Brick2:
                                                          fs-disk2:/storage/disk2b</div>
                                                          <div>Brick3:
                                                          fs-disk2:/storage/disk2d</div>
                                                          <div>Brick4:
                                                          fs-disk2:/storage/disk2e</div>
                                                          <div>Brick5:
                                                          fs-disk2:/storage/disk2f</div>
                                                          <div>Brick6:
                                                          fs-disk2:/storage/disk2g</div>
                                                          <div>Brick7:
                                                          fs-disk2:/storage/disk2h</div>
                                                          <div>Brick8:
                                                          fs-disk2:/storage/disk2i</div>
                                                          <div>
                                                          Brick9:
                                                          fs-disk2:/storage/disk2j</div>
                                                          <div>Brick10:
fs-disk2:/storage/disk2k</div>
                                                          <div>Brick11:
fs-disk2:/storage/disk2l</div>
                                                          <div>Brick12:
fs-disk2:/storage/disk2m</div>
                                                          <div>Brick13:
fs-disk2:/storage/disk2n</div>
                                                          <div>Brick14:
fs-disk2:/storage/disk2o</div>
                                                          <div>
                                                          Brick15:
                                                          fs-disk2:/storage/disk2p</div>
                                                          <div>Brick16:
fs-disk2:/storage/disk2q</div>
                                                          <div>Brick17:
fs-disk2:/storage/disk2r</div>
                                                          <div>Brick18:
fs-disk2:/storage/disk2s</div>
                                                          <div>Brick19:
fs-disk2:/storage/disk2t</div>
                                                          <div>
                                                          Brick20:
                                                          fs-disk2:/storage/disk2u</div>
                                                          <div>Brick21:
fs-disk2:/storage/disk2v</div>
                                                          <div>Brick22:
fs-disk2:/storage/disk2w</div>
                                                          <div>Brick23:
fs-disk2:/storage/disk2x</div>
                                                          <div>Brick24:
fs-disk3:/storage/disk3a</div>
                                                          <div>Brick25:
fs-disk3:/storage/disk3b</div>
                                                          <div>
                                                          Brick26:
                                                          fs-disk3:/storage/disk3c</div>
                                                          <div>Options
                                                          Reconfigured:</div>
                                                          <div>performance.write-behind:
                                                          on</div>
                                                          <div>performance.read-ahead:
                                                          on</div>
                                                          <div>performance.io-cache:
                                                          on</div>
                                                          <div>performance.stat-prefetch:
                                                          on</div>
                                                          <div>performance.quick-read:
                                                          on</div>
                                                          <div>cluster.min-free-disk:
                                                          500GB</div>
                                                          <div>nfs.disable:
                                                          off</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>sysctl.conf
                                                          settings for
                                                          10GBe</div>
                                                          <div>
                                                          <div>#
                                                          increase TCP
                                                          max buffer
                                                          size settable
                                                          using
                                                          setsockopt()</div>
                                                          <div>net.core.rmem_max
                                                          = 67108864&nbsp;</div>
                                                          <div>net.core.wmem_max
                                                          = 67108864&nbsp;</div>
                                                          <div>#
                                                          increase Linux
                                                          autotuning TCP
                                                          buffer limit</div>
                                                          <div>net.ipv4.tcp_rmem
                                                          = 4096 87380
                                                          67108864</div>
                                                          <div>net.ipv4.tcp_wmem
                                                          = 4096 65536
                                                          67108864</div>
                                                          <div>#
                                                          increase the
                                                          length of the
                                                          processor
                                                          input queue</div>
                                                          <div>net.core.netdev_max_backlog
                                                          = 250000</div>
                                                          <div>#
                                                          recommended
                                                          default
                                                          congestion
                                                          control is
                                                          htcp&nbsp;</div>
                                                          <div>net.ipv4.tcp_congestion_control=htcp</div>
                                                          <div>
                                                          # recommended
                                                          for hosts with
                                                          jumbo frames
                                                          enabled</div>
                                                          <div>net.ipv4.tcp_mtu_probing=1</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <br>
                                                          <br>
                                                          <div>
                                                          <span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px">
                                                          <div>
                                                          <span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px">
                                                          <div style="word-wrap:break-word">
                                                          <div>Thomas W.<br>
                                                          Sr. &nbsp;Systems
                                                          Administrator
                                                          COLA/IGES<br>
                                                          <a moz-do-not-send="true" href="mailto:twake@cola.iges.org" target="_blank">twake@cola.iges.org</a><br>
                                                          </div>
                                                          </div>
                                                          </span></span></div>
                                                          Affiliate&nbsp;<span style="font-size:12px">Computer Scientist GMU</span></span>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          <br>
_______________________________________________<br>
                                                          Gluster-users
                                                          mailing list<br>
                                                          <a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
                                                          <a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                        </div>
                                                        <br>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </div>
                                            </div>
                                          </blockquote>
                                        </div>
                                        <br>
                                      </div>
                                    </blockquote>
                                  </div>
                                  <br>
                                </div>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </blockquote>
                </div>
                <br>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </div>

_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>http://supercolony.gluster.org/mailman/listinfo/gluster-users</blockquote></div><br></div></body></html>