<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <font size="-1"><font face="serif">thanks for posting<br>
        I'd be curious to see what kind of disproportion you get
        between:  raw fs / single brick volume with local fuse
        mountpoint which effectively points back to the same raw fs<br>
        from my quick tests I saw massive gap between the two<br>
        thanks <br>
      </font></font><br>
    On 30/04/12 18:39, Wipe_Out wrote:
    <blockquote
cite="mid:CAJEk6a-E8aQePB6W3E8Hb2dBBhtk5UumXVWav5qB6UgN_PW1Ng@mail.gmail.com"
      type="cite">
      <div class="gmail_extra">Submitting again without
        the attachment because the list moderators don't appear to be
        clearing the queue.. So I pasted the CSV below..<br>
        <br>
        <div class="gmail_quote">---------- Forwarded message ----------<br>
          From: <b class="gmail_sendername">Wipe_Out</b> <span
            dir="ltr">&lt;<a moz-do-not-send="true"
              href="mailto:wipe_out@users.sourceforge.net">wipe_out@users.sourceforge.net</a>&gt;</span><br>
          Date: 29 April 2012 19:35<br>
          Subject: Some benchmarks for anyone that's interested..<br>
          To: <a moz-do-not-send="true"
            href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
          <br>
          <br>
          Hi all,
          <div><br>
          </div>
          <div>Here are some benchmarks I have just done in a small 2
            node cluster with 3 x 500GB drives.. OS on drive 1 and the
            other two used for testing.. Interconnect is 1Gb Ethernet..</div>
          <div><br>
          </div>
          <div>
            Tests are:</div>
          <div>Single Disk (direct, no gluster)</div>
          <div>Gluster Replicated</div>
          <div>Gluster Striped Replicated</div>
          <div>Gluster Distributed Replicated</div>
          <div>Gluster Stripe</div>
          <div><br>
          </div>
          <div>Glusterfs version is 3.3 Beta3..</div>
          <div><br>
          </div>
          <div>Simple test with bonnie++ with 8GB size (double the
            memory of the nodes).. Composite CSV attached which is
            probably the easiest way to compare results..</div>
          <div><br>
          </div>
          <div><b style="font-family:'Times New Roman';font-size:medium"><span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Single
                Disk </span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Output------ --Sequential Input- --Random-</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Concurrency
                  1 <span style="white-space:pre-wrap"> </span>-Per
                Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Machine
                   <span style="white-space:pre-wrap"> </span>Size
                K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.locald
                8G   288  98 76689  11 32637   6  2030  92 91483   5
                206.1   2</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                        <span style="white-space:pre-wrap"> </span>29634us
                <span style="white-space:pre-wrap"> </span>226ms <span
                  style="white-space:pre-wrap"> </span>318ms   95196us
                  27554us<span style="white-space:pre-wrap"> </span>1017ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Create------ --------Random Create--------</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.localdoma
                -Create-- --Read--- -Delete-- -Create-- --Read---
                -Delete--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                         <span style="white-space:pre-wrap"> </span>files
                 /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                            <span style="white-space:pre-wrap"> </span>16
                12687  46 +++++ +++ +++++ +++ 21140  76 +++++ +++ +++++
                +++</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                          <span style="white-space:pre-wrap"> </span>113us
                <span style="white-space:pre-wrap"> </span>652us <span
                  style="white-space:pre-wrap"> </span>494us <span
                  style="white-space:pre-wrap"> </span>100us  <span
                  style="white-space:pre-wrap"> </span>23us  <span
                  style="white-space:pre-wrap"> </span>51us</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1.96,1.96,localhost.localdomain,1,1335711943,8G,,288,98,76689,11,32637,6,2030,92,91483,5,206.1,2,16,,,,,12687,46,+++++,+++,+++++,+++,21140,76,+++++,+++,+++++,+++,29634us,226ms,318ms,95196us,27554us,1017ms,113us,652us,494us,100us,23us,51us</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"></span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Gluster
                Replicated</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Output------ --Sequential Input- --Random-</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Concurrency
                  1 <span style="white-space:pre-wrap"> </span>-Per
                Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Machine
                   <span style="white-space:pre-wrap"> </span>Size
                K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.locald
                8G<span style="white-space:pre-wrap"> </span>12  12
                63474  12 21709   5  2522  96 56940   3 157.4   2</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                          <span style="white-space:pre-wrap"> </span>711ms
                <span style="white-space:pre-wrap"> </span>189ms <span
                  style="white-space:pre-wrap"> </span>385ms   14804us
                <span style="white-space:pre-wrap"> </span>256ms<span
                  style="white-space:pre-wrap"> </span>1303ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Create------ --------Random Create--------</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.localdoma
                -Create-- --Read--- -Delete-- -Create-- --Read---
                -Delete--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                         <span style="white-space:pre-wrap"> </span>files
                 /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                            <span style="white-space:pre-wrap"> </span>16
                  336   2  5160   3   452   2   304   2  1548   1   309
                  1</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                         <span style="white-space:pre-wrap"> </span>1035ms<span
                  style="white-space:pre-wrap"> </span>4804us<span
                  style="white-space:pre-wrap"> </span>4355ms<span
                  style="white-space:pre-wrap"> </span>1440ms   10184us<span
                  style="white-space:pre-wrap"> </span>8349ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1.96,1.96,localhost.localdomain,1,1335706557,8G,,12,12,63474,12,21709,5,2522,96,56940,3,157.4,2,16,,,,,336,2,5160,3,452,2,304,2,1548,1,309,1,711ms,189ms,385ms,14804us,256ms,1303ms,1035ms,4804us,4355ms,1440ms,10184us,8349ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"></span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Striped
                Replicated</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Output------ --Sequential Input- --Random-</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Concurrency
                  1 <span style="white-space:pre-wrap"> </span>-Per
                Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Machine
                   <span style="white-space:pre-wrap"> </span>Size
                K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.locald
                8G<span style="white-space:pre-wrap"> </span>12  11
                40659   7 15032   3  1915  75 63220   3 248.0   2</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                          <span style="white-space:pre-wrap"> </span>744ms<span
                  style="white-space:pre-wrap"> </span>1539ms <span
                  style="white-space:pre-wrap"> </span>530ms <span
                  style="white-space:pre-wrap"> </span>178ms <span
                  style="white-space:pre-wrap"> </span>455ms <span
                  style="white-space:pre-wrap"> </span>165ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Create------ --------Random Create--------</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.localdoma
                -Create-- --Read--- -Delete-- -Create-- --Read---
                -Delete--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                         <span style="white-space:pre-wrap"> </span>files
                 /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                            <span style="white-space:pre-wrap"> </span>16
                  167   1  1103   0   231   0   151   1   807   1   194
                  0</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                         <span style="white-space:pre-wrap"> </span>1263ms
                  60415us<span style="white-space:pre-wrap"> </span>1308ms<span
                  style="white-space:pre-wrap"> </span>1194ms<span
                  style="white-space:pre-wrap"> </span>8964us <span
                  style="white-space:pre-wrap"> </span>850ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1.96,1.96,localhost.localdomain,1,1335709868,8G,,12,11,40659,7,15032,3,1915,75,63220,3,248.0,2,16,,,,,167,1,1103,0,231,0,151,1,807,1,194,0,744ms,1539ms,530ms,178ms,455ms,165ms,1263ms,60415us,1308ms,1194ms,8964us,850ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"></span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Distribute
                Replicate</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Output------ --Sequential Input- --Random-</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Concurrency
                  1 <span style="white-space:pre-wrap"> </span>-Per
                Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Machine
                   <span style="white-space:pre-wrap"> </span>Size
                K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.locald
                8G<span style="white-space:pre-wrap"> </span>12  11
                56218  11 20631   6  2126  85 55694   3 151.9   1</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                          <span style="white-space:pre-wrap"> </span>785ms
                <span style="white-space:pre-wrap"> </span>205ms <span
                  style="white-space:pre-wrap"> </span>387ms <span
                  style="white-space:pre-wrap"> </span>196ms <span
                  style="white-space:pre-wrap"> </span>287ms<span
                  style="white-space:pre-wrap"> </span>1277ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Create------ --------Random Create--------</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.localdoma
                -Create-- --Read--- -Delete-- -Create-- --Read---
                -Delete--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                         <span style="white-space:pre-wrap"> </span>files
                 /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                            <span style="white-space:pre-wrap"> </span>16
                  254   2   963   0   387   1   221   1  1397   1   307
                  1</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                          <span style="white-space:pre-wrap"> </span>473ms
                  20351us <span style="white-space:pre-wrap"> </span>198ms
                <span style="white-space:pre-wrap"> </span>984ms <span
                  style="white-space:pre-wrap"> </span>252ms <span
                  style="white-space:pre-wrap"> </span>487ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1.96,1.96,localhost.localdomain,1,1335720170,8G,,12,11,56218,11,20631,6,2126,85,55694,3,151.9,1,16,,,,,254,2,963,0,387,1,221,1,1397,1,307,1,785ms,205ms,387ms,196ms,287ms,1277ms,473ms,20351us,198ms,984ms,252ms,487ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap"></span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Stripe</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Output------ --Sequential Input- --Random-</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Concurrency
                  1 <span style="white-space:pre-wrap"> </span>-Per
                Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Machine
                   <span style="white-space:pre-wrap"> </span>Size
                K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.locald
                8G<span style="white-space:pre-wrap"> </span>10   9
                48514   9 18294   5  1992  79 50725   3 285.0   3</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                         <span style="white-space:pre-wrap"> </span>1036ms
                <span style="white-space:pre-wrap"> </span>751ms <span
                  style="white-space:pre-wrap"> </span>441ms <span
                  style="white-space:pre-wrap"> </span>200ms <span
                  style="white-space:pre-wrap"> </span>249ms <span
                  style="white-space:pre-wrap"> </span>252ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Version
                 1.96   <span style="white-space:pre-wrap"> </span>------Sequential
                Create------ --------Random Create--------</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">localhost.localdoma
                -Create-- --Read--- -Delete-- -Create-- --Read---
                -Delete--</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                         <span style="white-space:pre-wrap"> </span>files
                 /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
                %CP</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">
                            <span style="white-space:pre-wrap"> </span>16
                  280   1  2017   1   342   0   265   1  1466   2   269
                  0</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">Latency
                         <span style="white-space:pre-wrap"> </span>2082ms
                  28109us<span style="white-space:pre-wrap"> </span>7048ms<span
                  style="white-space:pre-wrap"> </span>6559ms<span
                  style="white-space:pre-wrap"> </span>9363us   17815ms</span><br>
              <span
style="font-size:15px;font-family:Arial;font-weight:normal;vertical-align:baseline;white-space:pre-wrap">1.96,1.96,localhost.localdomain,1,1335721555,8G,,10,9,48514,9,18294,5,1992,79,50725,3,285.0,3,16,,,,,280,1,2017,1,342,0,265,1,1466,2,269,0,1036ms,751ms,441ms,200ms,249ms,252ms,2082ms,28109us,7048ms,6559ms,9363us,17815ms</span></b>
          </div>
        </div>
        <br>
      </div>
      <div class="gmail_extra">--- CSV Start ---</div>
      <div class="gmail_extra">
        <div class="gmail_extra">Description,format_version,bonnie_version,name,concurrency,seed,file_size,io_chunk_size,putc,putc_cpu,put_block,put_block_cpu,rewrite,rewrite_cpu,getc,getc_cpu,get_block,get_block_cpu,seeks,seeks_cpu,num_files,max_size,min_size,num_dirs,file_chunk_size,seq_create,seq_create_cpu,seq_stat,seq_stat_cpu,seq_del,seq_del_cpu,ran_create,ran_create_cpu,ran_stat,ran_stat_cpu,ran_del,ran_del_cpu,putc_latency,put_block_latency,rewrite_latency,getc_latency,get_block_latency,seeks_latency,seq_create_latency,seq_stat_latency,seq_del_latency,ran_create_latency,ran_stat_latency,ran_del_latency</div>
        <div class="gmail_extra">Single
Disk,1.96,1.96,localhost.localdomain,1,1335711943,8G,,288,98,76689,11,32637,6,2030,92,91483,5,206.1,2,16,,,,,12687,46,+++++,+++,+++++,+++,21140,76,+++++,+++,+++++,+++,29634us,226ms,318ms,95196us,27554us,1017ms,113us,652us,494us,100us,23us,51us</div>
        <div class="gmail_extra">Gluster
Replicated,1.96,1.96,localhost.localdomain,1,1335706557,8G,,12,12,63474,12,21709,5,2522,96,56940,3,157.4,2,16,,,,,336,2,5160,3,452,2,304,2,1548,1,309,1,711ms,189ms,385ms,14804us,256ms,1303ms,1035ms,4804us,4355ms,1440ms,10184us,8349ms</div>
        <div class="gmail_extra">Gluster Striped
Replicated,1.96,1.96,localhost.localdomain,1,1335709868,8G,,12,11,40659,7,15032,3,1915,75,63220,3,248.0,2,16,,,,,167,1,1103,0,231,0,151,1,807,1,194,0,744ms,1539ms,530ms,178ms,455ms,165ms,1263ms,60415us,1308ms,1194ms,8964us,850ms</div>
        <div class="gmail_extra">Gluster Distributed
Replicated,1.96,1.96,localhost.localdomain,1,1335720170,8G,,12,11,56218,11,20631,6,2126,85,55694,3,151.9,1,16,,,,,254,2,963,0,387,1,221,1,1397,1,307,1,785ms,205ms,387ms,196ms,287ms,1277ms,473ms,20351us,198ms,984ms,252ms,487ms</div>
        <div class="gmail_extra">Gluster
Stripe,1.96,1.96,localhost.localdomain,1,1335721555,8G,,10,9,48514,9,18294,5,1992,79,50725,3,285.0,3,16,,,,,280,1,2017,1,342,0,265,1,1466,2,269,0,1036ms,751ms,441ms,200ms,249ms,252ms,2082ms,28109us,7048ms,6559ms,9363us,17815ms</div>
        <div class="gmail_extra">--- CSV End ---</div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
</pre>
    </blockquote>
  </body>
</html>