<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">In some basic testing using 2 RDMA gluster servers configured as a replicated pair, I'm seeing better significantly better performance with IPoIB versus RDMA only volumes (almost 2x in write performance and ~10-20% in read). <div><div><br></div><div>Has anyone had a similar experience?<div><br></div><div><div>glusterfs 3.4.0 built on Jul 12 2013 20:36:46</div><div>Repository revision: <a href="git://git.gluster.com/glusterfs.git">git://git.gluster.com/glusterfs.git</a></div><div>Copyright (c) 2006-2011 Gluster Inc. <<a href="http://www.gluster.com">http://www.gluster.com</a>></div><div>GlusterFS comes with ABSOLUTELY NO WARRANTY.</div><div>You may redistribute copies of GlusterFS under the terms of the GNU General Public License.</div></div><div><br></div><div><p style="font-size: 13px; line-height: 17.328125px; padding: 0px; margin-bottom: 10px; font-family: Arial, Helvetica, FreeSans, sans-serif; text-align: left; "><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; ">Volume Name: perftest<br>Type: Replicate<br>Volume ID: 10578349-a466-497a-b7de-1fcd34f967da<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: rdma<br>Bricks:<br>Brick1: <a href="http://cs1-i/gluster/perftest" data-mce-href="http://cs1-i/gluster/perftest" style="color: rgb(50, 108, 166) !important; outline: none; ">cs1-i:/gluster/perftest</a><br>Brick2: <a href="http://cs2-i/gluster/perftest" data-mce-href="http://cs2-i/gluster/perftest" style="color: rgb(50, 108, 166) !important; outline: none; ">cs2-i:/gluster/perftest</a></p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; "><a href="http://cs1-i/perftest.rdma" data-mce-href="http://cs1-i/perftest.rdma" style="color: rgb(50, 108, 166) !important; outline: none; ">cs1-i:/perftest.rdma</a> 3.2T 13G 3.2T 1% /tmp/perftest</p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; ">Run began: Wed Jul 17 08:31:39 2013</p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; ">Record Size 1024 KB<br>File size set to 5242880 KB<br>Command line used: iozone -i0 -i1 -r 1024#k -s 5#g<br>Output is in Kbytes/sec<br>Time Resolution = 0.000001 seconds.<br>Processor cache size set to 1024 Kbytes.<br>Processor cache line size set to 32 bytes.<br>File stride size set to 17 * record size.<br>random random bkwd record stride <br>KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread<br>5242880 1024 422514 496985 364946 324213</p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; "><br></p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; ">Volume Name: perftest<br>Type: Replicate<br>Volume ID: 10578349-a466-497a-b7de-1fcd34f967da<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: <a href="http://cs1-i/gluster/perftest" data-mce-href="http://cs1-i/gluster/perftest" style="color: rgb(50, 108, 166) !important; outline: none; ">cs1-i:/gluster/perftest</a><br>Brick2: <a href="http://cs2-i/gluster/perftest" data-mce-href="http://cs2-i/gluster/perftest" style="color: rgb(50, 108, 166) !important; outline: none; ">cs2-i:/gluster/perftest</a></p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; "><a href="http://cs1-i/perftest.tcp" data-mce-href="http://cs1-i/perftest.tcp" style="color: rgb(50, 108, 166) !important; outline: none; ">cs1-i:/perftest.tcp</a> 3.2T 13G 3.2T 1% /tmp/perftest</p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; "><br>Run began: Wed Jul 17 08:38:23 2013</p><p style="line-height: 17.328125px; padding: 0px; margin-bottom: 10px; ">Record Size 1024 KB<br>File size set to 5242880 KB<br>Command line used: iozone -i0 -i1 -r 1024#k -s 5#g<br>Output is in Kbytes/sec<br>Time Resolution = 0.000001 seconds.<br>Processor cache size set to 1024 Kbytes.<br>Processor cache line size set to 32 bytes.<br>File stride size set to 17 * record size.<br>random random bkwd record stride <br>KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread<br>5242880 1024 557479 549629 667576 666251</p><div><br></div></p></div></div></div></body></html>