<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial">Hello,<div>I test performance of Infiniband rdma and tcp, using glusterfs 3.2.5. There are 10 servers and 2 clients, all of them been connected with Infiniband and have the same hardwares. Two hash volumes, each have 10 bricks, <span style="line-height: 1.7;">All bricks are 16TB ext4 filesystem of raid5 on different servers. </span><span style="line-height: 1.7;">One volume's transport type is rdma and the other tcp. </span></div><div><span style="line-height: 1.7;"><br></span></div><div>One client mounts the rdma volume and the other tcp volume. Using iozone to test the performance.</div><div><div>root@client-1:/mnt/rdma# iozone -i 0 -i 1 -s 10g -t 10 -R</div><div>root@client-2:/mnt/tcp# iozone -i 0 -i 1 -s 10g -t 10 -R</div><div><br></div><div>Performance of rdma volume: </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Read: 1.1GB/s </div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Write: 1.0GB/s</div><div>While the performance tcp volume:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span><span style="line-height: 1.7;">Read: </span><span style="line-height: 1.7;">23.5MB/s </span></div><div><span style="line-height: 1.7;"><span class="Apple-tab-span" style="white-space:pre"> </span>Write: 52.8MB/s</span></div></div><div><span style="line-height: 1.7;">I have checked the network and brick filesystem, all of them are normal. I wonder why performance of infiniband over tcp are so poor. </span></div><div><span style="line-height: 1.7;">Thanks for any help.</span></div><div><span style="line-height: 1.7;"><br></span></div><div><span style="line-height: 1.7;">Best Regards,</span></div><div><span style="line-height: 1.7;">Luna</span></div><div><pre style="text-align: left;"><pre><br></pre></pre><pre><br></pre></div>
</div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>