Corey,<br><br>Make sure to test with direct I/O, otherwise the caching can give you unrealistic expectations of your actual throughput. Typically, using the ipoib driver is not recommended with Infiniband since you will introduce unnecessary overhead via TCP.<br><br>Knowing how you have Gluster configured is also essential to understanding whether any metrics you get from testing are within expectations. Including the output from `gluster volume info` is an essential piece of information.<br><br>Thanks,<br><br>Eco<br><br><div class='trimContent'> On Fri, Sep 7, 2012 at 1:45 AM, <span>Corey Kovacs <<a href='mailto:corey.kovacs@gmail.com' target='_blank'>corey.kovacs@gmail.com</a>></span> wrote: <br><blockquote>Folks,
<br>
<br>I finally got my hands on a 4x FDR (56Gb) Infiniband switch and 4 cards to do some testing of GlusterFS over that interface.
<br>
<br>So far, I am not getting the throughput I _think_ I should see.
<br>
<br>My config is made up of..
<br>
<br>4 dl360-g8's (three bricks and one client)
<br>4 4xFDR, dual port IB cards (one port configured in each card per host)
<br>1 4xFDR 36 port Mellanox Switch (managed and configured)
<br>GlusterFS 3.2.6
<br>RHEL6.3
<br>
<br> I have tested the IB cards and get about 6GB between hosts over raw IB. Using ipoib, I can get about 22Gb/sec. Not too shabby for a first go but I expected more (cards are in connected mode with MTU of 64k).
<br>
<br>My raw speed to the disks (though the buffer cache... I just realized I've not tested direct mode IO, I'll do that later today) is about 800MB/sec. I expect to see on the order of 2GB/sec (a little less than 3x800).
<br>
<br>When I write a large stream using dd, and watch the bricks I/O I see ~800MB/sec on each one, but at the end of the test, the report from dd indicates 800MB/sec.
<br>
<br>Am I missing something fundamental?
<br>
<br>Any pointers would be appreciated,
<br>
<br>
<br>Thanks!
<br>
<br>
<br>Corey
<br>
<br></blockquote><br></div>