One problem right away is that 3.3 doesn't support rdma. I know it still
builds the rdma packages but rdma isn't a supported connection method
for 3.3. It was set aside so the 33. release could make it on time. As I
understand it, 3.1 was/is suppposed to be the first 3.x version to
fully support it. I'd upgrade and test again. That or downgrade to 3.2.7
which does in fact support it right now.<br>
<br>I'm not even sure how you got things mounted using the "rdma" semantics with 3.3.<br><br>My experiences so far were sort of disappointing until I found out a few key items about GlusterFS which I'd taken for granted.<br>
<br>1. Stripes are not what you might think. The I/O for a stripe does
_not_ fan out as in a raid card. It's an unfortunate use of the term
only describing and allowing you to store files larger than the max size
of a single brick.<br>
2. I/O is done in sync mode so cache coherency isn't an issue and to ensure the integrity of the data written. <br>3. The performance of a distributed volume far exceeds that of a stripe for my use. Again, depends on the size of the bricks.<br>
<br>These are things which effect my experience and now that I at least
_think_ I understand them my results make much more sense to me.
Generally my throughput maxes out around 800-900MB/s which is the limit
of my disk storage right now. <br>
<br>As a test I created as large of a volume as I could using ramdisks. I
did this to see just how much of a limiter my disks actually are, and I
was very surprised to see the speed NOT increase across the file system
using an rdma target on version 3.2.6. The local I/O reached 1.9GB/s.
So, in addition to my spindle based limit, I do not believe I am using
any more than a single lane of my IB cards (which are FDR ( 56Gb)
Mellanox).<br>
<br>Using the ib test utilities I can indeed max the connection at
6GB/s, which is really amazing to see but so far I just can't seem to
get GlusterFS to make use of the total B/W available.<br><br>Also, you might try putting your cads into "Connected Mode" and increasing your MTU to 64k.<br><br><br>Corey<br><br><br><br>