<div dir="ltr">justin,<div><br></div><div>yeah, this fabirc is all bran new mellanox, and all nodes are running their v2 stack. </div><div><br></div><div>of for a beg report, sure thing. I was thinking i would tack on a comment here:</div>
<div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=982757">https://bugzilla.redhat.com/show_bug.cgi?id=982757</a><br></div><div><br></div><div>since thats about the silent failure. </div></div><div class="gmail_extra">
<br clear="all"><div><div dir="ltr">--<div>Matthew Nicholson<div>Research Computing Specialist</div><div>Harvard FAS Research Computing</div><div><a href="mailto:matthew_nicholson@harvard.edu" target="_blank">matthew_nicholson@harvard.edu</a></div>
<div><br></div></div></div></div>
<br><br><div class="gmail_quote">On Wed, Jul 10, 2013 at 3:00 PM, Justin Clift <span dir="ltr"><<a href="mailto:jclift@redhat.com" target="_blank">jclift@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 10/07/2013, at 7:49 PM, Matthew Nicholson wrote:<br>
> Well, first of all,thank for the responses. The volume WAS failing over the tcp just as predicted,though WHY is unclear as the fabric is know working (has about 28K compute cores on it all doing heavy MPI testing on it), and the OFED/verbs stack is consistent across all client/storage systems (actually, the OS image is identical).<br>
><br>
> Thats quiet sad RDMA isn't going to make 3.4. We put a good deal of hopes and effort around planning for 3.4 for this storage systems, specifically for RDMA support (well, with warnings to the team that it wasn't in/test for 3.3 and that all we could do was HOPE it was in 3.4 and in time for when we want to go live). we're getting "okay" performance out of IPoIB right now, and our bottle neck actually seems to be the fabric design/layout, as we're peaking at about 4.2GB/s writing 10TB over 160 threads to this distributed volume.<br>
<br>
</div>Out of curiosity, are you running the stock OS provided infiniband stack, or are you using the "vendor optimised" version? (eg "Mellanox OFED" if you're using Mellanox cards)<br>
<br>
Asking because although I've not personally done any perf measurements between them, Mellanox swears the new v2 of their OFED stack is much higher performance than both the stock drivers or their v1 stack. IPoIB is especially tuned.<br>
<br>
I'd really like to get around to testing that some time, but it won't be soon. :(<br>
<div class="im"><br>
<br>
> When it IS ready and in 3.4.1 (hopefully!), having good docs around it, and maybe even a simple printf for the tcp failover would be huge for us.<br>
<br>
</div>Would you be ok to create a Bugzilla ticket, asking for that printf item?<br>
<br>
<a href="https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=rdma" target="_blank">https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=rdma</a><br>
<br>
It doesn't have to be complicated or super in depth or anything. :)<br>
<br>
Asking because when something is a ticket, the "task" is much easier to hand<br>
to someone so it gets done.<br>
<br>
If that's too much effort though, just tell me what you'd like as the ticket<br>
summary line + body text and I'll go create it. :)<br>
<div class="im HOEnZb"><br>
Regards and best wishes,<br>
<br>
Justin Clift<br>
<br>
</div><div class="HOEnZb"><div class="h5">--<br>
Open Source and Standards @ Red Hat<br>
<br>
<a href="http://twitter.com/realjustinclift" target="_blank">twitter.com/realjustinclift</a><br>
<br>
</div></div></blockquote></div><br></div>