<div dir="ltr">Hi all,<br><div class="gmail_quote"><div dir="ltr"><div><br></div><div>After many days of experimentation, doc and mailing list reading, irc asking, etc, I think the crippled RDMA status in current versions of Gluster (3.3.x - 3.4.1) is a known issue. I'd like to confirm that, share my findings, and ask about any status updates/timelines.</div>
<div style="font-weight:bold"><b><br></b></div><div>After noticing that RDMA mounts were hanging with a new install of Gluster 3.4.1, I tested a series of different Gluster volumes. Simple (single brick), distributed, replicate, and distributed-replicate volumes were each tested for both tcp and rdma transport types. Detailed results are below, but the short version is that while <b>all volume types worked over tcp, only the simple (single brick) volume worked using rdma. All other volume types failed over rdma, </b>meaning that mount commands from the client hung forever.</div>
<div><br></div><div><b>Environment details:</b></div><div>OS: Debian Wheezy</div><div>Server type: Dell M610</div><div>Gluster version: 3.4.1, from Gluster Debian repository</div><div>Infiniband software: OFED 1.4.2, from Debian Wheezy stock packages</div>
<div>Infiniband card info: <a href="http://fpaste.org/45305/81273796/" target="_blank">http://fpaste.org/45305/81273796/</a></div><div>Loaded modules: <a href="http://fpaste.org/45306/73881138/" target="_blank">http://fpaste.org/45306/73881138/</a></div>
<div style="font-weight:bold"><b><br></b></div><b>RDMA successful configs:</b><div>Single brick</div><div><br></div><div><b>RDMA failed configs:</b></div><div><div>Distributed (2 bricks)</div><div>Replicate (2 bricks)</div>
<div>Distributed-Replicate (2 x 2 bricks)</div><div><br></div><div><b>TCP successful configs (all):</b></div><div>Single brick</div><div>Distributed (2 bricks)</div><div>Replicate (2 bricks)</div><div>Distributed-Replicate (2 x 2 bricks)<br>
</div><div><br></div><div><b>Example RDMA volume creation command:</b></div><div>gluster volume create dist-rdma transport rdma 192.168.255.120:/home/axiom/dist-rdma-1 192.168.255.120:/home/axiom/dist-rdma-2<br></div><div>
<br></div><div><b>Example RDMA mounting command:</b></div><div>mount -t glusterfs -o transport=rdma 192.168.255.120:/dist-rdma dist-rdma <br></div><div><br></div><div><b>Logs from example failed RDMA config (distributed/two bricks):</b></div>
<div>gluster volume info: <a href="http://fpaste.org/45298/38127208/" target="_blank">http://fpaste.org/45298/38127208/</a></div><div>gluster volume status: <a href="http://fpaste.org/45299/13812721/" target="_blank">http://fpaste.org/45299/13812721/</a></div>
<div>glusterd.vol.log excerpt: <a href="http://fpaste.org/45302/13812722/" target="_blank">http://fpaste.org/45302/13812722/</a></div><div>client log: <a href="http://fpaste.org/45303/38127234/" target="_blank">http://fpaste.org/45303/38127234/</a></div>
<div><br></div><div>These results somewhat agree with Justin Clift's findings during the GlusterFest (<a href="http://www.gluster.org/community/documentation/index.php/GlusterFest" target="_blank">http://www.gluster.org/community/documentation/index.php/GlusterFest</a>) testing, which evolved into this bug:</div>
<div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=978148" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=978148</a><br></div><div><br></div><div>However, in the bug report it's mentioned that only the distributed-replicate volume variant is failing, while I'm seeing distributed and replicate volumes fail also.</div>
<div><br></div><div>I'd be happy to create a new bug or update the existing bug if needed. Let me know if any additional information is needed.</div><div><br></div><div>Also, I dunno if there's a proper place to post a warning about RDMA's status, but it seems that a handful of people have banged their head against this problem. I'd suggest that if the resource don't exist to address this issue by 3.4.2 that a warning be issued when creating an RDMA volume, or perhaps that RDMA volume creation be disabled altogether.</div>
<div><br></div><div>Please let me know if we can be of any help in the future (testing, log output, etc).</div><div><br></div><div>Best,</div><div>Shane</div><span class="HOEnZb"><font color="#888888"><div><br></div>-- <br>
Shane StClair<br>Software Engineer<br>Axiom Consulting & Design<br>
<a href="http://www.axiomalaska.com" target="_blank">http://www.axiomalaska.com</a>
</font></span></div></div>
</div><br><br clear="all"><div><br></div>-- <br>Shane StClair<br>Software Engineer<br>Axiom Consulting & Design<br><a href="http://www.axiomalaska.com" target="_blank">http://www.axiomalaska.com</a>
</div>