<span style="font-family:arial,helvetica,sans-serif">Hi RedHat & GlusterFS users,</span><br style="font-family:arial,helvetica,sans-serif"><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">Last week-end, I worked on a GlusterFS cluster upgrade, from 3.0.3 to 3.3.0.</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">We were using hand-made volume files defining 2 volumes, a distributed one, and a replicated-distribute one; both using the "transport-type ib-verbs" option.</span><br style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">One of our objectives was to use the "gluster" CLI tool (which doesn't existed in 3.0.3 - from what I remember).</span><br style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">Here is what we did:</span><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">1 - Shutdown all glusterfs instances</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">2 - Install the Gluster 3.3.0</span><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">3 - Start glusterd on all hosts</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">4 - Create a trusted pool</span> with all our hosts<br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">5 - Create "compatible volumes" using the CLI tool; using the same bricks we were using with our hand-made volfiles and using the "rdma" transport (since ib-verbs was no longer an option...)</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">6 - Mount the volumes</span><br style="font-family:arial,helvetica,sans-serif"><br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">Of course, we tested that scenario on VMs. No issues with data. We tested everything except.... RDMA !</span><br style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,helvetica,sans-serif">When we finally made the upgrade, everything went fine, except mounting the volumes. We got this kind of error messages in the log files:</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">"E [rdma.c:4458:tcp_connect_finish] 0-zodiac-client-3: tcp connect to failed (Connection refused)"<br>(notice the 2 white spaces between "connect to" and "failed")<br style="font-family:arial,helvetica,sans-serif">
</span><pre style="font-family:arial,helvetica,sans-serif">That reminded me of an issue when we had a problem with the subnet manager running on the IB switch. But this time, the switch wasn't responsible; IPoIB was still running fine...<br>
<br>I scratched my head more than once, thinking about what I could possibly have forgotten. Then I searched for all information I could find about RDMA and 3.3.0.<br><br>Here is what I found:<br>- On page 123 of the "GlusterFS Administration Guide 3.3.0", a small note saying: "NOTE: with 3.3.0 release, transport type 'rdma' and 'tcp,rdma' are not fully supported."<br>
- On July 7, Ling Ho started a thread on this mailing-list, with very similar symptoms: <a href="http://www.mail-archive.com/gluster-users@gluster.org/msg09326.html">http://www.mail-archive.com/gluster-users@gluster.org/msg09326.html</a> ; but he doesn't got any answer.<br>
<br>In the upgrade urgency, we weren't sure rollbacking to 3.0.3 was a good option (since we don't precisely known what XFS attributes were modified by 3.3.0 on the backend FS). So we switched to TCP (over IPoIB).<br>
It's working. We are now running 3.3.0. But we are no longer taking advantage of RDMA.<br><br>So here are a few questions:<br>- Did I missed something that prevented me to use RDMA in 3.3.0 ?<br>- Is there a way to use RDMA in 3.3.0 ?<br>
- Is there any official communication about the 3.3.0 RDMA issue ?<br>- Is there a 3.3.x release with RDMA support planned ? For when ?<br>- Will the RDMA transport be dropped in future releases ?<br><br>Thanks !<br>(and yeah, despite that issue, I still love GlusterFS :-)<br style="font-family:arial,helvetica,sans-serif" clear="all">
</pre>P<span style="font-family:arial,helvetica,sans-serif">hilippe Muller</span><br style="font-family:arial,helvetica,sans-serif">