<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 07/16/2012 09:16 AM, Philippe Muller wrote:
<blockquote
cite="mid:CAGrsLzU=GrB8WKQxPwFg-J9wRUCzss6-j59+0scFcVrxw6eHKA@mail.gmail.com"
type="cite"><span style="font-family:arial,helvetica,sans-serif">Hi
RedHat & GlusterFS users,</span><br
style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">Last
week-end, I worked on a GlusterFS cluster upgrade, from 3.0.3 to
3.3.0.</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">We were using
hand-made volume files defining 2 volumes, a distributed one,
and a replicated-distribute one; both using the "transport-type
ib-verbs" option.</span><br
style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">One of our
objectives was to use the "gluster" CLI tool (which doesn't
existed in 3.0.3 - from what I remember).</span><br
style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">Here is what
we did:</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">1 - Shutdown
all glusterfs instances</span><br
style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">2 - Install
the Gluster 3.3.0</span><br
style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">3 - Start
glusterd on all hosts</span><br
style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">4 - Create a
trusted pool</span> with all our hosts<br
style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">5 - Create
"compatible volumes" using the CLI tool; using the same bricks
we were using with our hand-made volfiles and using the "rdma"
transport (since ib-verbs was no longer an option...)</span><br
style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">6 - Mount the
volumes</span><br style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">Of course, we
tested that scenario on VMs. No issues with data. We tested
everything except.... RDMA !</span><br
style="font-family:arial,helvetica,sans-serif">
<br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">When we
finally made the upgrade, everything went fine, except mounting
the volumes. We got this kind of error messages in the log
files:</span><br style="font-family:arial,helvetica,sans-serif">
<span style="font-family:arial,helvetica,sans-serif">"E
[rdma.c:4458:tcp_connect_finish] 0-zodiac-client-3: tcp connect
to failed (Connection refused)"<br>
(notice the 2 white spaces between "connect to" and "failed")<br
style="font-family:arial,helvetica,sans-serif">
</span>
<pre style="font-family:arial,helvetica,sans-serif">That reminded me of an issue when we had a problem with the subnet manager running on the IB switch. But this time, the switch wasn't responsible; IPoIB was still running fine...
I scratched my head more than once, thinking about what I could possibly have forgotten. Then I searched for all information I could find about RDMA and 3.3.0.
Here is what I found:
- On page 123 of the "GlusterFS Administration Guide 3.3.0", a small note saying: "NOTE: with 3.3.0 release, transport type 'rdma' and 'tcp,rdma' are not fully supported."
- On July 7, Ling Ho started a thread on this mailing-list, with very similar symptoms: <a moz-do-not-send="true" href="http://www.mail-archive.com/gluster-users@gluster.org/msg09326.html">http://www.mail-archive.com/gluster-users@gluster.org/msg09326.html</a> ; but he doesn't got any answer.
In the upgrade urgency, we weren't sure rollbacking to 3.0.3 was a good option (since we don't precisely known what XFS attributes were modified by 3.3.0 on the backend FS). So we switched to TCP (over IPoIB).
It's working. We are now running 3.3.0. But we are no longer taking advantage of RDMA.
So here are a few questions:
- Did I missed something that prevented me to use RDMA in 3.3.0 ?
- Is there a way to use RDMA in 3.3.0 ?
- Is there any official communication about the 3.3.0 RDMA issue ?
- Is there a 3.3.x release with RDMA support planned ? For when ?
- Will the RDMA transport be dropped in future releases ?
Thanks !
(and yeah, despite that issue, I still love GlusterFS :-)
</pre>
P<span style="font-family:arial,helvetica,sans-serif">hilippe
Muller</span><br style="font-family:arial,helvetica,sans-serif">
</blockquote>
I just came back from one week vacation. Yes, I didn't get any reply
from the list, and were not able to get RDMA working when the server
is configured for tcp,rdma. When I was doing testing, I had set up
the server using rdma only and totally missed this.<br>
<br>
I ended up using tcp with ipoverib. The performance is much better
than tcp over 10G/s. However, since I am in a mix environment, and
my I have to do some static routing on the gluster server. Basically
routing the ipoverib subnet to the 10G/s subnet which the bricks are
all set up with. Things have been working fine.<br>
<br>
...<br>
ling<br>
<br>
<br>
<br>
<br>
</body>
</html>