<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">I tried downgrading to 3.2 and the rdma
connection works, with substantially improved performance in some
cases over 3.3 using IP over IB.<br>
<br>
Is there a version of 3.3 that anyone is using in production with
rdma that I can try?<br>
<br>
Iain<br>
<br>
<br>
On 05/30/13 09:24, Iain Buchanan wrote:<br>
</div>
<blockquote cite="mid:51A70CD6.2030708@gmail.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">I'm now able to connect to volumes
using rdma when they are created using "tcp,rdma", but it
appears that the data is still being transferred over ethernet.
If I run "ifstat" while running iozone I can see a lot of data
being moved through the ethernet adapter (nothing else is on the
box) and the performance is basically identical to plain
ethernet. When I create the volume with just "rdma" I can't
even mount the volume again (see error below).<br>
<br>
I've modified all the volume files, adding the
transport.rdma.listen-port lines after completing set-up
(anywhere there is a "option transport-type rdma" I've added
this line - quite a few places).<br>
<br>
volume create storage transport tcp,rdma
my_server1:/data/area<br>
volume add-brick storage replica 2 my_server2:/data/area<br>
volume start storage<br>
mount -t glusterfs my_server:storage.rdma /mnt/storage<br>
<br>
I'm now back to the error I had before the workaround on
tcp,rdma - the lines are in the config files, and I've restarted
the service. I can see my alterations in the config dumped into
the log, but then it returns to the original errors:<br>
<br>
[2013-05-30 09:13:41.871408] E [rdma.c:4604:tcp_connect_finish]
0-storage-client-0: tcp connect to failed (Connection refused)<br>
[2013-05-30 09:13:41.871467] W [rdma.c:4187:gf_rdma_disconnect]
(-->/usr/sbin/glusterfs(main+0x34d) [0x7f563a24c3ed]
(-->/usr/lib/libglusterfs.so.0(+0x3bd17) [0x7f5639de5d17]
(-->/usr/lib/glusterfs/3.3.1/rpc-transport/rdma.so(+0x5231)
[0x7f5634012231]))) 0-storage-client-0: disconnect called
(peer:)<br>
<br>
I saw a message on the mailing list that seems to suggest plain
"rdma" doesn't work in the 3.3 series - is this correct?<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.gluster.org/pipermail/gluster-users/2013-January/035115.html">http://www.gluster.org/pipermail/gluster-users/2013-January/035115.html</a><br>
<br>
(I can run ib_read_bw etc. between the two servers without
problems.)<br>
<br>
Iain<br>
<br>
<br>
On 05/30/13 08:11, Iain Buchanan wrote:<br>
</div>
<blockquote cite="mid:51A6FBB2.5010007@gmail.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Just to confirm - putting the line
"option transport.rdma.listen-port 24008" into the first
"volume" block in the two files with "rdma" in their names
under /var/lib/glusterd/vols/<volumename> seems to have
fixed the issue. I'm now able to mount and I can run iozone
on a single node. I'll give it a go with two nodes and see if
rdma makes any difference.<br>
<br>
Thanks for your help Joe!<br>
<br>
Iain<br>
<br>
<br>
On 05/30/13 07:59, Joe Julian wrote:<br>
</div>
<blockquote cite="mid:51A6F8DD.9030705@julianfamily.org"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 05/29/2013 11:42 PM, Iain
Buchanan wrote:<br>
</div>
<blockquote cite="mid:51A6F4DB.5050704@gmail.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Thanks Joe,<br>
<br>
I tried mounting with the ".rdma" suffix after creating
using "transport tcp,rdma" and I get the same "tcp connect
to failed" messages in the log - I'm using the version
from the semiosis PPA at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://launchpad.net/%7Esemiosis/+archive/ubuntu-glusterfs-3.3">https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3</a><br>
<br>
Looking at the dates on there I don't think it includes
this fix. I've sent the maintainer a message asking if
they could update it. In the meantime would Niels de Vos'
workaround work? (Setting transport.rdma.listen-port to
24008 in the glusterd.vol?)<br>
</div>
</blockquote>
<br>
Yes, I had someone else try that and it worked. Remember, any
changes made to the volume through the CLI will reset that
configuration, causing it to stop working until you edit
again.<br>
<br>
<blockquote cite="mid:51A6F4DB.5050704@gmail.com" type="cite">
<div class="moz-cite-prefix"> <br>
Iain<br>
<br>
On 05/30/13 07:06, Joe Julian wrote:<br>
</div>
<blockquote cite="mid:51A6EC58.3070701@julianfamily.org"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 05/29/2013 10:33 PM, Iain
Buchanan wrote:<br>
</div>
<blockquote
cite="mid:B4A78FEC-456B-46A8-B876-69F4727B1DB2@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
Hi,
<div><br>
<div>I'm running GlusterFS 3.3.1-ubuntu1~precise9 and
I'm having some problems with the "rdma" and
"tcp,rdma" options I hope someone can help me with.
</div>
<div><br>
</div>
<div>1. What does "tcp,rdma" actually do - does it let
you mix both types of client? (I did a few tests
with iozone and found it gave identical performance
to the "tcp".)</div>
<div><br>
</div>
<div>2. I can't get "rdma" to work, even in the
simplest case with a single node.</div>
<div><font face="Courier"><span class="Apple-tab-span"
style="white-space:pre"> </span>volume create
storage transport transport rdma
my_server:/data/area</font></div>
<div><font face="Courier"><span class="Apple-tab-span"
style="white-space:pre"> </span>volume start
storage</font></div>
<div><font face="Courier"><span class="Apple-tab-span"
style="white-space:pre"> </span>mount -t
glusterfs my_server:storage /mnt/storage</font></div>
<div><br>
</div>
<div>The last line hangs. Looking in
/var/log/glusterfs I can see the log for the volume:</div>
<div><br>
</div>
<div>
<div><font face="Courier">[2013-05-30
06:24:19.605315] E
[rdma.c:4604:tcp_connect_finish]
0-storage-client-0: <b>tcp connect to failed
(Connection refused)</b></font></div>
<div><font face="Courier">[2013-05-30
06:24:19.605713] W
[rdma.c:4187:gf_rdma_disconnect]
(-->/usr/sbin/glusterfs(main+0x34d)
[0x7f374d38a3ed]
(-->/usr/lib/libglusterfs.so.0(+0x3bd17)
[0x7f374cf23d17]
(-->/usr/lib/glusterfs/3.3.1/rpc-transport/rdma.so(+0x5231)
[0x7f3743398231]))) 0-storage-client-0:
disconnect called (peer:)</font></div>
<div><font face="Courier">[2013-05-30
06:24:19.605763] W
[rdma.c:4521:gf_rdma_handshake_pollerr]
(-->/usr/sbin/glusterfs(main+0x34d)
[0x7f374d38a3ed]
(-->/usr/lib/libglusterfs.so.0(+0x3bd17)
[0x7f374cf23d17]
(-->/usr/lib/glusterfs/3.3.1/rpc-transport/rdma.so(+0x5150)
[0x7f3743398150]))) 0-rpc-transport/rdma:
storage-client-0: peer () disconnected, cleaning
up</font></div>
</div>
<div><br>
</div>
<div>This block repeats every few seconds - the line
"tcp connect to failed" looks like it has lost the
server name somehow? </div>
<div><br>
</div>
<div>Iain</div>
</div>
<br>
</blockquote>
If you've installed from the yum repo (<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://goo.gl/s077x">http://goo.gl/s077x</a>) that
shouldn't be happening. kkeithley applied the patch. If
not, rdma's broken in 3.3.[01].
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=849122">https://bugzilla.redhat.com/show_bug.cgi?id=849122</a><br>
<br>
To mount via rdma when using tcp,rdma, mount -t glusterfs
server1:myvol.rdma /mnt/foo<br>
<br>
<br>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>