<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">I'm now able to connect to volumes
using rdma when they are created using "tcp,rdma", but it appears
that the data is still being transferred over ethernet. If I run
"ifstat" while running iozone I can see a lot of data being moved
through the ethernet adapter (nothing else is on the box) and the
performance is basically identical to plain ethernet. When I
create the volume with just "rdma" I can't even mount the volume
again (see error below).<br>
<br>
I've modified all the volume files, adding the
transport.rdma.listen-port lines after completing set-up (anywhere
there is a "option transport-type rdma" I've added this line -
quite a few places).<br>
<br>
volume create storage transport tcp,rdma my_server1:/data/area<br>
volume add-brick storage replica 2 my_server2:/data/area<br>
volume start storage<br>
mount -t glusterfs my_server:storage.rdma /mnt/storage<br>
<br>
I'm now back to the error I had before the workaround on tcp,rdma
- the lines are in the config files, and I've restarted the
service. I can see my alterations in the config dumped into the
log, but then it returns to the original errors:<br>
<br>
[2013-05-30 09:13:41.871408] E [rdma.c:4604:tcp_connect_finish]
0-storage-client-0: tcp connect to failed (Connection refused)<br>
[2013-05-30 09:13:41.871467] W [rdma.c:4187:gf_rdma_disconnect]
(-->/usr/sbin/glusterfs(main+0x34d) [0x7f563a24c3ed]
(-->/usr/lib/libglusterfs.so.0(+0x3bd17) [0x7f5639de5d17]
(-->/usr/lib/glusterfs/3.3.1/rpc-transport/rdma.so(+0x5231)
[0x7f5634012231]))) 0-storage-client-0: disconnect called (peer:)<br>
<br>
I saw a message on the mailing list that seems to suggest plain
"rdma" doesn't work in the 3.3 series - is this correct?<br>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/pipermail/gluster-users/2013-January/035115.html">http://www.gluster.org/pipermail/gluster-users/2013-January/035115.html</a><br>
<br>
(I can run ib_read_bw etc. between the two servers without
problems.)<br>
<br>
Iain<br>
<br>
<br>
On 05/30/13 08:11, Iain Buchanan wrote:<br>
</div>
<blockquote cite="mid:51A6FBB2.5010007@gmail.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Just to confirm - putting the line
"option transport.rdma.listen-port 24008" into the first
"volume" block in the two files with "rdma" in their names under
/var/lib/glusterd/vols/<volumename> seems to have fixed
the issue. I'm now able to mount and I can run iozone on a
single node. I'll give it a go with two nodes and see if rdma
makes any difference.<br>
<br>
Thanks for your help Joe!<br>
<br>
Iain<br>
<br>
<br>
On 05/30/13 07:59, Joe Julian wrote:<br>
</div>
<blockquote cite="mid:51A6F8DD.9030705@julianfamily.org"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 05/29/2013 11:42 PM, Iain
Buchanan wrote:<br>
</div>
<blockquote cite="mid:51A6F4DB.5050704@gmail.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Thanks Joe,<br>
<br>
I tried mounting with the ".rdma" suffix after creating
using "transport tcp,rdma" and I get the same "tcp connect
to failed" messages in the log - I'm using the version from
the semiosis PPA at <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://launchpad.net/%7Esemiosis/+archive/ubuntu-glusterfs-3.3">https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.3</a><br>
<br>
Looking at the dates on there I don't think it includes this
fix. I've sent the maintainer a message asking if they
could update it. In the meantime would Niels de Vos'
workaround work? (Setting transport.rdma.listen-port to
24008 in the glusterd.vol?)<br>
</div>
</blockquote>
<br>
Yes, I had someone else try that and it worked. Remember, any
changes made to the volume through the CLI will reset that
configuration, causing it to stop working until you edit again.<br>
<br>
<blockquote cite="mid:51A6F4DB.5050704@gmail.com" type="cite">
<div class="moz-cite-prefix"> <br>
Iain<br>
<br>
On 05/30/13 07:06, Joe Julian wrote:<br>
</div>
<blockquote cite="mid:51A6EC58.3070701@julianfamily.org"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 05/29/2013 10:33 PM, Iain
Buchanan wrote:<br>
</div>
<blockquote
cite="mid:B4A78FEC-456B-46A8-B876-69F4727B1DB2@gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
Hi,
<div><br>
<div>I'm running GlusterFS 3.3.1-ubuntu1~precise9 and
I'm having some problems with the "rdma" and
"tcp,rdma" options I hope someone can help me with. </div>
<div><br>
</div>
<div>1. What does "tcp,rdma" actually do - does it let
you mix both types of client? (I did a few tests with
iozone and found it gave identical performance to the
"tcp".)</div>
<div><br>
</div>
<div>2. I can't get "rdma" to work, even in the simplest
case with a single node.</div>
<div><font face="Courier"><span class="Apple-tab-span"
style="white-space:pre"> </span>volume create
storage transport transport rdma
my_server:/data/area</font></div>
<div><font face="Courier"><span class="Apple-tab-span"
style="white-space:pre"> </span>volume start
storage</font></div>
<div><font face="Courier"><span class="Apple-tab-span"
style="white-space:pre"> </span>mount -t
glusterfs my_server:storage /mnt/storage</font></div>
<div><br>
</div>
<div>The last line hangs. Looking in /var/log/glusterfs
I can see the log for the volume:</div>
<div><br>
</div>
<div>
<div><font face="Courier">[2013-05-30 06:24:19.605315]
E [rdma.c:4604:tcp_connect_finish]
0-storage-client-0: <b>tcp connect to failed
(Connection refused)</b></font></div>
<div><font face="Courier">[2013-05-30 06:24:19.605713]
W [rdma.c:4187:gf_rdma_disconnect]
(-->/usr/sbin/glusterfs(main+0x34d)
[0x7f374d38a3ed]
(-->/usr/lib/libglusterfs.so.0(+0x3bd17)
[0x7f374cf23d17]
(-->/usr/lib/glusterfs/3.3.1/rpc-transport/rdma.so(+0x5231)
[0x7f3743398231]))) 0-storage-client-0: disconnect
called (peer:)</font></div>
<div><font face="Courier">[2013-05-30 06:24:19.605763]
W [rdma.c:4521:gf_rdma_handshake_pollerr]
(-->/usr/sbin/glusterfs(main+0x34d)
[0x7f374d38a3ed]
(-->/usr/lib/libglusterfs.so.0(+0x3bd17)
[0x7f374cf23d17]
(-->/usr/lib/glusterfs/3.3.1/rpc-transport/rdma.so(+0x5150)
[0x7f3743398150]))) 0-rpc-transport/rdma:
storage-client-0: peer () disconnected, cleaning
up</font></div>
</div>
<div><br>
</div>
<div>This block repeats every few seconds - the line
"tcp connect to failed" looks like it has lost the
server name somehow? </div>
<div><br>
</div>
<div>Iain</div>
</div>
<br>
</blockquote>
If you've installed from the yum repo (<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://goo.gl/s077x">http://goo.gl/s077x</a>) that
shouldn't be happening. kkeithley applied the patch. If not,
rdma's broken in 3.3.[01].
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=849122">https://bugzilla.redhat.com/show_bug.cgi?id=849122</a><br>
<br>
To mount via rdma when using tcp,rdma, mount -t glusterfs
server1:myvol.rdma /mnt/foo<br>
<br>
<br>
</blockquote>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>