<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi all,<br>
<br>
thanks for your suggestions,<br>
<br>
i think I have "solved" the performance issue now. I had a few too
many kernel patches included. I am back to the stock RHEL 5.8 kernel
with stock QLogic OFED and everything works ok (IPoIB). My original
intent was to explore cachefs on RHEL5 by building a 2.6.32 kernel
but while cachefs worked like a treat performance for gluster was as
bad as reported previously - so will go without cachefs for now and
reintroduce cachefs in an OS upgrade later on.<br>
<br>
I even have a nicely working rdma setup now and - using that -
performance is 900 MB/s + and that consistently so. <br>
<br>
Since I have two volumes exported by the same bricks it seems I only
can get one of them to use RDMA, the other will then refuse to mount
and only mount if not using rdma on that one - but that is not a
real problem for now as the second one is only used for backup
purposes.<br>
<br>
Michael, <br>
<br>
On 04/12/2012 01:13 AM, Fabricio Cannini wrote:
<blockquote
cite="mid:CAOp_GpoJe4P2-qxdjCqrb-ua4wqUofC4PT3NyzAt7UuL-5TwrA@mail.gmail.com"
type="cite">Hi there<br>
<br>
The only time i setup a gluster "distributed scratch" like Michael
is doing, <br>
( 3.0.5 Debian packages ) i too choose IPoIB simply because i
could not get rdma working at all. <br>
Time was short and IPoIB "Just worked" well enough for our demand
at the time, so i didn't looked into this issue. Plus, pinging and
ssh'ing into a node through the IB interface comes handy when
diagnosing and fixing networking issues.<br>
<br>
Em quarta-feira, 11 de abril de 2012, Sabuj Pattanayek<<a
moz-do-not-send="true" href="mailto:sabujp@gmail.com">sabujp@gmail.com</a>>
escreveu:<br>
> I wonder if it's possible to have both rdma and ipoib served
by a<br>
> single glusterfsd so I can test this? I guess so, since it's
just a<br>
> tcp mount?<br>
><br>
> On Wed, Apr 11, 2012 at 1:43 PM, Harry Mangalam <<a
moz-do-not-send="true" href="mailto:harry.mangalam@uci.edu">harry.mangalam@uci.edu</a>>
wrote:<br>
>> On Tuesday 10 April 2012 15:47:08 Bryan Whitehead wrote:<br>
>><br>
>>> with my infiniband setup I found my performance was
much better by<br>
>><br>
>>> setting up a TCP network over infiniband and then
using pure tcp as<br>
>><br>
>>> the transport with my gluster volume. For the life of
me I couldn't<br>
>><br>
>>> get rdma to beat tcp.<br>
>><br>
>> Thanks for that data point, Brian.<br>
>><br>
>> Very interesting. Is this a common experience? The RDMA
experience has not<br>
>> been a very smooth one for me and doing everything with
IPoIB would save a<br>
>> lot of headaches, especially if it's also higher
performance.<br>
>><br>
>> hjm<br>
>><br>
>> --<br>
>><br>
>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC
Irvine<br>
>><br>
>> [ZOT 2225] / 92697 Google Voice Multiplexer: (949)
478-4487<br>
>><br>
>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]<br>
>><br>
>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google
Maps)<br>
>><br>
>> --<br>
>><br>
>><br>
>> _______________________________________________<br>
>> Gluster-users mailing list<br>
>> <a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> <a moz-do-not-send="true"
href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
>><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a moz-do-not-send="true"
href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
<br>
</body>
</html>