<div dir="ltr">Hi Peter,<br>Glusterfs ib-verbs transport needs libibverbs to be installed. Is it installed on your system. Also, in case of ib-sdp are you able to ping the other node using the interface which has ib-sdp support?<br>
<br>regards,<br><div class="gmail_quote">On Fri, Jul 25, 2008 at 2:26 AM, Peter Pelekis <<a href="mailto:pete.pelekis@wsm.com">pete.pelekis@wsm.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I'm having some problems getting Glusterfs to run on my system. This is<br>
the first time using it so you I'm not to knowledgeable yet. The system<br>
I'm using has infiniband installed so I would like to use the IB for the<br>
transport for better performance. Heres what I have done so far. I<br>
first set up glusterfs to run over tcp on the Gig-E network and that<br>
worked fine so I decided to move to the IB network and thats when my<br>
problems started. Heres some information about the system.<br>
<br>
Glusterfs: glusterfs-1.3.9<br>
Fuse: fuse-2.7.3glfs10<br>
Cards: Qlogic QLE7240<br>
Drivers: InfiniPath-2.2<br>
OS: Centos5.2<br>
Kernel: 2.6.18-53<br>
4 Storage nodes<br>
46 Compute nodes<br>
<br>
When I first ran configure on glusterfs I got the following message<br>
after it ran configure.<br>
<br>
GlusterFS configure summary<br>
===========================<br>
Fuse client : yes<br>
Infiniband verbs : no<br>
epoll IO multiplex : yes<br>
<br>
If I look at lsmod I do have a ib_uverbs module installed and working<br>
but the configure script doesn't pick it up. I decided move on and use<br>
ib-sdp instead. I modified all my client and server volume files to<br>
reflect ib-sdp for the transport then I started the server on my 4<br>
storage nodes it started fine. I then went to one of my compute nodes<br>
to mount the volume.<br>
glusterfsd --transport=ib-sdp -s storage-ib1 /glusterfs<br>
<br>
It returned with the following error.<br>
glusterfs: could not open specfile<br>
<br>
Then I tried to specify the volume file to see if that would work.<br>
glusterfsd --spec-file=/usr/local/etc/glusterfs/glusterfs-client.vol<br>
/glusterfs<br>
<br>
It returned fine so I thought everything was working until I ran df I<br>
got the following error.<br>
df: `/glusterfs': Transport endpoint is not connected<br>
<br>
I had it log the error that its getting when I try and mount the filesystem.<br>
2008-07-24 17:28:01 E [ib-sdp-client.c:141:ib_sdp_connect] trans: error:<br>
not in progress - trace: Network is unreachable<br>
2008-07-24 17:28:01 W [client-protocol.c:332:client_protocol_xfer]<br>
trans: not connected at the moment to submit frame type(2) op(4)<br>
2008-07-24 17:28:01 E [client-protocol.c:4538:client_getspec_cbk] trans:<br>
no proper reply from server, returning ENOTCONN<br>
<br>
This where I'm now I looked around the devel list and didn't find<br>
anything on using the qlogic drivers just the ofed IB drivers. Heres<br>
what my configuration files. I would like to us ib-verbs if thats<br>
possible the performance should be much better then ib-sdp.<br>
<br>
volume posix-stripe<br>
type storage/posix<br>
option directory /export/glusterfs<br>
end-volume<br>
<br>
volume server<br>
type protocol/server<br>
option transport-type ib-sdp/server<br>
option auth.ip.posix-stripe.allow <a href="http://192.168.2." target="_blank">192.168.2.</a>*<br>
subvolumes posix-stripe<br>
end-volume<br>
<br>
volume client-stripe-1<br>
type protocol/client<br>
option transport-type ib-sdp/client<br>
option remote-host storage-ib1.cluster<br>
option remote-subvolume posix-stripe<br>
end-volume<br>
<br>
volume client-stripe-2<br>
type protocol/client<br>
option transport-type ib-sdp/client<br>
option remote-host storage-ib2.cluster<br>
option remote-subvolume posix-stripe<br>
end-volume<br>
<br>
volume client-stripe-3<br>
type protocol/client<br>
option transport-type ib-sdp/client<br>
option remote-host storage-ib3.cluster<br>
option remote-subvolume posix-stripe<br>
end-volume<br>
<br>
volume client-stripe-4<br>
type protocol/client<br>
option transport-type ib-sdp/client<br>
option remote-host storage-ib4.cluster<br>
option remote-subvolume posix-stripe<br>
end-volume<br>
<br>
volume stripe<br>
type cluster/stripe<br>
option block-size *:2MB # All files ending with .img are striped<br>
with 2MB stripe block size.<br>
subvolumes client-stripe-1 client-stripe-2 client-stripe-3<br>
client-stripe-4<br>
end-volume<br>
<br>
-Peter<br>
<br>
--<br>
Peter Pelekis 5444 Napa Street<br>
Western Scientific San Diego CA, 92110<br>
Senior Systems Engineer <a href="http://www.wsm.com" target="_blank">www.wsm.com</a><br>
<a href="mailto:pete.pelekis@wsm.com">pete.pelekis@wsm.com</a> Fax 619-220-6590<br>
Phone 619-220-6580 x212 Toll Free 800-443-6699<br>
GSA# GS-35F-5009H<br>
<br>
Visit us at:<br>
<br>
"I'm a great believer in luck, and I find the harder I work the more I have of it."<br>
-Thomas Jefferson<br>
<br>
"If everything seems under control, you're not going fast enough."<br>
-Mario Andretti<br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>A centipede was happy quite, until a toad in fun,<br>Said, "Prey, which leg comes after which?",<br>This raised his doubts to such a pitch,<br>
He fell flat into the ditch,<br>Not knowing how to run.<br>-Anonymous
</div>