Mark, <br><br>Finding the best practices isn't easy, sometimes. It would be great to discuss this with Harry if he's doing the same, or even something close. What I'm doing must be commonplace in the nextgen sequencing world.<br>
<br>Environment (for now): 4 bricks, replicated and distributed, gf1-1, gf1-2, gf1-1r, gf1-2r on the public side, gf1-ib-1, gf1-ib-2.... etc, on the private side, offering volume Gluster volume gf1.<br><br>We probably also want to use CTDB for this, since we want failover. <br>
<br>Question:<br><br>Do I have each volume brick act as a native Gluster client, use the IB side to mount the Gluster volume at, say, /mnt/gf1, then have the public side, using the public CTDB addresses, use NFS or CIFS to mount the Gluster-mounted directory? (Such as an NFS export or CIFS share might be defined to use /mnt/gf1/sequencer_files)?<br>
<br>Can a public network client on the public side even use NFS to mount Gluster volume gf1-1:/gf1 via Gluster?<br>(On the private side, HPC cluster nodes are in the private network and will mount volumes as FUSE clients.)<br>
<br>As you can see, I'm having some trouble with who sees what, what does CTDB do, etc?<br>========================================<br><br>Here's another question. Suppose there is a CTDB connection to one of the bricks offering NFS connections. If that brick fails, the connection should failover to one of the other CTDB nodes. And since the failed brick is in a replicated environment, the NFS connection will be to a still fully-functioning Gluster replicated/distributed storage cluster. <br>
<br>==============================================<br>I also have questions about RDMA. I understand that RDMA should be faster than TCP, yet there are conflicting reports about this in real-world terms. Also, for fast performance, do people use RDMA or SDP? <br>
I can setup IPoIB without any trouble in connected mode. But at the system side how do I configure RDMA connections instead? How do I then define the net address for RDMA? Further, if I'm using RDMA for the bricks to talk to each other, will the bricks still be available by TCP, say for ssh purposes, as well? Or do I have to create a different child IB net address that uses TCP? I guess I'm asking this: For any specific configurable IB device, would using it for RDMA and TCP be mutually exclusive?<br>
<br>You can see that at this low level, my network knowledge isn't deep.<br>A real example of both RDMA configuration and Gluster-using-rdma configuration would be a very good thing to see? <br><br>Any help greatly appreciated. I should go through the list to see if this is solved, but I've looked all over gmane, and the answer still isn't clear, at least to me.<br>
<br>Matt Temple<br><div class="gmail_extra"><br clear="all">------<br>Matt Temple<br>Director, Research Computing<br>Dana-Farber Cancer Institute.<br><br>
<br><br><div class="gmail_quote">On Wed, Oct 31, 2012 at 4:57 PM, John Mark Walker <span dir="ltr"><<a href="mailto:johnmark@redhat.com" target="_blank">johnmark@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div style="font-size:12pt;font-family:times new roman,new york,times,serif">Hi Matthew - did you get a response to this? Harry Mangalam from UC Irvine has been doing what sounds like similar things. I would be happy to put you guys in touch.<br>
<br>Also, I guess my main question is that I don't understand why or how getting "data from sequencers and other sources living on the public network /into/ the Gluster volume" is a problem. Are you saying that, as structured now, you can't get the data into GlusterFS? Or that you can but it's not performing well? <br>
<br>If your GlusterFS servers have gigE NICs on the public network, couldn't you just use the NFS server in GlusterFS? Wouldn't that also be available over the public network?<br><br>-JM<br><br><br><hr><blockquote style="padding-left:5px;font-size:12pt;font-style:normal;margin-left:5px;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal;border-left:2px solid rgb(16,16,255)">
We have a distributed/replicated gluster volume running on IPoIB. We don't yet know much about its performance in practice. The volume will be mounted natively to our HPC compute cluster and used for nextgen sequence analysis. The HPC compute cluster and the Gluster volume are in the same private IB network.<br>
<br>The problem is, we need a way to get data from sequencers and other sources living on the public network /into/ the Gluster volume. The Gluster bricks have gigE NICs on the public side in addition to the Infiniband connections. My first thought is to have each Gluster brick also act as a Gluster client, mount its own volume, then re-export the mount point by NFS or CIFS to the public network. <br>
Alternatively, I could set up some number of servers that are /not/ Gluster bricks, but are Gluster clients, and those servers would have IB and GigE -- then have those servers re-export the mounted Gluster volumes by NFS or CIFS. Neither of these models seems terribly efficient, but getting data into the volume won't be as intense and running analysis software against the volume.<br>
<br>1. Has anyone done this (or something similar)?<br>2. Did it work acceptably?<br>3. Does anyone have a better solution?<br><br>(There is an article in which there is a suggestion that Gluster volumes be accessible by multiple addresses natively, but it's not implemented anywhere as far as I know.)<br>
<br>Matt<br><br clear="all">------<br>Matt Temple<br>Director, Research Computing<br>Dana-Farber Cancer Institute.<br><br>
<br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
<br></div></div></blockquote></div><br></div>