<div dir="ltr"><div>We have two networks and I have 4 servers I'm trying to turn into a distributed/replicated Gluster storage cluster.<br><br></div><div>Each of the servers has two addresses, one ethernet, one IB.<br>
</div><div>The externet addresses are 155.42.48.1-4 and <br></div><div>the Infiniband addresses are 172.24.215.1-4<br></div><div><br>I login to the first server ("1") which is 155.52.48.1.<br><br></div><div>Let's say the servers are named 1,2,1R and 2R (on the ethernet side), but <br>
1-IB, 2-IB, 1-IB-R and 2-IB-R on he Infiniband side.<br></div><div>The Volume is to be mounted by other servers in the IB network by native Gluster clients.<br><br></div><div>so, I login to "1" over the ethernet network and do a peer probe.<br>
</div><div>All looks good from "1" with peer probe and peer status.<br><br></div><div>I probe the peers like this, successfully<br><br></div><div><font face="courier new,monospace">gluster peer probe 2-ib<br></font></div>
<div><font face="courier new,monospace">gluster peer probe 1-ib-r<br></font></div><div><font face="courier new,monospace">gluster peer prove 2-ib-r<br><br></font></div><div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif">Peer status looks right.<br>
</font></font>I issue the following command which fails, of course:<br><br><span style="font-size:15px;font-family:'Courier New';color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">gluster volume create gf2 replica 2 transport tcp,rdma \</span><br>
<span style="font-size:15px;font-family:'Courier New';color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">1-ib:/mnt/d0-0 1-ib-r:/mnt/d0-0 </span><span style="font-size:15px;font-family:'Courier New';color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">2-ib:/mnt/d0-0 2-ib-r:/mnt/d0-0</span></div>
<div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif"><br>From any of those nodes, peer status is wrong:<br>For instance, if I ssh over to 2-ib, <br>and ask for a peer status, it shows the peers to to be <br>
<br></font></font></div><div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif"><font face="courier new,monospace">1-ib-r<br>2-ib-r <br>and <b>155.52.48.1</b></font> (the last is the ethernet side of, not the IB side)<br>
<br></font></font></div><div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif">I guess my question is this: How do I get the first node to use the ib side when<br></font></font></div><div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif">connecting to the other nodes. (both of these networks are public.) Do I have to<br>
</font></font></div><div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif">do something interesting with routes?<br></font></font></div><div><font face="courier new,monospace"><font face="arial,helvetica,sans-serif"><br>
</font></font></div><div>I need to fix this. Don't most people with IB also have an internet side to things?<br></div><div>How do their networks and any local routing info get set up?<br></div><div>Am I ,missing something really obvious?<br>
<br></div><div>Matt<br></div><div><br></div><br><div><div>------<br>Matt Temple<br>Director, Research Computing<br>Dana-Farber Cancer Institute.<br></div><br></div>We have two
</div>