<div>The geo-replication module forks (gsyncd) two process' one in the <b><i>Master node</i></b> (where geo-replication start was invoked) and another in <b><i>Slave node</i></b>.</div><div>The two gsyncd' communicate with each other through a pair of pipe either on the same machine or through an ssh-tunnel (depending on where the <b><i>Slave node</i></b> resides),</div>
<div><br></div><div>The above log basically means that the communication-channel between Master and Slave is failing.</div><div><br></div><div>Here are the few reasons for which it may occur:</div><div><br></div><div>1) The ssh tunnel b/w the <b><i>Master & Slave gsyncd</i></b> is broken:</div>
<div> </div><div> The pre-requisite for geo-rep b/w Master and Slave to work is to have a passwordless SSH setup b/w them either as mentioned in <span style="border-collapse:collapse;font-family:Tahoma;font-size:13px"> <a href="http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Setting_Up_the_Environment_for_Geo-replication" style="color:rgb(0, 0, 204)" target="_blank">http://www.gluster.com/community/documentation/index.php/Gluster_3.2:_Setting_Up_the_Environment_for_Geo-replication</a> or as it is done normally.</span></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"><br></span></font></div><div><font face="Tahoma"><span style="border-collapse:collapse">Verify if the password-less SSH is working fine.</span></font></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"><br></span></font></div><div><font face="Tahoma"><span style="border-collapse:collapse">2) The </span></font><b style="border-collapse:collapse;font-family:Tahoma;font-size:13px"><i>Master gsyncd</i></b><font face="Tahoma"><span style="border-collapse:collapse"> could </span></font><b style="border-collapse:collapse;font-family:Tahoma;font-size:13px"><i>not</i></b><font face="Tahoma"><span style="border-collapse:collapse"> <b><i>spawn</i></b> the </span></font><b style="border-collapse:collapse;font-family:Tahoma;font-size:13px"><i>Slave gsyncd</i></b><font face="Tahoma"><span style="border-collapse:collapse"> session successfully:</span></font></div>
<div><span style="border-collapse:collapse;font-family:Tahoma;font-size:13px"><br></span></div><div><span style="border-collapse:collapse;font-family:Tahoma;font-size:13px"> a) This could be due to the SSH-tunnel not been setup as desired, or</span></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"> b) The Master gsyncd spawns the Slave gsyncd process by locating the gsyncd executable in the slave in a predefined location, if the gsyncd executable is not found in the expected location the above scenario might occur. Execute the following in the Master node to see where Master gsyncd expects the gsyncd executable in the Slave Node.</span></font></div>
<div><span style="border-collapse:collapse"><font face="Tahoma"> </font><font face="'courier new', monospace"> #gluster volume geo-replication <Master-Volume> <Slave-URI> config remote-gsyncd</font></span></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"><br></span></font></div><div><font face="Tahoma"><span style="border-collapse:collapse">The outuput might would be a location similar to:</span></font></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"><br></span></font></div><div><span style="border-collapse:collapse"><font face="Tahoma"> </font><font face="'courier new', monospace"> /usr/local/libexec/glusterfs/gsyncd </font></span></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"><br></span></font></div><div><font face="Tahoma"><span style="border-collapse:collapse">Verify in the Slave node if the above output is valid executable. If not configure the remote-gsyncd to point to the appropriate location by executing the following command in the Master node.</span></font></div>
<div><font face="Tahoma"><span style="border-collapse:collapse"><br></span></font></div><div><span style="border-collapse:collapse"><font face="Tahoma"> </font><font face="'courier new', monospace">#gluster volume geo-replication <Master-Volume> <Slave-URI> config remote-gsyncd <new_location></font></span></div>
<div><span style="border-collapse:collapse;font-family:Tahoma"><br></span></div><div><span style="border-collapse:collapse;font-family:Tahoma"><br></span></div><div>
<span style="border-collapse:collapse;font-family:Tahoma"> c) The <b><i>Slave gsyncd</i></b> process <b><i>died</i></b> unexpectedly after being spawned:</span></div><div><span style="border-collapse:collapse;font-family:Tahoma"><br>
</span></div><div><span style="border-collapse:collapse;font-family:Tahoma"> - If the Slave is a plain directory the gsyncd expects the directory to be created already. Verify if the directory has already been created.</span></div>
<div><span style="border-collapse:collapse;font-family:Tahoma"><br></span></div><div><span style="border-collapse:collapse;font-family:Tahoma"> - If the Slave is a Gluster Volume then verify if <i><b>Slave volume is started</b>.</i></span></div>
<div><span style="border-collapse:collapse;font-family:Tahoma"><i><br></i></span></div><div><span style="border-collapse:collapse;font-family:Tahoma"><i> - </i> If the Slave is a Gluster Volume , gsyncd does a fuse mount on the Slave Volume, if the mount fails (maybe due to <b><i>fuse</i></b> module not running) the <b><i>Slave gsyncd</i></b> <b><i>dies</i></b> if it cannot mount the Slave Volume.</span></div>
<div><span style="border-collapse:collapse;font-family:Tahoma"><br></span></div><div><span style="border-collapse:collapse;font-family:Tahoma"> For all the above scenarios looking at the Slave gsynd as well as auxialarry gluster log might throw more light on the issue, to locate the respective logs refer to the <span style="font-family:verdana, sans-serif;line-height:16px;border-collapse:separate"><font size="1"><b><i>Locating Log Files</i></b></font></span> section in </span><a href="http://gluster.com/community/documentation/index.php/Gluster_3.2:_Troubleshooting_Geo-replication" target="_blank">http://gluster.com/community/documentation/index.php/Gluster_3.2:_Troubleshooting_Geo-replication</a></div>
<div><br></div><div> If the problem persists post all the above logs by running the geo-replication session in DEBUG log level by executing the following command:</div><div><br></div><div> <font class="Apple-style-span" face="'courier new', monospace">#gluster volume geo-replication <Master-Volume> <Slave-URI> config-log-level DEBUG</font></div>
<div><br></div><div> So that the problem could be root caused. A lot of log improvements is done in the devel branch, will have a more sanitized logs in the future releases.</div>
<div><br></div><div>Regards,</div><div>Kaushik</div>