<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";
color:black;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Yes, the authorized_keys on the slave nodes have the master nodes keys prepended with "command=..."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Geo-replication is now functioning in my environment as described in the docs. The /var/lib/glusterd/geo-replication/…/gsyncd.conf file did not need any modifications
and I can confirm that the “remote_gsyncd = /nonexistent/gsyncd” entry in the gsyncd.conf file does work.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I ended up redoing the geo-replication from scratch; deleting the replication session, recreating common ssh keys via gsec_create, and banging my head into
the wall a few times. I’m not sure where the mis-configuration was, but I would consider this resolved for me.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks again for all your help Venky<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org]
<b>On Behalf Of </b>Venky Shankar<br>
<b>Sent:</b> Tuesday, July 01, 2014 2:15 AM<br>
<b>To:</b> gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] geo-replication status faulty<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 06/28/2014 12:33 AM, Chris Ferraro wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Thanks for the response Venky,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Here’s a rundown on my environment and what I did to get geo-replication set up. Let me know if you’d like any additional information.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>master volume - gluster_vol0 (node001 and node002)<o:p></o:p></pre>
<pre>slave volume - gluster_vol1 (node003 and node004)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Password-less SSH access is set up between the master volume node001 and slave node003<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>The root user uses /root/.ssh/id_rsa to auth from the master volume node001 to the slave node003<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>tested and working as expected<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Ran `gluster system:: execute gsec_create` from master volume node001 to create common pem file<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Created the geo-replication session with<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>`gluster volume geo-replication gluster_vol0 node003::gluster_vol1 create push-pem`<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>results of command were successful<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Checked status<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>`gluster volume geo-replication gluster_vol0 node003::gluster_vol1 status`<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>results showed node001 and node002 bricks from master and node003 as slave. status stopped.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Started geo-rep session<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>`gluster volume geo-replication gluster_vol0 node003::gluster_vol1 start`<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Starting geo-replication session between gluster_vol0 & node003::gluster_vol1 has been successful<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Check status<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>`gluster volume geo-replication gluster_vol0 node003::gluster_vol1 status`<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>status of all entries is faulty<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>## Other<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>· common_secre.pem.pub file exists on both slave nodes.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>· /usr/libexec/glusterfs/gsyncd exists on all master and slave nodes<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>· gsyncd.py process running on both master volume nodes<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>· /var/lib/glusterd/geo-replication/gsyncd.conf does not exist. Only gsyncd_template.conf<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>· Master geo-replication log shows following:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[2014-06-26 17:09:10.258044] E [syncdutils(/data/glusterfs/vol0/brick0/brick):223:log_raise_exception] <top>: connection to peer is broken<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.259278] W [syncdutils(/data/glusterfs/vol0/brick0/brick):227:log_raise_exception] <top>: !!!!!!!!!!!!!<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.260755] W [syncdutils(/data/glusterfs/vol0/brick0/brick):228:log_raise_exception] <top>: !!! getting "No such file or directory" errors is most likely due to MISCONFIGURATION, please consult <a href="https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/chap-User_Guide-Geo_Rep-Preparation-Settingup_Environment.html">https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/chap-User_Guide-Geo_Rep-Preparation-Settingup_Environment.html</a><o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.261250] W [syncdutils(/data/glusterfs/vol0/brick0/brick):231:log_raise_exception] <top>: !!!!!!!!!!!!!<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.263020] E [resource(/data/glusterfs/vol0/brick0/brick):204:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-DIu2bR/139ffeb96cd3f82a30d4e4ff1ff33f0b.sock root@node003 /nonexistent/gsyncd --session-owner 46d54a00-06a5-4e92-8ea4-eab0aa454c22 -N --listen --timeout 120 gluster://localhost:gluster_vol1" returned with 127, saying:<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.264806] E [resource(/data/glusterfs/vol0/brick0/brick):207:logerr] Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.266753] I [syncdutils(/data/glusterfs/vol0/brick0/brick):192:finalize] <top>: exiting.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks again for any help<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Chris<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>From: Venky Shankar [<a href="mailto:yknev.shankar@gmail.com">mailto:yknev.shankar@gmail.com</a>]<o:p></o:p></pre>
<pre>Sent: Thursday, June 26, 2014 11:13 PM<o:p></o:p></pre>
<pre>To: Chris Ferraro<o:p></o:p></pre>
<pre>Cc: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><o:p></o:p></pre>
<pre>Subject: Re: [Gluster-users] geo-replication status faulty<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Hey Chris,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>‘/nonexistent/gsyncd’ is purposely used in the ssh connection so as to avoid insecure access via ssh. Fiddling with remote_gsyncd should be avoided (it's a reserved option anyway).<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>As the log messages say, there seems to be a misconfiguration in the setup. Could you please list down the steps you're using for setting up a geo-rep session?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>On Fri, Jun 27, 2014 at 5:59 AM, Chris Ferraro <<a href="mailto:ChrisFerraro@fico.com">ChrisFerraro@fico.com</a><a href="mailto:ChrisFerraro@fico.com"><mailto:ChrisFerraro@fico.com></a>> wrote:<o:p></o:p></pre>
<pre>Venky Shankar, can you follow up on these questions? I too have this issue and cannot resolve the reference to ‘/nonexistent/gsyncd’.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>As Steve mentions, the nonexistent reference in the logs looks like the culprit especially seeing that the ssh command trying to be run is printed on an earlier line with the incorrect remote path.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I have followed the configuration steps as documented in the guide, but still hit this issue.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks for any help<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Chris<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre># Master geo replication log grab #<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[2014-06-26 17:09:08.794359] I [monitor(monitor):129:monitor] Monitor: ------------------------------------------------------------<o:p></o:p></pre>
<pre>[2014-06-26 17:09:08.795387] I [monitor(monitor):130:monitor] Monitor: starting gsyncd worker<o:p></o:p></pre>
<pre>[2014-06-26 17:09:09.358588] I [gsyncd(/data/glusterfs/vol0/brick0/brick):532:main_i] <top>: syncing: gluster://localhost:gluster_vol0 -> <a href="ssh://root@node003:gluster:/localhost:gluster_vol1">ssh://root@node003:gluster://localhost:gluster_vol1</a><o:p></o:p></pre>
<pre>[2014-06-26 17:09:09.537219] I [monitor(monitor):129:monitor] Monitor: ------------------------------------------------------------<o:p></o:p></pre>
<pre>[2014-06-26 17:09:09.540030] I [monitor(monitor):130:monitor] Monitor: starting gsyncd worker<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.137434] I [gsyncd(/data/glusterfs/vol0/brick1/brick):532:main_i] <top>: syncing: gluster://localhost:gluster_vol0 -> <a href="ssh://root@node003:gluster:/localhost:gluster_vol1">ssh://root@node003:gluster://localhost:gluster_vol1</a><o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.258044] E [syncdutils(/data/glusterfs/vol0/brick0/brick):223:log_raise_exception] <top>: connection to peer is broken<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.259278] W [syncdutils(/data/glusterfs/vol0/brick0/brick):227:log_raise_exception] <top>: !!!!!!!!!!!!!<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.260755] W [syncdutils(/data/glusterfs/vol0/brick0/brick):228:log_raise_exception] <top>: !!! getting "No such file or directory" errors is most likely due to MISCONFIGURATION, please consult <a href="https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/chap-User_Guide-Geo_Rep-Preparation-Settingup_Environment.html">https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/chap-User_Guide-Geo_Rep-Preparation-Settingup_Environment.html</a><o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.261250] W [syncdutils(/data/glusterfs/vol0/brick0/brick):231:log_raise_exception] <top>: !!!!!!!!!!!!!<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.263020] E [resource(/data/glusterfs/vol0/brick0/brick):204:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-DIu2bR/139ffeb96cd3f82a30d4e4ff1ff33f0b.sock root@node003 /nonexistent/gsyncd --session-owner 46d54a00-06a5-4e92-8ea4-eab0aa454c22 -N --listen --timeout 120 gluster://localhost:gluster_vol1" returned with 127, saying:<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.264806] E [resource(/data/glusterfs/vol0/brick0/brick):207:logerr] Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory<o:p></o:p></pre>
<pre>[2014-06-26 17:09:10.266753] I [syncdutils(/data/glusterfs/vol0/brick0/brick):192:finalize] <top>: exiting.<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.4817] E [syncdutils(/data/glusterfs/vol0/brick1/brick):223:log_raise_exception] <top>: connection to peer is broken<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.5966] W [syncdutils(/data/glusterfs/vol0/brick1/brick):227:log_raise_exception] <top>: !!!!!!!!!!!!!<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.6467] W [syncdutils(/data/glusterfs/vol0/brick1/brick):228:log_raise_exception] <top>: !!! getting "No such file or directory" errors is most likely due to MISCONFIGURATION, please consult <a href="https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/chap-User_Guide-Geo_Rep-Preparation-Settingup_Environment.html">https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/html/Administration_Guide/chap-User_Guide-Geo_Rep-Preparation-Settingup_Environment.html</a><o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.7822] W [syncdutils(/data/glusterfs/vol0/brick1/brick):231:log_raise_exception] <top>: !!!!!!!!!!!!!<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.8938] E [resource(/data/glusterfs/vol0/brick1/brick):204:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-N0sS2H/139ffeb96cd3f82a30d4e4ff1ff33f0b.sock root@node003 /nonexistent/gsyncd --session-owner 46d54a00-06a5-4e92-8ea4-eab0aa454c22 -N --listen --timeout 120 gluster://localhost:gluster_vol1" returned with 127, saying:<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.10252] E [resource(/data/glusterfs/vol0/brick1/brick):207:logerr] Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.12820] I [syncdutils(/data/glusterfs/vol0/brick1/brick):192:finalize] <top>: exiting.<o:p></o:p></pre>
<pre>[2014-06-26 17:09:11.274421] I [monitor(monitor):157:monitor] Monitor: worker(/data/glusterfs/vol0/brick0/brick) died in startup phase<o:p></o:p></pre>
<pre>[2014-06-26 17:09:12.18722] I [monitor(monitor):157:monitor] Monitor: worker(/data/glusterfs/vol0/brick1/brick) died in startup phase<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><a href="mailto:Gluster-users@gluster.org"><mailto:Gluster-users@gluster.org></a><o:p></o:p></pre>
<pre><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.<o:p></o:p></pre>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><tt><span style="font-size:10.0pt">Does authorized_keys on the slave nodes have the master nodes keys prepended with "command=..." ?</span></tt><span style="font-size:10.0pt;font-family:"Courier New""><br>
<br>
<tt>-venky</tt></span><o:p></o:p></p>
</div>
<br>This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.</body>
</html>