<div dir="ltr"><div>Hi,</div><div><br></div><div>Thanks for the advice, I finally have time to go back to this issue now.</div><div><br></div><div>It doesn't seem to be sticking on any particular part of the file system as far as I can tell.</div>
<div><br></div><div>One thing I've noticed is I always get an error about missing 'option transport-type'</div><div><br></div><div><br></div><div>2014-03-13 09:57:00.902189] E [resource:194:logerr] Popen: ssh> [2014-03-13 09:56:50.093951] W [rpc-transport.c:174:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"<br>
</div><div><br></div><div>on the master I have the following in glusterd.vol</div><div><br></div><div><div>volume management</div><div> type mgmt/glusterd</div><div> option working-directory /var/lib/glusterd</div><div>
option transport-type socket,rdma</div><div> option transport.socket.keepalive-time 10</div><div> option transport.socket.keepalive-interval 2</div><div> option transport.socket.read-fail-log off</div><div>end-volume</div>
</div><div><br></div><div><br></div><div>on the slave I have </div><div><br></div><div><div>volume management</div><div> type mgmt/glusterd</div><div> option working-directory /var/lib/glusterd</div><div> option transport-type socket,rdma</div>
<div> option transport.socket.keepalive-time 10</div><div> option transport.socket.keepalive-interval 2</div><div> option transport.socket.read-fail-log off</div><div><br></div><div> option mountbroker-root /var/mountbroker-root</div>
<div> option mountbroker-geo-replication.gluster-async geo-ftb-vol,geo-bak-vol,geo-j1h-vol</div><div> option geo-replication-log-group gluster-async</div><div><br></div><div>end-volume</div><div><br></div></div><div>
<br></div><div>What should I change to fix this error?</div><div><br></div><div><br></div><div><br></div><div>master log</div><div><br></div><div><div>[2014-03-13 09:56:47.888899] I [monitor(monitor):80:monitor] Monitor: ------------------------------------------------------------</div>
<div>[2014-03-13 09:56:47.889317] I [monitor(monitor):81:monitor] Monitor: starting gsyncd worker</div><div>[2014-03-13 09:56:47.995637] I [gsyncd:354:main_i] <top>: syncing: gluster://localhost:volname -> ssh://gluster-async@xx.xx.xx.xx:gluster://localhost:geo-ftb-vol</div>
<div>[2014-03-13 09:56:48.22799] D [repce:175:push] RepceClient: call 14516:140653524453120:1394704608.02 __repce_version__() ...</div><div>[2014-03-13 09:57:00.898520] E [syncdutils:173:log_raise_exception] <top>: connection to peer is broken</div>
<div>[2014-03-13 09:57:00.901844] E [resource:191:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-_wRYS3/gsycnd-ssh-%r@%h:%p gluster-async@xx.xx.xx.xx /nonexistent/gsyncd --session-owner acfda6fc-d995-4bf0-b13e-da789afb28c7 -N --listen --timeout 120 gluster://localhost:geo-ftb-vol" returned with 1, saying:<br>
</div><div>[2014-03-13 09:57:00.902189] E [resource:194:logerr] Popen: ssh> [2014-03-13 09:56:50.093951] W [rpc-transport.c:174:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"</div>
<div>[2014-03-13 09:57:00.902648] E [resource:194:logerr] Popen: ssh> [2014-03-13 09:56:52.136564] I [cli-rpc-ops.c:4318:gf_cli3_1_getwd_cbk] 0-cli: Received resp to getwd</div><div>[2014-03-13 09:57:00.902940] E [resource:194:logerr] Popen: ssh> [2014-03-13 09:56:52.136782] I [input.c:46:cli_batch] 0-: Exiting with: 0</div>
<div>[2014-03-13 09:57:00.903209] E [resource:194:logerr] Popen: ssh> failed with error.</div><div>[2014-03-13 09:57:00.903844] I [syncdutils:142:finalize] <top>: exiting.<br></div><div>[2014-03-13 09:57:00.906152] D [monitor(monitor):96:monitor] Monitor: worker seems to be connected (?? racy check)</div>
<div>[2014-03-13 09:57:01.907625] D [monitor(monitor):100:monitor] Monitor: worker died in startup phase</div><div>[2014-03-13 09:57:11.918355] I [monitor(monitor):80:monitor] Monitor: ------------------------------------------------------------</div>
<div>[2014-03-13 09:57:11.918920] I [monitor(monitor):81:monitor] Monitor: starting gsyncd worker</div><div>[2014-03-13 09:57:12.29169] I [gsyncd:354:main_i] <top>: syncing: gluster://localhost:volname -> ssh://gluster-async@xx.xx.xx.xx:gluster://localhost:geo-ftb-vol</div>
</div><div><br></div><div><br></div><div>-- lots of entries about syncing files ---</div><div><br></div><div><div>[2014-03-13 10:10:20.670299] E [syncdutils:190:log_raise_exception] <top>: FAIL:</div><div>Traceback (most recent call last):</div>
<div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 216, in twrap</div><div> tf(*aa)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 123, in tailer</div>
<div> poe, _ ,_ = select([po.stderr for po in errstore], [], [], 1)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 276, in select</div><div> return eintr_wrap(oselect.select, oselect.error, *a)</div>
<div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 269, in eintr_wrap</div><div> return func(*a)</div><div>error: (9, 'Bad file descriptor')</div><div>[2014-03-13 10:10:20.671988] I [syncdutils:142:finalize] <top>: exiting.</div>
<div>[2014-03-13 10:10:21.624923] D [monitor(monitor):100:monitor] Monitor: worker died in startup phase</div></div><div><br></div><div><br></div><div>slave log</div><div><br></div><div><div>[2014-03-13 10:08:44.478434] I [gsyncd(slave):354:main_i] <top>: syncing: gluster://localhost:geo-ftb-vol</div>
<div>[2014-03-13 10:08:55.6546] I [resource(slave):453:service_loop] GLUSTER: slave listening</div><div>[2014-03-13 10:09:31.698591] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</div><div>[2014-03-13 10:09:31.699101] I [syncdutils(slave):142:finalize] <top>: exiting.</div>
<div>[2014-03-13 10:09:49.26217] I [gsyncd(slave):354:main_i] <top>: syncing: gluster://localhost:geo-ftb-vol</div><div>[2014-03-13 10:10:00.252576] I [resource(slave):453:service_loop] GLUSTER: slave listening</div>
<div>[2014-03-13 10:10:20.783905] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</div><div>[2014-03-13 10:10:20.784468] I [syncdutils(slave):142:finalize] <top>: exiting.</div><div>[2014-03-13 10:10:37.405524] I [gsyncd(slave):354:main_i] <top>: syncing: gluster://localhost:geo-ftb-vol</div>
<div>[2014-03-13 10:10:46.988630] I [resource(slave):453:service_loop] GLUSTER: slave listening</div></div><div><br></div><div>Thanks</div><div><br></div><div>J.</div><div>
</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Feb 14, 2014 at 1:51 PM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Could you try again after changing the log-level to DEBUG using:<br>
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
# gluster volume geo-replication <master> <slave> config log-level DEBUG<br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Also, logs from both master and slave would help.<br>
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thanks,<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">-venky<br></div></div><div class="HOEnZb"><div class="h5">
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Feb 12, 2014 at 4:44 PM, John Ewing <span dir="ltr"><<a href="mailto:johnewing1@gmail.com" target="_blank">johnewing1@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><span style="font-family:arial,sans-serif;font-size:13px">No, its the latest 3.3 series release.</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
3.3.2 on both master and slave.</div><div style="font-family:arial,sans-serif;font-size:13px">Centos 6 on master , Amazon linux on slave.</div><div style="font-family:arial,sans-serif;font-size:13px">rsync 3.0.6 on both</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Using unprivileged ssh user setup with mountbroker.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">One thing I noticed was that the 3.3 manual says the base requirement is for rsync 3.0.0 and higher and the webpage now</div><div style="font-family:arial,sans-serif;font-size:13px">
says 3.0.7. Is this relevant ?</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div></div><div style="font-family:arial,sans-serif;font-size:13px"><div><img src="https://mail.google.com/mail/u/0/images/cleardot.gif"></div>
</div><div class="gmail_extra"><br><div class="gmail_quote"><div>On Wed, Feb 12, 2014 at 2:12 AM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br>
</div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-family:arial,helvetica,sans-serif">Is this from the latest master branch?</div>
</div><div class="gmail_extra">
<br><br><div class="gmail_quote"><div><div>On Tue, Feb 11, 2014 at 4:35 PM, John Ewing <span dir="ltr"><<a href="mailto:johnewing1@gmail.com" target="_blank">johnewing1@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">I am trying to use geo-replication but it is running slowly and I keep getting the<div>
following logged in the geo-replication log.</div>
<div><br></div><div><div>[2014-02-11 10:56:42.831517] I [monitor(monitor):80:monitor] Monitor: ------------------------------------------------------------</div>
<div>[2014-02-11 10:56:42.832226] I [monitor(monitor):81:monitor] Monitor: starting gsyncd worker</div><div>[2014-02-11 10:56:42.951199] I [gsyncd:354:main_i] <top>: syncing: gluster://localhost:xxxxxxx -> ssh://gluster-async@xx.xx.xx.xx:gluster://localhost:xxxxx</div>
<div>[2014-02-11 10:56:53.79632] I [master:284:crawl] GMaster: new master is acfda6fc-d995-4bf0-b13e-da789afb28c7</div><div>[2014-02-11 10:56:53.80282] I [master:288:crawl] GMaster: primary master with volume id acfda6fc-d995-4bf0-b13e-da789afb28c7 ...</div>
<div>[2014-02-11 10:56:57.453376] E [syncdutils:190:log_raise_exception] <top>: FAIL:</div><div>Traceback (most recent call last):</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 216, in twrap</div>
<div> tf(*aa)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 123, in tailer</div><div> poe, _ ,_ = select([po.stderr for po in errstore], [], [], 1)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 276, in select</div>
<div> return eintr_wrap(oselect.select, oselect.error, *a)</div><div> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 269, in eintr_wrap</div><div> return func(*a)</div><div>error: (9, 'Bad file descriptor')</div>
<div>[2014-02-11 10:56:57.462110] I [syncdutils:142:finalize] <top>: exiting.</div></div><div><br></div><div>I'm unsure what to do to debug and fix this. </div><div><br></div><div>Thanks</div><span><font color="#888888"><div>
<br></div><div>
John.</div></font></span></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>