<div dir="ltr">Correction: Different between the two local bricks is more likely a result of the ZFS snapshot process I've run on the larger of them.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 6, 2013 at 11:39 AM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">More stats in case you need them:<div><br></div><div>Grand total of the /data/docstore1 tree that's being sync'd is about 8.5 million files.</div>
<div>About 1 KB of new data has appeared on the destination server since we started this discussion yesterday...</div>
<div><br></div><div><br></div><div><div><font face="courier new, monospace">tony@backup-ds2:~$ df</font></div><div><font face="courier new, monospace">Filesystem 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/sda1 768989240 1841364 728085452 1% /</font></div><div><font face="courier new, monospace">udev 16450080 4 16450076 1% /dev</font></div><div>
<font face="courier new, monospace">tmpfs 6583556 348 6583208 1% /run</font></div><div><font face="courier new, monospace">none 5120 0 5120 0% /run/lock</font></div>
<div><font face="courier new, monospace">none 16458888 0 16458888 0% /run/shm</font></div><div><font face="courier new, monospace">data 6601579264 0 6601579264 0% /data</font></div>
<div><font face="courier new, monospace">data/docstore1 7651735936 1050156672 6601579264 14% /data/docstore1</font></div><div><font face="courier new, monospace">tony@backup-ds2:~$ date</font></div><div><font face="courier new, monospace">Thu Sep 5 09:36:24 EDT 2013</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">tony@backup-ds2:~$ df</font></div><div><font face="courier new, monospace">Filesystem 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/sda1 768989240 1842524 728084292 1% /</font></div><div><font face="courier new, monospace">udev 16450080 4 16450076 1% /dev</font></div><div>
<font face="courier new, monospace">tmpfs 6583556 348 6583208 1% /run</font></div><div><font face="courier new, monospace">none 5120 0 5120 0% /run/lock</font></div>
<div><font face="courier new, monospace">none 16458888 0 16458888 0% /run/shm</font></div><div><font face="courier new, monospace">data 6601577984 0 6601577984 0% /data</font></div>
<div><font face="courier new, monospace">data/docstore1 7651735808 1050157824 6601577984 14% /data/docstore1</font></div><div><font face="courier new, monospace">tony@backup-ds2:~$ date</font></div><div><font face="courier new, monospace">Fri Sep 6 11:34:07 EDT 2013</font></div>
<div><font face="courier new, monospace">tony@backup-ds2:~$ </font></div></div><div><br></div><div>The source data I'm attempting to geo-replicate is 3.3 TB. Mirroring between the local bricks seems to be working fine. They are within 20KB of drive usage between each other for the /data/docstore1 partition. I attribute that to the new geo-rep info on gfs6 making it larger by a bit? Also the data is constantly changing, so maybe it's a bit of replication lag.</div>
<div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 6, 2013 at 11:29 AM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">./evds3 contains 1,262,970 files. No more than 255 files or folders per subdirectory.<div><br></div><div>
Network latency is almost nonexistent - the servers are currently jacked into the same switch on a separate network segment for the initial sync:</div>
<div><br></div><div><div><font face="courier new, monospace">tony@gfs6:~$ ping backup-ds2.gluster</font></div><div><font face="courier new, monospace">PING backup-ds2.gluster (10.200.1.12) 56(84) bytes of data.</font></div>
<div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=1 ttl=64 time=0.245 ms</font></div><div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=2 ttl=64 time=0.197 ms</font></div>
<div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=3 ttl=64 time=0.174 ms</font></div><div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=4 ttl=64 time=0.221 ms</font></div>
<div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=5 ttl=64 time=0.163 ms</font></div></div><div><br></div><div><br></div><div><br></div></div><div><div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Fri, Sep 6, 2013 at 5:13 AM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">From the trace logs it looks like its still crawling (the lgetxattr() on the master and lgetxattr() on the slave). How many files are under '<span style="color:rgb(51,51,51);font-family:monospace,'Lucida Grande',Verdana,Arial,Helvetica,sans-serif;font-size:13px;white-space:pre-wrap">./evds3' ?</span></div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="color:rgb(51,51,51);font-family:monospace,'Lucida Grande',Verdana,Arial,Helvetica,sans-serif;font-size:13px;white-space:pre-wrap"><br>
</span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="color:rgb(51,51,51);font-family:monospace,'Lucida Grande',Verdana,Arial,Helvetica,sans-serif;font-size:13px;white-space:pre-wrap">Further more, what's the latency b/w the two sites?</span></div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 11:31 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">No rsync process is running. I tested for it several times over 10 seconds.<div><br></div><div>Here's the trace on feedback. I put it on TinyPaste instead of in everyone's inbox out of courtesy: <a href="http://tny.cz/505b09c8" target="_blank">http://tny.cz/505b09c8</a></div>
<div><br></div><div>Thanks,</div><div>Tony</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 12:24 PM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">So, nothing fishy in slave logs too.</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
Debugging this would need more information. Lets start with:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
1. Is there any rsync process that is running? (ps auxww | grep rsync)</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
2. I would need strace logs for the worker process (the process which scans the filesystem and invokes rsync to transfer data). Something like this would do:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> - get the pid of the worker process</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> # ps auxww | grep feedback (a gsyncd.py python process)</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> - trace the pid above</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
# strace -s 500 -o /tmp/worker.log -f -p <pid></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Let the trace run for about 20 seconds. Lets see what's in /tmp/worker.log<br>
</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
-venky</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 7:56 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Looks like the slave stopped logging things just before the sending brick did:<div><br></div><div><div>
<font face="courier new, monospace">[2013-09-01 14:56:45.13972] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 14:56:45.15433] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 15:57:34.592938] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 15:57:34.593383] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 15:57:45.374301] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 15:57:45.375871] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 16:02:09.115976] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:02:09.116446] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 16:02:19.869340] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:02:19.870598] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 16:32:58.701617] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:32:58.702126] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 16:33:09.456556] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:33:09.458009] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 21:47:13.442514] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 21:47:13.442899] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 21:47:24.240978] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 21:47:24.242424] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-02 02:12:14.26339] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:12:14.26809] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-02 02:12:24.818355] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:12:24.820008] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-02 02:16:14.525187] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:16:14.525675] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-02 02:16:25.263712] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:16:25.265168] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-02 02:37:39.315608] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:37:39.316071] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-02 02:37:50.78136] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:37:50.79577] I [resource(slave):453:service_loop] FILE: slave listening</font></div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra">
<div><div>
<br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 10:09 AM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Could you also provide the slave logs? (log location on the slave: /var/log/glusterfs/geo-replication-slaves)</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thanks,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">-venky</div></div><div class="gmail_extra"><br><br>
<div class="gmail_quote"><div><div>On Thu, Sep 5, 2013 at 7:29 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>
<div dir="ltr">I'm trying to create a new Geo-Rep of about 3 TB of data currently stored in a 2 brick mirror config. Obviously the geo-rep destination is a third server.<div><br></div><div>This is my 150th attempt. Okay, maybe not that far, but it's pretty darn bad.</div>
<div><br></div><div><div>Replication works fine until I hit around 1TB of data sync'd, then it just stalls. For the past two days it hasn't gone past 1050156672 bytes sync'd to the destination server.</div><div>
<br></div><div>I did some digging in the logs and it looks like the brick that's running the geo-rep process thinks it's syncing:</div><div><br></div><div><div><font face="courier new, monospace">[2013-09-05 09:45:37.354831] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000863.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.358669] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/0000083b.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.362251] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/0000087b.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.366027] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000834.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.369752] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000845.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.373528] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000864.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.377037] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/0000087f.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.391432] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000897.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.395059] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000829.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.398725] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000816.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.402559] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008cc.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.406450] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008d2.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.410310] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008df.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.414344] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008bd.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.438173] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/<a href="http://volume.info" target="_blank">volume.info</a> ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.441675] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/volume.enc ...</font></div></div><div><br></div><div>But, <b>those files never appear on the destination server,</b> however the containing folders are there, just empty.</div>
<div><br></div><div>Also, the other log file (...gluster.log) in the geo-replication log folder that matches the destination stopped updating when the syncing stopped apparently. It's last timestamp is from the 2nd, which is the last time data transferred.</div>
<div><br></div><div>The last bit from that log file is as such:</div><div><br></div><div><div><font face="courier new, monospace">+------------------------------------------------------------------------------+</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:50.109730] I [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-docstore1-client-1: changing port to 24009 (from 0)</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:50.109857] I [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-docstore1-client-0: changing port to 24009 (from 0)</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.097468] I [client-handshake.c:1614:select_server_supported_programs] 0-docstore1-client-1: Using Program GlusterFS 3.3.2, Num (1298437), Version (330)</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.097973] I [client-handshake.c:1411:client_setvolume_cbk] 0-docstore1-client-1: Connected to <a href="http://10.200.1.6:24009" target="_blank">10.200.1.6:24009</a>, attached to remote volume '/data/docstore1'.</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.098005] I [client-handshake.c:1423:client_setvolume_cbk] 0-docstore1-client-1: Server and Client lk-version numbers are not same, reopening the fds</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.098094] I [afr-common.c:3685:afr_notify] 0-docstore1-replicate-0: Subvolume 'docstore1-client-1' came back up; going online.</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:54.098274] I [client-handshake.c:453:client_set_lk_version_cbk] 0-docstore1-client-1: Server lk version = 1</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.098619] I [client-handshake.c:1614:select_server_supported_programs] 0-docstore1-client-0: Using Program GlusterFS 3.3.2, Num (1298437), Version (330)</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.099191] I [client-handshake.c:1411:client_setvolume_cbk] 0-docstore1-client-0: Connected to <a href="http://10.200.1.5:24009" target="_blank">10.200.1.5:24009</a>, attached to remote volume '/data/docstore1'.</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.099222] I [client-handshake.c:1423:client_setvolume_cbk] 0-docstore1-client-0: Server and Client lk-version numbers are not same, reopening the fds</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.105891] I [fuse-bridge.c:4191:fuse_graph_setup] 0-fuse: switched to graph 0</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:54.106039] I [client-handshake.c:453:client_set_lk_version_cbk] 0-docstore1-client-0: Server lk version = 1</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.106179] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.17</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:54.108766] I [afr-common.c:2022:afr_set_root_inode_on_first_lookup] 0-docstore1-replicate-0: added root inode</font></div>
</div><div><br></div><div><br></div><div>This is driving me nuts - I've been working on getting Geo-Replication working for over 2 months now without any success.</div><div><br></div><div>Status on the geo-rep shows OK:</div>
<div><br></div><div><div><font face="courier new, monospace">root@gfs6:~# gluster volume geo-replication docstore1 ssh://root@backup-ds2.gluster:/data/docstore1 status</font></div><div><font face="courier new, monospace">MASTER SLAVE STATUS </font></div>
<div><font face="courier new, monospace">--------------------------------------------------------------------------------</font></div><div><font face="courier new, monospace">docstore1 ssh://root@backup-ds2.gluster:/data/docstore1 OK </font></div>
<div><font face="courier new, monospace">root@gfs6:~# </font></div></div><div><br></div><div>Here's the config:</div><div><br></div><div><div><font face="courier new, monospace">root@gfs6:~# gluster volume geo-replication docstore1 ssh://root@backup-ds2.gluster:/data/docstore1 config</font></div>
<div><font face="courier new, monospace">log_level: DEBUG</font></div><div><font face="courier new, monospace">gluster_log_file: /var/log/glusterfs/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.gluster.log</font></div>
<div><font face="courier new, monospace">ssh_command: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem</font></div><div><font face="courier new, monospace">session_owner: 24f8c92d-723e-4513-9593-40ef4b7e766a</font></div>
<div><font face="courier new, monospace">remote_gsyncd: /usr/lib/glusterfs/glusterfs/gsyncd</font></div><div><font face="courier new, monospace">state_file: /var/lib/glusterd/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.status</font></div>
<div><font face="courier new, monospace">gluster_command_dir: /usr/sbin/</font></div><div><font face="courier new, monospace">pid_file: /var/lib/glusterd/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.pid</font></div>
<div><font face="courier new, monospace">log_file: /var/log/glusterfs/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.log</font></div><div><font face="courier new, monospace">gluster_params: xlator-option=*-dht.assert-no-child-down=true</font></div>
<div><font face="courier new, monospace">root@gfs6:~# </font></div></div><div><br></div><div>I'm running Ubuntu packages 3.3.2-ubuntu1-precise2 from the ppa. Any ideas for why it's stalling?</div><div><br></div>
</div></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img><br></div>
</font></span></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img><br>
</div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | 888.801.2020 • 304.536.1290<br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | 888.801.2020 • 304.536.1290<br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | 888.801.2020 • 304.536.1290<br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br></div>
</div>