<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">So the slave is slowly catching up. But I agree, it's too slow. The "slowness" you mention could be due to an inherent delay issue with Geo-replication. After geo-replication identifies a set of files that needs to be synced, it invokes rsync to transfer data. After a successful rsync, it sets extended attributes on "every" file/directory on the slave to mark it as synced. This does take up a significant amount of time and contributes to the "stalls" that you mention (even in our tests this has been a pain point).</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">There is a workaround for the above mentioned issue, which is to use rsync to sync extended attributes, but I think that's not available in the version you're using.</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Now, regarding manually rsync'ing the initial data set: Geo-replication replies on a extended attribute (xtime) comparison between the master and the slave to identify files that needs to be synced. Rsync without '-X' would not sync extended attributes, which implies that after geo-replication is started, crawling would still happen for the entire tree (data sync may not happen as rsync would identify no data differences). Now, in case you get tempted to use rsync (with -X), could you do the following:</div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"># mkdir /tmp/1</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
# <span style="font-size:13.333333969116211px;font-family:arial,sans-serif">glust</span><span style="font-size:13.333333969116211px;font-family:arial,sans-serif">erfs --volfile-id=<volume></span><span style="font-size:13.333333969116211px;font-family:arial,sans-serif"> --volfile-serve</span><span style="font-size:13.333333969116211px;font-family:arial,sans-serif">r=<server> --client-pid=-1 /tmp/1/</span></div>
<div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-size:13.333333969116211px;font-family:arial,sans-serif"># getfattr -m . -d /tmp/1</span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
<span style="font-size:13.333333969116211px;font-family:arial,sans-serif"># umount /tmp/1</span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
The third command would tell what all extended attributes would rsync try to sync in case '-X' is used.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">
Thanks,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">-venky</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><div class="gmail_quote">
On Tue, Sep 10, 2013 at 9:50 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Venky,<div><br></div><div>Just touching back with you to see where else I might want to look on this. The primary brick running the replication is still using a good deal of CPU though not quite what it was when the geo-rep was first started:</div>
<div><br></div><div><img src="cid:ii_14108aa5308dbe3b" alt="Inline image 2"><br></div><div><br></div><div>You can see just above Week 34 when I started the geo-rep process. The major drop just before the words "Week 36" is when it stopped sending the initial data to the target.</div>
<div><br></div><div>The replication target machine continues to slowly - very slowly - gain new data. Unfortunately I think it's only data that's been added recently. It doesn't appear to still be replicating the initial batch. That seems to have stopped right about 1TB total. I'm currently at 1.2 TB replicated and it's been over a week now since I eclipsed 1 TB.</div>
<div><br></div><div>Would I do better to stop the process, wipe the geo-rep information from Gluster, do an rsync of the files to the destination and then start the geo-rep process again?</div></div><div class=""><div class="h5">
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Sep 6, 2013 at 11:40 AM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">Correction: Different between the two local bricks is more likely a result of the ZFS snapshot process I've run on the larger of them.</div><div><div><div class="gmail_extra"><br>
<br><div class="gmail_quote">On Fri, Sep 6, 2013 at 11:39 AM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">More stats in case you need them:<div><br>
</div><div>Grand total of the /data/docstore1 tree that's being sync'd is about 8.5 million files.</div>
<div>About 1 KB of new data has appeared on the destination server since we started this discussion yesterday...</div>
<div><br></div><div><br></div><div><div><font face="courier new, monospace">tony@backup-ds2:~$ df</font></div><div><font face="courier new, monospace">Filesystem 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/sda1 768989240 1841364 728085452 1% /</font></div><div><font face="courier new, monospace">udev 16450080 4 16450076 1% /dev</font></div><div>
<font face="courier new, monospace">tmpfs 6583556 348 6583208 1% /run</font></div><div><font face="courier new, monospace">none 5120 0 5120 0% /run/lock</font></div>
<div><font face="courier new, monospace">none 16458888 0 16458888 0% /run/shm</font></div><div><font face="courier new, monospace">data 6601579264 0 6601579264 0% /data</font></div>
<div><font face="courier new, monospace">data/docstore1 7651735936 1050156672 6601579264 14% /data/docstore1</font></div><div><font face="courier new, monospace">tony@backup-ds2:~$ date</font></div><div><font face="courier new, monospace">Thu Sep 5 09:36:24 EDT 2013</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">tony@backup-ds2:~$ df</font></div><div><font face="courier new, monospace">Filesystem 1K-blocks Used Available Use% Mounted on</font></div>
<div><font face="courier new, monospace">/dev/sda1 768989240 1842524 728084292 1% /</font></div><div><font face="courier new, monospace">udev 16450080 4 16450076 1% /dev</font></div><div>
<font face="courier new, monospace">tmpfs 6583556 348 6583208 1% /run</font></div><div><font face="courier new, monospace">none 5120 0 5120 0% /run/lock</font></div>
<div><font face="courier new, monospace">none 16458888 0 16458888 0% /run/shm</font></div><div><font face="courier new, monospace">data 6601577984 0 6601577984 0% /data</font></div>
<div><font face="courier new, monospace">data/docstore1 7651735808 1050157824 6601577984 14% /data/docstore1</font></div><div><font face="courier new, monospace">tony@backup-ds2:~$ date</font></div><div><font face="courier new, monospace">Fri Sep 6 11:34:07 EDT 2013</font></div>
<div><font face="courier new, monospace">tony@backup-ds2:~$ </font></div></div><div><br></div><div>The source data I'm attempting to geo-replicate is 3.3 TB. Mirroring between the local bricks seems to be working fine. They are within 20KB of drive usage between each other for the /data/docstore1 partition. I attribute that to the new geo-rep info on gfs6 making it larger by a bit? Also the data is constantly changing, so maybe it's a bit of replication lag.</div>
<div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 6, 2013 at 11:29 AM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">./evds3 contains 1,262,970 files. No more than 255 files or folders per subdirectory.<div>
<br></div><div>
Network latency is almost nonexistent - the servers are currently jacked into the same switch on a separate network segment for the initial sync:</div>
<div><br></div><div><div><font face="courier new, monospace">tony@gfs6:~$ ping backup-ds2.gluster</font></div><div><font face="courier new, monospace">PING backup-ds2.gluster (10.200.1.12) 56(84) bytes of data.</font></div>
<div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=1 ttl=64 time=0.245 ms</font></div><div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=2 ttl=64 time=0.197 ms</font></div>
<div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=3 ttl=64 time=0.174 ms</font></div><div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=4 ttl=64 time=0.221 ms</font></div>
<div><font face="courier new, monospace">64 bytes from backup-ds2.gluster (10.200.1.12): icmp_req=5 ttl=64 time=0.163 ms</font></div></div><div><br></div><div><br></div><div><br></div></div><div><div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">On Fri, Sep 6, 2013 at 5:13 AM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div style="font-family:arial,helvetica,sans-serif">From the trace logs it looks like its still crawling (the lgetxattr() on the master and lgetxattr() on the slave). How many files are under '<span style="color:rgb(51,51,51);font-family:monospace,'Lucida Grande',Verdana,Arial,Helvetica,sans-serif;font-size:13px;white-space:pre-wrap">./evds3' ?</span></div>
<div style="font-family:arial,helvetica,sans-serif"><span style="color:rgb(51,51,51);font-family:monospace,'Lucida Grande',Verdana,Arial,Helvetica,sans-serif;font-size:13px;white-space:pre-wrap"><br>
</span></div><div style="font-family:arial,helvetica,sans-serif"><span style="color:rgb(51,51,51);font-family:monospace,'Lucida Grande',Verdana,Arial,Helvetica,sans-serif;font-size:13px;white-space:pre-wrap">Further more, what's the latency b/w the two sites?</span></div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 11:31 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">No rsync process is running. I tested for it several times over 10 seconds.<div>
<br></div><div>Here's the trace on feedback. I put it on TinyPaste instead of in everyone's inbox out of courtesy: <a href="http://tny.cz/505b09c8" target="_blank">http://tny.cz/505b09c8</a></div>
<div><br></div><div>Thanks,</div><div>Tony</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 12:24 PM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div style="font-family:arial,helvetica,sans-serif">
So, nothing fishy in slave logs too.</div>
<div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">
Debugging this would need more information. Lets start with:</div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">
1. Is there any rsync process that is running? (ps auxww | grep rsync)</div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">
2. I would need strace logs for the worker process (the process which scans the filesystem and invokes rsync to transfer data). Something like this would do:</div><div style="font-family:arial,helvetica,sans-serif">
<br></div><div style="font-family:arial,helvetica,sans-serif"> - get the pid of the worker process</div><div style="font-family:arial,helvetica,sans-serif"> # ps auxww | grep feedback (a gsyncd.py python process)</div>
<div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif"> - trace the pid above</div><div style="font-family:arial,helvetica,sans-serif">
# strace -s 500 -o /tmp/worker.log -f -p <pid></div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">Let the trace run for about 20 seconds. Lets see what's in /tmp/worker.log<br>
</div><div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">Thanks,</div><div style="font-family:arial,helvetica,sans-serif">
-venky</div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 7:56 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Looks like the slave stopped logging things just before the sending brick did:<div>
<br></div><div><div>
<font face="courier new, monospace">[2013-09-01 14:56:45.13972] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 14:56:45.15433] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 15:57:34.592938] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 15:57:34.593383] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 15:57:45.374301] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 15:57:45.375871] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 16:02:09.115976] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:02:09.116446] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 16:02:19.869340] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:02:19.870598] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 16:32:58.701617] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:32:58.702126] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 16:33:09.456556] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 16:33:09.458009] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-01 21:47:13.442514] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-01 21:47:13.442899] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-01 21:47:24.240978] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-01 21:47:24.242424] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-02 02:12:14.26339] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:12:14.26809] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-02 02:12:24.818355] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:12:24.820008] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-02 02:16:14.525187] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:16:14.525675] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-02 02:16:25.263712] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:16:25.265168] I [resource(slave):453:service_loop] FILE: slave listening</font></div><div><font face="courier new, monospace">[2013-09-02 02:37:39.315608] I [repce(slave):78:service_loop] RepceServer: terminating on reaching EOF.</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:37:39.316071] I [syncdutils(slave):142:finalize] <top>: exiting.</font></div><div><font face="courier new, monospace">[2013-09-02 02:37:50.78136] I [gsyncd(slave):354:main_i] <top>: syncing: file:///data/docstore1</font></div>
<div><font face="courier new, monospace">[2013-09-02 02:37:50.79577] I [resource(slave):453:service_loop] FILE: slave listening</font></div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra">
<div><div>
<br><br><div class="gmail_quote">On Thu, Sep 5, 2013 at 10:09 AM, Venky Shankar <span dir="ltr"><<a href="mailto:yknev.shankar@gmail.com" target="_blank">yknev.shankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div style="font-family:arial,helvetica,sans-serif">Could you also provide the slave logs? (log location on the slave: /var/log/glusterfs/geo-replication-slaves)</div><div style="font-family:arial,helvetica,sans-serif">
<br></div><div style="font-family:arial,helvetica,sans-serif">Thanks,</div><div style="font-family:arial,helvetica,sans-serif">-venky</div></div><div class="gmail_extra"><br><br>
<div class="gmail_quote"><div><div>On Thu, Sep 5, 2013 at 7:29 PM, Tony Maro <span dir="ltr"><<a href="mailto:tonym@evrichart.com" target="_blank">tonym@evrichart.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>
<div dir="ltr">I'm trying to create a new Geo-Rep of about 3 TB of data currently stored in a 2 brick mirror config. Obviously the geo-rep destination is a third server.<div><br></div><div>This is my 150th attempt. Okay, maybe not that far, but it's pretty darn bad.</div>
<div><br></div><div><div>Replication works fine until I hit around 1TB of data sync'd, then it just stalls. For the past two days it hasn't gone past 1050156672 bytes sync'd to the destination server.</div><div>
<br></div><div>I did some digging in the logs and it looks like the brick that's running the geo-rep process thinks it's syncing:</div><div><br></div><div><div><font face="courier new, monospace">[2013-09-05 09:45:37.354831] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000863.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.358669] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/0000083b.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.362251] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/0000087b.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.366027] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000834.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.369752] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000845.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.373528] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000864.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.377037] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/0000087f.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.391432] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000897.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.395059] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000829.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.398725] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/00000816.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.402559] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008cc.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.406450] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008d2.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.410310] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008df.enc ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.414344] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/00/00/08/000008bd.enc ...</font></div><div><font face="courier new, monospace">[2013-09-05 09:45:37.438173] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/<a href="http://volume.info" target="_blank">volume.info</a> ...</font></div>
<div><font face="courier new, monospace">[2013-09-05 09:45:37.441675] D [master:386:crawl] GMaster: syncing ./evds3/Sky_Main_66/volume.enc ...</font></div></div><div><br></div><div>But, <b>those files never appear on the destination server,</b> however the containing folders are there, just empty.</div>
<div><br></div><div>Also, the other log file (...gluster.log) in the geo-replication log folder that matches the destination stopped updating when the syncing stopped apparently. It's last timestamp is from the 2nd, which is the last time data transferred.</div>
<div><br></div><div>The last bit from that log file is as such:</div><div><br></div><div><div><font face="courier new, monospace">+------------------------------------------------------------------------------+</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:50.109730] I [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-docstore1-client-1: changing port to 24009 (from 0)</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:50.109857] I [rpc-clnt.c:1654:rpc_clnt_reconfig] 0-docstore1-client-0: changing port to 24009 (from 0)</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.097468] I [client-handshake.c:1614:select_server_supported_programs] 0-docstore1-client-1: Using Program GlusterFS 3.3.2, Num (1298437), Version (330)</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.097973] I [client-handshake.c:1411:client_setvolume_cbk] 0-docstore1-client-1: Connected to <a href="http://10.200.1.6:24009" target="_blank">10.200.1.6:24009</a>, attached to remote volume '/data/docstore1'.</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.098005] I [client-handshake.c:1423:client_setvolume_cbk] 0-docstore1-client-1: Server and Client lk-version numbers are not same, reopening the fds</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.098094] I [afr-common.c:3685:afr_notify] 0-docstore1-replicate-0: Subvolume 'docstore1-client-1' came back up; going online.</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:54.098274] I [client-handshake.c:453:client_set_lk_version_cbk] 0-docstore1-client-1: Server lk version = 1</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.098619] I [client-handshake.c:1614:select_server_supported_programs] 0-docstore1-client-0: Using Program GlusterFS 3.3.2, Num (1298437), Version (330)</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.099191] I [client-handshake.c:1411:client_setvolume_cbk] 0-docstore1-client-0: Connected to <a href="http://10.200.1.5:24009" target="_blank">10.200.1.5:24009</a>, attached to remote volume '/data/docstore1'.</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.099222] I [client-handshake.c:1423:client_setvolume_cbk] 0-docstore1-client-0: Server and Client lk-version numbers are not same, reopening the fds</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.105891] I [fuse-bridge.c:4191:fuse_graph_setup] 0-fuse: switched to graph 0</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:54.106039] I [client-handshake.c:453:client_set_lk_version_cbk] 0-docstore1-client-0: Server lk version = 1</font></div>
<div><font face="courier new, monospace">[2013-09-02 06:37:54.106179] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.17</font></div><div><font face="courier new, monospace">[2013-09-02 06:37:54.108766] I [afr-common.c:2022:afr_set_root_inode_on_first_lookup] 0-docstore1-replicate-0: added root inode</font></div>
</div><div><br></div><div><br></div><div>This is driving me nuts - I've been working on getting Geo-Replication working for over 2 months now without any success.</div><div><br></div><div>Status on the geo-rep shows OK:</div>
<div><br></div><div><div><font face="courier new, monospace">root@gfs6:~# gluster volume geo-replication docstore1 ssh://root@backup-ds2.gluster:/data/docstore1 status</font></div><div><font face="courier new, monospace">MASTER SLAVE STATUS </font></div>
<div><font face="courier new, monospace">--------------------------------------------------------------------------------</font></div><div><font face="courier new, monospace">docstore1 ssh://root@backup-ds2.gluster:/data/docstore1 OK </font></div>
<div><font face="courier new, monospace">root@gfs6:~# </font></div></div><div><br></div><div>Here's the config:</div><div><br></div><div><div><font face="courier new, monospace">root@gfs6:~# gluster volume geo-replication docstore1 ssh://root@backup-ds2.gluster:/data/docstore1 config</font></div>
<div><font face="courier new, monospace">log_level: DEBUG</font></div><div><font face="courier new, monospace">gluster_log_file: /var/log/glusterfs/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.gluster.log</font></div>
<div><font face="courier new, monospace">ssh_command: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem</font></div><div><font face="courier new, monospace">session_owner: 24f8c92d-723e-4513-9593-40ef4b7e766a</font></div>
<div><font face="courier new, monospace">remote_gsyncd: /usr/lib/glusterfs/glusterfs/gsyncd</font></div><div><font face="courier new, monospace">state_file: /var/lib/glusterd/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.status</font></div>
<div><font face="courier new, monospace">gluster_command_dir: /usr/sbin/</font></div><div><font face="courier new, monospace">pid_file: /var/lib/glusterd/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.pid</font></div>
<div><font face="courier new, monospace">log_file: /var/log/glusterfs/geo-replication/docstore1/ssh%3A%2F%2Froot%4010.200.1.12%3Afile%3A%2F%2F%2Fdata%2Fdocstore1.log</font></div><div><font face="courier new, monospace">gluster_params: xlator-option=*-dht.assert-no-child-down=true</font></div>
<div><font face="courier new, monospace">root@gfs6:~# </font></div></div><div><br></div><div>I'm running Ubuntu packages 3.3.2-ubuntu1-precise2 from the ppa. Any ideas for why it's stalling?</div><div><br></div>
</div></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img><br></div>
</font></span></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img><br>
</div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br>
</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br>
</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br>
</div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Thanks,<br><br><b>Tony Maro</b><br>Chief Information Officer<br>EvriChart • <a href="http://www.evrichart.com" target="_blank">www.evrichart.com</a><br>
Advanced Records Management<br>Office | <a href="tel:888.801.2020" value="+18888012020" target="_blank">888.801.2020</a> • <a href="tel:304.536.1290" value="+13045361290" target="_blank">304.536.1290</a><br><img src="http://www.evrichart.com/images/evrichart-150w.gif"><br>
</div>
</div>
</div></div></blockquote></div><br></div></div>