<div dir="ltr">Shawn,<div><br></div><div>Thanks for the detailed info. I have not yet looked into your logs, but will do so soon. There have been patches on rebalance which do fix issues related to ownership. But I am not (yet) sure about bugs which caused data loss. One question I have is -</div>
<div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">[2013-10-29 23:13:49.611069] I [dht-rebalance.c:647:dht_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">migra</span><span style="font-family:arial,sans-serif;font-size:13px">te_file] 0-mdfs-dht: /REDACTED/mdfs/KPA/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">kpacontentm</span><span style="font-family:arial,sans-serif;font-size:13px">inepix/docs/008/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">058: attempting to move from mdfs-replicate-1 to mdfs-replicate-6</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">[2013-10-29 23:13:49.611582] I [dht-rebalance.c:647:dht_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">migra</span><span style="font-family:arial,sans-serif;font-size:13px">te_file] 0-mdfs-dht: /REDACTED/mdfs/KPA/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">kpacontentm</span><span style="font-family:arial,sans-serif;font-size:13px">inepix/docs/008/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">058: attempting to move from mdfs-replicate-1 to mdfs-replicate-6</span><br>
</div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">Are these two lines from the same log file or separate log files? If they are from the same log, then it might be you need </span><span style="color:rgb(0,0,0)"><a href="http://review.gluster.org/4300">http://review.gluster.org/4300</a> (available in 3.4)</span></div>
<div><span style="color:rgb(0,0,0)"><br></span></div><div><span style="color:rgb(0,0,0)">It might also be that the "permission issues" is a cascaded effect of the same underlying problem - because the temporary file created by rebalance would have different permissions during the process of rebalance and failures might have left them in that state.</span></div>
<div><span style="color:rgb(0,0,0)"><br></span></div><div><span style="color:rgb(0,0,0)">Avati</span></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Nov 8, 2013 at 7:23 PM, Shawn Heisey <span dir="ltr"><<a href="mailto:gluster@elyograg.org" target="_blank">gluster@elyograg.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I'm starting a new thread on this, because I have more concrete information than I did the first time around. The full rebalance log from the machine where I started the rebalance can be found at the following link. It is slightly redacted - one search/replace was made to replace an identifying word with REDACTED.<br>
<br>
<a href="https://dl.dropboxusercontent.com/u/97770508/mdfs-rebalance-redacted.zip" target="_blank">https://dl.dropboxusercontent.<u></u>com/u/97770508/mdfs-rebalance-<u></u>redacted.zip</a><br>
<br>
The existing servers are running version 3.3.1-11.el6.x86_64, from kkeithley's epel repository. Those servers have CentOS 6.3 on them.<br>
<br>
The newer servers are running version 3.3.1-15.el6.x86_64, and are CentOS 6.4, fully upgraded as of October 28th, 2013. Both sets of servers have contrib, plus, epel, rpmforge, and kkeithley's glusterfs repo.<br>
<br>
One of our developers went through the log linked above and some of the source code. I am reproducing his detailed comments below. I have not looked at the log super-closely, except to produce a list of files that failed to migrate.<br>
<br>
What I'm hoping is that this information can either point us at some proof (bug, committed patch, etc) that the problem is fixed in 3.4.1, or we can use it to file a new new bug. I'm hoping that either an upgrade will fix it or that a workaround can be found.<br>
<br>
I'm still hoping to hire someone to look things over. Where can I find some good resources for this? I tried sending a message to Redhat Consulting, but something may have gone wrong with that process, because it's been two days with no response.<br>
<br>
Full quote from our developer:<br>
<br>
------------------------------<br>
Preface: I know what happened to the files and it's not what I thought it was. I don't know the exact cause but we're closer.<br>
<br>
Here's where I throw away my vague hunch of two days ago. I just realized that all the ZUMA files I saw on the new bricks were simply links created when users tried to access the files. We did indeed rebalance on the files in chronological order of their uploads. That was a two-day-long wrong-tree barking session because I didn't understand the architecture.<br>
<br>
When I looked at the individual cases of lost or corrupted files, one thing kept staring at me in the face until I recognized it:<br>
<br>
[2013-11-02 03:56:36.472170] I [dht-rebalance.c:647:dht_<u></u>migrate_file] 0-mdfs-dht: /REDACTED/mdfs/AKG/akgphotos/<u></u>docs/000/002: attempting to move from mdfs-replicate-2 to mdfs-replicate-12<br>
[2013-11-02 03:56:36.472186] I [dht-rebalance.c:647:dht_<u></u>migrate_file] 0-mdfs-dht: /REDACTED/mdfs/AKG/akgphotos/<u></u>docs/000/002: attempting to move from mdfs-replicate-2 to mdfs-replicate-12<br>
[2013-11-02 03:56:36.480567] I [dht-rebalance.c:647:dht_<u></u>migrate_file] 0-mdfs-dht: /REDACTED/mdfs/AKG/akgphotos/<u></u>docs/000/002: attempting to move from mdfs-replicate-2 to mdfs-replicate-12<br>
<br>
Three simultaneous processes on the same file! Of course that would have undefined results, and be the cause of all our problems. NFS may not be related after all.<br>
<br>
Tediously scrolling through the error log I found mostly errors where it refused to copy files from a more empty brick to a fuller brick, which makes perfect sense. The wheels started falling off about 26 hours into the rebalance.<br>
<br>
[2013-10-29 23:13:17.193108] C [client-handshake.c:126:rpc_<u></u>client_ping_timer_expired] 0-mdfs-client-1: server <a href="http://10.116.0.22:24025" target="_blank">10.116.0.22:24025</a> has not responded in the last 42 seconds, disconnecting.<br>
[2013-10-29 23:13:17.200616] E [rpc-clnt.c:373:saved_frames_<u></u>unwind] (-->/usr/lib64/libgfrpc.so.0(<u></u>rpc_clnt_notify+0x78) [0x36de60f808] (-->/usr/lib64/libgfrpc.so.0(<u></u>rpc_clnt_connection_cleanup+<u></u>0xb0) [0x36de60f4c0] (-->/usr/lib64/libgfrpc.so.0(<u></u>saved_frames_destroy+0xe) [0x36de60ef2e]))) 0-mdfs-client-1: forced unwinding frame type(GlusterFS 3.1) op(STAT(1)) called at 2013-10-29 23:12:20.641732 (xid=0x1763733x)<br>
[2013-10-29 23:13:17.200659] W [client3_1-fops.c:474:client3_<u></u>1_stat_cbk] 0-mdfs-client-1: remote operation failed: Transport endpoint is not connected<br>
[2013-10-29 23:13:17.200756] E [rpc-clnt.c:373:saved_frames_<u></u>unwind] (-->/usr/lib64/libgfrpc.so.0(<u></u>rpc_clnt_notify+0x78) [0x36de60f808] (-->/usr/lib64/libgfrpc.so.0(<u></u>rpc_clnt_connection_cleanup+<u></u>0xb0) [0x36de60f4c0] (-->/usr/lib64/libgfrpc.so.0(<u></u>saved_frames_destroy+0xe) [0x36de60ef2e]))) 0-mdfs-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2013-10-29 23:12:35.184481 (xid=0x1763734x)<br>
[2013-10-29 23:13:17.200792] W [client-handshake.c:275:<u></u>client_ping_cbk] 0-mdfs-client-1: timer must have expired<br>
[2013-10-29 23:13:17.217477] I [client.c:2090:client_rpc_<u></u>notify] 0-mdfs-client-1: disconnected<br>
.<br>
.<br>
.<br>
[2013-10-29 23:13:28.198386] I [client-handshake.c:1636:<u></u>select_server_supported_<u></u>programs] 0-mdfs-client-1: Using Program GlusterFS 3.3.1, Num (1298437), Version (330)<br>
[2013-10-29 23:13:28.198803] I [client-handshake.c:1433:<u></u>client_setvolume_cbk] 0-mdfs-client-1: Connected to <a href="http://10.116.0.22:24025" target="_blank">10.116.0.22:24025</a>, attached to remote volume '/bricks/d00v00/mdfs'.<br>
[2013-10-29 23:13:28.198826] I [client-handshake.c:1445:<u></u>client_setvolume_cbk] 0-mdfs-client-1: Server and Client lk-version numbers are not same, reopening the fds<br>
[2013-10-29 23:13:28.205629] I [client-handshake.c:1282:<u></u>client_post_handshake] 0-mdfs-client-1: 42745 fds open - Delaying child_up until they are re-opened<br>
[2013-10-29 23:13:28.205782] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>client_post_handshake+0x110) [0x7fb74bdf1c90] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>protocol_client_reopendir+<u></u>0x107) [0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.<u></u>so.0(inode_path+0x4a) [0x36dde2ca1a]))) 0-: Assertion failed: 0<br>
[2013-10-29 23:13:28.205852] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>client_post_handshake+0x110) [0x7fb74bdf1c90] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>protocol_client_reopendir+<u></u>0x107) [0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.<u></u>so.0(inode_path+0x4a) [0x36dde2ca1a]))) 0-mdfs-client-1: invalid inode<br>
[2013-10-29 23:13:28.205881] W [client-handshake.c:1108:<u></u>protocol_client_reopendir] 0-mdfs-client-1: couldn't build path from inode 00000000-0000-0000-0000-<u></u>000000000000<br>
[2013-10-29 23:13:28.205944] E [inode.c:1090:__inode_path] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>client_post_handshake+0x110) [0x7fb74bdf1c90] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>protocol_client_reopendir+<u></u>0x107) [0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.<u></u>so.0(inode_path+0x4a) [0x36dde2ca1a]))) 0-: Assertion failed: 0<br>
[2013-10-29 23:13:28.205992] W [inode.c:1091:__inode_path] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>client_post_handshake+0x110) [0x7fb74bdf1c90] (-->/usr/lib64/glusterfs/3.3.<u></u>1/xlator/protocol/client.so(<u></u>protocol_client_reopendir+<u></u>0x107) [0x7fb74bdf1907] (-->/usr/lib64/libglusterfs.<u></u>so.0(inode_path+0x4a) [0x36dde2ca1a]))) 0-mdfs-client-1: invalid inode<br>
[2013-10-29 23:13:28.206009] W [client-handshake.c:1108:<u></u>protocol_client_reopendir] 0-mdfs-client-1: couldn't build path from inode 00000000-0000-0000-0000-<u></u>000000000000<br>
<br>
The assertion failures actually go on for thousands of lines even though only 3 seconds have elapsed. Eventually it overcame whatever that handshake error means. Not long after:<br>
<br>
[2013-10-29 23:13:49.611069] I [dht-rebalance.c:647:dht_<u></u>migrate_file] 0-mdfs-dht: /REDACTED/mdfs/KPA/<u></u>kpacontentminepix/docs/008/<u></u>058: attempting to move from mdfs-replicate-1 to mdfs-replicate-6<br>
[2013-10-29 23:13:49.611582] I [dht-rebalance.c:647:dht_<u></u>migrate_file] 0-mdfs-dht: /REDACTED/mdfs/KPA/<u></u>kpacontentminepix/docs/008/<u></u>058: attempting to move from mdfs-replicate-1 to mdfs-replicate-6<br>
<br>
and there's our first lost file. From that point on, we were trashing files with simultaneous copy processes. I think a process hung up, and a new one was spawned, and when the first process woke back up it didn't know enough to stop.<br>
------------------------------<br>
<br>
Shawn<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
</blockquote></div><br></div>