Hi all!<br><br>After the start of pserver12 I ran the getfattr command on all 4 systems in order to check which files were out of sync. This came back with 63 files on pserver12 and none on the others. After starting the gluster server and client daemons on 12, the first batch was done automagically, as stated before. But not all of them as II would have expected.<br>
<br>Best, Martin<br><br><div class="gmail_quote">2011/4/29 Pranith Kumar. Karampuri <span dir="ltr"><<a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a>></span><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
This means that there is no differences in gfids. Could you let me know how the self heal is done after the pserver12 was brought up?.<br>
How did you find out that the self-heal is needed for 63 files?.<br>
<div class="im"><br>
Pranith.<br>
----- Original Message -----<br>
From: "Martin Schenker" <<a href="mailto:martin.schenker@profitbricks.com">martin.schenker@profitbricks.com</a>><br>
</div><div><div></div><div class="h5">To: "Pranith Kumar. Karampuri" <<a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a>>, <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
Sent: Friday, April 29, 2011 11:05:55 PM<br>
Subject: Re: [Gluster-users] Server outage, file sync/self-heal doesn't sync ALL files?!<br>
<br>
Sorry, I had manually sync due to imminent server upgrades.<br>
50 min. after the initial sync I was asked to bring the servers in a<br>
safe state for an upgrade and did a manual<br>
"touch-on-server13-client-mountpoint" which triggered an immediate<br>
self-heal on the rest of the files.<br>
<br>
All files were in sync across all four server after this action. Will<br>
run this command next time!!<br>
<br>
Best, Martin<br>
<br>
Am 29.04.2011 19:30, schrieb Pranith Kumar. Karampuri:<br>
> hi Martin,<br>
> Could you please send the output of -m "trusted*" instead of "trusted.afr" for the remaining 24 files from both the servers. I would like to see the gfids of these files on both the machines.<br>
><br>
> Pranith.<br>
> ----- Original Message -----<br>
> From: "Martin Schenker"<<a href="mailto:martin.schenker@profitbricks.com">martin.schenker@profitbricks.com</a>><br>
> To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
> Sent: Friday, April 29, 2011 8:39:46 PM<br>
> Subject: [Gluster-users] Server outage, file sync/self-heal doesn't sync ALL files?!<br>
><br>
> Hi all!<br>
><br>
> We have another incident over here.<br>
><br>
> One of the servers (pserver12) in a pair (12& 13) has been rebooted.<br>
> pserver13 showed 63 files not in sync after the outage for 2h.<br>
><br>
> Both server are clients as well.<br>
><br>
> Starting pserver12 brought up the self-heal mechanism, but only 39 files<br>
> were triggered within the first 10 min. Now the system seems dormant and<br>
> 24 files are left hanging.<br>
><br>
> On the other three servers no inconsistencies are seen.<br>
><br>
> tail of client log file:<br>
><br>
> 2011-04-29 14:48:23.820022] I<br>
> [afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]<br>
> 0-storage0-replicate-2: diff self-heal on /pserver13-17: 1960 blocks of<br>
> 22736 were different (8.62%)<br>
> [2011-04-29 14:48:23.887651] E [afr-common.c:110:afr_set_split_brain]<br>
> 0-storage0-replicate-2: invalid argument: inode<br>
> [2011-04-29 14:48:23.887740] I<br>
> [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk]<br>
> 0-storage0-replicate-2: background data self-heal completed on<br>
> /pserver13-17<br>
> [2011-04-29 14:48:24.272220] I<br>
> [afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]<br>
> 0-storage0-replicate-2: diff self-heal on /pserver13-19: 1960 blocks of<br>
> 22744 were different (8.62%)<br>
> [2011-04-29 14:48:24.341868] E [afr-common.c:110:afr_set_split_brain]<br>
> 0-storage0-replicate-2: invalid argument: inode<br>
> [2011-04-29 14:48:24.341959] I<br>
> [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk]<br>
> 0-storage0-replicate-2: background data self-heal completed on<br>
> /pserver13-19<br>
> [2011-04-29 14:48:24.758131] I<br>
> [afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]<br>
> 0-storage0-replicate-2: diff self-heal on /pserver13-23: 1952 blocks of<br>
> 22752 were different (8.58%)<br>
> [2011-04-29 14:48:24.766054] E [afr-common.c:110:afr_set_split_brain]<br>
> 0-storage0-replicate-2: invalid argument: inode<br>
> [2011-04-29 14:48:24.766137] I<br>
> [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk]<br>
> 0-storage0-replicate-2: background data self-heal completed on<br>
> /pserver13-23<br>
> [2011-04-29 14:48:24.884613] I<br>
> [afr-self-heal-algorithm.c:526:sh_diff_loop_driver_done]<br>
> 0-storage0-replicate-2: diff self-heal on /pserver13-10: 1952 blocks of<br>
> 22760 were different (8.58%)<br>
> [2011-04-29 14:48:24.895631] E [afr-common.c:110:afr_set_split_brain]<br>
> 0-storage0-replicate-2: invalid argument: inode<br>
> [2011-04-29 14:48:24.895721] I<br>
> [afr-self-heal-common.c:1527:afr_self_heal_completion_cbk]<br>
> 0-storage0-replicate-2: background data self-heal completed on<br>
> /pserver13-10<br>
> 0 root@pserver13:/var/log/glusterfs # date<br>
> Fri Apr 29 15:08:18 UTC 2011<br>
><br>
><br>
> Search for mismatch:<br>
><br>
> 0 root@pserver13:~ # getfattr -R -d -e hex -m "trusted.afr."<br>
> /mnt/gluster/brick?/storage | grep -v 0x000000000000000000000000 | grep<br>
> -B1 -A1 trusted | grep -c file<br>
> getfattr: Removing leading '/' from absolute path names<br>
> *24*<br>
><br>
><br>
> 0 root@pserver13:~ # getfattr -R -d -e hex -m "trusted.afr."<br>
> /mnt/gluster/brick?/storage | grep -v 0x000000000000000000000000 | grep<br>
> -B1 trusted<br>
> getfattr: Removing leading '/' from absolute path names<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-33<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-26<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file:<br>
> mnt/gluster/brick0/storage/images/1959/cd55c5f3-9aa1-bfd9-99a0-01c13a7d8559/hdd-images<br>
> trusted.afr.storage0-client-4=0x000000000000001600000001<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-24<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-8<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-21<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-22<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-30<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-20<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-9<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick0/storage/de-dc1-c1-pserver5-38<br>
> trusted.afr.storage0-client-4=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-18<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-2<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-23<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-4<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-3<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-34<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-37<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-12<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-27<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file:<br>
> mnt/gluster/brick1/storage/images/1831/9a039a81-60fe-5fa3-f562-8f6d3828382b/hdd-images/13169<br>
> trusted.afr.storage0-client-6=0x100000020000000000000000<br>
> --<br>
> # file:<br>
> mnt/gluster/brick1/storage/images/1959/cd55c5f3-9aa1-bfd9-99a0-01c13a7d8559/hdd-images<br>
> trusted.afr.storage0-client-6=0x000000000000001600000002<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-25<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
> --<br>
> # file: mnt/gluster/brick1/storage/de-dc1-c1-pserver5-7<br>
> trusted.afr.storage0-client-6=0x270000010000000000000000<br>
><br>
><br>
><br>
> I could trigger manually but why isn't the sync/self-heal not working on<br>
> all files shown as inconsistent? Or am I assuming something wrongly here?!?<br>
><br>
> Best, Martin<br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
><br>
<br>
</div></div></blockquote></div><br>