<div dir="ltr"><div>[2014-11-06 12:21:01.575385] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-HA-WIN-TT-1T-replicate-0: foreground data self heal is successfully completed, data self heal from HA-WIN-TT-1T-client-1 to sinks HA-WIN-TT-1T-client-0, with 966367641600 bytes on HA-WIN-TT-1T-client-0, 966367641600 bytes on HA-WIN-TT-1T-client-1, data - Pending matrix: [ [ 1 1 ] [ 14192 0 ] ] on <gfid:a2d3dc2d-6572-4e14-b1bf-4f1027433117></div><div>[2014-11-06 12:21:01.576454] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-1: remote operation failed: No such file or directory. Path: <gfid:b8026077-696e-45a8-b5a5-9c55e3813d38> (b8026077-696e-45a8-b5a5-9c55e3813d38)</div><div>[2014-11-06 12:21:01.576517] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: No such file or directory. Path: <gfid:b8026077-696e-45a8-b5a5-9c55e3813d38> (b8026077-696e-45a8-b5a5-9c55e3813d38)</div><div>[2014-11-06 12:21:01.577284] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-1: remote operation failed: No such file or directory. Path: <gfid:2f1ad6a8-11aa-483c-a561-75ce45e5f245> (2f1ad6a8-11aa-483c-a561-75ce45e5f245)</div><div>[2014-11-06 12:21:01.577349] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: No such file or directory. Path: <gfid:2f1ad6a8-11aa-483c-a561-75ce45e5f245> (2f1ad6a8-11aa-483c-a561-75ce45e5f245)</div><div>[2014-11-06 12:21:01.578105] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-1: remote operation failed: No such file or directory. Path: <gfid:66b3e4de-aea7-4fa2-9c0d-93fc1846dc35> (66b3e4de-aea7-4fa2-9c0d-93fc1846dc35)</div><div>[2014-11-06 12:21:01.578170] W [client-rpc-fops.c:2774:client3_3_lookup_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: No such file or directory. Path: <gfid:66b3e4de-aea7-4fa2-9c0d-93fc1846dc35> (66b3e4de-aea7-4fa2-9c0d-93fc1846dc35)</div><div>[2014-11-06 12:21:01.902786] I [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-HA-WIN-TT-1T-replicate-0: metadata self heal is successfully completed, foreground data self heal is successfully completed, data self heal from HA-WIN-TT-1T-client-1 to sinks HA-WIN-TT-1T-client-0, with 0 bytes on HA-WIN-TT-1T-client-0, 4 bytes on HA-WIN-TT-1T-client-1, data - Pending matrix: [ [ 0 0 ] [ 3 0 ] ] metadata self heal from source HA-WIN-TT-1T-client-1 to HA-WIN-TT-1T-client-0, metadata - Pending matrix: [ [ 0 0 ] [ 3 0 ] ], on <gfid:c7b624e4-1c66-4de7-aa89-95460ff098aa></div><div><br></div><div>ehm, just found this. what could it mean? any way to gfid to file transformation?</div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-11-06 14:40 GMT+02:00 Roman <span dir="ltr"><<a href="mailto:romeo.r@gmail.com" target="_blank">romeo.r@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">oh, never mind. it is synced now. took a LOT of time :)<div><br></div></div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">2014-11-06 13:12 GMT+02:00 Roman <span dir="ltr"><<a href="mailto:romeo.r@gmail.com" target="_blank">romeo.r@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hi,</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">another stupid/interesting situation:</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><div>root@stor1:~# gluster volume heal HA-WIN-TT-1T info</div><div>Brick stor1:/exports/NFS-WIN/1T/</div><div>/disk - Possibly undergoing heal</div><div>Number of entries: 1</div><div><br></div><div>Brick stor2:/exports/NFS-WIN/1T/</div><div>/test</div><div>/disk - Possibly undergoing heal</div><div>Number of entries: 2</div><div><br></div><div>due to testings I've brought down stor1 port on the switch and the made it up again.</div><div>then one of the volumes successfully restored and healed (with virtual machines)</div><div>while other still (about 2 hours atm) says, there is a healing process, meanwhile there is no traffic between the servers and client/server.</div><div><br></div><div>the /test is simple new file, i've made while stor1 was down.</div><div>the /disk is a simple virtual block-device made of /dev/null which is 900GB and is mounted on windows server via iscsitarget :). and it seem it wont stop healing forever, as it can't decide which file is right?</div><div><br></div><div>gluster client machine, where is volume for iscsi target is monted logs:</div><div><div>[2014-11-06 08:19:36.949092] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: Transport endpoint is not connected</div><div>[2014-11-06 08:19:36.949148] W [client-rpc-fops.c:1812:client3_3_fxattrop_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: Transport endpoint is not connected</div><div>[2014-11-06 08:19:36.951202] W [client-rpc-fops.c:1580:client3_3_finodelk_cbk] 0-HA-WIN-TT-1T-client-0: remote operation failed: Transport endpoint is not connected</div><div>[2014-11-06 08:19:57.682937] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on <a href="http://10.250.0.1:24007/" target="_blank">10.250.0.1:24007</a> failed (Connection timed out)</div><div>[2014-11-06 08:20:17.950981] E [socket.c:2161:socket_connect_finish] 0-glusterfs: connection to <a href="http://10.250.0.1:24007/" target="_blank">10.250.0.1:24007</a> failed (No route to host)</div><div>[2014-11-06 08:20:40.062928] E [socket.c:2161:socket_connect_finish] 0-HA-WIN-TT-1T-client-0: connection to <a href="http://10.250.0.1:24007/" target="_blank">10.250.0.1:24007</a> failed (Connection timed out)</div><div>[2014-11-06 08:30:15.638197] W [dht-diskusage.c:232:dht_is_subvol_filled] 0-HA-WIN-TT-1T-dht: disk space on subvolume 'HA-WIN-TT-1T-replicate-0' is getting full (95.00 %), consider adding more nodes</div><div>[2014-11-06 08:36:18.385659] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing</div><div>[2014-11-06 08:36:18.386573] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-HA-WIN-TT-1T-client-0: changing port to 49160 (from 0)</div><div>[2014-11-06 08:36:18.387182] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2014-11-06 08:36:18.387414] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Connected to <a href="http://10.250.0.1:49160/" target="_blank">10.250.0.1:49160</a>, attached to remote volume '/exports/NFS-WIN/1T'.</div><div>[2014-11-06 08:36:18.387433] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-0: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2014-11-06 08:36:18.387446] I [client-handshake.c:1314:client_post_handshake] 0-HA-WIN-TT-1T-client-0: 1 fds open - Delaying child_up until they are re-opened</div><div>[2014-11-06 08:36:18.387730] I [client-handshake.c:936:client_child_up_reopen_done] 0-HA-WIN-TT-1T-client-0: last fd open'd/lock-self-heal'd - notifying CHILD-UP</div><div>[2014-11-06 08:36:18.387862] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-0: Server lk version = 1</div></div><div><br></div><div>brick log on stor1:</div><div><br></div><div><div>[2014-11-06 08:38:04.269503] I [client-handshake.c:1677:select_server_supported_programs] 0-HA-WIN-TT-1T-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2014-11-06 08:38:04.269908] I [client-handshake.c:1462:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Connected to <a href="http://10.250.0.2:49160/" target="_blank">10.250.0.2:49160</a>, attached to remote volume '/exports/NFS-WIN/1T'.</div><div>[2014-11-06 08:38:04.269962] I [client-handshake.c:1474:client_setvolume_cbk] 0-HA-WIN-TT-1T-client-1: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2014-11-06 08:38:04.270560] I [client-handshake.c:450:client_set_lk_version_cbk] 0-HA-WIN-TT-1T-client-1: Server lk version = 1</div><div>[2014-11-06 08:39:33.277219] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 08:49:33.327786] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 08:59:33.375835] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 09:09:33.430726] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 09:19:33.486488] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 09:29:33.541596] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 09:39:33.595242] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 09:49:33.648526] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 09:59:33.702368] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 10:09:33.756633] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 10:19:33.810984] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 10:29:33.865172] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 10:39:33.918765] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 10:49:33.973283] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div><div>[2014-11-06 10:59:34.028836] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-HA-WIN-TT-1T-replicate-0: Another crawl is in progress for HA-WIN-TT-1T-client-0</div></div><div><br></div><div>same on stor2</div></div><span><font color="#888888"><div><br></div>-- <br><div>Best regards,<br>Roman.</div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div>Best regards,<br>Roman.</div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Best regards,<br>Roman.</div>
</div>