<div dir="ltr"><div>The file is :<br>/mnt/search-prod/index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.index<br><br>One client's log<br>============================================================<br>
[2014-02-16 04:02:02.089758] I [glusterfsd-mgmt.c:1565:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing<br>
[2014-02-18 11:06:04.621735] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-search-prod-client-3: remote operation failed: Stale NFS file handle. Path: /index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.data (808a5a65-36b8-4fdc-814a-5b832b36c394)<br>
[2014-02-18 11:06:04.621875] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-search-prod-client-4: remote operation failed: Stale NFS file handle. Path: /index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.data (808a5a65-36b8-4fdc-814a-5b832b36c394)<br>
[2014-02-18 11:06:04.621913] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-search-prod-client-5: remote operation failed: Stale NFS file handle. Path: /index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.data (808a5a65-36b8-4fdc-814a-5b832b36c394)<br>
[2014-02-18 11:18:04.070026] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-search-prod-client-5: remote operation failed: Stale NFS file handle. Path: /index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.index (58a65eb7-8fb7-432b-a9f1-f27f99439f18)<br>
[2014-02-18 11:18:04.070090] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-search-prod-client-4: remote operation failed: Stale NFS file handle. Path: /index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.index (58a65eb7-8fb7-432b-a9f1-f27f99439f18)<br>
[2014-02-18 11:18:04.070130] W [client3_1-fops.c:2630:client3_1_lookup_cbk] 0-search-prod-client-3: remote operation failed: Stale NFS file handle. Path: /index_pipeline_searchengine/leaf/data/2014-02-18/shard0/index_data.index (58a65eb7-8fb7-432b-a9f1-f27f99439f18)<br>
<br></div>The other client's log<br><div>============================================================<br>[2014-02-18 10:00:18.464929] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background entry self-heal completed on /index_pipeline_searchengine/root/data/2014-02-11<br>
[2014-02-18 10:00:19.409340] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background entry self-heal triggered. path: /index_pipeline_searchengine/root/data, reason: lookup detected pending operations<br>
[2014-02-18 10:00:19.503364] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background entry self-heal completed on /index_pipeline_searchengine/root/data<br>[2014-02-18 10:31:39.598236] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done, reason: lookup detected pending operations<br>
[2014-02-18 10:31:39.696935] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done<br>
[2014-02-18 10:31:42.312547] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done, reason: lookup detected pending operations<br>
[2014-02-18 10:31:42.406079] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done<br>
[2014-02-18 10:32:40.114804] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done, reason: lookup detected pending operations<br>
[2014-02-18 10:32:40.208238] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done<br>
[2014-02-18 10:33:40.162889] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done, reason: lookup detected pending operations<br>
[2014-02-18 10:33:40.260236] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done<br>
[2014-02-18 10:34:40.205376] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done, reason: lookup detected pending operations<br>
[2014-02-18 10:34:40.298777] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/channel_leaf/done/2014-02-18.done<br>
[2014-02-18 10:43:52.959485] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data data self-heal triggered. path: /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0/restriction_info, reason: lookup detected pending operations<br>
[2014-02-18 10:43:53.177817] I [afr-self-heal-data.c:712:afr_sh_data_fix] 0-search-prod-replicate-4: no active sinks for performing self-heal on file /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0/restriction_info<br>
[2014-02-18 10:43:53.240084] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data data self-heal completed on /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0/restriction_info<br>
[2014-02-18 10:58:07.853097] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0, reason: lookup detected pending operations<br>
[2014-02-18 10:58:07.948421] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0<br>
[2014-02-18 10:58:09.136128] I [afr-common.c:1340:afr_launch_self_heal] 0-search-prod-replicate-4: background meta-data self-heal triggered. path: /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0, reason: lookup detected pending operations<br>
[2014-02-18 10:58:09.232103] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-search-prod-replicate-4: background meta-data self-heal completed on /index_pipeline_searchengine/second_leaf/data/2014-02-18/shard0<br>
<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Feb 18, 2014 at 2:10 PM, Mingfan Lu <span dir="ltr"><<a href="mailto:mingfan.lu@gmail.com" target="_blank">mingfan.lu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Hi,<br></div> We saw such a issue.<br></div>
One client (fuse mount) updated one file, then the other client (also fuse mount) copied the same file while the reader found that the copied file was out-of-dated.<br>
</div> If the reader ran ls command to list the entries of the directory where the target file in,then it could copy the latest one.<br></div> Two clients's version:<br></div> glusterfs-3.3.0-1<br></div>
The server's version is glusterfs 3.3.0.5rhs<br>
<br></div> I remember that 3.3 could suport automatic self-heal in the first lookup, when calling "cp", it should trigger the self-heal to get the lastest file, but why not?<br><br></div> Any comments? I could try provide enough information what you need.<br>
</div>
</blockquote></div><br></div></div>