<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">hi,</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">There are bugs in replication. They exist in 2.0.x, and I tried 3.0.3, it also exists, so I report the bugs here with my solution.</font></font></font></span></p>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">settings:<span style="mso-tab-count: 1"> </span>2 server(server1 and server2), 1 client with replication</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">1)</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">client: </font></font></font></span><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir1]# pwd</font></font></font></span></div>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">/mnt/gluster/dir1</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir1]# ls</font></font></font></span></p>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">ddd</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 1"> </span>then kill server2;</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">client:</font></font></font></span><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 1"> </span>[root@centos dir1]# rm -rf ddd/</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 1"> </span><span style="mso-spacerun: yes"> </span>start server2;</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">client: [root@centos dir1]# ls</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">nothing.</font></font></font></span><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"> </font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">althrough the directory is empty in client. But ddd still exists in server2:</font></font></font></span></div>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">server2:</font></font></font></span><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir1]# ls</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">ddd</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"> </font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">If client continue to do lookup, mkdir or other operations on ddd in dir1, sometimes there would be segment fault or the connection between client and server would be lost. Not every time, but do sometimes.</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"> </font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">If kill server1 instead of server2, then client will recreate ddd in server2!</font></font></font></span></p>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"> </font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3">trace at client:</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3">————————————————————————————————————————————————————</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font color="#000000">[2010-03-26 17:18:35] N [trace.c:1397:trace_rmdir] trace2: 504: (loc {path=/dir1/ddd, ino=7209308})<br>
[2010-03-26 17:18:35] N [trace.c:833:trace_rmdir_cbk] trace2: 504: (op_ret=0, *prebuf = {st_ino=3604666, st_mode=40755, st_nlink=2, st_uid=0, st_gid=0, st_size=4096, st_blocks=16, st_atime=[Mar 26 16:16:39], st_mtime=[Mar 26 17:19:56], st_ctime=[Mar 26 17:19:56]}, *postbuf = {(null)}<br>
[2010-03-26 17:18:35] N [trace.c:1247:trace_xattrop] trace2: 504: (path=/dir1, ino=7209322 flags=0)<br>[2010-03-26 17:18:35] N [trace.c:1128:trace_xattrop_cbk] trace2: 504: (op_ret=0, op_errno=0)<br>[2010-03-26 17:18:35] N [trace.c:1176:trace_entrylk] trace2: 504: volume=afr, (loc= {path=/dir1, ino=7209322} basename=ddd, cmd=ENTRYLK_UNLOCK, type=ENTRYLK_WRLCK)<br>
[2010-03-26 17:18:35] N [trace.c:1113:trace_entrylk_cbk] trace2: 504: op_ret=0, op_errno=0<br>[2010-03-26 17:18:46] D [client-protocol.c:7041:notify] remote1: got GF_EVENT_CHILD_UP<br>[2010-03-26 17:18:46] D [client-protocol.c:7041:notify] remote1: got GF_EVENT_CHILD_UP<br>
[2010-03-26 17:18:46] N [client-protocol.c:6246:client_setvolume_cbk] remote1: Connected to <a href="http://192.168.0.182:7001">192.168.0.182:7001</a>, attached to remote volume 'trace'.<br>[2010-03-26 17:18:46] N [client-protocol.c:6246:client_setvolume_cbk] remote1: Connected to <a href="http://192.168.0.182:7001">192.168.0.182:7001</a>, attached to remote volume 'trace'.<br>
[2010-03-26 17:19:03] N [trace.c:1769:trace_opendir] trace1: 507:( loc {path=/dir1, ino=7209322}, fd=0x958b0f8)<br>[2010-03-26 17:19:03] N [trace.c:1769:trace_opendir] trace2: 507:( loc {path=/dir1, ino=7209322}, fd=0x958b0f8)<br>
[2010-03-26 17:19:03] N [trace.c:808:trace_opendir_cbk] trace2: 507: (op_ret=0, op_errno=22, fd=0x958b0f8)<br>[2010-03-26 17:19:03] N [trace.c:808:trace_opendir_cbk] trace1: 507: (op_ret=0, op_errno=22, fd=0x958b0f8)</font><br>
</span></div>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">From trace, we can see when user was still in dir1 which the "rm" operation took place in, "ls" wouldn’t call "lookup" but call "opendir" straightly. So the changelogs store in dir1 didn’t be noticed.</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"> </font></span></p>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">My solution: Every time the “opendir” is called, add another “lookup” call to lookup the directory which is opendir by the “opendir”.</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 39pt; TEXT-INDENT: 0cm; mso-char-indent-count: 0"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"></font></span></p>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">2)</font></font></font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">client:<span style="mso-tab-count: 1"> </span>[root@centos dir2]# pwd</font></font></font></span></div>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">/mnt/gluster/dir2</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir2]# ll ww/</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt; mso-para-margin-left: 4.0gd"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">-rw-r--r-- 1 root root 242 2010-03-26 file</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">kill server2:</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">client:<span style="mso-tab-count: 1"> </span>[root@centos dir2]# rm -rf ww</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">start server2:</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">client:<span style="mso-tab-count: 1"> </span>[root@centos dir2]# ll ww/</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 42pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">-rw-r--r-- 1 root root 242 2010-03-26 file</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font face="Calibri" color="#000000" size="3"> </font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">In my opinion, in the function afr_self_heal():</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 1"> </span>if (local->success_count && local->enoent_count) {</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 2"> </span>afr_self_heal_missing_entries (frame, this);</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 1"> </span>} else {</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 2"> </span>gf_log (this->name, GF_LOG_TRACE,</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 3"> </span>"proceeding to metadata check on %s",</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 3"> </span>local->loc.path);</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt 21pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000"><span style="mso-tab-count: 2"> </span>afr_sh_missing_entries_done (frame, this);</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">the judge is too simple.</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US" style="COLOR: red"><font size="3"><font face="Calibri"><font color="#000000">My solution: if (local->success_count && local->enoent_count) is ture, add some call to recursively lookup the common parent directory which exists in each server. Anyway, there must be one, for example the mount point. Lookup the first common parent directory, then let self-heal decide should whether delete or create.</font></font></font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3"> </font></span></p>
<p class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3"> </font></span></p>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3">Here’s my modified replicate in glusterfs-2.0.8 in the attachment:</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3">format:</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3">#ifndef AFR_ENTRY</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3"> My codes</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3">#else</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3"> original codes</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3">#endif</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3"></font></span> </div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3">Besides, in the attachment, I use Hexiaobin’s code about recursive delete in Afr-self-heal-entry.c which he may report it earlier.</font></span></div>
<div class="MsoListParagraph" style="MARGIN: 0cm 0cm 0pt"><span lang="EN-US"><font face="Calibri" size="3"></font></span> </div>