There is a problem in the format of the former attachment. I present a new one here,<br><br><div class="gmail_quote">2010/3/26 G Feng <span dir="ltr"><<a href="mailto:mgfeng@gmail.com">mgfeng@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">hi,</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">There are bugs in replication. They exist in 2.0.x, and I tried 3.0.3, it also exists, so I report the bugs here with my solution.</font></font></font></span></p>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">settings:<span> </span>2 server(server1 and server2), 1 client with replication</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">1)</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">client: </font></font></font></span><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir1]# pwd</font></font></font></span></div>
<p style="margin: 0cm 0cm 0pt 39pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">/mnt/gluster/dir1</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 39pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir1]# ls</font></font></font></span></p>
<div style="margin: 0cm 0cm 0pt 39pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">ddd</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt 39pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>then kill server2;</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">client:</font></font></font></span><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>[root@centos dir1]# rm -rf ddd/</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span><span> </span>start server2;</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">client: [root@centos dir1]# ls</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">nothing.</font></font></font></span><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"> </font></span></div>
<div style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"></font></span> </div>
<div style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">althrough the directory is empty in client. But ddd still exists in server2:</font></font></font></span></div>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">server2:</font></font></font></span><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir1]# ls</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">ddd</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"> </font></span></p>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">If client continue to do lookup, mkdir or other operations on ddd in dir1, sometimes there would be segment fault or the connection between client and server would be lost. Not every time, but do sometimes.</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"> </font></span></p>
<p style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">If kill server1 instead of server2, then client will recreate ddd in server2!</font></font></font></span></p>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"> </font></span></div>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3">trace at client:</font></span></div>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3">————————————————————————————————————————————————————</font></span></div>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000">[2010-03-26 17:18:35] N [trace.c:1397:trace_rmdir] trace2: 504: (loc {path=/dir1/ddd, ino=7209308})<br>
[2010-03-26 17:18:35] N [trace.c:833:trace_rmdir_cbk] trace2: 504: (op_ret=0, *prebuf = {st_ino=3604666, st_mode=40755, st_nlink=2, st_uid=0, st_gid=0, st_size=4096, st_blocks=16, st_atime=[Mar 26 16:16:39], st_mtime=[Mar 26 17:19:56], st_ctime=[Mar 26 17:19:56]}, *postbuf = {(null)}<br>
[2010-03-26 17:18:35] N [trace.c:1247:trace_xattrop] trace2: 504: (path=/dir1, ino=7209322 flags=0)<br>[2010-03-26 17:18:35] N [trace.c:1128:trace_xattrop_cbk] trace2: 504: (op_ret=0, op_errno=0)<br>[2010-03-26 17:18:35] N [trace.c:1176:trace_entrylk] trace2: 504: volume=afr, (loc= {path=/dir1, ino=7209322} basename=ddd, cmd=ENTRYLK_UNLOCK, type=ENTRYLK_WRLCK)<br>
[2010-03-26 17:18:35] N [trace.c:1113:trace_entrylk_cbk] trace2: 504: op_ret=0, op_errno=0<br>[2010-03-26 17:18:46] D [client-protocol.c:7041:notify] remote1: got GF_EVENT_CHILD_UP<br>[2010-03-26 17:18:46] D [client-protocol.c:7041:notify] remote1: got GF_EVENT_CHILD_UP<br>
[2010-03-26 17:18:46] N [client-protocol.c:6246:client_setvolume_cbk] remote1: Connected to <a href="http://192.168.0.182:7001" target="_blank">192.168.0.182:7001</a>, attached to remote volume 'trace'.<br>[2010-03-26 17:18:46] N [client-protocol.c:6246:client_setvolume_cbk] remote1: Connected to <a href="http://192.168.0.182:7001" target="_blank">192.168.0.182:7001</a>, attached to remote volume 'trace'.<br>
[2010-03-26 17:19:03] N [trace.c:1769:trace_opendir] trace1: 507:( loc {path=/dir1, ino=7209322}, fd=0x958b0f8)<br>[2010-03-26 17:19:03] N [trace.c:1769:trace_opendir] trace2: 507:( loc {path=/dir1, ino=7209322}, fd=0x958b0f8)<br>
[2010-03-26 17:19:03] N [trace.c:808:trace_opendir_cbk] trace2: 507: (op_ret=0, op_errno=22, fd=0x958b0f8)<br>[2010-03-26 17:19:03] N [trace.c:808:trace_opendir_cbk] trace1: 507: (op_ret=0, op_errno=22, fd=0x958b0f8)</font><br>
</span></div>
<p style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">From trace, we can see when user was still in dir1 which the "rm" operation took place in, "ls" wouldn’t call "lookup" but call "opendir" straightly. So the changelogs store in dir1 didn’t be noticed.</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"> </font></span></p>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">My solution: Every time the “opendir” is called, add another “lookup” call to lookup the directory which is opendir by the “opendir”.</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"></font></span> </div>
<div style="margin: 0cm 0cm 0pt 39pt; text-indent: 0cm;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"></font></font></font></span> </div>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"></font></span></p>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">2)</font></font></font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">client:<span> </span>[root@centos dir2]# pwd</font></font></font></span></div>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">/mnt/gluster/dir2</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">[root@centos dir2]# ll ww/</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">-rw-r--r-- 1 root root 242 2010-03-26 file</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">kill server2:</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">client:<span> </span>[root@centos dir2]# rm -rf ww</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">start server2:</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">client:<span> </span>[root@centos dir2]# ll ww/</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 42pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">-rw-r--r-- 1 root root 242 2010-03-26 file</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font color="#000000" face="Calibri" size="3"> </font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">In my opinion, in the function afr_self_heal():</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>if (local->success_count && local->enoent_count) {</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>afr_self_heal_missing_entries (frame, this);</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>} else {</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>gf_log (this->name, GF_LOG_TRACE,</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>"proceeding to metadata check on %s",</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>local->loc.path);</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt 21pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000"><span> </span>afr_sh_missing_entries_done (frame, this);</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">the judge is too simple.</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: red;" lang="EN-US"><font size="3"><font face="Calibri"><font color="#000000">My solution: if (local->success_count && local->enoent_count) is ture, add some call to recursively lookup the common parent directory which exists in each server. Anyway, there must be one, for example the mount point. Lookup the first common parent directory, then let self-heal decide should whether delete or create.</font></font></font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3"> </font></span></p>
<p style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3"> </font></span></p>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3">Here’s my modified replicate in glusterfs-2.0.8 in the attachment:</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3">format:</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3">#ifndef AFR_ENTRY</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3"> My codes</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3">#else</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3"> original codes</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3">#endif</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3"></font></span> </div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3">Besides, in the attachment, I use Hexiaobin’s code about recursive delete in Afr-self-heal-entry.c which he may report it earlier.</font></span></div>
<div style="margin: 0cm 0cm 0pt;"><span lang="EN-US"><font face="Calibri" size="3"></font></span> </div>
</blockquote></div><br>