Haha, I love how you named your server! I think a lot of us feel the same way :)<br><br><div class="gmail_quote">On Tue, Feb 9, 2010 at 11:18 AM, Fredrik Widlund <span dir="ltr"><<a href="mailto:fredrik.widlund@qbrick.com">fredrik.widlund@qbrick.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="purple">
<font face="Arial" color="Gray" size="1"></font><br>
<div>
<div>
<p class="MsoNormal">Hi,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I’ll try to make myself clearer than in the earlier thread, since I need some help here. I’m not sure if I am missing something.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I am not able to use atomic operations using glusterfs 3.0.2, on a Arch Linux 2.6.32.7-1 x86_64 server.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I’ve stripped everything down to the most simple configuration</p>
<p class="MsoNormal">Server: a posix store with locks</p>
<p class="MsoNormal">Client: a tcp client</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I mount the glusterfs on /gfs, and use a “writer” and a “reader” as below:</p>
<p class="MsoNormal">[writer] root@frustration> while :; do date > /gfs/.file; mv /gfs/.file /gfs/file; done</p>
<p class="MsoNormal">[reader] root@frustration> while :; do cat /gfs/file > /dev/null; done</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Leaving this running, I get a flood of</p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in">cat: /gfs/file: No such file or directory</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">This should not happen on a filesystem that supports atomic operations?!</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">What this means is that I am unable to replace a file as an atomic operation, creating a race-condition where the file is sometimes not present. Furthermore this actually “desyncs” the server in a more complex setup, requiring a restart
of the server part to make “desynced” files available again.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">This reproduces using XFS as well as EXT2 as the underlying fs.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">[glusterfs.vol]</p>
<p class="MsoNormal">volume media0</p>
<p class="MsoNormal"> type protocol/client</p>
<p class="MsoNormal"> option transport-type tcp</p>
<p class="MsoNormal"> option remote-host 127.0.0.1</p>
<p class="MsoNormal"> option remote-subvolume media0</p>
<p class="MsoNormal">end-volume</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">mounted with “glusterfs -f glusterfs.vol /gfs”</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">[glusterfsd.vol]</p>
<p class="MsoNormal">(se below debug log)</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">-x-</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file glusterfsd.vol</p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/features/locks.so: undefined symbol: notify -- neglecting</p>
<p class="MsoNormal">================================================================================</p>
<p class="MsoNormal">Version : glusterfs 3.0.2 built on Feb 8 2010 15:23:34</p>
<p class="MsoNormal">git: v3.0.2</p>
<p class="MsoNormal">Starting Time: 2010-02-09 18:18:23</p>
<p class="MsoNormal">Command line : glusterfs -f glusterfsd.vol --debug </p>
<p class="MsoNormal">PID : 19486</p>
<p class="MsoNormal">System name : Linux</p>
<p class="MsoNormal">Nodename : n100.p1.x0.local</p>
<p class="MsoNormal">Kernel Release : 2.6.32-ARCH</p>
<p class="MsoNormal">Hardware Identifier: x86_64</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Given volfile:</p>
<p class="MsoNormal">+------------------------------------------------------------------------------+</p>
<p class="MsoNormal"> 1: volume media0-store</p>
<p class="MsoNormal"> 2: type storage/posix</p>
<p class="MsoNormal"> 3: option directory /mnt</p>
<p class="MsoNormal"> 4: end-volume</p>
<p class="MsoNormal"> 5: </p>
<p class="MsoNormal"> 6: volume media0</p>
<p class="MsoNormal"> 7: type features/locks</p>
<p class="MsoNormal"> 8: subvolumes media0-store</p>
<p class="MsoNormal"> 9: end-volume</p>
<p class="MsoNormal"> 10: </p>
<p class="MsoNormal"> 11: volume server</p>
<p class="MsoNormal"> 12: type protocol/server</p>
<p class="MsoNormal"> 13: option transport-type tcp</p>
<p class="MsoNormal"> 14: option auth.addr.media0.allow *</p>
<p class="MsoNormal"> 15: subvolumes media0</p>
<p class="MsoNormal"> 16: end-volume</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">+------------------------------------------------------------------------------+</p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [glusterfsd.c:1370:main] glusterfs: running in pid 19486</p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so</p>
<p class="MsoNormal">[2010-02-09 18:18:23] N [glusterfsd.c:1396:main] glusterfs: Successfully started</p>
<p class="MsoNormal">[2010-02-09 18:18:43] D [addr.c:190:gf_auth] media0: allowed = "*", received addr = "127.0.0.1"</p>
<p class="MsoNormal">[2010-02-09 18:18:43] N [server-protocol.c:5812:mop_setvolume] server: accepted client from <a href="http://127.0.0.1:1023" target="_blank">127.0.0.1:1023</a></p>
<p class="MsoNormal">[2010-02-09 18:18:43] D [addr.c:190:gf_auth] media0: allowed = "*", received addr = "127.0.0.1"</p>
<p class="MsoNormal">[2010-02-09 18:18:43] N [server-protocol.c:5812:mop_setvolume] server: accepted client from <a href="http://127.0.0.1:1022" target="_blank">127.0.0.1:1022</a></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [server-resolve.c:238:resolve_path_deep] media0: RESOLVE RENAME() seeking deep resolution of /file</p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f79b</p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7b3</p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7cb</p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f79b</p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7b3</p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7cb</p>
<p class="MsoNormal">[2010-02-09 18:19:15] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502725</p>
<p class="MsoNormal">[2010-02-09 18:19:34] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502734</p>
<p class="MsoNormal">[2010-02-09 18:19:56] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502734</p>
<p class="MsoNormal">[…] repeated many times. Frequency of the “no dentry” entries depends on how aggressive the reader is.</p>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
<br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br>