<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<font face="Arial" color="Gray" size="1"></font><br>
<div>
<div class="Section1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ll try to make myself clearer than in the earlier thread, since I need some help here. I’m not sure if I am missing something.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am not able to use atomic operations using glusterfs 3.0.2, on a Arch Linux 2.6.32.7-1 x86_64 server.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve stripped everything down to the most simple configuration<o:p></o:p></p>
<p class="MsoNormal">Server: a posix store with locks<o:p></o:p></p>
<p class="MsoNormal">Client: a tcp client<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I mount the glusterfs on /gfs, and use a “writer” and a “reader” as below:<o:p></o:p></p>
<p class="MsoNormal">[writer] root@frustration> while :; do date > /gfs/.file; mv /gfs/.file /gfs/file; done<o:p></o:p></p>
<p class="MsoNormal">[reader] root@frustration> while :; do cat /gfs/file > /dev/null; done<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Leaving this running, I get a flood of<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in">cat: /gfs/file: No such file or directory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This should not happen on a filesystem that supports atomic operations?!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What this means is that I am unable to replace a file as an atomic operation, creating a race-condition where the file is sometimes not present. Furthermore this actually “desyncs” the server in a more complex setup, requiring a restart
of the server part to make “desynced” files available again.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This reproduces using XFS as well as EXT2 as the underlying fs.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[glusterfs.vol]<o:p></o:p></p>
<p class="MsoNormal">volume media0<o:p></o:p></p>
<p class="MsoNormal"> type protocol/client<o:p></o:p></p>
<p class="MsoNormal"> option transport-type tcp<o:p></o:p></p>
<p class="MsoNormal"> option remote-host 127.0.0.1<o:p></o:p></p>
<p class="MsoNormal"> option remote-subvolume media0<o:p></o:p></p>
<p class="MsoNormal">end-volume<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">mounted with “glusterfs -f glusterfs.vol /gfs”<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[glusterfsd.vol]<o:p></o:p></p>
<p class="MsoNormal">(se below debug log)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-x-<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file glusterfsd.vol<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/features/locks.so: undefined symbol: notify -- neglecting<o:p></o:p></p>
<p class="MsoNormal">================================================================================<o:p></o:p></p>
<p class="MsoNormal">Version : glusterfs 3.0.2 built on Feb 8 2010 15:23:34<o:p></o:p></p>
<p class="MsoNormal">git: v3.0.2<o:p></o:p></p>
<p class="MsoNormal">Starting Time: 2010-02-09 18:18:23<o:p></o:p></p>
<p class="MsoNormal">Command line : glusterfs -f glusterfsd.vol --debug <o:p></o:p></p>
<p class="MsoNormal">PID : 19486<o:p></o:p></p>
<p class="MsoNormal">System name : Linux<o:p></o:p></p>
<p class="MsoNormal">Nodename : n100.p1.x0.local<o:p></o:p></p>
<p class="MsoNormal">Kernel Release : 2.6.32-ARCH<o:p></o:p></p>
<p class="MsoNormal">Hardware Identifier: x86_64<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Given volfile:<o:p></o:p></p>
<p class="MsoNormal">+------------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal"> 1: volume media0-store<o:p></o:p></p>
<p class="MsoNormal"> 2: type storage/posix<o:p></o:p></p>
<p class="MsoNormal"> 3: option directory /mnt<o:p></o:p></p>
<p class="MsoNormal"> 4: end-volume<o:p></o:p></p>
<p class="MsoNormal"> 5: <o:p></o:p></p>
<p class="MsoNormal"> 6: volume media0<o:p></o:p></p>
<p class="MsoNormal"> 7: type features/locks<o:p></o:p></p>
<p class="MsoNormal"> 8: subvolumes media0-store<o:p></o:p></p>
<p class="MsoNormal"> 9: end-volume<o:p></o:p></p>
<p class="MsoNormal"> 10: <o:p></o:p></p>
<p class="MsoNormal"> 11: volume server<o:p></o:p></p>
<p class="MsoNormal"> 12: type protocol/server<o:p></o:p></p>
<p class="MsoNormal"> 13: option transport-type tcp<o:p></o:p></p>
<p class="MsoNormal"> 14: option auth.addr.media0.allow *<o:p></o:p></p>
<p class="MsoNormal"> 15: subvolumes media0<o:p></o:p></p>
<p class="MsoNormal"> 16: end-volume<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">+------------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [glusterfsd.c:1370:main] glusterfs: running in pid 19486<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:23] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:23] N [glusterfsd.c:1396:main] glusterfs: Successfully started<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:43] D [addr.c:190:gf_auth] media0: allowed = "*", received addr = "127.0.0.1"<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:43] N [server-protocol.c:5812:mop_setvolume] server: accepted client from 127.0.0.1:1023<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:43] D [addr.c:190:gf_auth] media0: allowed = "*", received addr = "127.0.0.1"<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:43] N [server-protocol.c:5812:mop_setvolume] server: accepted client from 127.0.0.1:1022<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [server-resolve.c:238:resolve_path_deep] media0: RESOLVE RENAME() seeking deep resolution of /file<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f79b<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7b3<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7cb<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f79b<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7b3<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:18:58] D [dict.c:303:dict_get] dict: @this=(nil) @key=0x7f2a38a9f7cb<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:19:15] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502725<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:19:34] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502734<o:p></o:p></p>
<p class="MsoNormal">[2010-02-09 18:19:56] D [inode.c:916:inode_path] media0/inode: no dentry for non-root inode 502734<o:p></o:p></p>
<p class="MsoNormal">[…] repeated many times. Frequency of the “no dentry” entries depends on how aggressive the reader is.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>