<div dir="ltr"><div>I have a replicated Gluster setup, 2 servers (fs-1 and fs-2) x 1 brick.  I have two clients (web-1 and web-2) which are connected and simultaneously execute tasks.  These clients mount the Gluster volume at /mnt/gfs.  One task they execute looks like this (note this is pseudocode, the actual task is PHP):<br>
</div><div class="gmail_quote"><div dir="ltr"><div>


<br></div><div>1. @symlink(/mnt/gfs/<span style="white-space:pre-wrap">slow265</span>, /mnt/gfs/<span style="white-space:pre-wrap">slow265.prod</span>);<br></div><div>2. if (!is_link(/mnt/gfs/<span style="white-space:pre-wrap">slow265.prod</span>)) {<br>



</div><div>3.   throw Exception;</div><div>4. }</div><div>5. symlink(/mnt/gfs/<span style="white-space:pre-wrap">slow265.prod</span>, /home/user/<span style="white-space:pre-wrap">slow265.prod</span>)<br>
</div><div><br></div><div>Note that line 1 may fail on either client because the link may have been created by the other client, but this is suppressed, the link is checked and an exception is thrown if the link does not exist.  These two tasks, when executed at the same time, usually succeed.  However, in a recent run, we saw an error on web-1 in line 5 because the local filesystem symlink creation failed, despite line 2 confirming that the target Gluster symlink existed.</div>


<div><br></div><div>I&#39;ve created a PHP script which can be run simultaneously on two clients to recreate the error: <a href="https://gist.github.com/pdrakeweb/7347198" target="_blank">https://gist.github.com/pdrakeweb/7347198</a><br>


</div><div><br></div><div>Running the same test script on a Gluster 3.0.8 setup does not cause the error to occur.  Running the same test on a local-only filesystem also does not cause the error to occur.  I&#39;d appreciate any insight people might have into what is going on here and whether this is a bug in 3.4.1.  Below are the related log entries from my Gluster servers and clients.</div>

<div><br></div><div>Entries from web-1&#39;s mnt-gfs.log file:</div><div><pre style="white-space:pre-wrap;word-wrap:break-word">[2013-11-05 05:25:24.686506] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-test-fs-cluster-1-client-1: remote operation failed: File exists. Path: /slow265.prod (00000000-0000-0000-0000-000000000000)
[2013-11-05 05:25:24.686584] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-test-fs-cluster-1-client-0: remote operation failed: File exists. Path: /slow265.prod (00000000-0000-0000-0000-000000000000)
[2013-11-05 05:25:24.686649] E [dht-helper.c:1052:dht_inode_ctx_get] (--&gt;/usr/lib/x86_64-linux-gnu/glusterfs/3.4git/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x75) [0x7f5e03dd4ff5] (--&gt;/usr/lib/x86_64-linux-gnu/glusterfs/3.4git/xlator/cluster/distribute.so(dht_layout_preset+0x59) [0x7f5e03dc1c89] (--&gt;/usr/lib/x86_64-linux-gnu/glusterfs/3.4git/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34) [0x7f5e03dc3b34]))) 0-test-fs-cluster-1-dht: invalid argument: inode
[2013-11-05 05:25:24.686687] E [dht-helper.c:1071:dht_inode_ctx_set] (--&gt;/usr/lib/x86_64-linux-gnu/glusterfs/3.4git/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x75) [0x7f5e03dd4ff5] (--&gt;/usr/lib/x86_64-linux-gnu/glusterfs/3.4git/xlator/cluster/distribute.so(dht_layout_preset+0x59) [0x7f5e03dc1c89] (--&gt;/usr/lib/x86_64-linux-gnu/glusterfs/3.4git/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x52) [0x7f5e03dc3b52]))) 0-test-fs-cluster-1-dht: invalid argument: inode
[2013-11-05 05:25:24.689670] W [fuse-bridge.c:1311:fuse_readlink_cbk] 0-glusterfs-fuse: 1736: /slow265.prod =&gt; -1 (Invalid argument)</pre><pre style="white-space:pre-wrap;word-wrap:break-word"><br></pre><div>Entries from web-2&#39;s mnt-gfs.log file:</div>

<div><pre style="white-space:pre-wrap;word-wrap:break-word"><pre style="word-wrap:break-word;white-space:pre-wrap">[2013-11-05 05:25:26.164593] W [client-rpc-fops.c:2469:client3_3_link_cbk] 0-test-fs-cluster-1-client-1: remote operation failed: File exists (00000000-0000-0000-0000-000000000000 -&gt; /slow265.prod)</pre>

</pre><pre style="white-space:pre-wrap;word-wrap:break-word">[2013-11-05 05:25:26.210652] W [client-rpc-fops.c:2469:client3_3_link_cbk] 0-test-fs-cluster-1-client-0: remote operation failed: File exists (00000000-0000-0000-0000-000000000000 -&gt; /slow265.prod)
</pre><pre style="white-space:pre-wrap;word-wrap:break-word"><br></pre><pre style="white-space:pre-wrap;word-wrap:break-word">Entries from fs-1&#39;s brick.log:</pre><pre style="white-space:pre-wrap;word-wrap:break-word">
<pre style="word-wrap:break-word;white-space:pre-wrap">[2013-11-05 05:25:24.832262] I [server-rpc-fops.c:575:server_mknod_cbk] 0-test-fs-cluster-1-server: 3337: MKNOD (null) (00000000-0000-0000-0000-000000000001/slow265.prod) ==&gt; (File exists)
<p>[2013-11-05 05:25:26.391611] I [server-rpc-fops.c:1211:server_link_cbk] 0-test-fs-cluster-1-server: 3301: LINK /slow265.prod (3658314e-7730-4771-8ac3-2d6fb20b1b13) -&gt; 00000000-0000-0000-0000-000000000001/slow265.prod ==&gt; (File exists)</p>



Entries from fs-2&#39;s brick.log:</pre><pre style="word-wrap:break-word;white-space:pre-wrap"><pre style="word-wrap:break-word;white-space:pre-wrap">[2013-11-05 05:25:24.554824] I [server-rpc-fops.c:575:server_mknod_cbk] 0-test-fs-cluster-1-server: 3290: MKNOD (null) (00000000-0000-0000-0000-000000000001/slow265.prod) ==&gt; (File exists)
<pre style="word-wrap:break-word;white-space:pre-wrap">[2013-11-05 05:25:26.160204] I [server-rpc-fops.c:1211:server_link_cbk] 0-test-fs-cluster-1-server: 3341: LINK /slow265.prod (3658314e-7730-4771-8ac3-2d6fb20b1b13) -&gt; 00000000-0000-0000-0000-000000000001/slow265.prod ==&gt; (File exists)</pre>

</pre></pre></pre></div>-- <br><div dir="ltr"><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"><b>Peter Drake </b>|<b> Cloud Software Engineer</b> |  </span><a href="http://acquia.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-family:Arial;color:rgb(6,124,187);background-color:transparent;font-weight:bold;vertical-align:baseline">Acquia</span></a><br>



<div><span style="font-family:Arial;color:rgb(6,124,187);background-color:transparent;font-weight:bold;vertical-align:baseline"></span><br><font face="arial, helvetica, sans-serif"><span style="vertical-align:baseline;background-color:transparent;font-weight:bold">O: </span><a href="tel:781.238.4236" value="+17812384236" target="_blank">781.238.4236</a></font></div>



<p dir="ltr" style="margin-top:0pt;margin-bottom:0pt"><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"><b>E:</b><font color="#1155cc"> </font><font color="#000000"><a href="mailto:peter.drake@acquia.com" target="_blank">peter.drake@acquia.com</a></font></span><span style="font-family:Arial;color:rgb(6,124,187);background-color:transparent;vertical-align:baseline">  </span><span style="vertical-align:baseline;background-color:transparent;font-family:Arial">|  Skype: pdrakeweb<br>



</span></p><p dir="ltr" style="margin-top:0pt;margin-bottom:0pt"><span style="vertical-align:baseline;background-color:transparent;font-family:Arial;font-weight:bold">W:</span><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"> <a href="http://www.acquia.com" target="_blank">http://www.acquia.com</a></span><a href="http://www.acquia.com/" style="color:rgb(17,85,204)" target="_blank"><span style="font-family:Arial;color:rgb(6,124,187);background-color:transparent;vertical-align:baseline"></span></a></p>



<p dir="ltr" style="margin-top:0pt;margin-bottom:0pt"><span style="vertical-align:baseline;background-color:transparent;font-family:Arial;font-weight:bold">Address:</span><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"> 25 Corporate Drive 4th Floor, Burlington, MA 01803</span></p>



<div><div><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"></span><br></div><p dir="ltr" style="margin-top:0pt;margin-bottom:0pt"><a href="http://www.acquia.com/about-us/newsroom/press-releases/inc-magazine-unveils-31st-annual-list-america-s-fastest-growing" style="color:rgb(17,85,204)" target="_blank"><span style="font-family:Arial;background-color:transparent;font-weight:bold;vertical-align:baseline">Acquia</span><span style="font-family:Arial;background-color:transparent;vertical-align:baseline"> ranked #1 Software Vendor on the 2012 Inc 500</span></a><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"></span></p>



</div><p dir="ltr" style="margin-top:0pt;margin-bottom:0pt"><a href="http://www.acquia.com/about-us/newsroom/press-releases/acquia-named-forbes-100-list-america-s-most-promising-companies-nov" style="color:rgb(17,85,204)" target="_blank"><span style="font-family:Arial;background-color:transparent;font-weight:bold;vertical-align:baseline">Acquia</span><span style="font-family:Arial;background-color:transparent;vertical-align:baseline"> named One of America’s Most Promising Companies by Forbes</span></a><span style="font-family:Arial;color:rgb(6,124,187);background-color:transparent;text-decoration:underline;vertical-align:baseline"></span></p>



<p dir="ltr" style="margin-top:0pt;margin-bottom:0pt"><span style="vertical-align:baseline;background-color:transparent;font-family:Arial;font-weight:bold">Drupal Sites:</span><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"> </span><span style="font-family:Arial;background-color:transparent;vertical-align:baseline"><a href="http://drupalshowcase.com" target="_blank">http://drupalshowcase.com</a></span><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"></span></p>



<div><span style="vertical-align:baseline;background-color:transparent;font-family:Arial;font-weight:bold">Twitter</span><span style="vertical-align:baseline;background-color:transparent;font-family:Arial"> <a href="http://www.twitter.com/Acquia" target="_blank">http://www.twitter.com/Acquia</a></span></div>
</div></div></div></div>
</div>