I'm getting the below error message on some of my gluster clients when under high I/O load:<div><br></div><div><div>009-02-23 12:43:50 E [unify.c:362:unify_lookup_cbk] unify: child(dht0): path(/bio/data/fast-hmmsearch-all/tmp5bgb6I_fast-hmmsearch-all_job/result.tigrfam.TIGR03461.hmmhits): No such file or directory</div>
<div>2009-02-23 12:43:50 E [unify.c:362:unify_lookup_cbk] unify: child(unify-switch-ns): path(/bio/data/fast-hmmsearch-all/tmp5bgb6I_fast-hmmsearch-all_job/result.tigrfam.TIGR03461.hmmhits): No such file or directory</div>
<div>2009-02-23 12:43:50 E [unify.c:362:unify_lookup_cbk] unify: child(unify-switch-ns): path(/bio/data/fast-hmmsearch-all/tmp5bgb6I_fast-hmmsearch-all_job/result.tigrfam.TIGR03461.hmmhits): No such file or directory</div>
<div><br></div><div>The thing is, the above files DO exist. But according to the nodes this happens on, they do not. There are no error messages in the gluster servers. Below is my client config. Any thoughts?</div><br>
</div><div><br></div><div><div>volume unify-switch-ns</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.51</div><div> option remote-subvolume posix-unify-switch-ns</div>
<div>end-volume</div><div><br></div><div>#volume distfs01-ns-readahead</div><div># type performance/read-ahead</div><div># option page-size 1MB</div><div># option page-count 8</div><div># subvolumes distfs01-ns-brick</div>
<div>#end-volume</div><div><br></div><div>#volume unify-switch-ns</div><div># type performance/write-behind</div><div># option block-size 1MB</div><div># option cache-size 3MB</div><div># subvolumes distfs01-ns-readahead</div>
<div>#end-volume</div><div><br></div><div>volume distfs01-unify</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.51</div><div> option remote-subvolume posix-unify</div>
<div>end-volume</div><div><br></div><div>volume distfs02-unify</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.52</div><div> option remote-subvolume posix-unify</div>
<div>end-volume</div><div><br></div><div>volume distfs03-unify</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.53</div><div> option remote-subvolume posix-unify</div>
<div>end-volume</div><div><br></div><div>volume distfs04-unify</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.54</div><div> option remote-subvolume posix-unify</div>
<div>end-volume</div><div><br></div><div>volume distfs01-stripe</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.51</div><div> option remote-subvolume posix-stripe</div>
<div>end-volume</div><div><br></div><div>volume distfs02-stripe</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.52</div><div> option remote-subvolume posix-stripe</div>
<div>end-volume</div><div><br></div><div>volume distfs03-stripe</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.53</div><div> option remote-subvolume posix-stripe</div>
<div>end-volume</div><div><br></div><div>volume distfs04-stripe</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host 10.8.101.54</div><div> option remote-subvolume posix-stripe</div>
<div>end-volume</div><div><br></div><div>volume stripe0</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>type cluster/stripe</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>option block-size *.jar,*.pin:1MB,*:2MB</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>subvolumes distfs01-stripe distfs02-stripe distfs03-stripe distfs04-stripe</div><div>end-volume</div><div><br></div><div>volume dht0</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>type cluster/dht</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>subvolumes distfs01-unify distfs02-unify distfs03-unify distfs04-unify</div><div>end-volume</div><div><br></div><div>volume unify</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>type cluster/unify</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>option namespace unify-switch-ns</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>option self-heal off</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>option scheduler switch</div>
<div># send *.phr/psq/pnd etc to stripe0, send the rest to hash</div><div># extensions have to be *.foo* and not simply *.foo or rsync's tmp file naming will prevent files from being matched</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>option scheduler.switch.case *.phr*:stripe0;*.psq*:stripe0;*.pnd*:stripe0;*.psd*:stripe0;*.pin*:stripe0;*.nsi*:stripe0;*.nin*:stripe0;*.nsd*:stripe0;*.nhr*:stripe0;*.nsq*:stripe0;*.tar*:stripe0;*.tar.gz*:stripe0;*.jar*:stripe0;*.img*:stripe0;*.perf*:stripe0;*.tgz*:stripe0;*.fasta*:stripe0;*.huge*:stripe0</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>subvolumes stripe0 dht0</div><div>end-volume</div><div><br></div><div>volume ioc</div><div> type performance/io-cache</div><div> subvolumes unify</div>
<div> option cache-size 3000MB</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>option cache-timeout 3600</div><div>end-volume</div><div><br></div><div>volume filter</div><div> type features/filter</div>
<div> option fixed-uid 0</div><div> option fixed-gid 900</div><div> subvolumes ioc</div><div>end-volume</div><div><br></div></div>