<br><div class="gmail_quote">On Sat, Aug 18, 2012 at 9:13 PM, Emmanuel Dreyfus <span dir="ltr"><<a href="mailto:manu@netbsd.org" target="_blank">manu@netbsd.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">> I understand this code checks that the layout cover the whole space,<br>
> is that right? Then it must be upset that layout->list[0] does not cover<br>
> anything. Since the error is transcient, I susepct a race condition:<br>
> the layout would be filled after that check. Is it possible? Where is<br>
> the layout crafted?<br>
<br>
</div>I improved my test by completely deleting and re-creating the<br>
volume before adding a brick. Here is what happens when I add a brick:<br>
<br>
1-vndfs-client-1: Connected to <a href="http://192.0.2.103:24027" target="_blank">192.0.2.103:24027</a>, attached to remote<br>
volume '/export/vnd1a'.<br>
1-vndfs-client-1: Server and Client lk-version numbers are not same,<br>
reopening the fds<br>
0-fuse: switched to graph 1<br></blockquote><div><br></div><div>Do you see all the protocol/client's connection succeeding on graph 1 before 'switched to graph 1' log? In the above excerpt you have shown "client-1" succeeding log. What about other protocol clients?<br>
</div><div><br></div><div>Avati</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
1-vndfs-client-1: Server lk version = 1<br>
1-vndfs-dht: missing disk layout on vndfs-client-0. err = -1<br>
1-dht_layout_merge: ==> layout[0] 0 - 0 err -1<br>
1-dht_layout_merge: ==> layout[1] 0 - 0 err 0<br>
1-vndfs-dht: missing disk layout on vndfs-client-1. err = -1<br>
1-dht_layout_merge: ==> layout[0] 0 - 0 err -1<br>
1-dht_layout_merge: ==> layout[1] 0 - 0 err -1<br>
<br>
I am not sure it is expected behavior. The broken layout does not<br>
raise EINVAL to process using the filesystem, but later similar<br>
treatment will.<br>
<br>
After playing a bit I tested the race condition with this patch:<br>
<br>
--- a/xlators/cluster/dht/src/dht-common.c<br>
+++ b/xlators/cluster/dht/src/dht-common.c<br>
@@ -477,6 +477,11 @@ unlock:<br>
ret = dht_layout_normalize (this, &local->loc, layout);<br>
<br>
if (ret != 0) {<br>
+ if (strcmp(local->loc.path, "/") == 0) {<br>
+ gf_log (this->name, GF_LOG_WARNING,<br>
+ "wAit 2s for DHT to settle...");<br>
+ sleep(2);<br>
+ }<br>
gf_log (this->name, GF_LOG_DEBUG,<br>
"fixing assignment on %s",<br>
local->loc.path);<br>
<br>
Here is the kind og log it procudes. I do not always see the EINVAL<br>
in the log, but it is never seen by processes using the filesystem.<br>
At least during the tests I did.<br>
<br>
[2012-08-19 06:04:06.288131] I [fuse-bridge.c:4195:fuse_graph_setup]<br>
0-fuse: switched to graph 1<br>
[2012-08-19 06:04:06.289052] I [client-handshake.c:453:<br>
client_set_lk_version_cbk] 1-vndfs-client-1: Server lk version = 1<br>
[2012-08-19 06:04:06.294234] W [dht-common.c:482:dht_lookup_dir_cbk]<br>
1-vndfs-dht: wait 2s for DHT to settle...<br>
[2012-08-19 06:04:08.306937] I [client.c:2151:notify] 0-vndfs-client-0:<br>
<div class="im"> current graph is no longer active, destroying rpc_client<br>
</div>[2012-08-19 06:04:08.308114] I [client.c:2090:client_rpc_notify]<br>
0-vndfs-client-0: disconnected<br>
[2012-08-19 06:04:08.309833] W [fuse-resolve.c:151:fuse_resolve_gfid_cbk]<br>
0-fuse: 4e4b4110-a585-4aae-b919-b2416355f5d1: failed to resolve<br>
(Invalid argument)<br>
[2012-08-19 06:04:08.310275] E [fuse-bridge.c:353:fuse_lookup_resume]<br>
<div class="im"> 0-fuse: failed to resolve path (null)<br>
<br>
</div>But this probably does not really fix the problem. I got an unreproductible<br>
ENOENT for a directory while copying a hierarchy for instance.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Emmanuel Dreyfus<br>
<a href="mailto:manu@netbsd.org">manu@netbsd.org</a><br>
</font></span></blockquote></div><br>