<div dir="ltr">Niels,<div>This is a good addition. While gluster clients do a reasonably good job at detecting dead/hung servers with ping-timeout, the server side detection has been rather weak. TCP_KEEPALIVE has helped to some extent, for cases where an idling client (which holds a lock) goes dead. However if an active client with pending data in server's socket buffer dies, we have been subject to long tcp retransmission to finish and give up.</div>
<div><br></div><div>The way I see it, this option is complementary to TCP_KEEPALIVE (keepalive works for idle and only idle connections, user_timeout works only when there is pending acknowledgements, thus covering the full spectrum). To that end, it might make sense to present the admin a single timeout configuration value rather than two. It would be very frustrating for the admin to configure one of them to, say, 30 seconds, and then find that the server does not clean up after 30 seconds of a hung client only because the connection was idle (or not idle). Configuring a second timeout for the other case can be very unintuitive.</div>
<div><br></div><div>In fact, I would suggest to have a single network timeout configuration, which gets applied to all the three: ping-timeout on the client, user_timeout on the server, keepalive on both. I think that is what a user would be expecting anyways. Each is for a slightly different technical situation, but all just internal details as far as a user is concerned.</div>
<div><br></div><div>Thoughts?</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, May 20, 2014 at 4:30 AM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
the last few days I've been looking at a problem [1] where a client<br>
locks a file over a FUSE-mount, and a 2nd client tries to grab that lock<br>
too. It is expected that the 2nd client gets blocked until the 1st<br>
client releases the lock. This all work as long as the 1st client<br>
cleanly releases the lock.<br>
<br>
Whenever the 1st client crashes (like a kernel panic) or the network is<br>
split and the 1st client is unreachable, the 2nd client may not get the<br>
lock until the bricks detect that the connection to the 1st client is<br>
dead. If there are pending Replies, the bricks may need 15-20 minutes<br>
until the re-transmissions of the replies have timed-out.<br>
<br>
The current default of 15-20 minutes is quite long for a fail-over<br>
scenario. Relatively recently [2], the Linux kernel got<br>
a TCP_USER_TIMEOUT socket option (similar to TCP_KEEPALIVE). This option<br>
can be used to configure a per-socket timeout, instead of a system-wide<br>
configuration through the net.ipv4.tcp_retries2 sysctl.<br>
<br>
The default network.ping-timeout is set to 42 seconds. I'd like to<br>
propose a network.tcp-timeout option that can be set per volume. This<br>
option should then set TCP_USER_TIMEOUT for the socket, which causes<br>
re-transmission failures to be fatal after the timeout has passed.<br>
<br>
Now the remaining question, what shall be the default timeout in seconds<br>
for this new network.tcp-timeout option? I'm currently thinking of<br>
making it high enough (like 5 minutes) to prevent false positives.<br>
<br>
Thoughts and comments welcome,<br>
Niels<br>
<br>
<br>
1 <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1099460" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1099460</a><br>
2 <a href="http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dca43c7" target="_blank">http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dca43c7</a><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br></div>