Hi all,<div><br></div><div>Finally, I figure out the Gerrit and submit a BUG and patch.</div><div>Hope I had followed a correct procedures, :-)</div><div><br></div><div>Bug description:</div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1144672">https://bugzilla.redhat.com/show_bug.cgi?id=1144672</a></div><div><br></div><div>Patch review request:</div><div><a href="http://review.gluster.org/#/c/8787/">http://review.gluster.org/#/c/8787/</a></div><div><br></div><div>Please review this patch when you are free to do it.</div><div>Thanks very much!</div><div><br></div><div>On Friday, September 19, 2014, Vijay Bellur <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Jaden,<br>
<br>
Thanks for your effort on this one.<br>
<br>
A lot of GlusterFS developers including me are busy with various things over the next 2 weeks. Please do expect some delays in our responses and reviews till then.<br>
<br>
Cheers,<br>
Vijay<br>
<br>
On 09/19/2014 12:44 PM, Jaden Liang wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
Here is a patch for this file flocks uncleanly disconnect issue of<br>
gluster-3.4.5.<br>
I am totally new guy in the gluster development work flow, and still<br>
trying to<br>
understand how to submit this patch to Gerrit. So I want to paste the patch<br>
here first to let devel team know, and submit it after I figure out the<br>
Gerrit :-).<br>
<br>
The major modification is adding an id for different tcp connection<br>
between a<br>
pair client and server to avoid a connection socket not close at the<br>
same time.<br>
<br>
diff --git a/rpc/rpc-lib/src/rpc-clnt.h b/rpc/rpc-lib/src/rpc-clnt.h<br>
index 263d5f7..718308d 100644<br>
--- a/rpc/rpc-lib/src/rpc-clnt.h<br>
+++ b/rpc/rpc-lib/src/rpc-clnt.h<br>
@@ -143,6 +143,7 @@ struct rpc_clnt_connection {<br>
struct timeval last_sent;<br>
struct timeval last_received;<br>
int32_t ping_started;<br>
+ uint32_t clnt_conn_id;<br>
};<br>
typedef struct rpc_clnt_connection rpc_clnt_connection_t;<br>
diff --git a/xlators/protocol/client/src/<u></u>client-handshake.c<br>
b/xlators/protocol/client/src/<u></u>client-handshake.c<br>
index d2083e6..1c2fc2f 100644<br>
--- a/xlators/protocol/client/src/<u></u>client-handshake.c<br>
+++ b/xlators/protocol/client/src/<u></u>client-handshake.c<br>
@@ -471,9 +471,10 @@ client_set_lk_version (xlator_t *this)<br>
conf = (clnt_conf_t *) this->private;<br>
req.lk_ver = client_get_lk_ver (conf);<br>
- ret = gf_asprintf (&req.uid, "%s-%s-%d",<br>
+ ret = gf_asprintf (&req.uid, "%s-%s-%d-%u",<br>
this->ctx->process_uuid, this->name,<br>
- this->graph->id);<br>
+ this->graph->id,<br>
+ (conf->rpc) ? conf->rpc->conn.clnt_conn_id : 0);<br>
if (ret == -1)<br>
goto err;<br>
@@ -1549,13 +1550,22 @@ client_setvolume (xlator_t *this, struct<br>
rpc_clnt *rpc)<br>
}<br>
}<br>
+ /* For different connections between a pair client and server,<br>
we use a<br>
+ * different clnt_conn_id to identify. Otherwise, there are<br>
some chances<br>
+ * lead to flocks not released in a uncleanly disconnection.<br>
+ * */<br>
+ if (conf->rpc) {<br>
+ conf->rpc->conn.clnt_conn_id = conf->clnt_conn_id++;<br>
+ }<br>
+<br>
/* With multiple graphs possible in the same process, we need a<br>
field to bring the uniqueness. Graph-ID should be enough to<br>
get the<br>
job done<br>
*/<br>
- ret = gf_asprintf (&process_uuid_xl, "%s-%s-%d",<br>
+ ret = gf_asprintf (&process_uuid_xl, "%s-%s-%d-%u",<br>
this->ctx->process_uuid, this->name,<br>
- this->graph->id);<br>
+ this->graph->id,<br>
+ (conf->rpc) ? conf->rpc->conn.clnt_conn_id : 0);<br>
if (-1 == ret) {<br>
gf_log (this->name, GF_LOG_ERROR,<br>
"asprintf failed while setting process_uuid");<br>
diff --git a/xlators/protocol/client/src/<u></u>client.c<br>
b/xlators/protocol/client/src/<u></u>client.c<br>
index ad95574..35fef49 100644<br>
--- a/xlators/protocol/client/src/<u></u>client.c<br>
+++ b/xlators/protocol/client/src/<u></u>client.c<br>
@@ -2437,6 +2437,7 @@ init (xlator_t *this)<br>
conf->lk_version = 1;<br>
conf->grace_timer = NULL;<br>
conf->grace_timer_needed = _gf_true;<br>
+ conf->clnt_conn_id = 0;<br>
ret = client_init_grace_timer (this, this->options, conf);<br>
if (ret)<br>
diff --git a/xlators/protocol/client/src/<u></u>client.h<br>
b/xlators/protocol/client/src/<u></u>client.h<br>
index 0a27c09..dea90d1 100644<br>
--- a/xlators/protocol/client/src/<u></u>client.h<br>
+++ b/xlators/protocol/client/src/<u></u>client.h<br>
@@ -116,6 +116,9 @@ typedef struct clnt_conf {<br>
*/<br>
gf_boolean_t filter_o_direct; /* if set, filter<br>
O_DIRECT from<br>
the flags list of<br>
open() */<br>
+ uint32_t clnt_conn_id; /* connection id for each<br>
connection<br>
+ in process_uuid, start<br>
with 0,<br>
+ increase once a new<br>
connection */<br>
} clnt_conf_t;<br>
typedef struct _client_fd_ctx {<br>
<br>
<br>
On Wednesday, September 17, 2014, Jaden Liang <<a>jaden1q84@gmail.com</a><br>
<mailto:<a>jaden1q84@gmail.com</a>>> wrote:<br>
<br>
Hi all,<br>
<br>
By several days tracking, we finally pinpointed the reason of<br>
glusterfs uncleanly<br>
detach file flocks in frequently network disconnection. We are now<br>
working on<br>
a patch to submit. And here is this issue details. Any suggestions<br>
will be<br>
appreciated!<br>
<br>
First of all, as I mentioned in<br>
<a href="http://supercolony.gluster.org/pipermail/gluster-devel/2014-September/042233.html" target="_blank">http://supercolony.gluster.<u></u>org/pipermail/gluster-devel/<u></u>2014-September/042233.html</a><br>
This issue happens in a frequently network disconnection.<br>
<br>
According to the sources, the server cleanup jobs is in<br>
server_connection_cleanup.<br>
When the RPCSVC_EVENT_DISCONNECT happens, it will come here:<br>
<br>
int<br>
server_rpc_notify ()<br>
{<br>
......<br>
case RPCSVC_EVENT_DISCONNECT:<br>
......<br>
if (!conf->lk_heal) {<br>
server_conn_ref (conn);<br>
server_connection_put (this, conn, &detached);<br>
if (detached)<br>
server_connection_cleanup (this, conn,<br>
<br>
INTERNAL_LOCKS |<br>
<br>
POSIX_LOCKS);<br>
server_conn_unref (conn);<br>
......<br>
}<br>
<br>
The server_connection_cleanup() will be called while variable<br>
'detached' is true.<br>
And the 'detached' is set by server_connection_put():<br>
server_connection_t*<br>
server_connection_put (xlator_t *this, server_connection_t *conn,<br>
gf_boolean_t *detached)<br>
{<br>
server_conf_t *conf = NULL;<br>
gf_boolean_t unref = _gf_false;<br>
<br>
if (detached)<br>
*detached = _gf_false;<br>
conf = this->private;<br>
pthread_mutex_lock (&conf->mutex);<br>
{<br>
conn->bind_ref--;<br>
if (!conn->bind_ref) {<br>
list_del_init (&conn->list);<br>
unref = _gf_true;<br>
}<br>
}<br>
pthread_mutex_unlock (&conf->mutex);<br>
if (unref) {<br>
gf_log (this->name, GF_LOG_INFO, "Shutting down<br>
connection %s",<br>
conn->id);<br>
if (detached)<br>
*detached = _gf_true;<br>
server_conn_unref (conn);<br>
conn = NULL;<br>
}<br>
return conn;<br>
}<br>
<br>
The 'detached' is only set _gf_true when 'conn->bind_ref' decrease<br>
to 0.<br>
This 'conn->bind_ref' is set in server_connection_get(), increase or<br>
set to 1.<br>
<br>
server_connection_t *<br>
server_connection_get (xlator_t *this, const char *id)<br>
{<br>
......<br>
list_for_each_entry (trav, &conf->conns, list) {<br>
if (!strcmp (trav->id, id)) {<br>
conn = trav;<br>
conn->bind_ref++;<br>
goto unlock;<br>
}<br>
}<br>
......<br>
}<br>
<br>
When the connection id is same, then the 'conn->bind_ref' will be<br>
increased.<br>
Therefore, the problem should be a reference mismatch increase or<br>
decrease. Then<br>
we add some logs to verify our guess.<br>
<br>
// 1st connection comes in. and there is no id<br>
'host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0'<br>
in the connection table. The 'conn->bind_ref' is set to 1.<br>
[2014-09-17 04:42:28.950693] D<br>
[server-helpers.c:712:server_<u></u>connection_get] 0-vs_vol_rep2-server:<br>
server connection id:<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0,<br>
conn->bind_ref:1, found:0<br>
[2014-09-17 04:42:28.950717] D<br>
[server-handshake.c:430:<u></u>server_setvolume] 0-vs_vol_rep2-server:<br>
Connected to<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0<br>
[2014-09-17 04:42:28.950758] I<br>
[server-handshake.c:567:<u></u>server_setvolume] 0-vs_vol_rep2-server:<br>
accepted client from<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0<br>
(version: 3.4.5) (peer: host-000c29e93d20:1015)<br>
......<br>
// Keep running several minutes.......<br>
......<br>
// Network disconnected here. The TCP socket of client side is<br>
disconnected by<br>
time-out, by the server-side socket still keep connected. AT THIS<br>
MOMENT,<br>
network restore. Client side reconnect a new TCP connection JUST<br>
BEFORE the<br>
last socket on server-side is reset. Note that at this point, there<br>
is 2 valid<br>
sockets on server side. The later new connection use the same conn<br>
id 'host-000<br>
c29e93d20-8661-2014/09/13-11:<u></u>02:26:995090-vs_vol_rep2-<u></u>client-2-0'<br>
look up in the<br>
connection table and increase the 'conn->bind_ref' to 2.<br>
<br>
[2014-09-17 04:46:16.135066] D<br>
[server-helpers.c:712:server_<u></u>connection_get] 0-vs_vol_rep2-server:<br>
server connection id:<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0,<br>
conn->bind_ref:2, found:1 // HERE IT IS, ref increase to 2!!!<br>
[2014-09-17 04:46:16.135113] D<br>
[server-handshake.c:430:<u></u>server_setvolume] 0-vs_vol_rep2-server:<br>
Connected to<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0<br>
[2014-09-17 04:46:16.135157] I<br>
[server-handshake.c:567:<u></u>server_setvolume] 0-vs_vol_rep2-server:<br>
accepted client from<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0<br>
(version: 3.4.5) (peer: host-000c29e93d20:1018)<br>
<br>
// After 13 seconds, the old connection is reset, decrease the<br>
'conn->bind_ref' to 1.<br>
<br>
[2014-09-17 04:46:28.688780] W<br>
[socket.c:2121:__socket_proto_<u></u>state_machine]<br>
0-tcp.vs_vol_rep2-server: ret = -1, error: Connection reset by peer,<br>
peer (host-000c29e93d20:1015)<br>
[2014-09-17 04:46:28.688790] I [socket.c:2274:socket_event_<u></u>handler]<br>
0-transport: socket_event_poll_in failed, ret=-1.<br>
[2014-09-17 04:46:28.688797] D [socket.c:2281:socket_event_<u></u>handler]<br>
0-transport: disconnecting now<br>
[2014-09-17 04:46:28.688831] I [server.c:762:server_rpc_<u></u>notify]<br>
0-vs_vol_rep2-server: disconnecting connectionfrom<br>
host-000c29e93d20-8661-2014/<u></u>09/13-11:02:26:995090-vs_vol_<u></u>rep2-client-2-0(host-<u></u>000c29e93d20:1015)<br>
[2014-09-17 04:46:28.688861] D<br>
[server-helpers.c:744:server_<u></u>connection_put] 0-vs_vol_rep2-server:<br>
conn->bind_ref:1<br>
<br>
In our production environment, there is some flocks in the 1st<br>
connection.<br>
According to the logs, there is no way to cleanup the flocks in the<br>
1st connection.<br>
And the 2nd new connection, the client-side can't flock again.<br>
<br>
Therefore, we think the major reason is different connections using<br>
the same conn id.<br>
The conn id is assembled in client_setvolume()<br>
<br>
ret = gf_asprintf (&process_uuid_xl, "%s-%s-%d",<br>
this->ctx->process_uuid, this->name,<br>
this->graph->id);<br>
<br>
The conn id contains 3 parts:<br>
this->ctx->process_uuid: hostname + pid + startup timestamp<br>
this->name: tranlator name<br>
this->graph->id: graph id<br>
<br>
It is apparently that the conn id is same unless the client side<br>
restart. So when<br>
network disconnects, there is some chance that socket on client side<br>
timeout and<br>
the one on server side is alive. At this moment, network restore,<br>
client reconnect<br>
before server old socket reset, that will cause the file flocks of<br>
old connection<br>
unclean.<br>
<br>
That is our total analysis of this flock leak issue. Now we are<br>
working on the patch.<br>
Hope someone could review it when it is finished.<br>
<br>
Any other comment is grateful, Thank you!<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a>Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>devel</a><br>
<br>
</blockquote>
<br>
</blockquote></div>