Darren,<div> Can you get us the process state dumps of the client when it is hung? (kill -USR1 <pid> of mount and gzip /tmp/glusterdump.<pid>). That will help us figuring out what exactly was happening.</div><div>
<br></div><div>Avati<br><br><div class="gmail_quote">On Tue, Jun 28, 2011 at 8:00 PM, Darren Austin <span dir="ltr"><<a href="mailto:darren-lists@widgit.com">darren-lists@widgit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">> Can you check the server (brick) logs to check the order of detected<br>
> disconnection and new/reconnection from the client?<br>
<br>
</div>Hi,<br>
It seems this wasn't due to keepalives - the system time on both server was a few seconds out. After a pointer from someone off-list, I synced the time and ran ntpd (which I wasn't doing as this was just a test system) and did some more tests.<br>
<br>
The partial-file syndrome I noted before seems to have gone away - at least in terms of the file not syncing back to the previously disconnected server after it finds it way back into the cluster. Once the keepalive timeout is reached, the client sends all the data to the second server.<br>
<br>
A quick question on that actually - when all servers are online, are the clients supposed to send the data to both at the same time? I see from monitoring the traffic that the client duplicates the writes - one to each server?<br>
<br>
Also, when one of the servers disconnects, is it notmal that the client "stalls" the write until the keepalive time expires and the online servers notice one has vanished?<br>
<br>
Finally, during my testing I encountered a replicable hard lock up of the client... here's the situation:<br>
Server1 and Server2 in the cluster, sharing 'data-volume' (which is /data on both servers).<br>
Client mounts server1:data-volume as /mnt.<br>
Client begins to write a large (1 or 2 GB) file to /mnt (I just used random data).<br>
Server1 goes down part way through the write (I simulated this by iptables -j DROP'ing everything from relevant IPs).<br>
Client "stalls" writes until the keepalive timeout, and then continues to send data to Server2.<br>
Server1 comes back online shortly after the keepalive timeout - but BEFORE the Client has written all the data toServer2.<br>
Server1 and Server2 reconnect and the writes on the Client completely hang.<br>
<br>
The mounted directory on the client becomes completely in-accessible when the two servers reconnect.<br>
I had to kill -9 the dd process doing the write (along with the glusterfs process on the client) in order to release the mountpoint.<br>
<br>
I've reproduced this issue several times now and the result is always the same. If the client is writing data to a server when one of the others comes back online after an outage, the client will hang.<br>
<br>
I've attached logs for one of the times I tested this - I hope it helps in diagnosing the problem :)<br>
<br>
Let me know if you need any more info.<br>
<font color="#888888"><br>
--<br>
Darren Austin - Systems Administrator, Widgit Software.<br>
Tel: +44 (0)1926 333680. Web: <a href="http://www.widgit.com/" target="_blank">http://www.widgit.com/</a><br>
26 Queen Street, Cubbington, Warwickshire, CV32 7NA.<br>
<br>
</font><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br></div>