[Gluster-users] locking in 3.6.1

[Scott Merrill]

After upgrading to Gluster 3.6.1, I'm seeing a lot more (stale?) locks
between my replicated servers.  This prevents me from executing commands
on either server.

>From one server:
root at gluster1:PRODUCTION:~> gluster volume status epa
Locking failed on gluster2.domain.local. Please check log file for details.
root at gluster1:PRODUCTION:~> gluster volume status store
Locking failed on gluster2.domain.local. Please check log file for details.


>From the server that reports the lock:
root at gluster2:PRODUCTION:~> gluster volume status epa
Another transaction is in progress. Please try again after sometime.
root at gluster2:PRODUCTION:~> gluster volume status store
Another transaction is in progress. Please try again after sometime.


I dutifully consulted the logs, as instructed, first from one server:
root at gluster1:PRODUCTION:~> tail
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log
[2014-11-24 21:06:34.402067] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:37.402400] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:40.402722] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:43.403091] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:46.403416] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:49.403757] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:52.404099] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:55.404440] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
[2014-11-24 21:06:58.404770] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/3714f2b1aabf9be7087fc323824b74dd.socket failed
(Invalid argument)
The message "I [MSGID: 106006]
[glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management:
nfs has disconnected from glusterd." repeated 39 times between
[2014-11-24 21:05:01.391554] and [2014-11-24 21:06:58.405175]


and then from the other:
root at gluster2:PRODUCTION:~> tail
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log
[2014-11-24 21:07:23.532144] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:26.532426] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:29.532721] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:32.533014] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:35.533368] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:38.533669] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:41.533998] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:44.534330] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:47.534668] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)
[2014-11-24 21:07:50.534984] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/9bdd01b8b5f546ce04b25ce7d68e3ace.socket failed
(Invalid argument)

(The NFS message from the first server is interesting, since all of my
volumes explicitly declare "nfs.disable: true".)


I can release the lock by `service glusterd restart`, but that seems
sub-optimal. It's terribly manual; and eventually some other lock will
stick, reproducing this situation.


How can I diagnose what is locking, and what can I do to remedy this?

Thanks!
Scott