<div dir="ltr"><div><font face="courier new, monospace">Hi Guys,</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">I am using gluster 3.4.2 and doing some test with replica=4, and got following 'unable to get lock error', while it seems the lock has been acquired by itself already. is this a deadlock?</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">any other volume command will be fail with error message like 'another transaction is in progress'.</font></div><div>
<font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><div>$ sudo gluster volume status x9db</div><div>Status of volume: testvol</div><div>Gluster process<span class="" style="white-space:pre"> </span>Port<span class="" style="white-space:pre"> </span>Online<span class="" style="white-space:pre"> </span>Pid</div>
<div>------------------------------------------------------------------------------</div><div>Brick ip1:/home/cluster/db<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2244</div>
<div>Brick ip2:/home/cluster/db<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2225</div><div>Brick ip3:/home/cluster/db<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2302</div>
<div>Brick ip4:/home/cluster/db<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2302</div><div>NFS Server on localhost<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>5411</div>
<div>Self-heal Daemon on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>5418</div><div>NFS Server on ip2<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2279</div>
<div>Self-heal Daemon ip2<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2284</div><div>NFS Server on ip3<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2316</div>
<div>Self-heal Daemon on ip3<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2321</div><div>NFS Server on ip4<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2316</div>
<div>Self-heal Daemon on ip4<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>2321</div><div><br></div><div>There are no active volume tasks</div>
<div><br></div><div>s$ sudo gluster volume info x9db</div><div><br></div><div>Volume Name: testvol</div><div>Type: Replicate</div><div>Volume ID: 6a1159aa-3116-45c3-8611-16aaab76265a</div><div>Status: Started</div><div>Number of Bricks: 1 x 4 = 4</div>
<div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: ip1:/home/cluster/test</div><div>Brick2: ip2:/home/cluster/test</div><div>Brick3: ip3:/home/cluster/test</div><div>Brick4: ip4:/home/cluster/test</div></font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">[2014-01-25 11:12:31.711699] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:42.143165] I [glusterd-handler.c:3260:__glusterd_handle_status_volume] 0-management: Received status volume req for volume x9db</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:46.061212] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.547211] I [glusterd-handshake.c:557:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:47.655510] I [glusterd-volume-ops.c:478:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume x9db</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.766356] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.766386] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.766398] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:47.766410] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.766426] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, hostname:172.16.1.108</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.848456] I [glusterd-handler.c:1956:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.883926] I [glusterd-handler.c:2987:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 172.16.1.105 (0), ret: 0</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:47.997692] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, host: 172.16.1.104</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:47.997715] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, host: 172.16.1.107</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:47.997729] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, host: 172.16.1.108</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:48.098660] I [glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:48.098696] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:48.098711] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:48.098718] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:48.098727] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, hostname:172.16.1.108</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:48.098738] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:48.098764] I [glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:49.036676] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:49.036722] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:49.036734] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:49.036746] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:49.036759] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:51.052062] I [glusterd-rpc-ops.c:345:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, host: 172.16.1.108, port: 0</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:51.152936] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:51.152983] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:51.153006] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:51.153017] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend</font></div><div><font face="courier new, monospace">[2014-01-25 11:12:51.153029] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:12:51.153042] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:13:18.695488] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:13:18.695534] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:13:18.695547] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend</font></div><div><font face="courier new, monospace">[2014-01-25 11:13:18.695561] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:13:18.695575] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:13:46.049416] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</font></div>
<div><font face="courier new, monospace">[2014-01-25 11:13:46.049481] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1</font></div><div><font face="courier new, monospace"><div>[2014-01-25 11:13:46.049985] E [glusterd-utils.c:374:glusterd_unlock] 0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140 ,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!</div>
<div>[2014-01-25 11:13:46.050019] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</div><div>[2014-01-25 11:13:46.050032] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1</div>
<div>[2014-01-25 11:17:37.710091] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</div><div>[2014-01-25 11:17:37.710159] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1</div>
<div>[2014-01-25 11:17:37.710605] E [glusterd-utils.c:374:glusterd_unlock] 0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140 ,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!</div><div>[2014-01-25 11:17:37.710638] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</div>
<div>[2014-01-25 11:17:37.710651] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1</div><div>[2014-01-25 11:18:25.706653] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</div>
<div>[2014-01-25 11:18:25.706716] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1</div><div>[2014-01-25 11:18:25.707379] E [glusterd-utils.c:374:glusterd_unlock] 0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140 ,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!</div>
<div>[2014-01-25 11:18:25.707413] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</div><div>[2014-01-25 11:18:25.707426] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1</div>
<div>[2014-01-25 11:20:59.713622] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div><div>[2014-01-25 11:21:03.963492] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by: b2c16491-ba2c-424c-bfce-19e3acabd140</div>
<div>[2014-01-25 11:21:03.963524] E [glusterd-syncop.c:1023:gd_sync_task_begin] 0-management: Unable to acquire lock</div><div>[2014-01-25 11:22:49.047317] E [rpc-clnt.c:207:call_bail] 0-management: bailing out frame type(glusterd mgmt) op(--(4)) xid = 0x29x sent = 2014-01-25 11:12:48.102044. timeout = 600</div>
<div>[2014-01-25 11:22:49.048264] E [glusterd-rpc-ops.c:664:__glusterd_cluster_unlock_cbk] 0-management: Received unlock RJT from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4</div><div>[2014-01-25 11:22:49.048287] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!</div>
<div>[2014-01-25 11:22:49.048299] C [glusterd-op-sm.c:3648:glusterd_op_txn_complete] 0-management: Unable to clear local lock, ret: -1</div><div>[2014-01-25 11:22:49.048350] E [glusterd-rpc-ops.c:126:glusterd_op_send_cli_response] 0-management: invalid operation</div>
<div>[2014-01-25 11:22:49.048392] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)</div><div>[2014-01-25 11:22:49.048405] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 31) to rpc-transport (socket.management)</div>
<div>[2014-01-25 11:22:49.048421] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed</div></font></div></div>