After Munaual reboot the gluster service :- [2014-11-25 06:54:11.630034] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.1 (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid) [2014-11-25 06:54:11.635944] I [glusterd.c:1122:init] 0-management: Using /var/lib/glusterd as working directory [2014-11-25 06:54:11.638156] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2014-11-25 06:54:11.638177] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2014-11-25 06:54:11.641459] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device) [2014-11-25 06:54:11.641479] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device [2014-11-25 06:54:11.641501] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2014-11-25 06:54:11.641555] W [rpcsvc.c:1521:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2014-11-25 06:54:11.641626] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled [2014-11-25 06:54:11.641641] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread [2014-11-25 06:54:11.648334] I [glusterd.c:367:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [2014-11-25 06:54:11.798836] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30501 [2014-11-25 06:54:11.849045] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-11-25 06:54:11.849073] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-11-25 06:54:11.849086] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2 [2014-11-25 06:54:11.849096] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3 [2014-11-25 06:54:11.849106] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4 [2014-11-25 06:54:11.849117] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5 [2014-11-25 06:54:11.849128] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6 [2014-11-25 06:54:11.849139] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7 [2014-11-25 06:54:12.783026] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2014-11-25 06:54:12.833169] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2014-11-25 06:54:12.883216] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2014-11-25 06:54:12.933303] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0 [2014-11-25 06:54:12.933372] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:12.933437] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:12.933451] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:12.941439] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:12.941482] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:12.941497] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:12.944146] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:12.944187] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:12.944197] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:12.946780] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:12.946824] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:12.946834] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:12.949846] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 918cef1f-ed83-4570-90ad-39c5ceec753c Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option transport.socket.listen-backlog 128 7: option transport.socket.read-fail-log off 8: option transport.socket.keepalive-interval 2 9: option transport.socket.keepalive-time 10 10: option transport-type rdma 11: option working-directory /var/lib/glusterd 12: end-volume 13: +------------------------------------------------------------------------------+ [2014-11-25 06:54:13.450703] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, host: compute3.3linux.com, port: 0 [2014-11-25 06:54:13.810377] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:13.810446] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:13.810462] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:13.812527] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:13.812617] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:13.812634] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:13.818535] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:13.818613] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:13.818627] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:14.761152] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30501 [2014-11-25 06:54:15.041594] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:15.041687] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:15.041702] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:20.824599] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-11-25 06:54:20.824696] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled [2014-11-25 06:54:20.824717] I [socket.c:3576:socket_init] 0-management: using system polling thread [2014-11-25 06:54:21.240569] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, host: compute1.3linux.com, port: 0 [2014-11-25 06:54:21.574630] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020 [2014-11-25 06:54:21.574675] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, hostname:compute3.3linux.com [2014-11-25 06:54:21.574695] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, hostname:compute2.3linux.com [2014-11-25 06:54:21.574710] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 918cef1f-ed83-4570-90ad-39c5ceec753c, hostname:compute4.3linux.com [2014-11-25 06:54:21.574725] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2014-11-25 06:54:21.574734] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, hostname:compute5.3linux.com [2014-11-25 06:54:21.574788] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, host: compute2.3linux.com, port: 0 [2014-11-25 06:54:21.901130] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, host: compute5.3linux.com, port: 0 [2014-11-25 06:54:22.226551] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2014-11-25 06:54:22.226889] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from compute4.3linux.com:/brick7 [2014-11-25 06:54:22.227216] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2014-11-25 06:54:22.227541] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from compute4.3linux.com:/brick8 [2014-11-25 06:54:22.227566] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2014-11-25 06:54:22.227593] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now [2014-11-25 06:54:22.227664] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020 [2014-11-25 06:54:22.227734] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to compute1.3linux.com (0), ret: 0 [2014-11-25 06:54:22.551176] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, host: compute5.3linux.com [2014-11-25 06:54:22.551211] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, host: compute1.3linux.com [2014-11-25 06:54:22.551229] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, host: compute2.3linux.com [2014-11-25 06:54:22.551245] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, host: compute3.3linux.com [2014-11-25 06:54:22.893624] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7 [2014-11-25 06:54:22.893659] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020 [2014-11-25 06:54:22.893678] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f [2014-11-25 06:54:22.893695] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30 [2014-11-25 06:54:22.893834] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30 [2014-11-25 06:54:22.893861] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, hostname:compute2.3linux.com [2014-11-25 06:54:22.893873] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, hostname:compute5.3linux.com [2014-11-25 06:54:22.893882] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 918cef1f-ed83-4570-90ad-39c5ceec753c, hostname:compute4.3linux.com [2014-11-25 06:54:22.893889] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2014-11-25 06:54:22.893898] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, hostname:compute1.3linux.com [2014-11-25 06:54:22.918527] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f [2014-11-25 06:54:22.918554] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, hostname:compute5.3linux.com [2014-11-25 06:54:22.918566] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 918cef1f-ed83-4570-90ad-39c5ceec753c, hostname:compute4.3linux.com [2014-11-25 06:54:22.918573] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2014-11-25 06:54:22.918582] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, hostname:compute3.3linux.com [2014-11-25 06:54:22.918591] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, hostname:compute1.3linux.com [2014-11-25 06:54:22.918717] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30501 [2014-11-25 06:54:23.353506] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30501 [2014-11-25 06:54:23.744838] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7 [2014-11-25 06:54:23.744889] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 918cef1f-ed83-4570-90ad-39c5ceec753c, hostname:compute4.3linux.com [2014-11-25 06:54:23.744905] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2014-11-25 06:54:23.744920] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, hostname:compute3.3linux.com [2014-11-25 06:54:23.744936] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, hostname:compute2.3linux.com [2014-11-25 06:54:23.744951] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, hostname:compute1.3linux.com [2014-11-25 06:54:23.745067] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30 [2014-11-25 06:54:23.745136] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to compute3.3linux.com (0), ret: 0 [2014-11-25 06:54:24.437530] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, host: compute5.3linux.com [2014-11-25 06:54:24.437565] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, host: compute1.3linux.com [2014-11-25 06:54:24.437579] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, host: compute2.3linux.com [2014-11-25 06:54:24.437589] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, host: compute3.3linux.com [2014-11-25 06:54:24.871837] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f [2014-11-25 06:54:24.871929] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to compute2.3linux.com (0), ret: 0 [2014-11-25 06:54:25.239612] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, host: compute5.3linux.com [2014-11-25 06:54:25.239636] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, host: compute1.3linux.com [2014-11-25 06:54:25.239647] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, host: compute2.3linux.com [2014-11-25 06:54:25.239658] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, host: compute3.3linux.com [2014-11-25 06:54:25.565279] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /brick8 on port 49153 [2014-11-25 06:54:25.569930] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /brick7 on port 49152 [2014-11-25 06:54:25.572193] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f [2014-11-25 06:54:25.572226] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30 [2014-11-25 06:54:25.572244] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7 [2014-11-25 06:54:25.572260] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020 [2014-11-25 06:54:25.572299] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30501 [2014-11-25 06:54:25.807473] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f [2014-11-25 06:54:25.807508] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30 [2014-11-25 06:54:25.807532] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7 [2014-11-25 06:54:25.807549] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020 [2014-11-25 06:54:25.808430] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7 [2014-11-25 06:54:25.808526] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to compute5.3linux.com (0), ret: 0 [2014-11-25 06:54:26.266730] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7, host: compute5.3linux.com [2014-11-25 06:54:26.266757] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020, host: compute1.3linux.com [2014-11-25 06:54:26.266775] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f, host: compute2.3linux.com [2014-11-25 06:54:26.266785] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30, host: compute3.3linux.com [2014-11-25 06:54:26.592503] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 268139cb-e7b7-48e9-99cf-e26c523516e7 [2014-11-25 06:54:26.592532] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 69b19f4e-03c8-4fe1-a644-2701c41d7020 [2014-11-25 06:54:26.592550] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2cf3428c-62be-42f4-86d3-ef67e7dfc97f [2014-11-25 06:54:26.592565] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 0d94d896-e994-4deb-b1a1-d6f68be86c30 ^C