<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hello,<div><br></div><div>There was an outage on one of our servers and after the reboot, mounting of the folder fails on that server with an error message:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">-bash-4.1# mount -t glusterfs srv1:/gv0 /var/www/html/image/</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Mount failed. Please check the log file for more details.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">-bash-4.1# </div></div><div><br></div><div>Looking at the log file glustershd.log file, I see the following:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Final graph:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">+------------------------------------------------------------------------------+</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 1: volume gv0-client-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 2: type protocol/client</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 3: option remote-host srv1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 4: option remote-subvolume /root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 5: option transport-type socket</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 6: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 7: option password 989d61f9-8393-4402-8d3f-988d18e832a6</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 8: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 9: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 10: volume gv0-client-1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 11: type protocol/client</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 12: option remote-host srv2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 13: option remote-subvolume /root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 14: option transport-type socket</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 15: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 16: option password 989d61f9-8393-4402-8d3f-988d18e832a6</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 17: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 18: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 19: volume gv0-replicate-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 20: type cluster/replicate</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 21: option node-uuid c531d907-2f86-4bec-9ae7-8318e28295bc</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 22: option background-self-heal-count 0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 23: option metadata-self-heal on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 24: option data-self-heal on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 25: option entry-self-heal on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 26: option self-heal-daemon on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 27: option iam-self-heal-daemon yes</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 28: subvolumes gv0-client-0 gv0-client-1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 29: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 30: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 31: volume glustershd</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 32: type debug/io-stats</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 33: subvolumes gv0-replicate-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 34: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 35: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">+------------------------------------------------------------------------------+</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.790905] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.798689] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.805120] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-0: Connected to 127.0.0.1:49152, attached to remote volume '/root/gluster-vol0'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.805163] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.805250] I [afr-common.c:4120:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0' came back up; going online.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.807784] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.808566] I [afr-self-heald.c:1687:afr_dir_exclusive_crawl] 0-gv0-replicate-0: Another crawl is in progress for gv0-client-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.808614] E [afr-self-heald.c:1479:afr_find_child_position] 0-gv0-replicate-0: getxattr failed on gv0-client-1 - (Transport endpoint is not connected)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.818679] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.828616] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.844354] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.8.0.6:49152, attached to remote volume '/root/gluster-vol0'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.844388] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.849128] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1</div></div><div><br></div><div>There is a nfs.log file that shows this:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Final graph:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">+------------------------------------------------------------------------------+</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 1: volume gv0-client-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 2: type protocol/client</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 3: option remote-host srv1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 4: option remote-subvolume /root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 5: option transport-type socket</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 6: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 7: option password 989d61f9-8393-4402-8d3f-988d18e832a6</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 8: option send-gids true</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 9: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 10: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 11: volume gv0-client-1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 12: type protocol/client</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 13: option remote-host srv2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 14: option remote-subvolume /root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 15: option transport-type socket</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 16: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 17: option password 989d61f9-8393-4402-8d3f-988d18e832a6</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 18: option send-gids true</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 19: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 20: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 21: volume gv0-replicate-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 22: type cluster/replicate</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 23: subvolumes gv0-client-0 gv0-client-1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 24: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 25: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 26: volume gv0-dht</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 27: type cluster/distribute</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 28: option lookup-unhashed on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 29: subvolumes gv0-replicate-0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 30: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 31: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 32: volume gv0-write-behind</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 33: type performance/write-behind</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 34: subvolumes gv0-dht</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 35: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 36: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 37: volume gv0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 38: type debug/io-stats</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 39: option latency-measurement off</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 40: option count-fop-hits off</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 41: subvolumes gv0-write-behind</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 42: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 43: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 44: volume nfs-server</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 45: type nfs/server</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 46: option rpc-auth.auth-glusterfs on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 47: option rpc-auth.auth-unix on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 48: option rpc-auth.auth-null on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 49: option rpc-auth.ports.insecure on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 50: option rpc-auth-allow-insecure on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 51: option transport-type socket</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 52: option transport.socket.listen-port 2049</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 53: option nfs.dynamic-volumes on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 54: option nfs.nlm on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 55: option nfs.drc off</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 56: option rpc-auth.addr.gv0.allow *</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 57: option nfs3.gv0.volume-id dc8dc3f2-f5bd-4047-9101-acad04695442</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 58: subvolumes gv0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 59: end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 60: </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">+------------------------------------------------------------------------------+</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.832422] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.835888] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.836157] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-0: Connected to 127.0.0.1:49152, attached to remote volume '/root/gluster-vol0'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.836174] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.836393] I [afr-common.c:4120:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0' came back up; going online.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.836430] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.839191] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.850953] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.851821] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.8.0.6:49152, attached to remote volume '/root/gluster-vol0'.</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.851843] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 03:54:30.853062] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1</div></div><div><br></div><div>srv1 (10.8.0.1) is also a VPN server that the srv2 (10.8.0.6) connects to. </div><div><br></div><div>The volume on srv1,srv2 seems to be up:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">-bash-4.1# gluster volume info</div><p style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;"> <br class="webkit-block-placeholder"></p><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Volume Name: gv0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Type: Replicate</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Status: Started</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Number of Bricks: 1 x 2 = 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Transport-type: tcp</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Bricks:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Brick1: srv1:/root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Brick2: srv2:/root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Options Reconfigured:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">cluster.lookup-unhashed: on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">performance.cache-refresh-timeout: 60</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">performance.cache-size: 1GB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">storage.health-check-interval: 30</div></div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ sudo gluster volume info</div><p style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;"> <br class="webkit-block-placeholder"></p><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Volume Name: gv0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Type: Replicate</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Status: Started</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Number of Bricks: 1 x 2 = 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Transport-type: tcp</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Bricks:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Brick1: srv1:/root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Brick2: srv2:/root/gluster-vol0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Options Reconfigured:</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">cluster.lookup-unhashed: on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">performance.cache-refresh-timeout: 60</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">performance.cache-size: 1GB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">storage.health-check-interval: 30</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div></div><div><br></div><div><br></div><div>But, still I am not able to mount the folder into the volume.</div><div><br></div><div>Please suggest how can we troubleshoot this issue.</div><div><br></div><div>Regards,</div><div>Guru.</div><div><br></div></body></html>