<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Hello Niels,</div><div><br></div><div>Thanks for pointing me to the right logs file. I see the following when I enable the log-level to DEBUG option:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.254925] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.1 (/usr/sbin/glusterfs --log-level=DEBUG --volfile-server=srv1 --volfile-id=/gv0 /var/www/html/image/)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.255810] D [glusterfsd.c:410:set_fuse_mount_options] 0-glusterfsd: fopen-keep-cache mode 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.255844] D [glusterfsd.c:466:set_fuse_mount_options] 0-: fuse direct io type 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.256118] D [options.c:1112:xlator_option_init_double] 0-fuse: option negative-timeout using set value 0.000000</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.269774] D [rpc-clnt.c:975:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.269872] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.5.1/rpc-transport/socket.so</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.274941] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.274962] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.274987] D [rpc-clnt.c:1427:rpcclnt_cbk_program_register] 0-glusterfs: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.281676] D [common-utils.c:248:gf_resolve_ip6] 0-resolver: returning ip-127.0.0.1 (port-24007) for hostname: srv1 and port: 24007</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285116] D [io-stats.c:2700:init] 0-gv0: dangling volume. check volfile </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285160] D [options.c:1109:xlator_option_init_bool] 0-gv0: option count-fop-hits using set value off</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285173] D [options.c:1109:xlator_option_init_bool] 0-gv0: option latency-measurement using set value off</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285204] D [options.c:1106:xlator_option_init_size] 0-gv0-quick-read: option cache-size using set value 1GB</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285311] D [quick-read.c:822:check_cache_size_ok] 0-gv0-quick-read: Max cache size is 1040621568</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285326] E [quick-read.c:827:check_cache_size_ok] 0-gv0-quick-read: Cache size 1073741824 is greater than the max size of 1040621568</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285336] E [xlator.c:403:xlator_init] 0-gv0-quick-read: Initialization of volume 'gv0-quick-read' failed, review your volfile again</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285345] E [graph.c:307:glusterfs_graph_init] 0-gv0-quick-read: initializing translator failed</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285354] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285889] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7fe1cb7766d5] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x320) [0x40bd50] (-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x106) [0x405146]))) 0-: received signum (0), shutting down</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285904] D [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[2014-10-05 07:57:49.285941] I [fuse-bridge.c:5475:fini] 0-fuse: Unmounting '/var/www/html/image/‘.</div></div><div><br></div><div>No modification was done to the .vol file:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">-bash-4.1# cat /etc/glusterfs/glusterd.vol </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">volume management</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> type mgmt/glusterd</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> option working-directory /var/lib/glusterd</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> option transport-type socket,rdma</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> option transport.socket.keepalive-time 10</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> option transport.socket.keepalive-interval 2</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> option transport.socket.read-fail-log off</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> option rpc-allow-insecure on</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"># option base-port 49152</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">end-volume</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">-bash-4.1# </div></div><div><br></div>I only added “option roc-allow-insecure on” as a troubleshooting step on both servers.<div><br></div><div>Any suggestions?</div><div><br></div><div>Thanks,</div><div>Guru.</div><div><br></div><div><br><div><div>On 5 Oct 2014, at 6:53 pm, Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Sun, Oct 05, 2014 at 03:09:21PM +1100, Gurdeep Singh (Guru) wrote:<br><blockquote type="cite">Hello,<br><br>There was an outage on one of our servers and after the reboot, mounting of the folder fails on that server with an error message:<br><br>-bash-4.1# mount -t glusterfs srv1:/gv0 /var/www/html/image/<br>Mount failed. Please check the log file for more details.<br>-bash-4.1#<span class="Apple-converted-space"> </span><br></blockquote><br>The log from this mount attempt would be called something like this:<br> /var/log/glusterfs/var-www-html-image-.log<br><br>Maybe you can find some hints there. If there is no obvious error, you<br>can add '-o log-level=DEBUG' to your mount command to make it more<br>verbose.<br><br>HTH,<br>Niels<br><br><blockquote type="cite"><br>Looking at the log file glustershd.log file, I see the following:<br><br>Final graph:<br>+------------------------------------------------------------------------------+<br> 1: volume gv0-client-0<br> 2: type protocol/client<br> 3: option remote-host srv1<br> 4: option remote-subvolume /root/gluster-vol0<br> 5: option transport-type socket<br> 6: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5<br> 7: option password 989d61f9-8393-4402-8d3f-988d18e832a6<br> 8: end-volume<br> 9:<span class="Apple-converted-space"> </span><br>10: volume gv0-client-1<br>11: type protocol/client<br>12: option remote-host srv2<br>13: option remote-subvolume /root/gluster-vol0<br>14: option transport-type socket<br>15: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5<br>16: option password 989d61f9-8393-4402-8d3f-988d18e832a6<br>17: end-volume<br>18:<span class="Apple-converted-space"> </span><br>19: volume gv0-replicate-0<br>20: type cluster/replicate<br>21: option node-uuid c531d907-2f86-4bec-9ae7-8318e28295bc<br>22: option background-self-heal-count 0<br>23: option metadata-self-heal on<br>24: option data-self-heal on<br>25: option entry-self-heal on<br>26: option self-heal-daemon on<br>27: option iam-self-heal-daemon yes<br>28: subvolumes gv0-client-0 gv0-client-1<br>29: end-volume<br>30:<span class="Apple-converted-space"> </span><br>31: volume glustershd<br>32: type debug/io-stats<br>33: subvolumes gv0-replicate-0<br>34: end-volume<br>35:<span class="Apple-converted-space"> </span><br>+------------------------------------------------------------------------------+<br>[2014-10-05 03:54:30.790905] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0)<br>[2014-10-05 03:54:30.798689] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2014-10-05 03:54:30.805120] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-0: Connected to 127.0.0.1:49152, attached to remote volume '/root/gluster-vol0'.<br>[2014-10-05 03:54:30.805163] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds<br>[2014-10-05 03:54:30.805250] I [afr-common.c:4120:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0' came back up; going online.<br>[2014-10-05 03:54:30.807784] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1<br>[2014-10-05 03:54:30.808566] I [afr-self-heald.c:1687:afr_dir_exclusive_crawl] 0-gv0-replicate-0: Another crawl is in progress for gv0-client-0<br>[2014-10-05 03:54:30.808614] E [afr-self-heald.c:1479:afr_find_child_position] 0-gv0-replicate-0: getxattr failed on gv0-client-1 - (Transport endpoint is not connected)<br>[2014-10-05 03:54:30.818679] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0)<br>[2014-10-05 03:54:30.828616] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2014-10-05 03:54:30.844354] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.8.0.6:49152, attached to remote volume '/root/gluster-vol0'.<br>[2014-10-05 03:54:30.844388] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds<br>[2014-10-05 03:54:30.849128] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1<br><br>There is a nfs.log file that shows this:<br><br>Final graph:<br>+------------------------------------------------------------------------------+<br> 1: volume gv0-client-0<br> 2: type protocol/client<br> 3: option remote-host srv1<br> 4: option remote-subvolume /root/gluster-vol0<br> 5: option transport-type socket<br> 6: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5<br> 7: option password 989d61f9-8393-4402-8d3f-988d18e832a6<br> 8: option send-gids true<br> 9: end-volume<br>10:<span class="Apple-converted-space"> </span><br>11: volume gv0-client-1<br>12: type protocol/client<br>13: option remote-host srv2<br>14: option remote-subvolume /root/gluster-vol0<br>15: option transport-type socket<br>16: option username 300c24e9-ac51-4735-b1ee-7acdd985ccd5<br>17: option password 989d61f9-8393-4402-8d3f-988d18e832a6<br>18: option send-gids true<br>19: end-volume<br>20:<span class="Apple-converted-space"> </span><br>21: volume gv0-replicate-0<br>22: type cluster/replicate<br>23: subvolumes gv0-client-0 gv0-client-1<br>24: end-volume<br>25:<span class="Apple-converted-space"> </span><br>26: volume gv0-dht<br>27: type cluster/distribute<br>28: option lookup-unhashed on<br>29: subvolumes gv0-replicate-0<br>30: end-volume<br>31:<span class="Apple-converted-space"> </span><br>32: volume gv0-write-behind<br>33: type performance/write-behind<br>34: subvolumes gv0-dht<br>35: end-volume<br>36:<span class="Apple-converted-space"> </span><br>37: volume gv0<br>38: type debug/io-stats<br>39: option latency-measurement off<br>40: option count-fop-hits off<br>41: subvolumes gv0-write-behind<br>42: end-volume<br>43:<span class="Apple-converted-space"> </span><br>44: volume nfs-server<br>45: type nfs/server<br>46: option rpc-auth.auth-glusterfs on<br>47: option rpc-auth.auth-unix on<br>48: option rpc-auth.auth-null on<br>49: option rpc-auth.ports.insecure on<br>50: option rpc-auth-allow-insecure on<br>51: option transport-type socket<br>52: option transport.socket.listen-port 2049<br>53: option nfs.dynamic-volumes on<br>54: option nfs.nlm on<br>55: option nfs.drc off<br>56: option rpc-auth.addr.gv0.allow *<br>57: option nfs3.gv0.volume-id dc8dc3f2-f5bd-4047-9101-acad04695442<br>58: subvolumes gv0<br>59: end-volume<br>60:<span class="Apple-converted-space"> </span><br>+------------------------------------------------------------------------------+<br>[2014-10-05 03:54:30.832422] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0)<br>[2014-10-05 03:54:30.835888] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2014-10-05 03:54:30.836157] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-0: Connected to 127.0.0.1:49152, attached to remote volume '/root/gluster-vol0'.<br>[2014-10-05 03:54:30.836174] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-0: Server and Client lk-version numbers are not same, reopening the fds<br>[2014-10-05 03:54:30.836393] I [afr-common.c:4120:afr_notify] 0-gv0-replicate-0: Subvolume 'gv0-client-0' came back up; going online.<br>[2014-10-05 03:54:30.836430] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-0: Server lk version = 1<br>[2014-10-05 03:54:30.839191] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49152 (from 0)<br>[2014-10-05 03:54:30.850953] I [client-handshake.c:1659:select_server_supported_programs] 0-gv0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)<br>[2014-10-05 03:54:30.851821] I [client-handshake.c:1456:client_setvolume_cbk] 0-gv0-client-1: Connected to 10.8.0.6:49152, attached to remote volume '/root/gluster-vol0'.<br>[2014-10-05 03:54:30.851843] I [client-handshake.c:1468:client_setvolume_cbk] 0-gv0-client-1: Server and Client lk-version numbers are not same, reopening the fds<br>[2014-10-05 03:54:30.853062] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gv0-client-1: Server lk version = 1<br><br>srv1 (10.8.0.1) is also a VPN server that the srv2 (10.8.0.6) connects to.<span class="Apple-converted-space"> </span><br><br>The volume on srv1,srv2 seems to be up:<br><br>-bash-4.1# gluster volume info<br><br>Volume Name: gv0<br>Type: Replicate<br>Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: srv1:/root/gluster-vol0<br>Brick2: srv2:/root/gluster-vol0<br>Options Reconfigured:<br>cluster.lookup-unhashed: on<br>performance.cache-refresh-timeout: 60<br>performance.cache-size: 1GB<br>storage.health-check-interval: 30<br><br>[guru@srv2 ~]$ sudo gluster volume info<br><br>Volume Name: gv0<br>Type: Replicate<br>Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: srv1:/root/gluster-vol0<br>Brick2: srv2:/root/gluster-vol0<br>Options Reconfigured:<br>cluster.lookup-unhashed: on<br>performance.cache-refresh-timeout: 60<br>performance.cache-size: 1GB<br>storage.health-check-interval: 30<br>[guru@srv2 ~]$<span class="Apple-converted-space"> </span><br><br><br>But, still I am not able to mount the folder into the volume.<br><br>Please suggest how can we troubleshoot this issue.<br><br>Regards,<br>Guru.<br><br></blockquote><br><blockquote type="cite">_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></blockquote></div><br></div></body></html>