<p dir="ltr">Could you also provide the output of this command:<br>
$ mount | column -t<br><br></p>
<p dir="ltr">--<br>
Gene Liverman<br>
Systems Administrator<br>
Information Technology Services<br>
University of West Georgia<br>
<a href="mailto:gliverma@westga.edu">gliverma@westga.edu</a></p>
<p dir="ltr">ITS: Making Technology Work for You!<br></p>
<p dir="ltr">This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return mail, delete this message, and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal or actionable by law.<br>
</p>
<div class="gmail_quote">On Oct 23, 2014 10:07 AM, "Niels de Vos" <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The only way I can manage to hit this issue too, is by mounting an<br>
NFS-export on the Gluster server that starts the Gluster/NFS process.<br>
There is not crash happening on my side, Gluster/NFS just fails to<br>
start.<br>
<br>
Steps to reproduce:<br>
1. mount -t nfs nas.example.net:/export /mnt<br>
2. systemctl start glusterd<br>
<br>
After this, the error about being unable to register NLM4 is in<br>
/var/log/glusterfs/nfs.log.<br>
<br>
This is expected, because the Linux kernel NFS-server requires an NLM<br>
service in portmap/rpcbind (nlockmgr). You can verify what process<br>
occupies the service slot in rpcbind like this:<br>
<br>
1. list the rpc-programs and their port numbers<br>
<br>
# rpcinfo -p<br>
<br>
2. check the process that listens on the TCP-port for nlockmgr (port<br>
32770 was returned by the command from point 1)<br>
<br>
# netstat -nlpt | grep -w 32770<br>
<br>
If the right column in the output lists 'glusterfs', then the<br>
Gluster/NFS process could register successfully and is handling the NLM4<br>
calls. However, if the right columnt contains a single '-', the Linux<br>
kernel module 'lockd' is handling the NLM4 calls. Gluster/NFS can not<br>
work together with the Linux kernel NFS-client (mountpoint) or the Linux<br>
kernel NFS-server.<br>
<br>
Does this help? If something is unclear, post the output if the above<br>
commands and tell us what further details you want to see clarified.<br>
<br>
Cheers,<br>
Niels<br>
<br>
<br>
On Mon, Oct 20, 2014 at 12:53:46PM +0200, Demeter Tibor wrote:<br>
><br>
> Hi,<br>
><br>
> Thank you for you reply.<br>
><br>
> I did your recommendations, but there are no changes.<br>
><br>
> In the nfs.log there are no new things.<br>
><br>
><br>
> [root@node0 glusterfs]# reboot<br>
> Connection to 172.16.0.10 closed by remote host.<br>
> Connection to 172.16.0.10 closed.<br>
> [tdemeter@sirius-31 ~]$ ssh <a href="mailto:root@172.16.0.10">root@172.16.0.10</a><br>
> <a href="mailto:root@172.16.0.10">root@172.16.0.10</a>'s password:<br>
> Last login: Mon Oct 20 11:02:13 2014 from 192.168.133.106<br>
> [root@node0 ~]# systemctl status nfs.target<br>
> nfs.target - Network File System Server<br>
> Loaded: loaded (/usr/lib/systemd/system/nfs.target; disabled)<br>
> Active: inactive (dead)<br>
><br>
> [root@node0 ~]# gluster volume status engine<br>
> Status of volume: engine<br>
> Gluster process Port Online Pid<br>
> ------------------------------------------------------------------------------<br>
> Brick gs00.itsmart.cloud:/gluster/engine0 50160 Y 3271<br>
> Brick gs01.itsmart.cloud:/gluster/engine1 50160 Y 595<br>
> NFS Server on localhost N/A N N/A<br>
> Self-heal Daemon on localhost N/A Y 3286<br>
> NFS Server on gs01.itsmart.cloud 2049 Y 6951<br>
> Self-heal Daemon on gs01.itsmart.cloud N/A Y 6958<br>
><br>
> Task Status of Volume engine<br>
> ------------------------------------------------------------------------------<br>
> There are no active volume tasks<br>
><br>
> [root@node0 ~]# systemctl status<br>
> Display all 262 possibilities? (y or n)<br>
> [root@node0 ~]# systemctl status nfs-lock<br>
> nfs-lock.service - NFS file locking service.<br>
> Loaded: loaded (/usr/lib/systemd/system/nfs-lock.service; enabled)<br>
> Active: inactive (dead)<br>
><br>
> [root@node0 ~]# systemctl stop nfs-lock<br>
> [root@node0 ~]# systemctl restart gluster<br>
> glusterd.service glusterfsd.service gluster.mount<br>
> [root@node0 ~]# systemctl restart gluster<br>
> glusterd.service glusterfsd.service gluster.mount<br>
> [root@node0 ~]# systemctl restart glusterfsd.service<br>
> [root@node0 ~]# systemctl restart glusterd.service<br>
> [root@node0 ~]# gluster volume status engine<br>
> Status of volume: engine<br>
> Gluster process Port Online Pid<br>
> ------------------------------------------------------------------------------<br>
> Brick gs00.itsmart.cloud:/gluster/engine0 50160 Y 5140<br>
> Brick gs01.itsmart.cloud:/gluster/engine1 50160 Y 2037<br>
> NFS Server on localhost N/A N N/A<br>
> Self-heal Daemon on localhost N/A N N/A<br>
> NFS Server on gs01.itsmart.cloud 2049 Y 6951<br>
> Self-heal Daemon on gs01.itsmart.cloud N/A Y 6958<br>
><br>
><br>
> Any other idea?<br>
><br>
> Tibor<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> ----- Eredeti üzenet -----<br>
> > On Mon, Oct 20, 2014 at 09:04:2.8AM +0200, Demeter Tibor wrote:<br>
> > > Hi,<br>
> > ><br>
> > > This is the full nfs.log after delete & reboot.<br>
> > > It is refers to portmap registering problem.<br>
> > ><br>
> > > [root@node0 glusterfs]# cat nfs.log<br>
> > > [2014-10-20 06:48:43.221136] I [glusterfsd.c:1959:main]<br>
> > > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2<br>
> > > (/usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p<br>
> > > /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S<br>
> > > /var/run/567e0bba7ad7102eae3049e2ad6c3ed7.socket)<br>
> > > [2014-10-20 06:48:43.224444] I [socket.c:3561:socket_init]<br>
> > > 0-socket.glusterfsd: SSL support is NOT enabled<br>
> > > [2014-10-20 06:48:43.224475] I [socket.c:3576:socket_init]<br>
> > > 0-socket.glusterfsd: using system polling thread<br>
> > > [2014-10-20 06:48:43.224654] I [socket.c:3561:socket_init] 0-glusterfs: SSL<br>
> > > support is NOT enabled<br>
> > > [2014-10-20 06:48:43.224667] I [socket.c:3576:socket_init] 0-glusterfs:<br>
> > > using system polling thread<br>
> > > [2014-10-20 06:48:43.235876] I<br>
> > > [rpcsvc.c:2127:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured<br>
> > > rpc.outstanding-rpc-limit with value 16<br>
> > > [2014-10-20 06:48:43.254087] I [socket.c:3561:socket_init]<br>
> > > 0-socket.nfs-server: SSL support is NOT enabled<br>
> > > [2014-10-20 06:48:43.254116] I [socket.c:3576:socket_init]<br>
> > > 0-socket.nfs-server: using system polling thread<br>
> > > [2014-10-20 06:48:43.255241] I [socket.c:3561:socket_init]<br>
> > > 0-socket.nfs-server: SSL support is NOT enabled<br>
> > > [2014-10-20 06:48:43.255264] I [socket.c:3576:socket_init]<br>
> > > 0-socket.nfs-server: using system polling thread<br>
> > > [2014-10-20 06:48:43.257279] I [socket.c:3561:socket_init]<br>
> > > 0-socket.nfs-server: SSL support is NOT enabled<br>
> > > [2014-10-20 06:48:43.257315] I [socket.c:3576:socket_init]<br>
> > > 0-socket.nfs-server: using system polling thread<br>
> > > [2014-10-20 06:48:43.258135] I [socket.c:3561:socket_init] 0-socket.NLM:<br>
> > > SSL support is NOT enabled<br>
> > > [2014-10-20 06:48:43.258157] I [socket.c:3576:socket_init] 0-socket.NLM:<br>
> > > using system polling thread<br>
> > > [2014-10-20 06:48:43.293724] E<br>
> > > [rpcsvc.c:1314:rpcsvc_program_register_portmap] 0-rpc-service: Could not<br>
> > > register with portmap<br>
> > > [2014-10-20 06:48:43.293760] E [nfs.c:332:nfs_init_versions] 0-nfs: Program<br>
> > > NLM4 registration failed<br>
> ><br>
> > The above line suggests that there already is a service registered at<br>
> > portmapper for the NLM4 program/service. This happens when the kernel<br>
> > module 'lockd' is loaded. The kernel NFS-client and NFS-server depend on<br>
> > this, but unfortunately it conflicts with the Gluster/nfs server.<br>
> ><br>
> > Could you verify that the module is loaded?<br>
> > - use 'lsmod | grep lockd' to check the modules<br>
> > - use 'rpcinfo | grep nlockmgr' to check the rpcbind registrations<br>
> ><br>
> > Make sure that you do not mount any NFS exports on the Gluster server.<br>
> > Unmount all NFS mounts.<br>
> ><br>
> > You mentioned you are running CentOS-7, which is systemd based. You<br>
> > should be able to stop any conflicting NFS services like this:<br>
> ><br>
> > # systemctl stop nfs-lock.service<br>
> > # systemctl stop nfs.target<br>
> > # systemctl disable nfs.target<br>
> ><br>
> > If all these services cleanup themselves, you should be able to start<br>
> > the Gluster/nfs service:<br>
> ><br>
> > # systemctl restart glusterd.service<br>
> ><br>
> > In case some bits are still lingering around, it might be easier to<br>
> > reboot after disabling the 'nfs.target'.<br>
> ><br>
> > > [2014-10-20 06:48:43.293771] E [nfs.c:1312:init] 0-nfs: Failed to<br>
> > > initialize protocols<br>
> > > [2014-10-20 06:48:43.293777] E [xlator.c:403:xlator_init] 0-nfs-server:<br>
> > > Initialization of volume 'nfs-server' failed, review your volfile again<br>
> > > [2014-10-20 06:48:43.293783] E [graph.c:307:glusterfs_graph_init]<br>
> > > 0-nfs-server: initializing translator failed<br>
> > > [2014-10-20 06:48:43.293789] E [graph.c:502:glusterfs_graph_activate]<br>
> > > 0-graph: init failed<br>
> > > pending frames:<br>
> > > frame : type(0) op(0)<br>
> > ><br>
> > > patchset: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a><br>
> > > signal received: 11<br>
> > > time of crash: 2014-10-20 06:48:43configuration details:<br>
> > > argp 1<br>
> > > backtrace 1<br>
> > > dlfcn 1<br>
> > > fdatasync 1<br>
> > > libpthread 1<br>
> > > llistxattr 1<br>
> > > setfsid 1<br>
> > > spinlock 1<br>
> > > epoll.h 1<br>
> > > xattr.h 1<br>
> > > st_atim.tv_nsec 1<br>
> > > package-string: glusterfs 3.5.2<br>
> > > [root@node0 glusterfs]# systemctl status portma<br>
> > > portma.service<br>
> > > Loaded: not-found (Reason: No such file or directory)<br>
> > > Active: inactive (dead)<br>
> > ><br>
> > ><br>
> > ><br>
> > > Also I have checked the rpcbind service.<br>
> > ><br>
> > > [root@node0 glusterfs]# systemctl status rpcbind.service<br>
> > > rpcbind.service - RPC bind service<br>
> > > Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled)<br>
> > > Active: active (running) since h 2014-10-20 08:48:39 CEST; 2min 52s ago<br>
> > > Process: 1940 ExecStart=/sbin/rpcbind -w ${RPCBIND_ARGS} (code=exited,<br>
> > > status=0/SUCCESS)<br>
> > > Main PID: 1946 (rpcbind)<br>
> > > CGroup: /system.slice/rpcbind.service<br>
> > > └─1946 /sbin/rpcbind -w<br>
> > ><br>
> > > okt 20 08:48:39 node0.itsmart.cloud systemd[1]: Starting RPC bind<br>
> > > service...<br>
> > > okt 20 08:48:39 node0.itsmart.cloud systemd[1]: Started RPC bind service.<br>
> > ><br>
> > > The restart does not solve this problem.<br>
> > ><br>
> > ><br>
> > > I think this is the problem. Why are "exited" the portmap status?<br>
> ><br>
> > The 'portmap' service has been replaced with 'rpcbind' since RHEL-6.<br>
> > They have the same functionality, 'rpcbind' just happens to be the newer<br>
> > version.<br>
> ><br>
> > Did you file a bug for this already? As Vijay mentions, this crash seems<br>
> > to happen because the Gluster/nfs service fails to initialize correctly<br>
> > and then fails to cleanup correctly. The cleanup should get fixed, and<br>
> > we should also give an easier to understand error message.<br>
> ><br>
> > Thanks,<br>
> > Niels<br>
> ><br>
> > ><br>
> > ><br>
> > > On node1 is ok:<br>
> > ><br>
> > > [root@node1 ~]# systemctl status rpcbind.service<br>
> > > rpcbind.service - RPC bind service<br>
> > > Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled)<br>
> > > Active: active (running) since p 2014-10-17 19:15:21 CEST; 2 days ago<br>
> > > Main PID: 1963 (rpcbind)<br>
> > > CGroup: /system.slice/rpcbind.service<br>
> > > └─1963 /sbin/rpcbind -w<br>
> > ><br>
> > > okt 17 19:15:21 node1.itsmart.cloud systemd[1]: Starting RPC bind<br>
> > > service...<br>
> > > okt 17 19:15:21 node1.itsmart.cloud systemd[1]: Started RPC bind service.<br>
> > ><br>
> > ><br>
> > ><br>
> > > Thanks in advance<br>
> > ><br>
> > > Tibor<br>
> > ><br>
> > ><br>
> > ><br>
> > > ----- Eredeti üzenet -----<br>
> > > > On 10/19/2014 06:56 PM, Niels de Vos wrote:<br>
> > > > > On Sat, Oct 18, 2014 at 01:24:12PM +0200, Demeter Tibor wrote:<br>
> > > > >> Hi,<br>
> > > > >><br>
> > > > >> [root@node0 ~]# tail -n 20 /var/log/glusterfs/nfs.log<br>
> > > > >> [2014-10-18 07:41:06.136035] E [graph.c:307:glusterfs_graph_init]<br>
> > > > >> 0-nfs-server: initializing translator failed<br>
> > > > >> [2014-10-18 07:41:06.136040] E [graph.c:502:glusterfs_graph_activate]<br>
> > > > >> 0-graph: init failed<br>
> > > > >> pending frames:<br>
> > > > >> frame : type(0) op(0)<br>
> > > > >><br>
> > > > >> patchset: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a><br>
> > > > >> signal received: 11<br>
> > > > >> time of crash: 2014-10-18 07:41:06configuration details:<br>
> > > > >> argp 1<br>
> > > > >> backtrace 1<br>
> > > > >> dlfcn 1<br>
> > > > >> fdatasync 1<br>
> > > > >> libpthread 1<br>
> > > > >> llistxattr 1<br>
> > > > >> setfsid 1<br>
> > > > >> spinlock 1<br>
> > > > >> epoll.h 1<br>
> > > > >> xattr.h 1<br>
> > > > >> st_atim.tv_nsec 1<br>
> > > > >> package-string: glusterfs 3.5.2<br>
> > > > ><br>
> > > > > This definitely is a gluster/nfs issue. For whatever reasone, the<br>
> > > > > gluster/nfs server crashes :-/ The log does not show enough details,<br>
> > > > > some more lines before this are needed.<br>
> > > > ><br>
> > > ><br>
> > > > I wonder if the crash is due to a cleanup after the translator<br>
> > > > initialization failure. The complete logs might help in understanding<br>
> > > > why the initialization failed.<br>
> > > ><br>
> > > > -Vijay<br>
> > > ><br>
> > > ><br>
> ><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>