<div dir="ltr">Hi Michael,<div><br></div><div>Thanks for the information - you were right on both counts. Disabling the kernel NFS & mounting from a separate instance resolved the issue</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 8:08 AM, Michael Brown <span dir="ltr"><<a href="mailto:michael@netdirect.ca" target="_blank">michael@netdirect.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>I found that when I was exporting NFS
from gluster I had to be careful not to load the kernel nfs
modules (such as lockd) or things would get messed up. I think
that's happened to you.<br>
<br>
Is gluster lockd in use? Do this:<br>
<br>
<font face="Consolas">[root@fearless1 ~]# rpcinfo -p | grep 100021<div class="im"><br>
100021 4 tcp 38468 nlockmgr<br></div>
100021 1 udp 872 nlockmgr<br>
100021 1 tcp 874 nlockmgr<br>
[root@fearless1 ~]# ss -lp sport = :38468<br>
State Recv-Q Send-Q Local Address:Port Peer
Address:Port<br>
LISTEN 0 10
*:38468 *:*
users:(("glusterfs",28675,13))<br>
</font><br>
Yes it is!<br>
<br>
If not:<br>
* ensure no nfs mounts are on gluster servers<br>
* rmmod nfs<br>
* rmmod lockd<br>
* 'gluster volume volname start force' should ensure all the
daemons are up.<div><div class="h5"><br>
<br>
On 13-06-05 10:59 AM, Charles Cooke wrote:<br>
</div></div></div>
<blockquote type="cite"><div><div class="h5">
<div dir="ltr">Hello,
<div><br>
</div>
<div>We had some issues with context switches using the
GlusterFS FUSE client - & we are processing a large number
of small files.</div>
<div><br>
</div>
<div>We decided to switch to NFS to see if that would
increase performance, but are having issues with the NLM
locking functionality.</div>
<div><br>
</div>
<div>Using <a href="http://wiki.samba.org/index.php/Ping_pong" target="_blank">http://wiki.samba.org/index.php/Ping_pong</a>
- we are able to verify that a single F_SETLKW call is
blocking indefinitely.</div>
<div><br>
</div>
<div>ping_pong -r glusterfs_test.txt 1
<div>Hangs on this line blocking for a wait lock:</div>
<div>
<div> return fcntl(fd,F_SETLKW,&lock);</div>
</div>
<div><br>
</div>
<div>The root cause appears to be this in the
glusterfs/nfs.log:</div>
<div><b>RPC Program procedure not available for procedure 5 in
NLM4</b></div>
<div><b><br>
</b></div>
<div>Is there any more information on this error, or how it
can be resolved?<br>
<div class="gmail_extra"><br>
<div class="gmail_extra">
rpcinfo -p</div>
<div class="gmail_extra"> program vers proto port</div>
<div class="gmail_extra"> 100000 2 tcp 111
portmapper</div>
<div class="gmail_extra"> 100000 2 udp 111
portmapper</div>
<div class="gmail_extra">
100021 3 udp 45739 nlockmgr</div>
<div class="gmail_extra"> 100021 3 tcp 58923
nlockmgr</div>
<div class="gmail_extra"> 100005 3 tcp 38465
mountd</div>
<div class="gmail_extra"> 100005 1 tcp 38466
mountd</div>
<div class="gmail_extra"> 100003 3 tcp 38467 nfs</div>
<div class="gmail_extra"> 100021 4 tcp 38468
nlockmgr</div>
<div class="gmail_extra"> 100024 1 udp 791
status</div>
<div class="gmail_extra">
100024 1 tcp 794 status</div>
<div class="gmail_extra"> 100021 1 udp 783
nlockmgr</div>
<div class="gmail_extra"> 100021 1 tcp 785
nlockmgr</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<br>
</div>
<div class="gmail_extra">/var/log/messages</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: lockd: nlm_lookup_host(0a8b08af, p=6, v=4)</div>
<div class="gmail_extra">Jun 5 14:53:48 ip-10-139-8-175
kernel: lockd: host garbage collection</div>
<div class="gmail_extra">Jun 5 14:53:48 ip-10-139-8-175
kernel: lockd: nlmsvc_mark_resources</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: lockd: delete host 10.139.8.175</div>
<div class="gmail_extra">
Jun 5 10:53:48 ip-10-139-8-175 kernel: lockd:
nsm_unmonitor(10.139.8.175)</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: lockd: creating host entry</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: lockd: get host 10.139.8.175</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: lockd: nsm_monitor(10.139.8.175)</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: nsm: xdr_decode_stat_res status 0 state 39</div>
<div class="gmail_extra">
Jun 5 10:53:48 ip-10-139-8-175 kernel: lockd: call
procedure 2 on 10.139.8.175</div>
<div class="gmail_extra">Jun 5 10:53:48 ip-10-139-8-175
kernel: lockd: nlm_bind_host(0a8b08af)</div>
<div class="gmail_extra">Jun 5 14:53:48 ip-10-139-8-175
kernel: lockd: server returns status 50331648</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">/var/log/glusterfs/nfs.log</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147647] T
[nfs-fops.c:131:nfs_create_frame] 0-nfs: uid: 0, gid 0,
gids: 7</div>
<div class="gmail_extra">
[2013-06-05 14:53:48.147664] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 0</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147680] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 1</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147695] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 2</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147710] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 3</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147736] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 4</div>
<div class="gmail_extra">
[2013-06-05 14:53:48.147752] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 6</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147767] T
[nfs-fops.c:133:nfs_create_frame] 0-nfs: gid: 10</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147798] T
[rpc-clnt.c:1308:rpc_clnt_record] 0-nfs_test-client-0:
Auth Info: pid: 1, uid: 0, gid: 0, owner:
304069702d31302d-3133392d382d3137-35</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147821] T
[rpc-clnt.c:1188:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 196, payload: 88, rpc hdr:
108</div>
<div class="gmail_extra">[2013-06-05 14:53:48.147861] T
[rpc-clnt.c:1507:rpc_clnt_submit] 0-rpc-clnt: submitted
request (XID: 0x27x Program: GlusterFS 3.1, ProgVers:
330, Proc: 26) to rpc-transport (nfs_test-client-0)</div>
<div class="gmail_extra">[2013-06-05 14:53:48.148249] T
[rpc-clnt.c:685:rpc_clnt_reply_init]
0-nfs_test-client-0: received rpc message (RPC XID:
0x27x Program: GlusterFS 3.1, ProgVers: 330, Proc: 26)
from rpc-transport (nfs_test-client-0)</div>
<div class="gmail_extra">[2013-06-05 14:53:48.148293] T
[rpc-clnt.c:1308:rpc_clnt_record] 0-nfs_test-client-1:
Auth Info: pid: 1, uid: 0, gid: 0, owner:
304069702d31302d-3133392d382d3137-35</div>
<div class="gmail_extra">[2013-06-05 14:53:48.148314] T
[rpc-clnt.c:1188:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 196, payload: 88, rpc hdr:
108</div>
<div class="gmail_extra">[2013-06-05 14:53:48.148349] T
[rpc-clnt.c:1507:rpc_clnt_submit] 0-rpc-clnt: submitted
request (XID: 0x48x Program: GlusterFS 3.1, ProgVers:
330, Proc: 26) to rpc-transport (nfs_test-client-1)</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149264] T
[rpc-clnt.c:685:rpc_clnt_reply_init]
0-nfs_test-client-1: received rpc message (RPC XID:
0x48x Program: GlusterFS 3.1, ProgVers: 330, Proc: 26)
from rpc-transport (nfs_test-client-1)</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149300] D
[fd-lk.c:456:fd_lk_insert_and_merge] 0-fd-lk: new lock
requrest: owner = 304069702d31302d-3133392d382d3137-35,
fl_type = F_WRLCK, fs_start = 0, fs_end = 0, user_flock:
l_type = F_WRLCK, l_start = 0, l_len = 1</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149320] D
[fd-lk.c:419:print_lock_list] 0-fd-lk: lock list:</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149340] D
[fd-lk.c:431:print_lock_list] 0-fd-lk: owner =
304069702d31302d-3133392d382d3137-35, cmd = F_SETLKW
fl_type = F_WRLCK, fs_start = 0, fs_end = 0, user_flock:
l_type = F_WRLCK, l_start = 0, l_len = 1,</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149371] T
[rpc-clnt.c:1308:rpc_clnt_record] 0-NLM-client: Auth
Info: pid: 1, uid: 0, gid: 0, owner:
304069702d31302d-3133392d382d3137-35</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149392] T
[rpc-clnt.c:1188:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 156, payload: 116, rpc hdr:
40</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149430] T
[rpc-clnt.c:1507:rpc_clnt_submit] 0-rpc-clnt: submitted
request (XID: 0x19x Program: NLMv4, ProgVers: 4, Proc:
5) to rpc-transport (NLM-client)</div>
<div class="gmail_extra">
[2013-06-05 14:53:48.149452] T
[nfs3.c:447:nfs3_call_state_wipe] 0-nfs-nfsv3: fd
0x1c9f53fc ref: 4</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149495] T
[rpcsvc.c:470:rpcsvc_handle_rpc_call] 0-rpcsvc: Client
port: 1009</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149516] T
[rpcsvc-auth.c:305:rpcsvc_auth_request_init]
0-rpc-service: Auth handler: AUTH_NULL</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149533] T
[rpcsvc.c:382:rpcsvc_request_create] 0-rpc-service:
received rpc-message (XID: 0x13, Ver: 2, Program:
100021, ProgVers: 4, Proc: 5) from rpc-transport
(socket.NLM)</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149551] E
[rpcsvc.c:203:rpcsvc_program_actor] 0-rpc-service: RPC
Program procedure not available for procedure 5 in NLM4</div>
<div class="gmail_extra">[2013-06-05 14:53:48.149598] D
[rpcsvc.c:1113:rpcsvc_error_reply]
(-->/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x27)
[0x2b1874ed5e17]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x16c)
[0x2b1874ed442c]
(-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x143)
[0x2b1874ed40b3]))) 0-: sending a RPC error reply</div>
<div><br>
</div>
-- <br>
<div dir="ltr">
<div>Thank You,</div>
<div><br>
</div>
<div>Charles Cooke</div>
<div>Senior Cloud Software Engineer</div>
<div>Coupa Software | <a href="http://www.coupa.com" target="_blank">www.coupa.com</a></div>
<div><a href="mailto:charles.cooke@coupa.com" target="_blank">charles.cooke@coupa.com</a></div>
<div>Cell: <a href="tel:902.452.4858" value="+19024524858" target="_blank">902.452.4858</a> </div>
<div><a href="http://www.coupa.com/solutions/new-in-coupa-9" target="_blank"><br>
</a></div>
<div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<br>
<pre cols="72">--
Michael Brown | `One of the main causes of the fall of
Systems Consultant | the Roman Empire was that, lacking zero,
Net Direct Inc. | they had no way to indicate successful
☎: <a href="tel:%2B1%20519%20883%201172%20x5106" value="+15198831172" target="_blank">+1 519 883 1172 x5106</a> | termination of their C programs.' - Firth
</pre>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div>Thank You,</div><div><br></div><div>Charles Cooke</div><div>Senior Cloud Software Engineer</div><div>Coupa Software | <a href="http://www.coupa.com" target="_blank">www.coupa.com</a></div>
<div><a href="mailto:charles.cooke@coupa.com" target="_blank">charles.cooke@coupa.com</a></div><div>Cell: 902.452.4858 </div><div><a href="http://www.coupa.com/solutions/new-in-coupa-9" target="_blank"><br></a></div><div>
<a href="http://www.coupa.com/solutions/new-in-coupa-9" target="_blank"><img src="http://i.imgur.com/4I8kDFc.jpg"></a><br></div><div></div></div>
</div>