Hi,<br><br>starting glusterfs with direct-io disabled (glusterfs --disable-direct-io) ensures working of NFS re-export of glusterfs.<br><br>regards,<br><br><div class="gmail_quote">On Fri, Apr 27, 2007 at 12:47 AM, <a href="mailto:gpvcs@tiscali.fr">gpvcs@tiscali.fr</a> <span dir="ltr"><<a href="mailto:gpvcs@tiscali.fr">gpvcs@tiscali.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Avati,<br>
<br>
I first thought that I found the error when suddenly it was<br>
working on my machine but I never managed to reproduce it.<br>
Then I realized that in cluster mode, one of my machine was<br>
running using the default fuse but it did it again on my<br>
standalone machine.<br>
One think for sure is that with default fuse (API 7.6)<br>
NFS mount fail always. You have to use a latter fuse.<br>
<br>
After many try I started to trace NFS and Fuse in kernel mode:<br>
<br>
cd /proc/sys/sunrpc<br>
echo 32767 > nfs_debug<br>
echo 32767 > nfsd_debug<br>
<br>
and did my test again:<br>
<br>
exportfs:<br>
localhost:/gfs/fs1 on /gfs/d610.nfsV3 type nfs<br>
(rw,nfsvers=3,noac,hard,intr,tcp,lock,posix,addr=127.0.0.1)<br>
<br>
cd /gfs/nfsV3<br>
ls > 11<br>
<br>
<br>
The trace between a successfull writing other NFS and error<br>
change here:<br>
Apr 26 16:21:06 fturi kernel: NFS call write 6 @ 0<br>
Apr 26 16:21:06 fturi kernel: nfsd_dispatch: vers 3 proc 7<br>
Apr 26 16:21:06 fturi kernel: nfsd: WRITE(3) 24: 02010001<br>
00000000 00000003 00000000 00000001 00000000 6 bytes at 0 stable<br>
Apr 26 16:21:06 fturi kernel: nfsd: fh_verify(24: 02010001<br>
00000000 00000003 00000000 00000001 00000000)<br>
<br>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>><br>
Apr 26 16:21:06 fturi kernel: # NO PERM 531 nfsd: write<br>
complete err=-1<br>
Apr 26 16:21:06 fturi kernel: NFS reply write: -1<br>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<br>
<br>
The NO PERM message comes from this function in fuse<br>
2.6.3/kernel/file.c<br>
<br>
static int fuse_get_user_pages(struct fuse_req *req, const<br>
char __user *buf,<br>
unsigned nbytes, int write)<br>
{<br>
unsigned long user_addr = (unsigned long) buf;<br>
unsigned offset = user_addr & ~PAGE_MASK;<br>
int npages;<br>
<br>
/* This doesn't work with nfsd */<br>
if (!current->mm)<br>
{printk(KERN_CRIT "# NO PERM<br>
%d",__LINE__);return -EPERM;};<br>
<br>
nbytes = min(nbytes, (unsigned)<br>
FUSE_MAX_PAGES_PER_REQ << PAGE_SHIFT);<br>
npages = (nbytes + offset + PAGE_SIZE - 1) >><br>
PAGE_SHIFT;<br>
npages = max(npages, 1);<br>
npages = min(npages, FUSE_MAX_PAGES_PER_REQ);<br>
down_read(&current->mm->mmap_sem);<br>
npages = get_user_pages(current, current->mm,<br>
user_addr, npages, write,<br>
0, req->pages, NULL);<br>
up_read(&current->mm->mmap_sem);<br>
if (npages < 0)<br>
return npages;<br>
<br>
req->num_pages = npages;<br>
req->page_offset = offset;<br>
return 0;<br>
}<br>
<br>
<br>
I Just added the dprintk to make sure it was there. There is<br>
a comment about: /* doesnt work with nfsd */<br>
<br>
So, its look like its fuse refusing (error=-1 = NOPERM) the<br>
writting other NFS.<br>
I'm puzzled by the fact it works under another linux. maybe<br>
the fuse_get_user_pages is not called in your case of the<br>
structure "current->mm" is filled up in your case.<br>
<br>
This is the trace collected by the debug mode of server.vol<br>
and client.vol ( debug on top of the brick):<br>
<br>
glusterfsd is empty<br>
glusterfs contains:<br>
[Apr 26 16:21:02] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:21:02] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/11)<br>
<br>
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_<br>
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,<br>
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26<br>
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26<br>
15:54:32]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/11)<br>
<br>
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_<br>
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,<br>
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26<br>
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26<br>
15:54:32]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:967/trace_open()]<br>
trace:trace_open (*this=0x8395650, path=/11, flags=32769,<br>
mode=0)<br>
[Apr 26 16:21:04] [DEBUG/trace.c:88/trace_open_cbk()]<br>
trace:trace_open_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *ctx=0x8396f58), *buf=0x8396c60 {st_dev=64768,<br>
st_ino=<br>
64356, st_mode=33206, st_nlink=1, st_uid=99, st_gid=99,<br>
st_rdev=0, st_size=0, st_blksize=4096, st_blocks=8,<br>
st_atime=[Apr 26 15:48:19], st_mtime=[Apr 26 15:54:32], st_cti<br>
me=[Apr 26 15:54:32]})<br>
[Apr 26 16:21:04] [DEBUG/trace.c:1089/trace_release()]<br>
trace:trace_release (*this=0x8395650, *ctx=0x8396f58)<br>
[Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/11)<br>
<br>
[Apr 26 16:21:04] [DEBUG/trace.c:386/trace_release_cbk()]<br>
trace:trace_release_cbk (*this=0x8395650, op_ret=0, op_errno=95)<br>
[Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8397098 {st_dev=64768, st_ino=64356, st_<br>
mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,<br>
st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26<br>
15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26<br>
15:54:32]})<br>
[Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]<br>
trace:trace_getattr (*this=0x8395650, path=/)<br>
<br>
[Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]<br>
trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,<br>
op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_<br>
mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,<br>
st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr<br>
26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap<br>
r 26 09:42:58]})<br>
[Apr 26 16:28:31] [DEBUG/trace.c:1155/trace_getxattr()]<br>
trace:trace_getxattr (*this=0x8395650, path=/,<br>
name=system.posix_acl_access, size=0)<br>
[Apr 26 16:28:31] [DEBUG/trace.c:513/trace_getxattr_cbk()]<br>
trace:trace_getxattr_cbk (*this=0x8395650, op_ret=-1,<br>
op_errno=95)<br>
<br>
<br>
Cdt FT<br>
<br>
<br>
<br>
<br>
<br>
---------- Initial Header -----------<br>
<br>
>From : Anand Avati <<a href="mailto:avati@zresearch.com">avati@zresearch.com</a>><br>
To : <a href="mailto:gpvcs@tiscali.fr">gpvcs@tiscali.fr</a><br>
Cc : <a href="mailto:gluster-devel@nongnu.org">gluster-devel@nongnu.org</a><br>
Date : Wed, 18 Apr 2007 08:55:26 -0700<br>
Subject : Re: Re:[Gluster-devel] NFS re-export works for me<br>
<br>
I was able to successfully mount a glusterfs export via NFS<br>
too. I am<br>
using slackware 10.2 and did nothing special apart from using<br>
fuse-2.6.0. Maybe your problem could be because of the AIX<br>
client.<br>
please try from a GNU/Linux client and see if it helps.<br>
Attaching the<br>
log file by loading debug/trace will help get a better picture.<br>
<br>
regards,<br>
avati<br>
<br>
On Wed, Apr 18, 2007 at 11:25:49AM -0400, Brent A Nelson wrote:<br>
> rw,fsid=10 are the export options.<br>
><br>
> 2.6.15 is the kernel (it's a Ubuntu Dapper with its<br>
kernel, probably<br>
> closer to 2.6.16). Fuse 2.6.3 was added from an extra<br>
repository.<br>
><br>
> I used no options at all on the Ubuntu Breezy NFS client<br>
mount. Looking<br>
> at the mtab, no special options are listed there, either,<br>
so the options<br>
> are whatever is default for this platform.<br>
><br>
> Thanks,<br>
><br>
> Brent<br>
><br>
> On Wed, 18 Apr 2007, <a href="mailto:gpvcs@tiscali.fr">gpvcs@tiscali.fr</a> wrote:<br>
><br>
> >Hi Brent,<br>
> ><br>
> >That's a very good news for me. Could you please tell me<br>
> >which linux kernel version are you using (uname -a) and<br>
> >/etc/exports line and mount command you used ?<br>
> >I'm not lucky with NFS and will very appreciate thoses<br>
details.<br>
> ><br>
> >Have a nice day, Francois<br>
> >---------- Initial Header -----------<br>
> ><br>
> >From : gluster-devel-bounces+gpvcs=<a href="http://tiscali.fr" target="_blank">tiscali.fr</a>@<a href="http://nongnu.org" target="_blank">nongnu.org</a><br>
> >To : <a href="mailto:gluster-devel@nongnu.org">gluster-devel@nongnu.org</a><br>
> >Cc :<br>
> >Date : Wed, 18 Apr 2007 00:51:51 -0400 (EDT)<br>
> >Subject : [Gluster-devel] NFS re-export works for me<br>
> ><br>
> >I tried a quick NFS re-export to another Linux box (with the<br>
> >fsid= option<br>
> >and fuse 2.6.3 on the NFS server, of course). I didn't test<br>
> >it much, but<br>
> >copies to and from the NFS filesystem worked just fine. I<br>
> >think writes<br>
> >were very slow, but they didn't give me any trouble.<br>
> ><br>
> >No special mount options (such as NFS version or transport)<br>
> >were needed.<br>
> ><br>
> >Thanks,<br>
> ><br>
> >Brent<br>
> ><br>
> ><br>
> >_______________________________________________<br>
> >Gluster-devel mailing list<br>
> ><a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
> ><a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
> ><br>
><br>
> ------------------------ ALICE C'EST ENCORE MIEUX AVEC<br>
CANAL+ LE BOUQUET !<br>
> ---------------<br>
> Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE<br>
BOUQUET, en cliquant<br>
> ici <a href="http://alicebox.fr" target="_blank">http://alicebox.fr</a><br>
> Soumis à conditions.<br>
> ><br>
> ><br>
<br>
> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
> <a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br>
<br>
--<br>
ultimate_answer_t<br>
deep_thought (void)<br>
{<br>
sleep (years2secs (7500000));<br>
return 42;<br>
}<br>
<br>
<br>
------------------------ ALICE C'EST ENCORE MIEUX AVEC CANAL+ LE BOUQUET ! ---------------<br>
Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE BOUQUET, en cliquant ici <a href="http://alicebox.fr" target="_blank">http://alicebox.fr</a><br>
Soumis à conditions.<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>