Hi Nicolas,<br><br><div class="gmail_quote">On Tue, Feb 3, 2009 at 12:01 AM, nicolas prochazka <span dir="ltr"><<a href="mailto:prochazka.nicolas@gmail.com">prochazka.nicolas@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I inspect the log and i find something interesting : <br>All is ok, <br>i have stop 10.98.98.2 and i restart it : <br><br>2009-02-02 15:00:32 D [client-protocol.c:6498:notify] brick_10.98.98.2: got GF_EVENT_CHILD_UP<br>2009-02-02 15:00:32 D [socket.c:924:socket_connect] brick_10.98.98.2: connect () called on transport already connected<br>
2009-02-02 15:00:32 N [client-protocol.c:5786:client_setvolume_cbk] brick_10.98.98.2: connection and handshake succeeded<br>2009-02-02 15:00:40 D [fuse-bridge.c:1945:fuse_statfs] glusterfs-fuse: 17399: STATFS<br>2009-02-02 15:00:40 D [fuse-bridge.c:368:fuse_entry_cbk] glusterfs-fuse: 17400: LOOKUP() / => 1 (1)<br>
200t9-02-02 15:00:42 D [client-protocol.c:5854:client_protocol_reconnect] brick_10.98.98.2: breaking reconnect chain<br><br>All seems to be ok but now i have this log : <br>( a lot of times ) <br><br>2009-02-02 15:07:05 D [client-protocol.c:2799:client_fstat] brick_10.98.98.2: (2148533016): failed to get remote fd. returning EBADFD<br>
<br>then <i>stop 10.98.98.1 ( I tought that 10.98.98.2 is ok but EBADFD seems to be not ! )</i></blockquote><div><br>This is a known issue in afr for files which remain open across the time frame when a server goes down and comes back. Ideally afr should've issued reopen for those files once the server comes back. But currently its not doing so.<br>
<br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><i></i><br><br>2009-02-02 15:10:30 D [page.c:644:ioc_frame_return] io-cache: locked local(0x6309d0)<br>
2009-02-02 15:10:30 D [client-protocol.c:2799:client_fstat] brick_10.98.98.2: (2148533016): failed to get remote fd. returning EBADFD<br>
2009-02-02 15:10:30 D [page.c:646:ioc_frame_return] io-cache: unlocked local(0x6309d0)<br>2009-02-02 15:10:30 D [io-cache.c:798:ioc_need_prune] io-cache: locked table(0x614320)<br>2009-02-02 15:10:30 D [io-cache.c:802:ioc_need_prune] io-cache: unlocked table(0x614320)<br>
2009-02-02 15:10:30 D [client-protocol.c:2799:client_fstat] brick_10.98.98.1: (2148533016): failed to get remote fd. returning EBADFD<br>2009-02-02 15:10:30 D [io-cache.c:425:ioc_cache_validate_cbk] io-cache: cache for inode(0x7fdce0002780) is invalid. flushing all pages<br>
<br><br>Now my client have problems with two servers ( fd )<br><br>so perhaps there is a problem, why 10.98.98.2 is online but client tells EBADFD.<br><br>Regard, <br><font color="#888888">Nicolas</font><div><div></div><div class="Wj3C7c">
<br><br><br><div class="gmail_quote">On Mon, Feb 2, 2009 at 3:30 PM, nicolas prochazka <span dir="ltr"><<a href="mailto:prochazka.nicolas@gmail.com" target="_blank">prochazka.nicolas@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">hi again, <br>last test and last log before stop for me : <br>I do a change, i add option read-subvolume brick_10.98.98.2 in client conf 10.98.98.48<br>
and option read-subvolume brick_10.98.98.1 in client conf 10.98.98.44<br>
<br>run 10.98.98.1 and 10.98.98.2 as server<br>run 10.98.98.44 and 10.98.98.48 as client<br><br>1 - stop 10.98.98.2<br>10.98.98.48 always run and go read to 10.98.98.1<br>10.98.98.44 always run , 10.98.98.1<br><br>2 - rerun 10.98.98.2 , waiting 5 minutes<br>
<br>3 - stop 10.98.98.1<br>process 10.98.98.44 / 48 are hanging<br><br>I think, client can not re read to 10.98.98.2 , is it normal ? 10.98.98.2 is become ready after crash.<br><br> <br>Regards, <br>Nico<div><div></div>
<div><br><br><br><div class="gmail_quote">
On Mon, Feb 2, 2009 at 2:25 PM, nicolas prochazka <span dir="ltr"><<a href="mailto:prochazka.nicolas@gmail.com" target="_blank">prochazka.nicolas@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
hello <br>I always trying to debugging my strange and block problem.<br>I run client with log but there's a lot and a lot (100 mo ) so i can not send you, just info : <br><br>Server 10.98.98.1 and 10.98.98.2<br>client 10.98.98.44 10.98.98.48<br>
<br>Test : ( all tests is performe with big file ( > 10G ) sometimes the test hangs process, sometimes, big file become corrupte ( there's seem that's some data is lacking )<br><br>run all system. : ok <br>stop : 10.98.98.2 : client seems ok<br>
run 10.98.98.2 : sometime it block<br>stop 10.98.98.1 : client 10.98.98.44 is blocking : last log is : <br><br>2009-02-02 13:53:59 D [io-cache.c:798:ioc_need_prune] io-cache: locked table(0x614320)<br>2009-02-02 13:53:59 D [io-cache.c:802:ioc_need_prune] io-cache: unlocked table(0x614320)<br>
2009-02-02 13:53:59 D [client-protocol.c:1701:client_readv] brick_10.98.98.2: (2148533016): failed to get remote fd, returning EBADFD<br><br>and if i rerun 10.98.98.1 , client run again ( ls works ) and log : <br><br>2009-02-02 14:03:18 D [fuse-bridge.c:1945:fuse_statfs] glusterfs-fuse: 40423: STATFS<br>
2009-02-02 14:03:18 D [fuse-bridge.c:1945:fuse_statfs] glusterfs-fuse: 40424: STATFS<br>2009-02-02 14:03:33 D [fuse-bridge.c:1945:fuse_statfs] glusterfs-fuse: 40425: STATFS<br><br>On client 10.98.98.48 , not block.<div><div>
</div><div><br><br>
<br><br><div class="gmail_quote">On Fri, Jan 30, 2009 at 10:14 AM, nicolas prochazka <span dir="ltr"><<a href="mailto:prochazka.nicolas@gmail.com" target="_blank">prochazka.nicolas@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello, <br>first thing, thanks a lot for all yours works.<br>second,<br>Your tests is ok for me but when i replace echo or tail by opening a file with certains type of program, <br>as qemu for example, there's a lot of problem. Process hangs, I also try with --disable-direct-io-mode then process do not hang but file seems to be corrupted.<br>
It's very strange problem. <br><br>Regards, <br>Nicolas Prochazka.<br><br><div class="gmail_quote">2009/1/30 Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@zresearch.com" target="_blank">raghavendra@zresearch.com</a>></span><div>
<div></div><div><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">nicolas,<br><br>I've two servers n1 and n2 which are being afred from client side. I am using the same configuration you finalized on for which you are facing the problem. n1 is the first child of afr.<br>
<br>on n1:<br>
ifconfig eth0 down (eth0 is the interface I am using for communicating with server on n1)<br><br>on glusterfs mount:<br>1. ls (hangs for transport-timeout seconds but completes successfully after timeout)<br>2. I also had a file opened with tail -f /mnt/glusterfs/file before bringing down eth0 on n1.<br>
3. echo "content" >> /mnt/glusterfs/file, appends to file and I was able to observe the content through tail -f.<br><br>on n1:<br>bring up eth0<br><br>on glusterfs mount:<br>1. ls (completes successfully without any problem).<br>
2. echo "content-2" >> /mnt/glusterfs/file (also appends content-2 to file and shown in the output of tail -f)<br><br>From the above tests, it seems the bug is not reproducible in our setup. Is this the similar procedure you followed to reproduce the bug? I am using glusterfs--mainline--3.0--patch-883.<br>
<br>regards,<div><div></div><div><br><br><div class="gmail_quote">On Fri, Jan 30, 2009 at 12:05 AM, Anand Avati <span dir="ltr"><<a href="mailto:avati@zresearch.com" target="_blank">avati@zresearch.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Raghu/ Krishna,<br>
can you guys look into this? It seems like a serious flaw..<br>
<br>
avati<br>
<br>
On Thu, Jan 29, 2009 at 7:13 PM, nicolas prochazka<br>
<div><<a href="mailto:prochazka.nicolas@gmail.com" target="_blank">prochazka.nicolas@gmail.com</a>> wrote:<br>
> hello again,<br>
</div><div><div></div><div>> to be more precise,<br>
> now i can do 'ls /glustermountpoint ' after timeout in all cases, that's<br>
> good<br>
> but, for files which be opened before the crash of first server, that do not<br>
> work, process seems to be block.<br>
><br>
> Regards,<br>
> Nicolas.<br>
</div></div></blockquote></div><br><br clear="all"><br></div></div>-- <br><font color="#888888">Raghavendra G<br><br>
</font></blockquote></div></div></div><br>
</blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>