<div dir="ltr">Hi Keith,<br><br>Work is being done on AFR to make it stable. Please wait for a stable release.<br><br><div class="gmail_quote">On Tue, Sep 23, 2008 at 6:41 PM, Keith Freedman <span dir="ltr"><<a href="mailto:freedman@freeformit.com">freedman@freeformit.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">as a followup.. I have shutdown the "broken" one in the pair since<br>
it kept crashing.<br>
the working one is running on it's own but gluster dies every 10 mins or so.<br>
seems 1.4pre5 doesn't like being an AFR client all on it's own?<br>
<br>
I'm going to see if it works with only itself as the AFR subvolumes list<br>
<br>
<br>
<br>
<br>
2008-09-23 07:24:00 E [afr.c:3434:afr_statfs_cbk] home: (child=home2)<br>
op_ret=-1 op_errno=107(Transport endpoint is not connected)<br>
2008-09-23 07:24:03 E [afr.c:3434:afr_statfs_cbk] home: (child=home2)<br>
op_ret=-1 op_errno=107(Transport endpoint is not connected)<br>
2008-09-23 07:24:28 E [afr.c:4759:afr_create_cbk] home:<br>
(path=/glusterfile/tmp/1222179868.H882395P21565.HOSTNAME child=home2)<br>
op_ret=-1 op_errno=107(Transport endpoint is not connected)<br>
<div class="Ih2E3d">pending frames:<br>
<br>
Signal received: 11<br>
configuration details:argp 1<br>
backtrace 1<br>
dlfcn 1<br>
fdatasync 1<br>
libpthread 1<br>
llistxattr 1<br>
setfsid 1<br>
spinlock 1<br>
epoll.h 1<br>
xattr.h 1<br>
tv_nsec 1<br>
package-string: glusterfs 1.4.0pre5<br>
</div>/lib64/libc.so.6[0x300d0322a0]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_incver_cbk+0x38)[0xe5250c]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so(client_incver+0xb9)[0xa29072]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_lock_cbk+0x6d8)[0xe52d75]<br>
/usr/local/lib/libglusterfs.so.0[0x125c5b]<br>
/usr/local/lib/libglusterfs.so.0(mop_lock_impl+0x103)[0x12a6aa]<br>
/usr/local/lib/libglusterfs.so.0(default_lock+0x126)[0x125d88]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_incver_internal_fd+0x33a)[0xe530d1]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so(afr_close+0x26d)[0xe5c64d]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x7299fa8]<br>
/lib64/libfuse.so.2[0x10824b2]<br>
/usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so[0x729cc35]<br>
/lib64/libpthread.so.0[0x300dc0729a]<br>
/lib64/libc.so.6(clone+0x6d)[0x300d0e439d]<br>
---------<br>
<div><div></div><div class="Wj3C7c"><br>
<br>
At 07:09 AM 9/23/2008, Keith Freedman wrote:<br>
>I had a pair of servers running 1.4pre5 in AFR.<br>
>they've been running fine for over a week, and suddenly today one of<br>
>them had decided it just will crash anytime it tries to AFR a file.<br>
><br>
>strange is, it seems to get updates form the other server.<br>
>it's not up long enough to do any thorough testing, but when I do<br>
>this from the "good" server:<br>
>echo `hostname` `date` > /gluster/shared/file<br>
>I can read the correct hostname and date from the "bad" server, but<br>
>when I do the same thing on the "bad" server, it crashes instantly.<br>
><br>
>running FC9 with default fuse:<br>
>fuse-2.7.4-8_10.fc9.i386<br>
><br>
>I'm going to re-install fuse thinking that perhaps something got<br>
>corrupted, but it's odd it happened while the servers been goign just<br>
>fine for days.<br>
><br>
>I turned on debugging and here's what it's producing<br>
>where the log ends is where the server crashed while I was tailing<br>
>the logfile:<br>
>2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode:<br>
>retiring inode(0) lru=21/0 active=21 purge=29<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse:<br>
>223: LOOKUP /uservideo/public_html/Guests/Images/Misc/.htaccess<br>
>2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0)<br>
>2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode:<br>
>activating inode(0), lru=21/0 active=22 purge=29<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk]<br>
>glusterfs-fuse: 222: FLUSH() ERR => 0<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release]<br>
>glusterfs-fuse: 224: CLOSE 0x8fecf58<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:562:fuse_getattr]<br>
>glusterfs-fuse: 225: FGETATTR 20971566<br>
>(/user2/public_html/shopping/var/run/classes/kernel/Profiler.php/0x8fece28)<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:496:fuse_attr_cbk]<br>
>glusterfs-fuse: 225: FSTAT()<br>
>/user2/public_html/shopping/var/run/classes/kernel/Profiler.php => 20971566<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:398:fuse_entry_cbk]<br>
>glusterfs-fuse: 223: LOOKUP()<br>
>/uservideo/public_html/Guests/Images/Misc/.htaccess => -1 (No such<br>
>file or directory)<br>
>2008-09-23 06:56:31 D [inode.c:311:__inode_retire] fuse/inode:<br>
>retiring inode(0) lru=21/0 active=21 purge=30<br>
>2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode:<br>
>activating inode(11010602), lru=20/0 active=22 purge=30<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1429:fuse_open] glusterfs-fuse:<br>
>226: OPEN /uservideo/public_html/Guests/Images/Misc/userLogo.jpg<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk]<br>
>glusterfs-fuse: 224: CLOSE() ERR => 0<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse:<br>
>227: FLUSH 0x8fece28<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:603:fuse_fd_cbk] glusterfs-fuse:<br>
>226: OPEN() /uservideo/public_html/Guests/Images/Misc/userLogo.jpg<br>
>=> 0x8fecd50<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1487:fuse_readv] glusterfs-fuse:<br>
>228: READ (0x8fecd50, size=4096, offset=0)<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk]<br>
>glusterfs-fuse: 227: FLUSH() ERR => 0<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1455:fuse_readv_cbk]<br>
>glusterfs-fuse: 228: READ => 3513/4096,0/3513<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release]<br>
>glusterfs-fuse: 229: CLOSE 0x8fece28<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:437:fuse_lookup] glusterfs-fuse:<br>
>230: LOOKUP /user2/public_html/shopping/var/run/classes/kernel/Database.php<br>
>2008-09-23 06:56:31 D [inode.c:443:__inode_create] fuse/inode: create inode(0)<br>
>2008-09-23 06:56:31 D [inode.c:268:__inode_activate] fuse/inode:<br>
>activating inode(0), lru=20/0 active=23 purge=30<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1572:fuse_flush] glusterfs-fuse:<br>
>231: FLUSH 0x8fecd50<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk]<br>
>glusterfs-fuse: 229: CLOSE() ERR => 0<br>
>2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode:<br>
>passivating inode(20971566) lru=21/0 active=22 purge=30<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:370:fuse_entry_cbk]<br>
>glusterfs-fuse: 230: LOOKUP()<br>
>/user2/public_html/shopping/var/run/classes/kernel/Database.php => 20971567<br>
>2008-09-23 06:56:31 D [inode.c:287:__inode_passivate] fuse/inode:<br>
>passivating inode(20971567) lru=22/0 active=21 purge=30<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:857:fuse_err_cbk]<br>
>glusterfs-fuse: 231: FLUSH() ERR => 0<br>
>2008-09-23 06:56:31 D [fuse-bridge.c:1599:fuse_release]<br>
>glusterfs-fuse: 232: CLOSE 0x8fecd50<br>
>2008-09-23 06:56:31 D [fuse-b<br>
><br>
>here's some more from when the server rebooted<br>
>+-----<br>
>2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home1'<br>
>2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt<br>
>to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/storage/posix.so<br>
>2008-09-23 07:04:39 D [spec.y:219:section_type] parser:<br>
>Type:home1:storage/posix<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:home1:directory:/gluster/home<br>
>2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home1<br>
>2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for<br>
>'posix-locks-home1'<br>
>2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt<br>
>to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so<br>
>2008-09-23 07:04:39 D [xlator.c:318:xlator_set_type]<br>
>posix-locks-home1: dlsym(notify) on<br>
>/usr/local/lib/glusterfs/1.4.0pre5/xlator/features/posix-locks.so:<br>
>undefined symbol: notify -- neglecting<br>
>2008-09-23 07:04:39 D [spec.y:219:section_type] parser:<br>
>Type:posix-locks-home1:features/posix-locks<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:posix-locks-home1:mandatory:on<br>
>2008-09-23 07:04:39 D [spec.y:352:section_sub] parser:<br>
>child:posix-locks-home1->home1<br>
>2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:posix-locks-home1<br>
>2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home2'<br>
>2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt<br>
>to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so<br>
>2008-09-23 07:04:39 D [spec.y:219:section_type] parser:<br>
>Type:home2:protocol/client<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:home2:transport-type:tcp/client<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:home2:remote-host:<a href="http://72.36.173.218" target="_blank">72.36.173.218</a><br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:home2:remote-subvolume:posix-locks-home1<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:home2:transport-timeout:10<br>
>2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home2<br>
>2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'server'<br>
>2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt<br>
>to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/server.so<br>
>2008-09-23 07:04:39 D [spec.y:219:section_type] parser:<br>
>Type:server:protocol/server<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:server:transport-type:tcp/server<br>
>2008-09-23 07:04:39 D [spec.y:352:section_sub] parser:<br>
>child:server->posix-locks-home1<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:server:auth.addr.posix-locks-home1.allow:<a href="http://72.36.173.218" target="_blank">72.36.173.218</a>,<a href="http://127.0.0.1" target="_blank">127.0.0.1</a><br>
>2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:server<br>
>2008-09-23 07:04:39 D [spec.y:194:new_section] parser: New node for 'home'<br>
>2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt<br>
>to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so<br>
>2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] home: strict<br>
>option validation is not enforced -- neglecting<br>
>2008-09-23 07:04:39 D [spec.y:219:section_type] parser: Type:home:cluster/afr<br>
>2008-09-23 07:04:39 D [spec.y:285:section_option] parser:<br>
>Option:home:read-subvolume:posix-locks-home1<br>
>2008-09-23 07:04:39 D [spec.y:352:section_sub] parser:<br>
>child:home->posix-locks-home1<br>
>2008-09-23 07:04:39 D [spec.y:352:section_sub] parser: child:home->home2<br>
>2008-09-23 07:04:39 D [spec.y:367:section_end] parser: end:home<br>
>2008-09-23 07:04:39 D [xlator.c:289:xlator_set_type] xlator: attempt<br>
>to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so<br>
>2008-09-23 07:04:39 D [xlator.c:324:xlator_set_type] fuse: strict<br>
>option validation is not enforced -- neglecting<br>
>2008-09-23 07:04:39 D [glusterfs.c:771:main] glusterfs: running in pid 1145<br>
>2008-09-23 07:04:39 D [fuse-options.c:140:fuse_options_validate]<br>
>fuse-options: using mount-point = /home<br>
>2008-09-23 07:04:39 D [fuse-options.c:147:fuse_options_validate]<br>
>fuse-options: using attr-timeout = 1<br>
>2008-09-23 07:04:39 D [fuse-options.c:159:fuse_options_validate]<br>
>fuse-options: using entry-timeout = 1<br>
>2008-09-23 07:04:39 D [fuse-options.c:171:fuse_options_validate]<br>
>fuse-options: using direct-io-mode = 1<br>
>2008-09-23 07:04:39 D [client-protocol.c:4383:init] home2: setting<br>
>transport-timeout to 10<br>
>2008-09-23 07:04:39 D [transport.c:104:transport_load] transport:<br>
>attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so<br>
>2008-09-23 07:04:39 D [client-protocol.c:4427:init] home2: defaulting<br>
>limits.transaction-size to 268435456<br>
>2008-09-23 07:04:39 D [afr.c:6397:init] home: self-heal is enabled (default)<br>
>2008-09-23 07:04:39 D [afr.c:6421:init] home: config: reads will be<br>
>done on posix-locks-home1<br>
>2008-09-23 07:04:39 D [afr.c:6309:notify] home: GF_EVENT_CHILD_UP<br>
>from posix-locks-home1<br>
>2008-09-23 07:04:39 D [afr.c:6241:afr_check_xattr_cbk] home:<br>
>'posix-locks-home1' supports Extended attribute<br>
>2008-09-23 07:04:39 D [inode.c:928:inode_table_new] fuse: creating<br>
>new inode table with lru_limit=0<br>
>2008-09-23 07:04:39 D [inode.c:443:__inode_create] fuse/inode: create inode(0)<br>
>2008-09-23 07:04:39 D [client-protocol.c:4653:notify] home2: got<br>
>GF_EVENT_PARENT_UP, attempting connect on transport<br>
>2008-09-23 07:04:39 D [transport.c:104:transport_load] transport:<br>
>attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/transport/socket.so<br>
>2008-09-23 07:04:39 E [name.c:344:af_inet_server_get_local_sockaddr]<br>
>server: getaddrinfo failed (Name or service not known)<br>
>2008-09-23 07:04:39 W [common-utils.c:158:gf_print_bytes] glusterfs:<br>
>Total data (in bytes): transfered (0), received (0)<br>
>pending frames:<br>
><br>
>Signal received: 11<br>
>configuration details:argp 1<br>
>backtrace 1<br>
>dlfcn 1<br>
>fdatasync 1<br>
>libpthread 1<br>
>llistxattr 1<br>
>setfsid 1<br>
>spinlock 1<br>
>epoll.h 1<br>
>xattr.h 1<br>
>tv_nsec 1<br>
>package-string: glusterfs 1.4.0pre5<br>
><br>
><br>
><br>
>_______________________________________________<br>
>Gluster-users mailing list<br>
><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
><a href="http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>A centipede was happy quite, until a toad in fun,<br>Said, "Prey, which leg comes after which?",<br>This raised his doubts to such a pitch,<br>
He fell flat into the ditch,<br>Not knowing how to run.<br>-Anonymous<br>
</div>