[2010-02-26 14:55:01] D [glusterfsd.c:424:_get_specfp] glusterfs: loading volume file /etc/glusterfs/glusterfs.vol [2010-02-26 14:55:01] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/performance/write-behind.so: undefined symbol: notify -- neglecting [2010-02-26 14:55:01] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/performance/read-ahead.so: undefined symbol: notify -- neglecting [2010-02-26 14:55:01] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/performance/io-cache.so: undefined symbol: notify -- neglecting [2010-02-26 14:55:01] D [xlator.c:744:xlator_set_type] xlator: dlsym(dumpops) on /usr/lib/glusterfs/3.0.2/xlator/performance/io-cache.so: undefined symbol: dumpops -- neglecting [2010-02-26 14:55:01] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/performance/quick-read.so: undefined symbol: notify -- neglecting [2010-02-26 14:55:01] D [xlator.c:739:xlator_set_type] xlator: dlsym(notify) on /usr/lib/glusterfs/3.0.2/xlator/performance/stat-prefetch.so: undefined symbol: notify -- neglecting [2010-02-26 14:55:01] D [xlator.c:744:xlator_set_type] xlator: dlsym(dumpops) on /usr/lib/glusterfs/3.0.2/xlator/performance/stat-prefetch.so: undefined symbol: dumpops -- neglecting [2010-02-26 14:55:01] D [xlator.c:754:xlator_set_type] statprefetch: Strict option validation not enforced -- neglecting ================================================================================ Version : glusterfs 3.0.2 built on Feb 9 2010 17:57:23 git: v3.0.2 Starting Time: 2010-02-26 14:55:01 Command line : glusterfs --debug -f /etc/glusterfs/glusterfs.vol /srv/gluster-export PID : 12613 System name : Linux Nodename : node01 Kernel Release : 2.6.26-2-amd64 Hardware Identifier: x86_64 Given volfile: +------------------------------------------------------------------------------+ 1: 2: ### client defs, 3 distributed volumes, then replicated 3: # TRANSPORT-TYPE tcp 4: volume node01-1 5: type protocol/client 6: option transport-type tcp 7: option remote-host node01 8: option transport.socket.nodelay on 9: option transport.remote-port 6996 10: option remote-subvolume brick1 11: end-volume 12: 13: volume node01-2 14: type protocol/client 15: option transport-type tcp 16: option remote-host node01 17: option transport.socket.nodelay on 18: option transport.remote-port 6996 19: option remote-subvolume brick2 20: end-volume 21: 22: volume node01-3 23: type protocol/client 24: option transport-type tcp 25: option remote-host node01 26: option transport.socket.nodelay on 27: option transport.remote-port 6996 28: option remote-subvolume brick3 29: end-volume 30: 31: volume node01-4 32: type protocol/client 33: option transport-type tcp 34: option remote-host node01 35: option transport.socket.nodelay on 36: option transport.remote-port 6996 37: option remote-subvolume brick4 38: end-volume 39: 40: volume node02-1 41: type protocol/client 42: option transport-type tcp 43: option remote-host node02 44: option transport.socket.nodelay on 45: option transport.remote-port 6996 46: option remote-subvolume brick1 47: end-volume 48: 49: volume node02-2 50: type protocol/client 51: option transport-type tcp 52: option remote-host node02 53: option transport.socket.nodelay on 54: option transport.remote-port 6996 55: option remote-subvolume brick2 56: end-volume 57: 58: volume node02-3 59: type protocol/client 60: option transport-type tcp 61: option remote-host node02 62: option transport.socket.nodelay on 63: option transport.remote-port 6996 64: option remote-subvolume brick3 65: end-volume 66: 67: volume node02-4 68: type protocol/client 69: option transport-type tcp 70: option remote-host node02 71: option transport.socket.nodelay on 72: option transport.remote-port 6996 73: option remote-subvolume brick4 74: end-volume 75: 76: volume node03-1 77: type protocol/client 78: option transport-type tcp 79: option remote-host node03 80: option transport.socket.nodelay on 81: option transport.remote-port 6996 82: option remote-subvolume brick1 83: end-volume 84: 85: volume node03-2 86: type protocol/client 87: option transport-type tcp 88: option remote-host node03 89: option transport.socket.nodelay on 90: option transport.remote-port 6996 91: option remote-subvolume brick2 92: end-volume 93: 94: volume node03-3 95: type protocol/client 96: option transport-type tcp 97: option remote-host node03 98: option transport.socket.nodelay on 99: option transport.remote-port 6996 100: option remote-subvolume brick3 101: end-volume 102: 103: volume node03-4 104: type protocol/client 105: option transport-type tcp 106: option remote-host node03 107: option transport.socket.nodelay on 108: option transport.remote-port 6996 109: option remote-subvolume brick4 110: end-volume 111: 112: volume node04-1 113: type protocol/client 114: option transport-type tcp 115: option remote-host node04 116: option transport.socket.nodelay on 117: option transport.remote-port 6996 118: option remote-subvolume brick1 119: end-volume 120: 121: volume node04-2 122: type protocol/client 123: option transport-type tcp 124: option remote-host node04 125: option transport.socket.nodelay on 126: option transport.remote-port 6996 127: option remote-subvolume brick2 128: end-volume 129: 130: volume node04-3 131: type protocol/client 132: option transport-type tcp 133: option remote-host node04 134: option transport.socket.nodelay on 135: option transport.remote-port 6996 136: option remote-subvolume brick3 137: end-volume 138: 139: volume node04-4 140: type protocol/client 141: option transport-type tcp 142: option remote-host node04 143: option transport.socket.nodelay on 144: option transport.remote-port 6996 145: option remote-subvolume brick4 146: end-volume 147: 148: ### Our 3 replicas 149: volume repstore1 150: type cluster/distribute 151: #option local-volume-name `hostname`-1 `hostname`-4 # note the backquote, so 'hostname' output will be used as the option 152: #option local-volume-name `hostname`-1 # note the backquote, so 'hostname' output will be used as the option 153: subvolumes node01-1 node02-1 node03-1 node04-1 node04-4 154: end-volume 155: 156: volume repstore2 157: type cluster/distribute 158: #option local-volume-name `hostname`-2 # note the backquote, so 'hostname' output will be used as the option 159: subvolumes node01-2 node02-2 node03-2 node04-2 node02-2 160: end-volume 161: 162: volume repstore3 163: type cluster/distribute 164: #option local-volume-name `hostname`-3 # note the backquote, so 'hostname' output will be used as the option 165: subvolumes node01-3 node02-3 node03-3 node04-3 node03-3 166: end-volume 167: 168: volume replicate 169: type cluster/replicate 170: subvolumes repstore1 repstore2 repstore3 171: end-volume 172: 173: volume writebehind 174: type performance/write-behind 175: option cache-size 4MB 176: subvolumes replicate 177: end-volume 178: 179: volume readahead 180: type performance/read-ahead 181: option page-count 4 182: subvolumes writebehind 183: end-volume 184: 185: volume iocache 186: type performance/io-cache 187: option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB 188: option cache-timeout 1 189: subvolumes readahead 190: end-volume 191: 192: volume quickread 193: type performance/quick-read 194: option cache-timeout 1 195: option max-file-size 64kB 196: subvolumes iocache 197: end-volume 198: 199: volume statprefetch 200: type performance/stat-prefetch 201: subvolumes quickread 202: end-volume 203: +------------------------------------------------------------------------------+ [2010-02-26 14:55:01] D [glusterfsd.c:1370:main] glusterfs: running in pid 12613 [2010-02-26 14:55:01] D [read-ahead.c:884:init] readahead: Using conf->page_count = 4 [2010-02-26 14:55:01] D [write-behind.c:2476:init] writebehind: disabling write-behind for first 1 bytes [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node04-4: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node04-4: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node04-4: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node04-3: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node04-3: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node04-3: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node04-2: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node04-2: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node04-2: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node04-1: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node04-1: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node04-1: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node04-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] W [client-protocol.c:6568:init] node03-4: Volume is dangling. [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node03-4: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node03-4: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node03-4: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node03-3: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node03-3: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node03-3: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node03-2: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node03-2: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node03-2: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node03-1: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node03-1: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node03-1: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node03-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] W [client-protocol.c:6568:init] node02-4: Volume is dangling. [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node02-4: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node02-4: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node02-4: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node02-3: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node02-3: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node02-3: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node02-2: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node02-2: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node02-2: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node02-1: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node02-1: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node02-1: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node02-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] W [client-protocol.c:6568:init] node01-4: Volume is dangling. [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node01-4: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node01-4: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node01-4: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-4: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node01-3: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node01-3: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node01-3: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-3: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node01-2: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node01-2: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node01-2: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-2: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:6585:init] node01-1: defaulting frame-timeout to 30mins [2010-02-26 14:55:01] D [client-protocol.c:6596:init] node01-1: defaulting ping-timeout to 42 [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] W [xlator.c:655:validate_xlator_volume_options] node01-1: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [transport.c:145:transport_load] transport: attempt to load file /usr/lib/glusterfs/3.0.2/transport/socket.so [2010-02-26 14:55:01] D [xlator.c:284:_volume_option_value_validate] node01-1: no range check required for 'option remote-port 6996' [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-4: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-4: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-1: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-4: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-4: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-2: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node01-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node02-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node04-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-3: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-4: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] D [client-protocol.c:7009:notify] node03-4: got GF_EVENT_PARENT_UP, attempting connect on transport [2010-02-26 14:55:01] N [glusterfsd.c:1396:main] glusterfs: Successfully started [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node01-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node01-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node01-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node01-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node01-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node02-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node02-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node02-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node01-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node02-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node02-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node04-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node04-3: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-4: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node01-1: Connected to 10.135.32.50:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:01] N [afr.c:2625:notify] replicate: Subvolume 'repstore1' came back up; going online. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node01-1: Connected to 10.135.32.50:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:01] N [afr.c:2625:notify] replicate: Subvolume 'repstore1' came back up; going online. [2010-02-26 14:55:01] D [client-protocol.c:7023:notify] node03-4: got GF_EVENT_CHILD_UP [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-1: Connected to 10.135.32.52:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:01] N [afr.c:2625:notify] replicate: Subvolume 'repstore1' came back up; going online. [2010-02-26 14:55:01] D [fuse-bridge.c:3092:fuse_thread_proc] fuse: pthread_cond_timedout returned non zero value ret: -1 errno: 0 [2010-02-26 14:55:01] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.9 [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node02-1: Connected to 10.135.32.51:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:01] N [afr.c:2625:notify] replicate: Subvolume 'repstore1' came back up; going online. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node01-2: Connected to 10.135.32.50:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node01-2: Connected to 10.135.32.50:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-2: Connected to 10.135.32.52:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node04-3: Connected to 10.135.32.53:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node04-3: Connected to 10.135.32.53:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-2: Connected to 10.135.32.52:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node02-2: Connected to 10.135.32.51:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node02-2: Connected to 10.135.32.51:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node01-3: Connected to 10.135.32.50:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node01-3: Connected to 10.135.32.50:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node02-1': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node04-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-3: Connected to 10.135.32.52:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-3: Connected to 10.135.32.52:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node04-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node02-3: Connected to 10.135.32.51:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node02-3: Connected to 10.135.32.51:6996, attached to remote volume 'brick3'. [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-4: Connected to 10.135.32.52:6996, attached to remote volume 'brick4'. [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] N [client-protocol.c:6228:client_setvolume_cbk] node03-4: Connected to 10.135.32.52:6996, attached to remote volume 'brick4'. [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node01-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node01-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node01-1': avail_percent is: 99.00 and avail_space is: 144278073344 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node01-1': avail_percent is: 99.00 and avail_space is: 144278073344 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node01-2': avail_percent is: 99.00 and avail_space is: 144278077440 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node01-2': avail_percent is: 99.00 and avail_space is: 144278077440 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node03-1': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node02-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node03-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node03-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node03-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore3: on subvolume 'node02-3': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:01] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node02-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node03-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node02-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node04-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node04-1: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node04-4: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node04-4: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node03-1: Connected to 10.135.32.52:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node02-1: Connected to 10.135.32.51:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node04-1: Connected to 10.135.32.53:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node04-1: Connected to 10.135.32.53:6996, attached to remote volume 'brick1'. [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node04-4: Connected to 10.135.32.53:6996, attached to remote volume 'brick4'. [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node04-4: Connected to 10.135.32.53:6996, attached to remote volume 'brick4'. [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node03-1': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node02-1': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node04-1': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node04-1': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node04-4': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore1: on subvolume 'node04-4': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node04-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] D [client-protocol.c:7023:notify] node04-2: got GF_EVENT_CHILD_UP [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node04-2: Connected to 10.135.32.53:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:04] N [client-protocol.c:6228:client_setvolume_cbk] node04-2: Connected to 10.135.32.53:6996, attached to remote volume 'brick2'. [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node04-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:04] D [dht-diskusage.c:71:dht_du_info_cbk] repstore2: on subvolume 'node04-2': avail_percent is: 99.00 and avail_space is: 72032043008 [2010-02-26 14:55:15] D [dht-layout.c:576:dht_layout_normalize] repstore2: found anomalies in /. holes=1 overlaps=1 [2010-02-26 14:55:15] D [dht-common.c:164:dht_lookup_dir_cbk] repstore2: fixing assignment on / [2010-02-26 14:55:15] D [dht-layout.c:576:dht_layout_normalize] repstore3: found anomalies in /. holes=1 overlaps=1 [2010-02-26 14:55:15] D [dht-common.c:164:dht_lookup_dir_cbk] repstore3: fixing assignment on / [2010-02-26 14:55:15] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:55:15] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:55:15] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:55:15] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:55:15] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:55:15] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:55:15] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:55:15] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:55:15] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 3: LOOKUP() / => -1 (Stale NFS file handle) [2010-02-26 14:55:30] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:55:30] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:55:30] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:55:30] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:55:30] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:55:30] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:55:30] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:55:30] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:55:30] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 5: LOOKUP() / => -1 (Stale NFS file handle) [2010-02-26 14:55:45] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:55:45] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:55:45] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:55:45] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:55:45] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:55:45] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:55:45] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:55:45] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:55:45] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 7: LOOKUP() / => -1 (Stale NFS file handle) [2010-02-26 14:56:01] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:56:01] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:56:01] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore2: subvol: node02-2; inode layout - 0 - 858993458; disk layout - 858993459 - 1717986917 [2010-02-26 14:56:01] D [dht-common.c:274:dht_revalidate_cbk] repstore2: mismatching layouts for / [2010-02-26 14:56:01] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:56:01] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:56:01] D [dht-layout.c:674:dht_layout_dir_mismatch] repstore3: subvol: node03-3; inode layout - 858993459 - 1717986917; disk layout - 1717986918 - -1717986920 [2010-02-26 14:56:01] D [dht-common.c:274:dht_revalidate_cbk] repstore3: mismatching layouts for / [2010-02-26 14:56:01] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 9: LOOKUP() / => -1 (Stale NFS file handle)