<div dir="ltr"><div class="gmail_extra"><div><div>On Sat, Mar 23, 2013 at 9:08 AM, Brian Foster <span dir="ltr"><<a href="mailto:bfoster@redhat.com" target="_blank">bfoster@redhat.com</a>></span> wrote:<br></div></div>
<div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">On 03/22/2013 05:11 PM, Patrick Regan wrote:<br>
> I have a 8-node 2 replica Gluster volume mounted with the fuse client.<br>
> We also have an in-house Perl script we use for doing block string<br>
> substitutions. If we run this script on directories on the volume, I<br>
> get the following error on almost every file:<br>
><br>
> cp: skipping file $FILEPATH as it was replaced while being copied<br>
><br>
<br>
</div>That error is preceded with the following comment in coreutils:<br>
<br>
/* Compare the source dev/ino from the open file to the incoming,<br>
saved ones obtained via a previous call to stat. */<br>
<br>
... which basically means the source inode has changed between the time<br>
this particular function runs a stat on the source path and some<br>
previous call had run the same thing. Had you confirmed which cp command<br>
in your script results in this error (e.g., it looks like you have a cp<br>
to temp and a cp to orig)?<br></blockquote><div><br></div><div style>Actually both cp's cause the error to occur. Both the copy to the temp, and the copy from the temp to the original.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I know nothing about perl (and the lack of indentation in the script<br>
post is making my eyes cross :P), but have you thought about the level<br>
of serialization you're getting in the perl script between the cp<br>
commands and programmatic file access? In other words, is it possible<br>
the work your script invokes via print and `` operators is racing with<br>
the script itself?<br></blockquote><div><br></div><div style>My apologies for the lack of indentation. I should have known whitespace would have gotten stripped. Here are links to both the perl and the shell scripts:</div>
<div style><br></div><div style><a href="http://pastebin.com/Q9mxSjS0">http://pastebin.com/Q9mxSjS0</a> - Perl<br></div><div style><a href="http://pastebin.com/RaS3cxQa">http://pastebin.com/RaS3cxQa</a> - Bash</div><div style>
<br></div><div style>I too am not that much of a Perl expert, so I'll ask a few of my developers here. I'll report back with anything else I can find.</div><div style><br></div><div style>Even if it was a race condition, why does it happen with the FUSE client, but not NFS?</div>
<div style><br></div><div style>Thanks,</div><div style>Pat</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Brian<br>
<div><div class="h5"><br>
> If I mount the volume with NFS, this issue does not arise.<br>
><br>
> I have also tried to replicate the issue using a roughly equivalent<br>
> shell script, but the shell script does not produce the same result on<br>
> either the fuse or the nfs client.<br>
><br>
> I'll paste my volume log followed by the perl script, followed by my<br>
> rough shell script.<br>
><br>
> I would appreciate any feedback.<br>
><br>
> Thanks!<br>
><br>
> -------------<br>
><br>
> [2013-03-22 16:32:52.893362] I [glusterfsd.c:1666:main]<br>
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version<br>
> 3.3.1<br>
> [2013-03-22 16:32:52.932157] I [io-cache.c:1549:check_cache_size_ok]<br>
> 0-usrweb-quick-read: Max cache size is 2124763136<br>
> [2013-03-22 16:32:52.932634] I [io-cache.c:1549:check_cache_size_ok]<br>
> 0-usrweb-io-cache: Max cache size is 2124763136<br>
> [2013-03-22 16:32:53.035759] I [client.c:2142:notify]<br>
> 0-usrweb-client-0: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:32:53.041228] I [client.c:2142:notify]<br>
> 0-usrweb-client-1: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:32:53.046270] I [client.c:2142:notify]<br>
> 0-usrweb-client-2: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:32:53.050863] I [client.c:2142:notify]<br>
> 0-usrweb-client-3: parent translators are ready, attempting connect on<br>
> transport<br>
> Given volfile:<br>
> +------------------------------------------------------------------------------+<br>
> 1: volume usrweb-client-0<br>
> 2: type protocol/client<br>
> 3: option remote-host ak001<br>
> 4: option remote-subvolume /srv/gluster/volusrweb<br>
> 5: option transport-type tcp<br>
> 6: end-volume<br>
> 7:<br>
> 8: volume usrweb-client-1<br>
> 9: type protocol/client<br>
> 10: option remote-host ak002<br>
> 11: option remote-subvolume /srv/gluster/volusrweb<br>
> 12: option transport-type tcp<br>
> 13: end-volume<br>
> 14:<br>
> 15: volume usrweb-client-2<br>
> 16: type protocol/client<br>
> 17: option remote-host ak003<br>
> 18: option remote-subvolume /srv/gluster/volusrweb<br>
> 19: option transport-type tcp<br>
> 20: end-volume<br>
> 21:<br>
> 22: volume usrweb-client-3<br>
> 23: type protocol/client<br>
> 24: option remote-host ak004<br>
> 25: option remote-subvolume /srv/gluster/volusrweb<br>
> 26: option transport-type tcp<br>
> 27: end-volume<br>
> 28:<br>
> 29: volume usrweb-replicate-0<br>
> 30: type cluster/replicate<br>
> 31: subvolumes usrweb-client-0 usrweb-client-1<br>
> 32: end-volume<br>
> 33:<br>
> 34: volume usrweb-replicate-1<br>
> 35: type cluster/replicate<br>
> 36: subvolumes usrweb-client-2 usrweb-client-3<br>
> 37: end-volume<br>
> 38:<br>
> 39: volume usrweb-dht<br>
> 40: type cluster/distribute<br>
> 41: subvolumes usrweb-replicate-0 usrweb-replicate-1<br>
> 42: end-volume<br>
> 43:<br>
> 44: volume usrweb-write-behind<br>
> 45: type performance/write-behind<br>
> 46: subvolumes usrweb-dht<br>
> 47: end-volume<br>
> 48:<br>
> 49: volume usrweb-read-ahead<br>
> 50: type performance/read-ahead<br>
> 51: subvolumes usrweb-write-behind<br>
> 52: end-volume<br>
> 53:<br>
> 54: volume usrweb-io-cache<br>
> 55: type performance/io-cache<br>
> 56: subvolumes usrweb-read-ahead<br>
> 57: end-volume<br>
> 58:<br>
> 59: volume usrweb-quick-read<br>
> 60: type performance/quick-read<br>
> 61: subvolumes usrweb-io-cache<br>
> 62: end-volume<br>
> 63:<br>
> 64: volume usrweb-md-cache<br>
> 65: type performance/md-cache<br>
> 66: subvolumes usrweb-quick-read<br>
> 67: end-volume<br>
> 68:<br>
> 69: volume usrweb<br>
> 70: type debug/io-stats<br>
> 71: option latency-measurement off<br>
> 72: option count-fop-hits off<br>
> 73: subvolumes usrweb-md-cache<br>
> 74: end-volume<br>
><br>
> +------------------------------------------------------------------------------+<br>
> [2013-03-22 16:32:53.057710] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 0-usrweb-client-2: changing port to 24010 (from 0)<br>
> [2013-03-22 16:32:53.057895] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 0-usrweb-client-1: changing port to 24010 (from 0)<br>
> [2013-03-22 16:32:53.058025] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 0-usrweb-client-0: changing port to 24010 (from 0)<br>
> [2013-03-22 16:32:53.059259] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 0-usrweb-client-3: changing port to 24010 (from 0)<br>
> [2013-03-22 16:32:56.933277] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 0-usrweb-client-2: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:32:56.933966] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 0-usrweb-client-2:<br>
> Connected to <a href="http://10.0.8.242:24010" target="_blank">10.0.8.242:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:32:56.934012] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 0-usrweb-client-2:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:32:56.934161] I [afr-common.c:3628:afr_notify]<br>
> 0-usrweb-replicate-1: Subvolume 'usrweb-client-2' came back up; going<br>
> online.<br>
> [2013-03-22 16:32:56.934400] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 0-usrweb-client-2:<br>
> Server lk version = 1<br>
> [2013-03-22 16:32:56.938890] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 0-usrweb-client-1: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:32:56.939504] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 0-usrweb-client-1:<br>
> Connected to <a href="http://10.0.8.241:24010" target="_blank">10.0.8.241:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:32:56.939553] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 0-usrweb-client-1:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:32:56.939685] I [afr-common.c:3628:afr_notify]<br>
> 0-usrweb-replicate-0: Subvolume 'usrweb-client-1' came back up; going<br>
> online.<br>
> [2013-03-22 16:32:56.939940] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 0-usrweb-client-1:<br>
> Server lk version = 1<br>
> [2013-03-22 16:32:56.944004] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 0-usrweb-client-0: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:32:56.944408] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 0-usrweb-client-0:<br>
> Connected to <a href="http://10.0.8.240:24010" target="_blank">10.0.8.240:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:32:56.944502] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 0-usrweb-client-0:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:32:56.945219] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 0-usrweb-client-0:<br>
> Server lk version = 1<br>
> [2013-03-22 16:32:56.949911] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 0-usrweb-client-3: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:32:56.950480] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 0-usrweb-client-3:<br>
> Connected to <a href="http://10.0.8.243:24010" target="_blank">10.0.8.243:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:32:56.950524] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 0-usrweb-client-3:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:32:56.998230] I [fuse-bridge.c:4191:fuse_graph_setup]<br>
> 0-fuse: switched to graph 0<br>
> [2013-03-22 16:32:56.998706] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 0-usrweb-client-3:<br>
> Server lk version = 1<br>
> [2013-03-22 16:32:56.999145] I [fuse-bridge.c:3376:fuse_init]<br>
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13<br>
> kernel 7.10<br>
> [2013-03-22 16:32:57.000718] I<br>
> [afr-common.c:1965:afr_set_root_inode_on_first_lookup]<br>
> 0-usrweb-replicate-0: added root inode<br>
> [2013-03-22 16:32:57.001914] I<br>
> [afr-common.c:1965:afr_set_root_inode_on_first_lookup]<br>
> 0-usrweb-replicate-1: added root inode<br>
> [2013-03-22 16:42:00.074968] I [glusterfsd-mgmt.c:65:mgmt_cbk_spec]<br>
> 0-mgmt: Volume file changed<br>
> [2013-03-22 16:42:02.114686] I [io-cache.c:1549:check_cache_size_ok]<br>
> 1-usrweb-quick-read: Max cache size is 2124763136<br>
> [2013-03-22 16:42:02.114945] I [io-cache.c:1549:check_cache_size_ok]<br>
> 1-usrweb-io-cache: Max cache size is 2124763136<br>
> [2013-03-22 16:42:02.265593] I [client.c:2142:notify]<br>
> 1-usrweb-client-0: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.270654] I [client.c:2142:notify]<br>
> 1-usrweb-client-1: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.275012] I [client.c:2142:notify]<br>
> 1-usrweb-client-2: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.279133] I [client.c:2142:notify]<br>
> 1-usrweb-client-3: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.283267] I [client.c:2142:notify]<br>
> 1-usrweb-client-4: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.287310] I [client.c:2142:notify]<br>
> 1-usrweb-client-5: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.291472] I [client.c:2142:notify]<br>
> 1-usrweb-client-6: parent translators are ready, attempting connect on<br>
> transport<br>
> [2013-03-22 16:42:02.295771] I [client.c:2142:notify]<br>
> 1-usrweb-client-7: parent translators are ready, attempting connect on<br>
> transport<br>
> Given volfile:<br>
> +------------------------------------------------------------------------------+<br>
> 1: volume usrweb-client-0<br>
> 2: type protocol/client<br>
> 3: option remote-host ak001<br>
> 4: option remote-subvolume /srv/gluster/volusrweb<br>
> 5: option transport-type tcp<br>
> 6: end-volume<br>
> 7:<br>
> 8: volume usrweb-client-1<br>
> 9: type protocol/client<br>
> 10: option remote-host ak002<br>
> 11: option remote-subvolume /srv/gluster/volusrweb<br>
> 12: option transport-type tcp<br>
> 13: end-volume<br>
> 14:<br>
> 15: volume usrweb-client-2<br>
> 16: type protocol/client<br>
> 17: option remote-host ak003<br>
> 18: option remote-subvolume /srv/gluster/volusrweb<br>
> 19: option transport-type tcp<br>
> 20: end-volume<br>
> 21:<br>
> 22: volume usrweb-client-3<br>
> 23: type protocol/client<br>
> 24: option remote-host ak004<br>
> 25: option remote-subvolume /srv/gluster/volusrweb<br>
> 26: option transport-type tcp<br>
> 27: end-volume<br>
> 28:<br>
> 29: volume usrweb-client-4<br>
> 30: type protocol/client<br>
> 31: option remote-host ak005<br>
> 32: option remote-subvolume /srv/gluster/volusrweb<br>
> 33: option transport-type tcp<br>
> 34: end-volume<br>
> 35:<br>
> 36: volume usrweb-client-5<br>
> 37: type protocol/client<br>
> 38: option remote-host ak006<br>
> 39: option remote-subvolume /srv/gluster/volusrweb<br>
> 40: option transport-type tcp<br>
> 41: end-volume<br>
> 42:<br>
> 43: volume usrweb-client-6<br>
> 44: type protocol/client<br>
> 45: option remote-host ak007<br>
> 46: option remote-subvolume /srv/gluster/volusrweb<br>
> 47: option transport-type tcp<br>
> 48: end-volume<br>
> 49:<br>
> 50: volume usrweb-client-7<br>
> 51: type protocol/client<br>
> 52: option remote-host ak008<br>
> 53: option remote-subvolume /srv/gluster/volusrweb<br>
> 54: option transport-type tcp<br>
> 55: end-volume<br>
> 56:<br>
> 57: volume usrweb-replicate-0<br>
> 58: type cluster/replicate<br>
> 59: subvolumes usrweb-client-0 usrweb-client-1<br>
> 60: end-volume<br>
> 61:<br>
> 62: volume usrweb-replicate-1<br>
> 63: type cluster/replicate<br>
> 64: subvolumes usrweb-client-2 usrweb-client-3<br>
> 65: end-volume<br>
> 66:<br>
> 67: volume usrweb-replicate-2<br>
> 68: type cluster/replicate<br>
> 69: subvolumes usrweb-client-4 usrweb-client-5<br>
> 70: end-volume<br>
> 71:<br>
> 72: volume usrweb-replicate-3<br>
> 73: type cluster/replicate<br>
> 74: subvolumes usrweb-client-6 usrweb-client-7<br>
> 75: end-volume<br>
> 76:<br>
> 77: volume usrweb-dht<br>
> 78: type cluster/distribute<br>
> 79: subvolumes usrweb-replicate-0 usrweb-replicate-1<br>
> usrweb-replicate-2 usrweb-replicate-3<br>
> 80: end-volume<br>
> 81:<br>
> 82: volume usrweb-write-behind<br>
> 83: type performance/write-behind<br>
> 84: subvolumes usrweb-dht<br>
> 85: end-volume<br>
> 86:<br>
> 87: volume usrweb-read-ahead<br>
> 88: type performance/read-ahead<br>
> 89: subvolumes usrweb-write-behind<br>
> 90: end-volume<br>
> 91:<br>
> 92: volume usrweb-io-cache<br>
> 93: type performance/io-cache<br>
> 94: subvolumes usrweb-read-ahead<br>
> 95: end-volume<br>
> 96:<br>
> 97: volume usrweb-quick-read<br>
> 98: type performance/quick-read<br>
> 99: subvolumes usrweb-io-cache<br>
> 100: end-volume<br>
> 101:<br>
> 102: volume usrweb-md-cache<br>
> 103: type performance/md-cache<br>
> 104: subvolumes usrweb-quick-read<br>
> 105: end-volume<br>
> 106:<br>
> 107: volume usrweb<br>
> 108: type debug/io-stats<br>
> 109: option latency-measurement off<br>
> 110: option count-fop-hits off<br>
> 111: subvolumes usrweb-md-cache<br>
> 112: end-volume<br>
><br>
> +------------------------------------------------------------------------------+<br>
> [2013-03-22 16:42:02.301090] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-0: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:04.158794] E<br>
> [client-handshake.c:1717:client_query_portmap_cbk] 1-usrweb-client-5:<br>
> failed to get the port number for remote subvolume<br>
> [2013-03-22 16:42:04.159003] I [client.c:2090:client_rpc_notify]<br>
> 1-usrweb-client-5: disconnected<br>
> [2013-03-22 16:42:04.159074] E<br>
> [client-handshake.c:1717:client_query_portmap_cbk] 1-usrweb-client-7:<br>
> failed to get the port number for remote subvolume<br>
> [2013-03-22 16:42:04.159223] I [client.c:2090:client_rpc_notify]<br>
> 1-usrweb-client-7: disconnected<br>
> [2013-03-22 16:42:04.161765] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-3: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:04.161868] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-2: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:04.162355] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-1: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:04.162449] E<br>
> [client-handshake.c:1717:client_query_portmap_cbk] 1-usrweb-client-6:<br>
> failed to get the port number for remote subvolume<br>
> [2013-03-22 16:42:04.162529] E<br>
> [client-handshake.c:1717:client_query_portmap_cbk] 1-usrweb-client-4:<br>
> failed to get the port number for remote subvolume<br>
> [2013-03-22 16:42:04.162625] I [client.c:2090:client_rpc_notify]<br>
> 1-usrweb-client-4: disconnected<br>
> [2013-03-22 16:42:04.162708] E [afr-common.c:3665:afr_notify]<br>
> 1-usrweb-replicate-2: All subvolumes are down. Going offline until<br>
> atleast one of them comes back up.<br>
> [2013-03-22 16:42:04.162765] I [client.c:2090:client_rpc_notify]<br>
> 1-usrweb-client-6: disconnected<br>
> [2013-03-22 16:42:04.162805] E [afr-common.c:3665:afr_notify]<br>
> 1-usrweb-replicate-3: All subvolumes are down. Going offline until<br>
> atleast one of them comes back up.<br>
> [2013-03-22 16:42:05.715027] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-0: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:05.715447] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-0:<br>
> Connected to <a href="http://10.0.8.240:24010" target="_blank">10.0.8.240:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:05.715501] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-0:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:05.715767] I [afr-common.c:3628:afr_notify]<br>
> 1-usrweb-replicate-0: Subvolume 'usrweb-client-0' came back up; going<br>
> online.<br>
> [2013-03-22 16:42:05.715836] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-0:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:07.722528] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-5: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:07.727863] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-7: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:07.732813] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-3: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:07.733314] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-3:<br>
> Connected to <a href="http://10.0.8.243:24010" target="_blank">10.0.8.243:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:07.733358] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-3:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:07.733524] I [afr-common.c:3628:afr_notify]<br>
> 1-usrweb-replicate-1: Subvolume 'usrweb-client-3' came back up; going<br>
> online.<br>
> [2013-03-22 16:42:07.733731] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-3:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:07.738423] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-2: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:07.738819] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-2:<br>
> Connected to <a href="http://10.0.8.242:24010" target="_blank">10.0.8.242:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:07.738860] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-2:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:07.739276] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-2:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:07.744321] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-1: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:07.744874] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-1:<br>
> Connected to <a href="http://10.0.8.241:24010" target="_blank">10.0.8.241:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:07.744916] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-1:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:07.798486] I [fuse-bridge.c:4191:fuse_graph_setup]<br>
> 0-fuse: switched to graph 1<br>
> [2013-03-22 16:42:07.799081] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-1:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:07.799444] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-6: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:07.799591] I [rpc-clnt.c:1657:rpc_clnt_reconfig]<br>
> 1-usrweb-client-4: changing port to 24010 (from 0)<br>
> [2013-03-22 16:42:10.771491] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-5: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:10.776211] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-5:<br>
> Connected to <a href="http://10.0.8.245:24010" target="_blank">10.0.8.245:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:10.776323] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-5:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:10.776500] I [afr-common.c:3628:afr_notify]<br>
> 1-usrweb-replicate-2: Subvolume 'usrweb-client-5' came back up; going<br>
> online.<br>
> [2013-03-22 16:42:10.776836] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-5:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:10.777113] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-7: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:10.777615] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-7:<br>
> Connected to <a href="http://10.0.8.247:24010" target="_blank">10.0.8.247:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:10.777676] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-7:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:10.777866] I [afr-common.c:3628:afr_notify]<br>
> 1-usrweb-replicate-3: Subvolume 'usrweb-client-7' came back up; going<br>
> online.<br>
> [2013-03-22 16:42:10.778129] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-7:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:11.784150] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-6: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:11.787025] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-6:<br>
> Connected to <a href="http://10.0.8.246:24010" target="_blank">10.0.8.246:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:11.787141] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-6:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:11.788193] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-6:<br>
> Server lk version = 1<br>
> [2013-03-22 16:42:11.789908] I<br>
> [client-handshake.c:1636:select_server_supported_programs]<br>
> 1-usrweb-client-4: Using Program GlusterFS 3.3.1, Num (1298437),<br>
> Version (330)<br>
> [2013-03-22 16:42:11.790334] I<br>
> [client-handshake.c:1433:client_setvolume_cbk] 1-usrweb-client-4:<br>
> Connected to <a href="http://10.0.8.244:24010" target="_blank">10.0.8.244:24010</a>, attached to remote volume<br>
> '/srv/gluster/volusrweb'.<br>
> [2013-03-22 16:42:11.790398] I<br>
> [client-handshake.c:1445:client_setvolume_cbk] 1-usrweb-client-4:<br>
> Server and Client lk-version numbers are not same, reopening the fds<br>
> [2013-03-22 16:42:11.790726] I<br>
> [client-handshake.c:453:client_set_lk_version_cbk] 1-usrweb-client-4:<br>
> Server lk version = 1<br>
><br>
> ------------------------------<br>
><br>
> #!/usr/bin/perl<br>
><br>
> open(FIND, "find . -print |") || die "Coundn't run find: $!\n";<br>
><br>
> $counter = 0;<br>
> FILE:<br>
> while ($filename = <FIND>)<br>
> {<br>
> chop $filename;<br>
> next FILE unless -T$filename;<br>
> if(!open(TEXTFILE, "$filename"))<br>
> {<br>
> print STDERR "Can't open $filename - continuing...\n";<br>
> next FILE;<br>
> }<br>
> while (<TEXTFILE>)<br>
> {<br>
> $tempfile[$counter] = $_;<br>
> $counter = $counter + 1;<br>
> if (index($tempfile[$counter - 1],$ARGV[0]) >= 0)<br>
> {<br>
> $totalchanges = $totalchanges +1;<br>
> $dummy=`cp -p $filename blocktemp.txt\n`;<br>
> print $dummy;<br>
> print "$filename\n";<br>
> open(TEMPFILE, "> blocktemp.txt");<br>
> $tempfile[$counter -1] =~ s/$ARGV[0]/$ARGV[1]/g;<br>
> for($i = 0; $i < $counter; $i++)<br>
> {<br>
> print TEMPFILE "$tempfile[$i]";<br>
> }<br>
> while (<TEXTFILE>)<br>
> {<br>
> $storage = $_;<br>
> if (index($storage,$ARGV[0]) >= 0)<br>
> {<br>
> $storage =~ s/$ARGV[0]/$ARGV[1]/g;<br>
> $totalchanges = $totalchanges +1;<br>
> }<br>
> print TEMPFILE "$storage";<br>
> }<br>
> close(TEXTFILE);<br>
> close(TEMPFILE);<br>
> $dummy2=`cp -p blocktemp.txt $filename`;<br>
> print $dummy2;<br>
> $dummy3=`rm blocktemp.txt`;<br>
> print $dummy3;<br>
> }<br>
> }<br>
> close(TEXTFILE);<br>
> $counter = 0;<br>
> }<br>
> print "$totalchanges lines changed\n";<br>
><br>
> ---------------------------<br>
><br>
> #!/bin/bash<br>
><br>
> FILES=$(find . -type f)<br>
><br>
> for f in $FILES<br>
> do<br>
> cp -p $f tempfile.tmp<br>
> sed -i "s/$1/$2/g" tempfile.tmp<br>
> cp -p tempfile.tmp $f<br>
> rm tempfile.tmp<br>
> done<br>
><br>
> --<br>
> Patrick Regan<br>
</div></div>> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
><br>
<br>
</blockquote></div><br></div></div>