Hi,<br><br>Does client logs contain log messages like "connection timed out"? It might be that statfs call is taking long time to complete on server and the client connection is timing out. Can you attach the client-logfile? Can you also try with larger transport-timeout in protocol/client?<br>
<br>regards,<br><div class="gmail_quote">2009/5/13 "José M. Martín" <span dir="ltr"><<a href="mailto:jmartin@onsager.ugr.es">jmartin@onsager.ugr.es</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello,<br>
<br>
I'm having problems when I set option export-statfs-size to "no" in<br>
brick-posix. Clients can't connect to the servers.<br>
If I comment that option, all works. Is there any bug or am I doing<br>
something wrong?<br>
<br>
I'm using glusterfs-2.0.0.<br>
<br>
Thank you.<br>
<br>
<br>
LOG SERVER<br>
<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]<br>
ns-posix: allowed = "192.168.*", received addr = "192.168.3.56"<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1023" target="_blank">192.168.3.56:1023</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]<br>
ns-posix: allowed = "192.168.*", received addr = "192.168.3.56"<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1022" target="_blank">192.168.3.56:1022</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]<br>
brick: allowed = "192.168.*", received addr = "192.168.3.56"<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1019" target="_blank">192.168.3.56:1019</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]<br>
brick: allowed = "192.168.*", received addr = "192.168.3.56"<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1018" target="_blank">192.168.3.56:1018</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]<br>
stripe: allowed = "192.168.*", received addr = "192.168.3.56"<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1009" target="_blank">192.168.3.56:1009</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [addr.c:174:gf_auth]<br>
stripe: allowed = "192.168.*", received addr = "192.168.3.56"<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1008" target="_blank">192.168.3.56:1008</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[server-protocol.c:5235:server_statfs] brick: 3: STATFS '/ (0)'<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[server-protocol.c:5235:server_statfs] brick: 3: STATFS '/ (0)'<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:90:__socket_rwv] server: EOF from peer <a href="http://192.168.3.56:1023" target="_blank">192.168.3.56:1023</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:562:__socket_proto_state_machine] server: read (Transport<br>
endpoint is not connected) in state 1 (<a href="http://192.168.3.56:1023" target="_blank">192.168.3.56:1023</a>)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1023" target="_blank">192.168.3.56:1023</a> disconnected<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]<br>
server: transport 0x7f50bc000910 destroyed<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:90:__socket_rwv] server: EOF from peer <a href="http://192.168.3.56:1022" target="_blank">192.168.3.56:1022</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:562:__socket_proto_state_machine] server: read (Transport<br>
endpoint is not connected) in state 1 (<a href="http://192.168.3.56:1022" target="_blank">192.168.3.56:1022</a>)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1022" target="_blank">192.168.3.56:1022</a> disconnected<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-helpers.c:723:server_connection_destroy] server: destroyed<br>
connection of calypso56-14634-2009/05/13-10:42:50:315134-ns-remote<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]<br>
server: transport 0x7f50bc000c30 destroyed<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E<br>
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E<br>
[socket.c:562:__socket_proto_state_machine] server: read (Connection<br>
reset by peer) in state 1 (<a href="http://192.168.3.56:1019" target="_blank">192.168.3.56:1019</a>)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1019" target="_blank">192.168.3.56:1019</a> disconnected<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]<br>
server: transport 0x7f50bc000f50 destroyed<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E<br>
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 E<br>
[socket.c:562:__socket_proto_state_machine] server: read (Connection<br>
reset by peer) in state 1 (<a href="http://192.168.3.56:1018" target="_blank">192.168.3.56:1018</a>)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1018" target="_blank">192.168.3.56:1018</a> disconnected<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-helpers.c:723:server_connection_destroy] server: destroyed<br>
connection of calypso56-14634-2009/05/13-10:42:50:315134-brick2<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]<br>
server: transport 0x7f50bc001270 destroyed<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:90:__socket_rwv] server: EOF from peer <a href="http://192.168.3.56:1009" target="_blank">192.168.3.56:1009</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:562:__socket_proto_state_machine] server: read (Transport<br>
endpoint is not connected) in state 1 (<a href="http://192.168.3.56:1009" target="_blank">192.168.3.56:1009</a>)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1009" target="_blank">192.168.3.56:1009</a> disconnected<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]<br>
server: transport 0x7f50bc001590 destroyed<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:90:__socket_rwv] server: EOF from peer <a href="http://192.168.3.56:1008" target="_blank">192.168.3.56:1008</a><br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D<br>
[socket.c:562:__socket_proto_state_machine] server: read (Transport<br>
endpoint is not connected) in state 1 (<a href="http://192.168.3.56:1008" target="_blank">192.168.3.56:1008</a>)<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1008" target="_blank">192.168.3.56:1008</a> disconnected<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 N<br>
[server-helpers.c:723:server_connection_destroy] server: destroyed<br>
connection of calypso56-14634-2009/05/13-10:42:50:315134-stripe2<br>
May 13 10:45:00 nix02 logger: 2009-05-13 10:44:59 D [socket.c:1332:fini]<br>
server: transport 0x7f50bc0018b0 destroyed<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1013" target="_blank">192.168.3.56:1013</a><br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1012" target="_blank">192.168.3.56:1012</a><br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1003" target="_blank">192.168.3.56:1003</a><br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:7502:mop_setvolume] server: accepted client from<br>
<a href="http://192.168.3.56:1002" target="_blank">192.168.3.56:1002</a><br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E<br>
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E<br>
[socket.c:562:__socket_proto_state_machine] server: read (Connection<br>
reset by peer) in state 1 (<a href="http://192.168.3.56:1013" target="_blank">192.168.3.56:1013</a>)<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1013" target="_blank">192.168.3.56:1013</a> disconnected<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E<br>
[socket.c:102:__socket_rwv] server: readv failed (Connection reset by peer)<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 E<br>
[socket.c:562:__socket_proto_state_machine] server: read (Connection<br>
reset by peer) in state 1 (<a href="http://192.168.3.56:1012" target="_blank">192.168.3.56:1012</a>)<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1012" target="_blank">192.168.3.56:1012</a> disconnected<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-helpers.c:723:server_connection_destroy] server: destroyed<br>
connection of calypso56-14634-2009/05/13-10:42:50:315134-brick5<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1003" target="_blank">192.168.3.56:1003</a> disconnected<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-protocol.c:8272:notify] server: <a href="http://192.168.3.56:1002" target="_blank">192.168.3.56:1002</a> disconnected<br>
May 13 10:45:12 nix05 logger: 2009-05-13 10:45:11 N<br>
[server-helpers.c:723:server_connection_destroy] server: destroyed<br>
connection of calypso56-14634-2009/05/13-10:42:50:315134-stripe5<br>
<br>
<br>
<br>
SERVER CONFIG<br>
<br>
### Export volume "brick" with the contents of<br>
"/mnt/raid0/glusterfs/export" directory.<br>
volume brick-posix<br>
type storage/posix # POSIX FS translator<br>
option directory /mnt/raid0/glusterfs/exports # Export this<br>
directory<br>
#When more than one directory is exported from a single mountpoint in<br>
backend<br>
#(ie, there is another volume exporting directory from this mount point), it<br>
#is possible that 'df' on the client shows more diskspace than that of<br>
it actually has.<br>
#In that case, one can say 'no' for exporting statfs-size from redundant<br>
exports.<br>
option export-statfs-size no<br>
end-volume<br>
<br>
volume stripe-posix<br>
type storage/posix<br>
option directory /mnt/raid0/glusterfs/stripe<br>
# option export-statfs-size no<br>
# it was observed that when a very big file (file size > 100GB) is<br>
getting unlinked,<br>
# and the backend storage is of ext3 or some other journaling fs,<br>
glusterfs mountpoint<br>
# becomes un-usable till unlink() call is returned. This can be solved<br>
by enabling this<br>
# option in posix layer. Advised to use it when the file sizes are<br>
serveral GBs.<br>
option background-unlink yes<br>
end-volume<br>
<br>
volume ns-posix<br>
type storage/posix<br>
option directory /mnt/raid0/glusterfs/ns<br>
# option export-statfs-size no<br>
end-volume<br>
<br>
# lock<br>
volume brick-locks<br>
type features/locks<br>
subvolumes brick-posix<br>
end-volume<br>
volume stripe<br>
type features/locks<br>
subvolumes stripe-posix<br>
end-volume<br>
<br>
# io-cache<br>
volume brick-cache<br>
type performance/io-cache<br>
option cache-size 1GB<br>
subvolumes brick-locks<br>
end-volume<br>
<br>
# iothreads<br>
volume brick<br>
type performance/io-threads<br>
option thread-count 16<br>
subvolumes brick-cache<br>
end-volume<br>
<br>
### Add network serving capability to above brick.<br>
volume server<br>
type protocol/server<br>
option transport-type tcp<br>
option transport.socket.listen-port 6993 # Default is 6996<br>
subvolumes brick stripe ns-posix<br>
# NOTE: Access to any volume through protocol/server is denied by<br>
# default. You need to explicitly grant access through # "auth"<br>
# option.<br>
option auth.addr.brick.allow 192.168.* # Allow access to "brick" volume<br>
option auth.addr.stripe.allow 192.168.*<br>
option auth.addr.ns-posix.allow 192.168.*<br>
end-volume<br>
<br>
<br>
<br>
CLIENT CONFIG<br>
<br>
<br>
volume brick1<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume brick # name of the remote volume<br>
option remote-host nix01<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume brick2<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume brick # name of the remote volume<br>
option remote-host nix02<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume brick3<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume brick # name of the remote volume<br>
option remote-host nix03<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume brick4<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume brick # name of the remote volume<br>
option remote-host nix04<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume brick5<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume brick # name of the remote volume<br>
option remote-host nix05<br>
option remote-port 6993<br>
end-volume<br>
<br>
############################################################################<br>
# DHT<br>
#<br>
volume dht<br>
type cluster/distribute<br>
# option lookup-unhashed yes<br>
# option min-free-disk 10%<br>
subvolumes brick1 brick2 brick3 brick4 brick5<br>
end-volume<br>
<br>
############################################################################<br>
# CLIENTE STRIPE<br>
#<br>
volume stripe1<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume stripe # name of the remote volume<br>
option remote-host nix01<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume stripe2<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume stripe # name of the remote volume<br>
option remote-host nix02<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume stripe3<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume stripe # name of the remote volume<br>
option remote-host nix03<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume stripe4<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume stripe # name of the remote volume<br>
option remote-host nix04<br>
option remote-port 6993<br>
end-volume<br>
<br>
volume stripe5<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume stripe # name of the remote volume<br>
option remote-host nix05<br>
option remote-port 6993<br>
end-volume<br>
############################################################################<br>
# STRIPE<br>
#<br>
volume stripe<br>
type cluster/stripe<br>
option block-size 1MB<br>
subvolumes stripe1 stripe2 stripe3 stripe4 stripe5<br>
end-volume<br>
<br>
############################################################################<br>
# NAMESPACE<br>
#<br>
volume ns-remote<br>
type protocol/client<br>
option transport-type tcp # for TCP/IP transport<br>
option remote-subvolume ns-posix # name of the remote volume<br>
option remote-host nix02<br>
option remote-port 6993<br>
end-volume<br>
############################################################################<br>
# UNIFY SWITCH<br>
#<br>
volume bricks<br>
type cluster/unify<br>
option scheduler switch<br>
option namespace ns-remote<br>
option self-heal off<br>
option switch.case *.big*:stripe;*:dht<br>
subvolumes dht stripe<br>
end-volume<br>
<br>
volume writebehind<br>
type performance/write-behind<br>
option aggregate-size 128KB # default is 0bytes<br>
option cache-size 1MB # default is equal to aggregate-size<br>
option flush-behind on # default is 'off'<br>
subvolumes bricks<br>
end-volume<br>
<br>
volume readahead<br>
type performance/read-ahead<br>
option page-size 1024kB # 256KB is the default option<br>
option page-count 32 # 2 is default option<br>
option force-atime-update off # default is off<br>
subvolumes writebehind<br>
end-volume<br>
<br>
volume iothreads<br>
type performance/io-threads<br>
option thread-count 8 # deault is 1<br>
subvolumes readahead<br>
end-volume<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>