<div dir="ltr">Hi posting to the user list as well since I think there are 2 different issues.<br><br>1. Mount point gets dropped after 12-24 hours, seems like the connection gets stale since if you do an ls or such it hangs "forever".<br>
<br>2. Bad client config which spits out a lot of EIO's. We are onto this and will fix the config asap.<br><br>However it is the no1 I'm really concerned about. We did heavy loadtests with IOZone with the bad config (it actually works but unify does not like it) and we got no errors from IOZone, to the contrary we got quite nice throughput!<br>
<br>What would happen to glusterfs if the network between client and master goes away for just a sec once a day ? I'm suspecting that this could be an issue. it would be nice if glusterfs could "auto remount" like NFS.<br>
<br>Kindly<br><br>//Marcus<br><br><br><br><br><br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Marcus Herou</b> <span dir="ltr"><<a href="mailto:marcus.herou@tailsweep.com">marcus.herou@tailsweep.com</a>></span><br>
Date: Sun, Sep 14, 2008 at 1:43 PM<br>Subject: Re: {Disarmed} Re: [Gluster-devel] GlusterFS drops mount point<br>To: "Amar S. Tumballi" <<a href="mailto:amar@zresearch.com">amar@zresearch.com</a>><br>Cc: Brian Taber <<a href="mailto:btaber@diversecg.com">btaber@diversecg.com</a>>, <a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<br><br><div dir="ltr">Thanks a bunch!<br><br>So this would lead to that the mount point would "loose" it's connection ?<br><br>Kindly<br><font color="#888888"><br>//Marcus</font><div><div></div><div class="Wj3C7c">
<br><br><br><div class="gmail_quote">On Sat, Sep 13, 2008 at 7:49 PM, Amar S. Tumballi <span dir="ltr"><<a href="mailto:amar@zresearch.com" target="_blank">amar@zresearch.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div dir="ltr">this will always lead to EIO as it fails to meet the criteria for unify's functioning.<br>
<br>Unify wants a file to be present only on one of its subvolumes, and in this case, you have done afr of (v1 v2), (v2 v3), (v3 v1), which means, if a file is present on (v1 v2) pair, it will be seen by other two afrs too, (v2 in second pair, and v1 in third pair), so unify sees file to be present on all of its subvolume, and gets confused which file to open, and returns EIO.<br>
<br>the fix is, you need to export two volumes (instead of currently present 1) per server, and make pairs of (v1-1 v2-2), (v2-1 v3-2) (v3-1 v1-2), hope i am clear<br><br>Regards,<div><div></div><div><br>
<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div dir="ltr"><br>Client:<br>volume v1<br> type protocol/client<br> option transport-type tcp/client<br> option remote-host <a href="http://192.168.10.30" target="_blank">192.168.10.30</a><br> option remote-subvolume home<br>
end-volume<br><br>volume v2<br>
type protocol/client<br> option transport-type tcp/client<br> option remote-host <a href="http://192.168.10.31" target="_blank">192.168.10.31</a><br> option remote-subvolume home<br>end-volume<br><br>volume v3<br> type protocol/client<br>
option transport-type tcp/client<br> option remote-host <a href="http://192.168.10.32" target="_blank">192.168.10.32</a><br> option remote-subvolume home<br>end-volume<br><br>volume afr-1<br> type cluster/afr<br> subvolumes v1 v2<br>
end-volume<br><br>volume afr-2<br> type cluster/afr<br> subvolumes v2 v3<br>end-volume<br><br>volume afr-3<br> type cluster/afr<br> subvolumes v3 v1<br>end-volume<br><br>volume ns1<br> type protocol/client<br> option transport-type tcp/client<br>
option remote-host <a href="http://192.168.10.30" target="_blank">192.168.10.30</a><br> option remote-subvolume home-namespace<br>end-volume<br><br>volume ns2<br> type protocol/client<br> option transport-type tcp/client<br>
option remote-host <a href="http://192.168.10.31" target="_blank">192.168.10.31</a><br>
option remote-subvolume home-namespace<br>end-volume<br><br>volume ns3<br> type protocol/client<br> option transport-type tcp/client<br> option remote-host <a href="http://192.168.10.32" target="_blank">192.168.10.32</a><br>
option remote-subvolume home-namespace<br>
end-volume<br><br>volume namespace<br> type cluster/afr<br> subvolumes ns1 ns2 ns3<br>end-volume<br><br>volume v<br> type cluster/unify<br> option scheduler rr<br> option namespace namespace<br> subvolumes afr-1 afr-2 afr-3<br>
end-volume<br><br>I really hope we have misconfigured something since that is the easiest fix :)<br><br>Kindly<br><font color="#888888"><br>//Marcus</font><div><div></div><div><br><br><br><br><div class="gmail_quote">
On Sat, Sep 13, 2008 at 12:50 AM, Amar S. Tumballi <span dir="ltr"><<a href="mailto:amar@zresearch.com" target="_blank">amar@zresearch.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div dir="ltr">Also which version of GlusterFS? <br><br><div class="gmail_quote"><span dir="ltr"><a href="mailto:ber@diversecg.com" target="_blank">ber@diversecg.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>
may be configuration issue... lets start with config, what does you config look like on client and server?<br>
<br>
Marcus Herou wrote:<br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>
Lots of these on server<br></div>
2008-09-12 20:48:14 E [protocol.c:271:gf_block_unserialize_transport] server: EOF from peer (*MailScanner has detected a possible fraud attempt from "<a href="http://192.168.10.4:1007" target="_blank">192.168.10.4:1007</a>" claiming to be* *MailScanner warning: numerical links are often malicious:* <a href="http://192.168.10.4:1007" target="_blank">192.168.10.4:1007</a> <<a href="http://192.168.10.4:1007" target="_blank">http://192.168.10.4:1007</a>>)<div>
<br>
...<br>
2008-09-12 20:50:12 E [server-protocol.c:4153:server_closedir] server: not getting enough data, returning EINVAL<br>
...<br>
2008-09-12 20:50:12 E [server-protocol.c:4148:server_closedir] server: unresolved fd 6<br>
...<br></div>
2008-09-12 20:51:47 E [protocol.c:271:gf_block_unserialize_transport] server: EOF from peer (*MailScanner has detected a possible fraud attempt from "<a href="http://192.168.10.10:1015" target="_blank">192.168.10.10:1015</a>" claiming to be* *MailScanner warning: numerical links are often malicious:* <a href="http://192.168.10.10:1015" target="_blank">192.168.10.10:1015</a> <<a href="http://192.168.10.10:1015" target="_blank">http://192.168.10.10:1015</a>>)<div>
<div></div><div><br>
...<br>
<br>
And lots of these on client<br>
<br>
2008-09-12 19:54:45 E [afr.c:2201:afr_open] home-namespace: self heal failed, returning EIO<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3954: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3956: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3958: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3987: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3989: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3991: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:45 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3993: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 C [client-protocol.c:212:call_bail] home3: bailing transport<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3970: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3971: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3972: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 3974: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 4001: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 4002: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:54:54 E [client-protocol.c:4827:client_protocol_cleanup] home3: forced unwinding frame type(2) op(5) reply=@0x809abb0<br>
2008-09-12 19:54:54 E [client-protocol.c:4239:client_lock_cbk] home3: no proper reply from server, returning ENOTCONN<br>
2008-09-12 19:54:54 E [afr.c:1933:afr_selfheal_lock_cbk] home-afr-3: (path=/rsyncer/.ssh/authorized_keys2 child=home3) op_ret=-1 op_errno=107<br>
2008-09-12 19:54:54 E [afr.c:2201:afr_open] home-afr-3: self heal failed, returning EIO<br>
2008-09-12 19:54:54 E [fuse-bridge.c:715:fuse_fd_cbk] glusterfs-fuse: 4004: (12) /rsyncer/.ssh/authorized_keys2 => -1 (5)<br>
2008-09-12 19:55:01 E [unify.c:335:unify_lookup] home: returning ESTALE for /rsyncer/.ssh/authorized_keys2: file count is 4<br>
2008-09-12 19:55:01 E [unify.c:339:unify_lookup] home: /rsyncer/.ssh/authorized_keys2: found on home-namespace<br>
2008-09-12 19:55:01 E [unify.c:339:unify_lookup] home: /rsyncer/.ssh/authorized_keys2: found on home-afr-2<br>
2008-09-12 19:55:01 E [unify.c:339:unify_lookup] home: /rsyncer/.ssh/authorized_keys2: found on home-afr-1<br>
2008-09-12 19:55:01 E [unify.c:339:unify_lookup] home: /rsyncer/.ssh/authorized_keys2: found on home-afr-3<br>
<br>
<br>
Both server and client are spitting out tons of these. Thought "E" was Error level, seems like DEBUG ?<br>
<br>
Kindly<br>
<br>
//Marcus<br>
<br>
<br>
<br>
<br></div></div><div>
On Fri, Sep 12, 2008 at 8:01 PM, Brian Taber <<a href="mailto:btaber@diversecg.com" target="_blank">btaber@diversecg.com</a> <mailto:<a href="mailto:btaber@diversecg.com" target="_blank">btaber@diversecg.com</a>>> wrote:<br>
<br>
What do you see in your server and client logs for gluster?<br>
<br>
-------------------------<br>
Brian Taber<br>
Owner/IT Specialist<br>
Diverse Computer Group<br>
Office: 774-206-5592<br>
Cell: 508-496-9221<br></div>
<a href="mailto:btaber@diversecg.com" target="_blank">btaber@diversecg.com</a> <mailto:<a href="mailto:btaber@diversecg.com" target="_blank">btaber@diversecg.com</a>><div><br>
<br>
<br>
<br>
<br>
Marcus Herou wrote:<br>
> Hi.<br>
><br>
> We have just recently installed a 3 node cluster with 16 SATA<br>
disks each.<br>
><br>
> We are using Hardy and the glusterfs-3.10 Ubuntu package on both<br>
client(s)<br>
> and server.<br>
><br>
> We have only created one export (/home) yet since we want to<br>
test it a while<br>
> before putting it into a live high performance environment.<br>
><br>
> The problem is currently that the client looses /home once a day<br>
or so. This<br>
> is really bad since this is a machine which all other connect to<br>
with ssh<br>
> keys thus making them unable to log in.<br>
><br>
> Anyone seen something similar ?<br>
><br>
> Kindly<br>
><br>
> //Marcus<br>
> _______________________________________________<br>
> Gluster-devel mailing list<br></div>
> <a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a> <mailto:<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a>><div><br>
> <a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
><br>
<br>
<br>
<br>
<br>
-- <br>
Marcus Herou CTO and co-founder Tailsweep AB<br>
+46702561312<br>
</div><a href="mailto:marcus.herou@tailsweep.com" target="_blank">marcus.herou@tailsweep.com</a> <mailto:<a href="mailto:marcus.herou@tailsweep.com" target="_blank">marcus.herou@tailsweep.com</a>><div>
<br>
<a href="http://www.tailsweep.com/" target="_blank">http://www.tailsweep.com/</a><br>
<a href="http://blogg.tailsweep.com/" target="_blank">http://blogg.tailsweep.com/</a><br>
</div></blockquote>
_______________________________________________<div><br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></blockquote></div><br><br clear="all"><br>-- <br>Amar Tumballi<br>Gluster/GlusterFS Hacker<br>[bulde on #gluster/<a href="http://irc.gnu.org" target="_blank">irc.gnu.org</a>]<br><a href="http://www.zresearch.com" target="_blank">http://www.zresearch.com</a> - Commoditizing Super Storage!<br>
</div>
</blockquote></div><br><br clear="all"><br>-- <br>Marcus Herou CTO and co-founder Tailsweep AB<br>+46702561312<br><a href="mailto:marcus.herou@tailsweep.com" target="_blank">marcus.herou@tailsweep.com</a><br></div></div>
<div><a href="http://www.tailsweep.com/" target="_blank">http://www.tailsweep.com/</a><br>
<a href="http://blogg.tailsweep.com/" target="_blank">http://blogg.tailsweep.com/</a><br>
</div></div>
</blockquote></div><br><br clear="all"><br>-- <br>Amar Tumballi<br>Gluster/GlusterFS Hacker<br>[bulde on #gluster/<a href="http://irc.gnu.org" target="_blank">irc.gnu.org</a>]<br><a href="http://www.zresearch.com" target="_blank">http://www.zresearch.com</a> - Commoditizing Super Storage!<br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>Marcus Herou CTO and co-founder Tailsweep AB<br>+46702561312<br><a href="mailto:marcus.herou@tailsweep.com" target="_blank">marcus.herou@tailsweep.com</a><br><a href="http://www.tailsweep.com/" target="_blank">http://www.tailsweep.com/</a><br>
<a href="http://blogg.tailsweep.com/" target="_blank">http://blogg.tailsweep.com/</a><br>
</div></div></div>
</div><br><br clear="all"><br>-- <br>Marcus Herou CTO and co-founder Tailsweep AB<br>+46702561312<br><a href="mailto:marcus.herou@tailsweep.com">marcus.herou@tailsweep.com</a><br><a href="http://www.tailsweep.com/">http://www.tailsweep.com/</a><br>
<a href="http://blogg.tailsweep.com/">http://blogg.tailsweep.com/</a><br>
</div>