Hi,<br><br>Please find the comments inlined.<br><br><div class="gmail_quote">On Mon, Dec 1, 2008 at 8:54 PM, Manhong Dai <span dir="ltr"><<a href="mailto:daimh@umich.edu">daimh@umich.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi,<br>
<br>
<br>
After a month's file operations, which included coping 20 million of<br>
small files and about 20 thousand of cluster jobs, I am overall<br>
satisfied except two stability glitches.<br>
<br>
<br>
1. A small portion (about 1%?) of jobs got an error of "transport<br>
endpoint not connected", and output file is incomplete. This error<br>
happened on random computing nodes, and it doesn't affect subsequent<br>
jobs on the same node. An example of error message of glusterfsd is<br>
2008-11-19 23:09:51 E [protocol.c:271:gf_block_unserialize_transport]<br>
server: EOF from peer (<a href="http://172.20.102.2:1022" target="_blank">172.20.102.2:1022</a>)<br>
<br>
Error of glusterfs is either (looks to be caused by brick)<br>
2008-11-19 23:09:52 C [client-protocol.c:212:call_bail] muskie-brick:<br>
bailing transport<br>
2008-11-19 23:09:52 E [client-protocol.c:4834:client_protocol_cleanup]<br>
muskie-brick: forced unwinding frame type(1) op(14) reply=@0x67e2150<br>
2008-11-19 23:09:52 E [client-protocol.c:3254:client_write_cbk]<br>
muskie-brick: no proper reply from server, returning ENOTCONN<br>
2008-11-19 23:09:56 E [write-behind.c:602:wb_writev] wb: delayed error :<br>
107<br>
<br>
or (caused by namespace)<br>
2008-11-28 20:47:53 C [client-protocol.c:212:call_bail] muskie-ns:<br>
bailing transport<br>
2008-11-28 20:47:53 E [client-protocol.c:4834:client_protocol_cleanup]<br>
muskie-ns: forced unwinding frame type(1) op(40) reply=@0x1b447cc0<br>
2008-11-28 20:47:53 E [client-protocol.c:4613:client_checksum_cbk]<br>
muskie-ns: no proper reply from server, returning ENOTCONN<br>
2008-11-28 20:47:53 E [client-protocol.c:325:client_protocol_xfer]<br>
muskie-ns: transport_submit failed<br>
<br>
</blockquote><div><br>what is the transport timeout you are using? If the transport-timeout is small and the server is busy serving other requests, there is a good possibility that the operations are bailing out and resulting in ENOTCONN errors.<br>
<br>Are you using io-threads on server side? Can you send the configuration files?<br><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
2. Right now the process 'glusterfs' takes 1785M virt mem, and 1500 RES<br>
mem, according to top. I hope this is not a memory leak, or at least<br>
there should be a way to reduce memory usage without remounting it.<br>
<br>
<br>
<br>
If somebody can shed some light on these issues, I appreciate it. Just<br>
let me know if you need more detailed information.<br>
<br>
<br>
Best,<br>
Manhong<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>