<div dir="ltr">oops, spoke too soon, but i did get a bit farther into the test<div><br></div><div><div> 16384 4 4021 4155 11464 11468 4237 4157 4128 4175 4139 4018 4166 11449 11447</div>
<div> 16384 8 5285 5392 11466 11465 5459 5391 5358 5409 5354 5274 4925 11372 11446</div><div> 16384 16 7549 7711 11467 11465 7802 7702 7662 7756 7690 7524 7704 11447 11446</div>
<div> 16384 32 8718 8900 11441 10001 8998 8591 8597 8396 7724 8686 8544</div><div>Error freading block 382 3a900000</div><div>read: File descriptor in bad state</div><br>
<div class="gmail_quote">On Thu, Aug 7, 2008 at 9:11 AM, Michael Di Domenico <span dir="ltr"><<a href="mailto:mdidomenico4@gmail.com">mdidomenico4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div dir="ltr">As a followup, I added a fourth storage brick and a seperate namespace server into the mix, and now this problem seems to have gone away. I'm not sure if it was the 4th storage brick or the seperate name server that corrected it just yet.<div>
<div></div><div class="Wj3C7c"><br>
<br><div class="gmail_quote">On Sun, Aug 3, 2008 at 8:20 PM, Michael Di Domenico <span dir="ltr"><<a href="mailto:mdidomenico4@gmail.com" target="_blank">mdidomenico4@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>[root@green gluster]# /opt/gluster/sbin/glusterfs -V</div><div>glusterfs 1.3.10 built on Aug 1 2008 12:08:22</div><div>Repository revision: glusterfs--mainline--2.5--patch-788</div><div>Copyright (c) 2006, 2007, 2008 Z RESEARCH Inc. <<a href="http://www.zresearch.com" target="_blank">http://www.zresearch.com</a>></div>
<div>GlusterFS comes with ABSOLUTELY NO WARRANTY.</div><div>You may redistribute copies of GlusterFS under the terms of the GNU General Public License.</div><div><br></div><div>[root@green gluster]# ./iozone -a<br></div>
<div>
<div>...output snipped....</div><div>Error freading block 7 bc700000</div><div><div>read: File descriptor in bad state</div><div><br></div></div><div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1701:fuse_flush] glusterfs-fuse: 25151: FLUSH 0x1f33dbb0</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25151: (16) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1728:fuse_release] glusterfs-fuse: 25152: CLOSE 0x1f33dbb0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1746:fuse_fsync] glusterfs-fuse: 25153: FSYNC 0x1f33fc80</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25153: (18) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1701:fuse_flush] glusterfs-fuse: 25154: FLUSH 0x1f33fc80</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25152: (17) ERR => 0</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25154: (16) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1728:fuse_release] glusterfs-fuse: 25155: CLOSE 0x1f33fc80</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1553:fuse_open] glusterfs-fuse: 25156: OPEN /iozone.tmp</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25155: (17) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:687:fuse_fd_cbk] glusterfs-fuse: 25156: (12) /iozone.tmp => 0x1f33d8d0</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1553:fuse_open] glusterfs-fuse: 25157: OPEN /iozone.tmp</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:687:fuse_fd_cbk] glusterfs-fuse: 25157: (12) /iozone.tmp => 0x1f340b10</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1746:fuse_fsync] glusterfs-fuse: 25158: FSYNC 0x1f340b10</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25158: (18) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1701:fuse_flush] glusterfs-fuse: 25159: FLUSH 0x1f340b10</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25159: (16) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1728:fuse_release] glusterfs-fuse: 25160: CLOSE 0x1f340b10</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25161: READ (0x1f33d8d0, size=131072, offset=0)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25162: READ (0x1f33d8d0, size=131072, offset=131072)</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25161: READ => 131072/131072,0/1048576</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25160: (17) ERR => 0</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25163: READ (0x1f33d8d0, size=131072, offset=262144)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25162: READ => 131072/131072,131072/1048576</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25164: READ (0x1f33d8d0, size=131072, offset=393216)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25163: READ => 131072/131072,262144/1048576</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25165: READ (0x1f33d8d0, size=131072, offset=524288)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25164: READ => 131072/131072,393216/1048576</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25166: READ (0x1f33d8d0, size=131072, offset=655360)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25165: READ => 131072/131072,524288/1048576</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25167: READ (0x1f33d8d0, size=131072, offset=786432)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25166: READ => 131072/131072,655360/1048576</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25168: READ (0x1f33d8d0, size=131072, offset=917504)</div>
<div>2008-08-03 20:18:30 D [fuse-bridge.c:1582:fuse_readv_cbk] glusterfs-fuse: 25167: READ => 131072/131072,786432/1048576</div><div>2008-08-03 20:18:30 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25169: READ (0x1f33d8d0, size=131072, offset=1048576)</div>
<div>2008-08-03 20:19:17 W [client-protocol.c:205:call_bail] remote1: activating bail-out. pending frames = 1. last sent = 2008-08-03 20:18:30. last received = 2008-08-03 20:18:30 transport-timeout = 42</div><div>2008-08-03 20:19:17 C [client-protocol.c:212:call_bail] remote1: bailing transport</div>
<div>2008-08-03 20:19:17 D [tcp.c:137:cont_hand] tcp: forcing poll/read/write to break on blocked socket (if any)</div><div>2008-08-03 20:19:17 W [client-protocol.c:205:call_bail] remote2: activating bail-out. pending frames = 1. last sent = 2008-08-03 20:18:30. last received = 2008-08-03 20:18:30 transport-timeout = 42</div>
<div>2008-08-03 20:19:17 C [client-protocol.c:212:call_bail] remote2: bailing transport</div><div>2008-08-03 20:19:17 D [tcp.c:137:cont_hand] tcp: forcing poll/read/write to break on blocked socket (if any)</div><div>2008-08-03 20:19:17 E [protocol.c:376:gf_block_unserialize_transport] remote1: full_read of block failed: peer (<a href="http://192.168.0.254:6996" target="_blank">192.168.0.254:6996</a>)</div>
<div>2008-08-03 20:19:17 W [client-protocol.c:4777:client_protocol_cleanup] remote1: cleaning up state in transport object 0x1f3349a0</div><div>2008-08-03 20:19:17 E [client-protocol.c:4827:client_protocol_cleanup] remote1: forced unwinding frame type(1) op(13) reply=@0x1f33c070</div>
<div>2008-08-03 20:19:17 E [client-protocol.c:3193:client_readv_cbk] remote1: no proper reply from server, returning ENOTCONN</div><div>2008-08-03 20:19:17 E [fuse-bridge.c:1587:fuse_readv_cbk] glusterfs-fuse: 25168: READ => -1 (107)</div>
<div>2008-08-03 20:19:17 D [tcp.c:87:tcp_disconnect] remote1: connection disconnected</div><div>2008-08-03 20:19:17 W [client-protocol.c:4777:client_protocol_cleanup] remote2: cleaning up state in transport object 0x1f337300</div>
<div>2008-08-03 20:19:17 E [client-protocol.c:4827:client_protocol_cleanup] remote2: forced unwinding frame type(1) op(13) reply=@0x1f33d3b0</div><div>2008-08-03 20:19:17 E [client-protocol.c:3193:client_readv_cbk] remote2: no proper reply from server, returning ENOTCONN</div>
<div>2008-08-03 20:19:17 E [fuse-bridge.c:1587:fuse_readv_cbk] glusterfs-fuse: 25169: READ => -1 (107)</div><div>2008-08-03 20:19:17 D [tcp.c:87:tcp_disconnect] remote2: connection disconnected</div><div>2008-08-03 20:19:17 D [fuse-bridge.c:1613:fuse_readv] glusterfs-fuse: 25170: READ (0x1f33d8d0, size=4096, offset=917504)</div>
<div>2008-08-03 20:19:17 E [client-protocol.c:1105:client_readv] remote1: : returning EBADFD</div><div>2008-08-03 20:19:17 E [fuse-bridge.c:1587:fuse_readv_cbk] glusterfs-fuse: 25170: READ => -1 (77)</div><div>2008-08-03 20:19:17 D [fuse-bridge.c:1728:fuse_release] glusterfs-fuse: 25171: CLOSE 0x1f33eae0</div>
<div>2008-08-03 20:19:17 W [client-protocol.c:1296:client_close] remote2: no valid fd found, returning</div><div>2008-08-03 20:19:17 D [fuse-bridge.c:1701:fuse_flush] glusterfs-fuse: 25172: FLUSH 0x1f33d8d0</div><div>2008-08-03 20:19:17 D [dict.c:357:dict_destroy] dict: @this=(nil)</div>
<div>2008-08-03 20:19:17 E [client-protocol.c:1238:client_flush] remote1: : returning EBADFD</div><div>2008-08-03 20:19:17 W [stripe.c:198:stripe_stack_unwind_cbk] stripe0: remote1 returned errno 77</div><div>2008-08-03 20:19:17 D [dict.c:357:dict_destroy] dict: @this=(nil)</div>
<div>2008-08-03 20:19:17 E [client-protocol.c:1238:client_flush] remote2: : returning EBADFD</div><div>2008-08-03 20:19:17 W [stripe.c:198:stripe_stack_unwind_cbk] stripe0: remote2 returned errno 77</div><div>2008-08-03 20:19:17 W [client-protocol.c:1296:client_close] remote1: no valid fd found, returning</div>
<div>2008-08-03 20:19:17 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25171: (17) ERR => 0</div><div>2008-08-03 20:19:17 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25172: (16) ERR => 0</div><div>2008-08-03 20:19:17 D [fuse-bridge.c:1728:fuse_release] glusterfs-fuse: 25173: CLOSE 0x1f33d8d0</div>
<div>2008-08-03 20:19:17 W [client-protocol.c:1296:client_close] remote2: no valid fd found, returning</div><div>2008-08-03 20:19:17 W [client-protocol.c:1296:client_close] remote1: no valid fd found, returning</div><div>
2008-08-03 20:19:17 D [fuse-bridge.c:939:fuse_err_cbk] glusterfs-fuse: 25173: (17) ERR => 0</div><div>2008-08-03 20:19:17 D [client-protocol.c:4750:client_protocol_reconnect] remote1: attempting reconnect</div><div>2008-08-03 20:19:17 D [tcp-client.c:77:tcp_connect] remote1: socket fd = 6</div>
<div>2008-08-03 20:19:17 D [tcp-client.c:107:tcp_connect] remote1: finalized on port `1023'</div><div>2008-08-03 20:19:17 D [tcp-client.c:128:tcp_connect] remote1: defaulting remote-port to 6996</div><div>2008-08-03 20:19:17 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: n1</div>
<div>2008-08-03 20:19:17 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:<a href="http://192.168.0.254" target="_blank">192.168.0.254</a>[0] for hostname: n1</div><div>2008-08-03 20:19:17 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache</div>
<div>2008-08-03 20:19:17 D [tcp-client.c:161:tcp_connect] remote1: connect on 6 in progress (non-blocking)</div><div>2008-08-03 20:19:17 D [tcp-client.c:205:tcp_connect] remote1: connection on 6 success</div><div>2008-08-03 20:19:17 D [client-protocol.c:5355:notify] remote1: got GF_EVENT_CHILD_UP</div>
<div>2008-08-03 20:19:17 D [client-protocol.c:4750:client_protocol_reconnect] remote2: attempting reconnect</div><div>2008-08-03 20:19:17 D [tcp-client.c:77:tcp_connect] remote2: socket fd = 7</div><div>2008-08-03 20:19:17 D [tcp-client.c:107:tcp_connect] remote2: finalized on port `1022'</div>
<div>2008-08-03 20:19:17 D [tcp-client.c:128:tcp_connect] remote2: defaulting remote-port to 6996</div><div>2008-08-03 20:19:17 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: n2</div>
<div>2008-08-03 20:19:17 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:<a href="http://192.168.0.253" target="_blank">192.168.0.253</a>[0] for hostname: n2</div><div>2008-08-03 20:19:17 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache</div>
<div>2008-08-03 20:19:17 D [tcp-client.c:161:tcp_connect] remote2: connect on 7 in progress (non-blocking)</div><div>2008-08-03 20:19:17 D [client-protocol.c:5096:client_protocol_handshake_reply] remote1: reply frame has callid: 424242</div>
<div>2008-08-03 20:19:17 D [client-protocol.c:5130:client_protocol_handshake_reply] remote1: SETVOLUME on remote-host succeeded</div><div>2008-08-03 20:19:17 D [tcp-client.c:205:tcp_connect] remote2: connection on 7 success</div>
<div>2008-08-03 20:19:17 D [client-protocol.c:5355:notify] remote2: got GF_EVENT_CHILD_UP</div><div>2008-08-03 20:19:17 D [client-protocol.c:5096:client_protocol_handshake_reply] remote2: reply frame has callid: 424242</div>
<div>2008-08-03 20:19:17 D [client-protocol.c:5130:client_protocol_handshake_reply] remote2: SETVOLUME on remote-host succeeded</div><div>2008-08-03 20:19:18 D [client-protocol.c:4756:client_protocol_reconnect] remote1: breaking reconnect chain</div>
<div>2008-08-03 20:19:18 D [client-protocol.c:4756:client_protocol_reconnect] remote2: breaking reconnect chain</div></div></div><div><div></div><div><br><div class="gmail_quote">On Fri, Aug 1, 2008 at 4:36 PM, Amar S. Tumballi <span dir="ltr"><<a href="mailto:amar@zresearch.com" target="_blank">amar@zresearch.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Version of glusterfs used will help us a lot. (Get it by 'glustersf -V')<br><br>Also, any snippet of log files (mostly the last few lines) which contains errors logs related to this would help too.<br>
<br>Regards,<br>Amar<br><br><div class="gmail_quote">2008/8/1 Michael Di Domenico <span dir="ltr"><<a href="mailto:mdidomenico4@gmail.com" target="_blank">mdidomenico4@gmail.com</a>></span><br><blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
<div><div></div><div>
<div dir="ltr"><div>I've just setup a simple gluster storage system on Centos 5.2 x64 w/ gluster 1.3.10</div><div><br></div><div>I have three storage bricks and one client</div><div><br></div><div>Everytime i run iozone across this setup, i seem to get a bad file descriptor around the 4k mark.</div>
<div><br></div><div>Any thoughts why? I'm sure more info is wanted, i'm just not sure what else to include at this point.</div><div><br></div><div>thanks</div><div><br></div><div><div>[root@green gluster]# cat /opt/gluster/etc/glusterfs/gluster.client </div>
<div>volume remote1</div><div> type protocol/client</div><div> option transport-type tcp/client</div><div> option remote-host n1</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>
volume remote2</div>
<div> type protocol/client</div><div> option transport-type tcp/client</div><div> option remote-host n2</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume remote3</div><div>
type protocol/client</div>
<div> option transport-type tcp/client</div><div> option remote-host n3</div><div> option remote-subvolume brick</div><div>end-volume</div><div><br></div><div>volume stripe0</div><div> type cluster/stripe</div><div> option block-size *:1MB</div>
<div> subvolumes remote1 remote2 remote3</div><div>end-volume</div><div><br></div><div>[root@green gluster]# cat /opt/gluster/etc/glusterfs/gluster.server </div><div>volume brick</div><div> type storage/posix</div><div>
option directory /gluster</div><div>end-volume</div><div><br></div><div>volume server</div><div> type protocol/server</div><div> option transport-type tcp/server</div><div> option auth.ip.brick.allow *</div><div> subvolumes brick</div>
<div>end-volume</div></div><div><br></div><div>--- iozone snipped...</div><div><br></div><div> 4096 4 4109 4228 7336 10409 4245 4209 4163 4232 4178 4098 4215 10218 6782</div>
<div> 4096 8 5951 6093 11445 11450 6096 6085 6028 6104 6054 5935 6077 11409 6792</div><div> 4096 16 7609 7757 11395 7353 7825 7747 7705 7793 7776 7595 7753 11396 7078</div>
<div> 4096 32 8915 9074 11313 10268 9166 9074 8977 9118 9127 8922 9083</div><div><br></div><div>Error freading block 62 9d800000</div><div>read: File descriptor in bad state</div>
<div>[root@green gluster]# </div><div>[root@green gluster]# </div></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Amar Tumballi<br>Gluster/GlusterFS Hacker<br>[bulde on #gluster/<a href="http://irc.gnu.org" target="_blank">irc.gnu.org</a>]<br><a href="http://www.zresearch.com" target="_blank">http://www.zresearch.com</a> - Commoditizing Super Storage!<br>
</div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div>