<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">So a little further down the road I have destroyed/rebuilt the gluster volume and restarted glusterd on each of the nodes. Shortly after the instances directory is created and the console.log file is placed, I receive the following fuse errors on the client:<div><br></div><div><div> [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-4: remote operation failed: Invalid argument</div><div>[2014-03-19 19:06:35.326487] W [client-rpc-fops.c:873:client3_3_writev_cbk] 0-novashared-client-5: remote operation failed: Invalid argument</div><div>[2014-03-19 19:06:35.328750] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106504: WRITE => -1 (Invalid argument)</div><div>[2014-03-19 19:06:35.328855] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106505: FTRUNCATE() ERR => -1 (Bad file descriptor)</div><div>[2014-03-19 19:06:35.329506] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106507: WRITE => -1 (Bad file descriptor)</div><div>[2014-03-19 19:06:35.329775] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106508: FTRUNCATE() ERR => -1 (Bad file descriptor)</div><div>[2014-03-19 19:06:35.330495] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106510: WRITE => -1 (Bad file descriptor)</div><div>[2014-03-19 19:06:35.330758] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106511: FTRUNCATE() ERR => -1 (Bad file descriptor)</div><div>[2014-03-19 19:06:35.331538] W [fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse: 106513: WRITE => -1 (Bad file descriptor)</div><div>[2014-03-19 19:06:35.331792] W [fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse: 106514: FTRUNCATE() ERR => -1 (Bad file descriptor)</div><div><br></div><div><br></div><div>One one of the servers the normal messages apply:</div><div><br></div><div><div>[2014-03-13 21:47:33.735042] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0 (version: 3.4.2)</div><div>[2014-03-18 21:17:42.763044] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0</div><div>[2014-03-18 21:17:42.763076] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0</div><div>[2014-03-18 21:17:42.763101] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0</div><div>[2014-03-18 21:50:02.206635] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0 (version: 3.4.2)</div><div>[2014-03-18 21:50:23.037886] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0 (version: 3.4.2)</div><div>[2014-03-18 21:50:23.051419] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0</div><div>[2014-03-18 21:50:23.051446] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0</div><div>[2014-03-18 21:50:23.051466] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0</div><div>[2014-03-18 21:56:22.215866] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0</div><div>[2014-03-18 21:56:22.215898] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0</div><div>[2014-03-18 21:56:22.215923] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0</div><div>[2014-03-18 21:57:04.821397] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0 (version: 3.4.2)</div><div>[2014-03-19 16:48:32.135593] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0</div><div>[2014-03-19 16:48:32.135629] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0</div><div>[2014-03-19 16:48:32.135652] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0</div><div>[2014-03-19 16:48:47.712973] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0 (version: 3.4.2)</div><div>[2014-03-19 18:07:44.332240] I [server.c:762:server_rpc_notify] 0-novashared-server: disconnecting connectionfrom cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0</div><div>[2014-03-19 18:07:44.332272] I [server-helpers.c:729:server_connection_put] 0-novashared-server: Shutting down connection cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0</div><div>[2014-03-19 18:07:44.332296] I [server-helpers.c:617:server_connection_destroy] 0-novashared-server: destroyed connection of cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0</div><div>[2014-03-19 19:00:51.369205] I [server-handshake.c:567:server_setvolume] 0-novashared-server: accepted client from cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0 (version: 3.4.2)</div></div><div><br></div><div><br></div><div>These are all on 3.4.2 with Ubuntu 13.10. </div><div><br></div><div><br></div><div>-ryan</div><div><br></div><div><br></div><div><div>On Mar 19, 2014, at 11:15 AM, Carlos Capriotti <<a href="mailto:capriotti.carlos@gmail.com">capriotti.carlos@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">This rings a bell...<br><br>I mapped a remote nfs share to one of my nodes and mapped the local nfs<br>share, to the "local" gluster volume as well and started to copy (with cp)<br>files from one mount to the other.<br><br>Somewhere down the line all stalled, the server was irresponsive, even<br>restarting glusterd solved nothing. Had to reboot the nodes.<br><br>CentOS 6.5 and Gluster 3.4.2.<br><br>Since then, I haven't tried those copies, but, I also, I am now using the<br>native glusterfs client just in case.<br><br><br><br><br>On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <<a href="mailto:rpowell1@uchicago.edu">rpowell1@uchicago.edu</a>> wrote:<br><br><blockquote type="cite">-----BEGIN PGP SIGNED MESSAGE-----<br>Hash: SHA512<br><br>I have not seen that with the fuse client. I had it happen repeatedly<br>when we were using gluster/NFS for /var/lib/nova/instances and/or<br>/var/lib/nova/instances/_base. The access to instances-XXXXXX/disk<br>(or its qcow2 backing image) would hang completely or be very slow<br>under heavy load. The VM interprets the delay as a problem on<br>/dev/vda and remounts RO.<br><br><br>On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:<br><blockquote type="cite">I was wondering if anyone had seen similar issues before, when<br>booting a vm with a root volume on local disk things work fine.<br><br>When I try to do the same with /var/lib/nova/instances mounted via<br>gluster/fuse I get the following: <a href="http://paste.ubuntu.com/7116594/">http://paste.ubuntu.com/7116594/</a><br><br>I have of course insured perms were correct with chown -R nova:nova<br>/var/lib/nova/instances, additionally app armor is completely<br>unloaded/disabled.<br><br>Has anyone encountered this before? Gluster version is 3.4.2<br><br><br>-ryan<br><br><br><br><br>_______________________________________________ Gluster-users<br>mailing list <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br><br></blockquote><br>- --<br>Raymond Powell<br>Systems Administrator<br>Laboratory for Advanced Computing<br>University of Chicago, KCBD 10160<br>900 East 57th Street<br>Chicago, IL 60637<br>-----BEGIN PGP SIGNATURE-----<br>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)<br>Comment: GPGTools -<span class="Apple-converted-space"> </span><a href="http://gpgtools.org/">http://gpgtools.org</a><br>Comment: Using GnuPG with Thunderbird -<span class="Apple-converted-space"> </span><a href="http://www.enigmail.net/">http://www.enigmail.net/</a><br><br>iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9<br>lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy<br>pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B<br>YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm<br>eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg<br>huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/<br>HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD<br>lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa<br>sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA<br>o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF<br>JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe<br>+UhAq6+74ie1MokNssB8<br>=un7j<br>-----END PGP SIGNATURE-----<br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br><br></blockquote>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br></div></body></html>