<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Resending it as I suppose it was not
delievered :(<br>
</div>
<blockquote cite="mid:53318BE5.40600@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Hi Ryan/Narayan,<br>
<br>
Apologies for top posting.<br>
After today's call, we had a chat with Vijay and he added a
hint.<br>
<br>
You need to enable<i><b> remote-dio</b></i> for that volume that
serves for virt-store.<br>
> gluster volume set <vol-name> remote-dio enable<br>
<br>
Hope this could solve your issue most probably.<br>
Could you test it with this option 'on' and let us know the
results ?<br>
<br>
-- sas<br>
<br>
On 03/20/2014 12:40 AM, Aydelott, Ryan M. wrote:<br>
</div>
<blockquote
cite="mid:E036C3FD-05E9-47DF-9278-76B880E7B742@anl.gov"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
So a little further down the road I have destroyed/rebuilt the
gluster volume and restarted glusterd on each of the nodes.
Shortly after the instances directory is created and the
console.log file is placed, I receive the following fuse errors
on the client:
<div><br>
</div>
<div>
<div> [client-rpc-fops.c:873:client3_3_writev_cbk]
0-novashared-client-4: remote operation failed: Invalid
argument</div>
<div>[2014-03-19 19:06:35.326487] W
[client-rpc-fops.c:873:client3_3_writev_cbk]
0-novashared-client-5: remote operation failed: Invalid
argument</div>
<div>[2014-03-19 19:06:35.328750] W
[fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
106504: WRITE => -1 (Invalid argument)</div>
<div>[2014-03-19 19:06:35.328855] W
[fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
106505: FTRUNCATE() ERR => -1 (Bad file descriptor)</div>
<div>[2014-03-19 19:06:35.329506] W
[fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
106507: WRITE => -1 (Bad file descriptor)</div>
<div>[2014-03-19 19:06:35.329775] W
[fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
106508: FTRUNCATE() ERR => -1 (Bad file descriptor)</div>
<div>[2014-03-19 19:06:35.330495] W
[fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
106510: WRITE => -1 (Bad file descriptor)</div>
<div>[2014-03-19 19:06:35.330758] W
[fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
106511: FTRUNCATE() ERR => -1 (Bad file descriptor)</div>
<div>[2014-03-19 19:06:35.331538] W
[fuse-bridge.c:2167:fuse_writev_cbk] 0-glusterfs-fuse:
106513: WRITE => -1 (Bad file descriptor)</div>
<div>[2014-03-19 19:06:35.331792] W
[fuse-bridge.c:648:fuse_truncate_cbk] 0-glusterfs-fuse:
106514: FTRUNCATE() ERR => -1 (Bad file descriptor)</div>
<div><br>
</div>
<div><br>
</div>
<div>One one of the servers the normal messages apply:</div>
<div><br>
</div>
<div>
<div>[2014-03-13 21:47:33.735042] I
[server-handshake.c:567:server_setvolume]
0-novashared-server: accepted client from
cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0
(version: 3.4.2)</div>
<div>[2014-03-18 21:17:42.763044] I
[server.c:762:server_rpc_notify] 0-novashared-server:
disconnecting connectionfrom
cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0</div>
<div>[2014-03-18 21:17:42.763076] I
[server-helpers.c:729:server_connection_put]
0-novashared-server: Shutting down connection
cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0</div>
<div>[2014-03-18 21:17:42.763101] I
[server-helpers.c:617:server_connection_destroy]
0-novashared-server: destroyed connection of
cm6-p-3475-2014/03/13-21:47:33:539329-novashared-client-0-0</div>
<div>[2014-03-18 21:50:02.206635] I
[server-handshake.c:567:server_setvolume]
0-novashared-server: accepted client from
cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0
(version: 3.4.2)</div>
<div>[2014-03-18 21:50:23.037886] I
[server-handshake.c:567:server_setvolume]
0-novashared-server: accepted client from
cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0
(version: 3.4.2)</div>
<div>[2014-03-18 21:50:23.051419] I
[server.c:762:server_rpc_notify] 0-novashared-server:
disconnecting connectionfrom
cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0</div>
<div>[2014-03-18 21:50:23.051446] I
[server-helpers.c:729:server_connection_put]
0-novashared-server: Shutting down connection
cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0</div>
<div>[2014-03-18 21:50:23.051466] I
[server-helpers.c:617:server_connection_destroy]
0-novashared-server: destroyed connection of
cm6-p-5973-2014/03/18-21:50:22:966318-novashared-client-0-0</div>
<div>[2014-03-18 21:56:22.215866] I
[server.c:762:server_rpc_notify] 0-novashared-server:
disconnecting connectionfrom
cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0</div>
<div>[2014-03-18 21:56:22.215898] I
[server-helpers.c:729:server_connection_put]
0-novashared-server: Shutting down connection
cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0</div>
<div>[2014-03-18 21:56:22.215923] I
[server-helpers.c:617:server_connection_destroy]
0-novashared-server: destroyed connection of
cm6-p-5910-2014/03/18-21:50:02:8221-novashared-client-0-0</div>
<div>[2014-03-18 21:57:04.821397] I
[server-handshake.c:567:server_setvolume]
0-novashared-server: accepted client from
cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0
(version: 3.4.2)</div>
<div>[2014-03-19 16:48:32.135593] I
[server.c:762:server_rpc_notify] 0-novashared-server:
disconnecting connectionfrom
cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0</div>
<div>[2014-03-19 16:48:32.135629] I
[server-helpers.c:729:server_connection_put]
0-novashared-server: Shutting down connection
cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0</div>
<div>[2014-03-19 16:48:32.135652] I
[server-helpers.c:617:server_connection_destroy]
0-novashared-server: destroyed connection of
cm6-p-6815-2014/03/18-21:57:04:750942-novashared-client-0-0</div>
<div>[2014-03-19 16:48:47.712973] I
[server-handshake.c:567:server_setvolume]
0-novashared-server: accepted client from
cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0
(version: 3.4.2)</div>
<div>[2014-03-19 18:07:44.332240] I
[server.c:762:server_rpc_notify] 0-novashared-server:
disconnecting connectionfrom
cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0</div>
<div>[2014-03-19 18:07:44.332272] I
[server-helpers.c:729:server_connection_put]
0-novashared-server: Shutting down connection
cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0</div>
<div>[2014-03-19 18:07:44.332296] I
[server-helpers.c:617:server_connection_destroy]
0-novashared-server: destroyed connection of
cm6-p-20177-2014/03/19-16:48:47:641763-novashared-client-0-0</div>
<div>[2014-03-19 19:00:51.369205] I
[server-handshake.c:567:server_setvolume]
0-novashared-server: accepted client from
cm6-p-21755-2014/03/19-19:00:51:286018-novashared-client-0-0
(version: 3.4.2)</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>These are all on 3.4.2 with Ubuntu 13.10. </div>
<div><br>
</div>
<div><br>
</div>
<div>-ryan</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>On Mar 19, 2014, at 11:15 AM, Carlos Capriotti <<a
moz-do-not-send="true"
href="mailto:capriotti.carlos@gmail.com">capriotti.carlos@gmail.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;">This
rings a bell...<br>
<br>
I mapped a remote nfs share to one of my nodes and
mapped the local nfs<br>
share, to the "local" gluster volume as well and started
to copy (with cp)<br>
files from one mount to the other.<br>
<br>
Somewhere down the line all stalled, the server was
irresponsive, even<br>
restarting glusterd solved nothing. Had to reboot the
nodes.<br>
<br>
CentOS 6.5 and Gluster 3.4.2.<br>
<br>
Since then, I haven't tried those copies, but, I also,
I am now using the<br>
native glusterfs client just in case.<br>
<br>
<br>
<br>
<br>
On Wed, Mar 19, 2014 at 3:51 PM, Ray Powell <<a
moz-do-not-send="true"
href="mailto:rpowell1@uchicago.edu">rpowell1@uchicago.edu</a>>
wrote:<br>
<br>
<blockquote type="cite">-----BEGIN PGP SIGNED
MESSAGE-----<br>
Hash: SHA512<br>
<br>
I have not seen that with the fuse client. I had it
happen repeatedly<br>
when we were using gluster/NFS for
/var/lib/nova/instances and/or<br>
/var/lib/nova/instances/_base. The access to
instances-XXXXXX/disk<br>
(or its qcow2 backing image) would hang completely or
be very slow<br>
under heavy load. The VM interprets the delay as a
problem on<br>
/dev/vda and remounts RO.<br>
<br>
<br>
On 3/18/14, 5:04 PM, Aydelott, Ryan M. wrote:<br>
<blockquote type="cite">I was wondering if anyone had
seen similar issues before, when<br>
booting a vm with a root volume on local disk things
work fine.<br>
<br>
When I try to do the same with
/var/lib/nova/instances mounted via<br>
gluster/fuse I get the following: <a
moz-do-not-send="true"
href="http://paste.ubuntu.com/7116594/">http://paste.ubuntu.com/7116594/</a><br>
<br>
I have of course insured perms were correct with
chown -R nova:nova<br>
/var/lib/nova/instances, additionally app armor is
completely<br>
unloaded/disabled.<br>
<br>
Has anyone encountered this before? Gluster version
is 3.4.2<br>
<br>
<br>
-ryan<br>
<br>
<br>
<br>
<br>
_______________________________________________
Gluster-users<br>
mailing list <a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</blockquote>
<br>
- --<br>
Raymond Powell<br>
Systems Administrator<br>
Laboratory for Advanced Computing<br>
University of Chicago, KCBD 10160<br>
900 East 57th Street<br>
Chicago, IL 60637<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)<br>
Comment: GPGTools -<span class="Apple-converted-space"> </span><a
moz-do-not-send="true" href="http://gpgtools.org/">http://gpgtools.org</a><br>
Comment: Using GnuPG with Thunderbird -<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="http://www.enigmail.net/">http://www.enigmail.net/</a><br>
<br>
iQIcBAEBCgAGBQJTKa8HAAoJEBIjAczqLCW8fYMQAIMsxPVG0n+ujRJeCYvphUy9<br>
lHe7bkTLiJIkMs51hzXCfsqT3tEkvixYbjaAfjuHTdg0q5sK4zu+5MUM4KqG7oZy<br>
pYMzX2VrYnkTqQRFTOQbzntSvKlDjuSebuGpbI8sCxfgOR6/25tuzcJDCvn7sp4B<br>
YxVFaLZx5Xd6U24BHT9MVVJgAFXCVoalj2XMy+WHGhM07VvEXHxf62oW2VymLHOm<br>
eYiQGpjDWwb218/6NtJ2z47xr7HIQF1sW1Hhu/YRsnNzAYPh1FL3NwvnkFvNwGdg<br>
huTFpQegV9bmg78A3eEnIEEjDm1haX/E1NOo2bHdRYF2+Fwz+rHgkQJxFfEHxOV/<br>
HDylUIPeMue9qEgttRg345NuypoVrBN3sFGf72UOfgwwElr+Z0t83y6uquTcNfOD<br>
lWIPa7d0vyLWfm6+mjR1hDHsLx6H/Y0MyAKFgYUIerdO8+TgAnLsv4FfBGlbm+Xa<br>
sutQxYPxBeAlSnKXmwrP9keB1ttiMV8IptHbuM5cQna5jdPeouwpluquZsP30bpA<br>
o/a7nPaogBx+X6BsM4aBjRLSjqtu+mywNh1AqdY1dIpvDtD9lXrO5L+Z2SLZUDeF<br>
JvFWkp8Epdn0k7vn6WKT6RZvjSji9WgTIdbFrKEjfEqAz/+9UbrG2D/vYL2TlgAe<br>
+UhAq6+74ie1MokNssB8<br>
=un7j<br>
-----END PGP SIGNATURE-----<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</blockquote>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>