<div dir="ltr">Seems like there were on-going write operations. On errors we log and network disconnect has resulted in these logs.<br><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 27, 2014 at 7:21 PM, Sergio Traldi <span dir="ltr"><<a href="mailto:sergio.traldi@pd.infn.it" target="_blank">sergio.traldi@pd.infn.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
One server Redhat 6 with this rpms set:<br>
<br>
[ ~]# rpm -qa | grep gluster | sort<br>
glusterfs-3.5.2-1.el6.x86_64<br>
glusterfs-api-3.5.2-1.el6.x86_<u></u>64<br>
glusterfs-cli-3.5.2-1.el6.x86_<u></u>64<br>
glusterfs-fuse-3.5.2-1.el6.<u></u>x86_64<br>
glusterfs-geo-replication-3.5.<u></u>2-1.el6.x86_64<br>
glusterfs-libs-3.5.2-1.el6.<u></u>x86_64<br>
glusterfs-server-3.5.2-1.el6.<u></u>x86_64<br>
<br>
I have a gluster volume with 1 server and 1 brick:<br>
<br>
[ ~]# gluster volume info volume-nova-pp<br>
Volume Name: volume-nova-pp<br>
Type: Distribute<br>
Volume ID: b5ec289b-9a54-4df1-9c21-<u></u>52ca556aeead<br>
Status: Started<br>
Number of Bricks: 1<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: 192.168.61.100:/brick-nova-pp/<u></u>mpathc<br>
Options Reconfigured:<br>
storage.owner-gid: 162<br>
storage.owner-uid: 162<br>
<br>
There are four clients attached to this volume with same O.S. and same fuse gluster rpms set:<br>
[ ~]# rpm -qa | grep gluster | sort<br>
glusterfs-3.5.0-2.el6.x86_64<br>
glusterfs-api-3.5.0-2.el6.x86_<u></u>64<br>
glusterfs-fuse-3.5.0-2.el6.<u></u>x86_64<br>
glusterfs-libs-3.5.0-2.el6.<u></u>x86_6<br>
<br>
Last week, but it happens also two weeks ago, I found the disk almost full and I found the gluster logs /var/log/glusterfs/var-lib-<u></u>nova-instances.log of 68GB:<br>
In the log there was the starting problem:<br>
<br>
[2014-10-10 07:29:43.730792] W [socket.c:522:__socket_rwv] 0-glusterfs: readv on <a href="http://192.168.61.100:24007" target="_blank">192.168.61.100:24007</a> failed (No data available)<br>
[2014-10-10 07:29:54.022608] E [socket.c:2161:socket_connect_<u></u>finish] 0-glusterfs: connection to <a href="http://192.168.61.100:24007" target="_blank">192.168.61.100:24007</a> failed (Connection refused)<br>
[2014-10-10 07:30:05.271825] W [client-rpc-fops.c:866:<u></u>client3_3_writev_cbk] 0-volume-nova-pp-client-0: remote operation failed: Input/output error<br>
[2014-10-10 07:30:08.783145] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 3661260: WRITE => -1 (Input/output error)<br>
[2014-10-10 07:30:08.783368] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 3661262: WRITE => -1 (Input/output error)<br>
[2014-10-10 07:30:08.806553] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 3661649: WRITE => -1 (Input/output error)<br>
[2014-10-10 07:30:08.844415] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 3662235: WRITE => -1 (Input/output error)<br>
<br>
and a lot of these lines:<br>
<br>
[2014-10-15 14:41:15.895105] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 951700230: WRITE => -1 (Transport endpoint is not connected)<br>
[2014-10-15 14:41:15.896205] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 951700232: WRITE => -1 (Transport endpoint is not connected)<br>
<br>
This second line log with different "sector" number has been written every millisecond so in about 1 minute we have 1GB write in O.S. disk.<br>
<br>
I search for a solution but I didn't find nobody having the same problem.<br>
<br>
I think there was a network problem but why does gluster write in logs million of:<br>
[2014-10-15 14:41:15.895105] W [fuse-bridge.c:2201:fuse_<u></u>writev_cbk] 0-glusterfs-fuse: 951700230: WRITE => -1 (Transport endpoint is not connected) ?<br>
<br>
Thanks in advance.<br>
Cheers<br>
Sergio<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>devel</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br>
</div></div></div></div>