<div dir="ltr">The content of file zp2-brick2.log is at <a href="http://ur1.ca/iku0l" style="color:rgb(0,102,204);text-decoration:none;font-family:'Liberation Sans','Lucida Grande','Luxi Sans','Bitstream Vera Sans',helvetica,verdana,arial,sans-serif">http://ur1.ca/iku0l</a> (<a href="http://fpaste.org/145714/44849041/">http://fpaste.org/145714/44849041/</a> )<div><br></div><div>I can't open the file <span style="font-size:13px;font-family:arial,sans-serif">/zp2/brick2/.glusterfs/health_</span><span style="font-size:13px;font-family:arial,sans-serif">check since it hangs due to no disk present.</span></div><div><span style="font-size:13px;font-family:arial,sans-serif"><br></span></div><div><span style="font-size:13px;font-family:arial,sans-serif">Let me know the filename pattern, so that I can find it.</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 28, 2014 at 1:42 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Oct 28, 2014 at 01:10:56PM +0530, Kiran Patil wrote:<br>
> I applied the patches, compiled and installed the gluster.<br>
><br>
> # glusterfs --version<br>
> glusterfs 3.7dev built on Oct 28 2014 12:03:10<br>
> Repository revision: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a><br>
> Copyright (c) 2006-2013 Red Hat, Inc. <<a href="http://www.redhat.com/" target="_blank">http://www.redhat.com/</a>><br>
> GlusterFS comes with ABSOLUTELY NO WARRANTY.<br>
> It is licensed to you under your choice of the GNU Lesser<br>
> General Public License, version 3 or any later version (LGPLv3<br>
> or later), or the GNU General Public License, version 2 (GPLv2),<br>
> in all cases as published by the Free Software Foundation.<br>
><br>
> # git log<br>
> commit 990ce16151c3af17e4cdaa94608b737940b60e4d<br>
> Author: Lalatendu Mohanty <<a href="mailto:lmohanty@redhat.com">lmohanty@redhat.com</a>><br>
> Date: Tue Jul 1 07:52:27 2014 -0400<br>
><br>
> Posix: Brick failure detection fix for ext4 filesystem<br>
> ...<br>
> ...<br>
><br>
> I see below messages<br>
<br>
</span>Many thanks Kiran!<br>
<br>
Do you have the messages from the brick that uses the zp2 mountpoint?<br>
<br>
There also should be a file with a timestamp when the last check was<br>
done successfully. If the brick is still running, this timestamp should<br>
get updated every storage.health-check-interval seconds:<br>
/zp2/brick2/.glusterfs/health_check<br>
<span class="HOEnZb"><font color="#888888"><br>
Niels<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> File /var/log/glusterfs/etc-glusterfs-glusterd.vol.log :<br>
><br>
> The message "I [MSGID: 106005]<br>
> [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick<br>
> 192.168.1.246:/zp2/brick2 has disconnected from glusterd." repeated 39<br>
> times between [2014-10-28 05:58:09.209419] and [2014-10-28 06:00:06.226330]<br>
> [2014-10-28 06:00:09.226507] W [socket.c:545:__socket_rwv] 0-management:<br>
> readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> argument)<br>
> [2014-10-28 06:00:09.226712] I [MSGID: 106005]<br>
> [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick<br>
> 192.168.1.246:/zp2/brick2 has disconnected from glusterd.<br>
> [2014-10-28 06:00:12.226881] W [socket.c:545:__socket_rwv] 0-management:<br>
> readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> argument)<br>
> [2014-10-28 06:00:15.227249] W [socket.c:545:__socket_rwv] 0-management:<br>
> readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> argument)<br>
> [2014-10-28 06:00:18.227616] W [socket.c:545:__socket_rwv] 0-management:<br>
> readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> argument)<br>
> [2014-10-28 06:00:21.227976] W [socket.c:545:__socket_rwv] 0-management:<br>
> readv on<br>
><br>
> .....<br>
> .....<br>
><br>
> [2014-10-28 06:19:15.142867] I<br>
> [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:<br>
> Received get vol req<br>
> The message "I [MSGID: 106005]<br>
> [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick<br>
> 192.168.1.246:/zp2/brick2 has disconnected from glusterd." repeated 12<br>
> times between [2014-10-28 06:18:09.368752] and [2014-10-28 06:18:45.373063]<br>
> [2014-10-28 06:23:38.207649] W [glusterfsd.c:1194:cleanup_and_exit] (--><br>
> 0-: received signum (15), shutting down<br>
><br>
><br>
> dmesg output:<br>
><br>
> SPLError: 7869:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has<br>
> encountered an uncorrectable I/O failure and has been suspended.<br>
><br>
> SPLError: 7868:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has<br>
> encountered an uncorrectable I/O failure and has been suspended.<br>
><br>
> SPLError: 7869:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has<br>
> encountered an uncorrectable I/O failure and has been suspended.<br>
><br>
> The brick is still online.<br>
><br>
> # gluster volume status<br>
> Status of volume: repvol<br>
> Gluster process Port Online Pid<br>
> ------------------------------------------------------------------------------<br>
> Brick 192.168.1.246:/zp1/brick1 49152 Y 4067<br>
> Brick 192.168.1.246:/zp2/brick2 49153 Y 4078<br>
> NFS Server on localhost 2049 Y 4092<br>
> Self-heal Daemon on localhost N/A Y 4097<br>
><br>
> Task Status of Volume repvol<br>
> ------------------------------------------------------------------------------<br>
> There are no active volume tasks<br>
><br>
> # gluster volume info<br>
><br>
> Volume Name: repvol<br>
> Type: Replicate<br>
> Volume ID: ba1e7c6d-1e1c-45cd-8132-5f4fa4d2d22b<br>
> Status: Started<br>
> Number of Bricks: 1 x 2 = 2<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: 192.168.1.246:/zp1/brick1<br>
> Brick2: 192.168.1.246:/zp2/brick2<br>
> Options Reconfigured:<br>
> storage.health-check-interval: 30<br>
><br>
> Let me know if you need further information.<br>
><br>
> Thanks,<br>
> Kiran.<br>
><br>
> On Tue, Oct 28, 2014 at 11:44 AM, Kiran Patil <<a href="mailto:kiran@fractalio.com">kiran@fractalio.com</a>> wrote:<br>
><br>
> > I changed git fetch git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a> to git fetch<br>
> > <a href="http://review.gluster.org/glusterfs" target="_blank">http://review.gluster.org/glusterfs</a> and now it works.<br>
> ><br>
> > Thanks,<br>
> > Kiran.<br>
> ><br>
> > On Tue, Oct 28, 2014 at 11:13 AM, Kiran Patil <<a href="mailto:kiran@fractalio.com">kiran@fractalio.com</a>> wrote:<br>
> ><br>
> >> Hi Niels,<br>
> >><br>
> >> I am getting "fatal: Couldn't find remote ref refs/changes/13/8213/9"<br>
> >> error.<br>
> >><br>
> >> Steps to reproduce the issue.<br>
> >><br>
> >> 1) # git clone git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a><br>
> >> Initialized empty Git repository in /root/gluster-3.6/glusterfs/.git/<br>
> >> remote: Counting objects: 84921, done.<br>
> >> remote: Compressing objects: 100% (48307/48307), done.<br>
> >> remote: Total 84921 (delta 57264), reused 63233 (delta 36254)<br>
> >> Receiving objects: 100% (84921/84921), 23.23 MiB | 192 KiB/s, done.<br>
> >> Resolving deltas: 100% (57264/57264), done.<br>
> >><br>
> >> 2) # cd glusterfs<br>
> >> # git branch<br>
> >> * master<br>
> >><br>
> >> 3) # git fetch git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a> refs/changes/13/8213/9<br>
> >> && git checkout FETCH_HEAD<br>
> >> fatal: Couldn't find remote ref refs/changes/13/8213/9<br>
> >><br>
> >> Note: I also tried the above steps on git repo<br>
> >> <a href="https://github.com/gluster/glusterfs" target="_blank">https://github.com/gluster/glusterfs</a> and the result is same as above.<br>
> >><br>
> >> Please let me know if I miss any steps.<br>
> >><br>
> >> Thanks,<br>
> >> Kiran.<br>
> >><br>
> >> On Mon, Oct 27, 2014 at 5:53 PM, Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:<br>
> >><br>
> >>> On Mon, Oct 27, 2014 at 05:19:13PM +0530, Kiran Patil wrote:<br>
> >>> > Hi,<br>
> >>> ><br>
> >>> > I created replicated vol with two bricks on the same node and copied<br>
> >>> some<br>
> >>> > data to it.<br>
> >>> ><br>
> >>> > Now removed the disk which has hosted one of the brick of the volume.<br>
> >>> ><br>
> >>> > Storage.health-check-interval is set to 30 seconds.<br>
> >>> ><br>
> >>> > I could see the disk is unavailable using zpool command of zfs on<br>
> >>> linux but<br>
> >>> > the gluster volume status still displays the brick process running<br>
> >>> which<br>
> >>> > should have been shutdown by this time.<br>
> >>> ><br>
> >>> > Is this a bug in 3.6 since it is mentioned as feature "<br>
> >>> ><br>
> >>> <a href="https://github.com/gluster/glusterfs/blob/release-3.6/doc/features/brick-failure-detection.md" target="_blank">https://github.com/gluster/glusterfs/blob/release-3.6/doc/features/brick-failure-detection.md</a><br>
> >>> "<br>
> >>> > or am I doing any mistakes here?<br>
> >>><br>
> >>> The initial detection of brick failures did not work for all<br>
> >>> filesystems. It may not work for ZFS too. A fix has been posted, but it<br>
> >>> has not been merged into the master branch yet. When the change has been<br>
> >>> merged, it can get backported to 3.6 and 3.5.<br>
> >>><br>
> >>> You may want to test with the patch applied, and add your "+1 Verified"<br>
> >>> to the change in case it makes it functional for you:<br>
> >>> - <a href="http://review.gluster.org/8213" target="_blank">http://review.gluster.org/8213</a><br>
> >>><br>
> >>> Cheers,<br>
> >>> Niels<br>
> >>><br>
> >>> ><br>
> >>> > [root@fractal-c92e gluster-3.6]# gluster volume status<br>
> >>> > Status of volume: repvol<br>
> >>> > Gluster process Port Online Pid<br>
> >>> ><br>
> >>> ------------------------------------------------------------------------------<br>
> >>> > Brick 192.168.1.246:/zp1/brick1 49154 Y 17671<br>
> >>> > Brick 192.168.1.246:/zp2/brick2 49155 Y 17682<br>
> >>> > NFS Server on localhost 2049 Y 17696<br>
> >>> > Self-heal Daemon on localhost N/A Y 17701<br>
> >>> ><br>
> >>> > Task Status of Volume repvol<br>
> >>> ><br>
> >>> ------------------------------------------------------------------------------<br>
> >>> > There are no active volume tasks<br>
> >>> ><br>
> >>> ><br>
> >>> > [root@fractal-c92e gluster-3.6]# gluster volume info<br>
> >>> ><br>
> >>> > Volume Name: repvol<br>
> >>> > Type: Replicate<br>
> >>> > Volume ID: d4f992b1-1393-43b8-9fda-2e2b6e3b5039<br>
> >>> > Status: Started<br>
> >>> > Number of Bricks: 1 x 2 = 2<br>
> >>> > Transport-type: tcp<br>
> >>> > Bricks:<br>
> >>> > Brick1: 192.168.1.246:/zp1/brick1<br>
> >>> > Brick2: 192.168.1.246:/zp2/brick2<br>
> >>> > Options Reconfigured:<br>
> >>> > storage.health-check-interval: 30<br>
> >>> ><br>
> >>> > [root@fractal-c92e gluster-3.6]# zpool status zp2<br>
> >>> > pool: zp2<br>
> >>> > state: UNAVAIL<br>
> >>> > status: One or more devices are faulted in response to IO failures.<br>
> >>> > action: Make sure the affected devices are connected, then run 'zpool<br>
> >>> > clear'.<br>
> >>> > see: <a href="http://zfsonlinux.org/msg/ZFS-8000-HC" target="_blank">http://zfsonlinux.org/msg/ZFS-8000-HC</a><br>
> >>> > scan: none requested<br>
> >>> > config:<br>
> >>> ><br>
> >>> > NAME STATE READ WRITE CKSUM<br>
> >>> > zp2 UNAVAIL 0 0 0 insufficient replicas<br>
> >>> > sdb UNAVAIL 0 0 0<br>
> >>> ><br>
> >>> > errors: 2 data errors, use '-v' for a list<br>
> >>> ><br>
> >>> ><br>
> >>> > Thanks,<br>
> >>> > Kiran.<br>
> >>><br>
> >>> > _______________________________________________<br>
> >>> > Gluster-devel mailing list<br>
> >>> > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> >>> > <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
> >>><br>
> >>><br>
> >><br>
> ><br>
</div></div></blockquote></div><br></div>