<div dir="ltr">I am not seeing below message in any log files under /var/log/glusterfs directroy and its subdirectories.<div><br></div><div><pre class="" style="font-size:9pt;margin-top:0px;margin-bottom:0px;color:rgb(53,53,53)">health-check failed, going down</pre></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 31, 2014 at 3:16 PM, Kiran Patil <span dir="ltr"><<a href="mailto:kiran@fractalio.com" target="_blank">kiran@fractalio.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I set zfs pool failmode to continue, which should disable only write and not read as explained below<div><br></div><div><div>failmode=wait | continue | panic</div><div><br></div><div> Controls the system behavior in the event of catastrophic pool failure. This condition is typically a result of a loss of connec-</div><div> tivity to the underlying storage device(s) or a failure of all devices within the pool. The behavior of such an event is deter-</div><div> mined as follows:</div><div><br></div><div> wait Blocks all I/O access until the device connectivity is recovered and the errors are cleared. This is the default</div><div> behavior.</div><div><br></div><div> continue Returns EIO to any new write I/O requests but allows reads to any of the remaining healthy devices. Any write</div><div> requests that have yet to be committed to disk would be blocked.</div><div><br></div><div> panic Prints out a message to the console and generates a system crash dump.</div></div><div><br></div><div><br></div><div>Now, I rebuilt the glusterfs master and tried to see if failed driver results in failed brick and in turn kill brick process and the brick is not going offline.<br></div><div><br></div><div><span class=""><div># gluster volume status</div><div>Status of volume: repvol</div><div>Gluster process<span style="white-space:pre-wrap"> </span>Port<span style="white-space:pre-wrap"> </span>Online<span style="white-space:pre-wrap"> </span>Pid</div><div>------------------------------------------------------------------------------</div></span><div>Brick 192.168.1.246:/zp1/brick1<span style="white-space:pre-wrap"> </span>49152<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>2400</div><div>Brick 192.168.1.246:/zp2/brick2<span style="white-space:pre-wrap"> </span>49153<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>2407</div><div>NFS Server on localhost<span style="white-space:pre-wrap"> </span>2049<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>30488</div><div>Self-heal Daemon on localhost<span style="white-space:pre-wrap"> </span>N/A<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>30495</div><span class=""><div> </div><div>Task Status of Volume repvol</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div></span></div><div><br></div><div><br></div><div>The /var/log/gluster/mnt.log output:</div><div><br></div><div><div>[2014-10-31 09:18:15.934700] W [rpc-clnt-ping.c:154:rpc_clnt_ping_cbk] 0-repvol-client-1: socket disconnected</div><div>[2014-10-31 09:18:15.934725] I [client.c:2215:client_rpc_notify] 0-repvol-client-1: disconnected from repvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available</div><div>[2014-10-31 09:18:15.935238] I [rpc-clnt.c:1765:rpc_clnt_reconfig] 0-repvol-client-1: changing port to 49153 (from 0)</div></div><div><br></div><div>Now if I copy a file to /mnt it copied without any hang and brick still shows online.</div><div><br></div><div>Thanks,</div><div>Kiran.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 28, 2014 at 3:44 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, Oct 28, 2014 at 02:08:32PM +0530, Kiran Patil wrote:<br>
> The content of file zp2-brick2.log is at <a href="http://ur1.ca/iku0l" target="_blank">http://ur1.ca/iku0l</a> (<br>
> <a href="http://fpaste.org/145714/44849041/" target="_blank">http://fpaste.org/145714/44849041/</a> )<br>
><br>
> I can't open the file /zp2/brick2/.glusterfs/health_check since it hangs<br>
> due to no disk present.<br>
><br>
> Let me know the filename pattern, so that I can find it.<br>
<br>
</span>Hmm, if there is a hang while reading from the disk, it will not get<br>
detected in the current solution. We implemented failure detection on<br>
top of the detection that is done by the filesystem. Suspending a<br>
filesystem with fsfreeze or similar should probably not be seen as a<br>
failure.<br>
<br>
In your case, it seems that the filesystem suspends itself when the disk<br>
went away. I have no idea if it is possible to configure ZFS to not<br>
suspend, but return an error to the reading/writing application. Please<br>
check with such an option.<br>
<br>
If you find such an option, please update the wiki page and recommend<br>
enabling it:<br>
- <a href="http://gluster.org/community/documentation/index.php/GlusterOnZFS" target="_blank">http://gluster.org/community/documentation/index.php/GlusterOnZFS</a><br>
<br>
<br>
Thanks,<br>
Niels<br>
<div><div><br>
<br>
><br>
> On Tue, Oct 28, 2014 at 1:42 PM, Niels de Vos <<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>> wrote:<br>
><br>
> > On Tue, Oct 28, 2014 at 01:10:56PM +0530, Kiran Patil wrote:<br>
> > > I applied the patches, compiled and installed the gluster.<br>
> > ><br>
> > > # glusterfs --version<br>
> > > glusterfs 3.7dev built on Oct 28 2014 12:03:10<br>
> > > Repository revision: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a><br>
> > > Copyright (c) 2006-2013 Red Hat, Inc. <<a href="http://www.redhat.com/" target="_blank">http://www.redhat.com/</a>><br>
> > > GlusterFS comes with ABSOLUTELY NO WARRANTY.<br>
> > > It is licensed to you under your choice of the GNU Lesser<br>
> > > General Public License, version 3 or any later version (LGPLv3<br>
> > > or later), or the GNU General Public License, version 2 (GPLv2),<br>
> > > in all cases as published by the Free Software Foundation.<br>
> > ><br>
> > > # git log<br>
> > > commit 990ce16151c3af17e4cdaa94608b737940b60e4d<br>
> > > Author: Lalatendu Mohanty <<a href="mailto:lmohanty@redhat.com" target="_blank">lmohanty@redhat.com</a>><br>
> > > Date: Tue Jul 1 07:52:27 2014 -0400<br>
> > ><br>
> > > Posix: Brick failure detection fix for ext4 filesystem<br>
> > > ...<br>
> > > ...<br>
> > ><br>
> > > I see below messages<br>
> ><br>
> > Many thanks Kiran!<br>
> ><br>
> > Do you have the messages from the brick that uses the zp2 mountpoint?<br>
> ><br>
> > There also should be a file with a timestamp when the last check was<br>
> > done successfully. If the brick is still running, this timestamp should<br>
> > get updated every storage.health-check-interval seconds:<br>
> > /zp2/brick2/.glusterfs/health_check<br>
> ><br>
> > Niels<br>
> ><br>
> > ><br>
> > > File /var/log/glusterfs/etc-glusterfs-glusterd.vol.log :<br>
> > ><br>
> > > The message "I [MSGID: 106005]<br>
> > > [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick<br>
> > > 192.168.1.246:/zp2/brick2 has disconnected from glusterd." repeated 39<br>
> > > times between [2014-10-28 05:58:09.209419] and [2014-10-28<br>
> > 06:00:06.226330]<br>
> > > [2014-10-28 06:00:09.226507] W [socket.c:545:__socket_rwv] 0-management:<br>
> > > readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> > > argument)<br>
> > > [2014-10-28 06:00:09.226712] I [MSGID: 106005]<br>
> > > [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick<br>
> > > 192.168.1.246:/zp2/brick2 has disconnected from glusterd.<br>
> > > [2014-10-28 06:00:12.226881] W [socket.c:545:__socket_rwv] 0-management:<br>
> > > readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> > > argument)<br>
> > > [2014-10-28 06:00:15.227249] W [socket.c:545:__socket_rwv] 0-management:<br>
> > > readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> > > argument)<br>
> > > [2014-10-28 06:00:18.227616] W [socket.c:545:__socket_rwv] 0-management:<br>
> > > readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid<br>
> > > argument)<br>
> > > [2014-10-28 06:00:21.227976] W [socket.c:545:__socket_rwv] 0-management:<br>
> > > readv on<br>
> > ><br>
> > > .....<br>
> > > .....<br>
> > ><br>
> > > [2014-10-28 06:19:15.142867] I<br>
> > > [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd:<br>
> > > Received get vol req<br>
> > > The message "I [MSGID: 106005]<br>
> > > [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick<br>
> > > 192.168.1.246:/zp2/brick2 has disconnected from glusterd." repeated 12<br>
> > > times between [2014-10-28 06:18:09.368752] and [2014-10-28<br>
> > 06:18:45.373063]<br>
> > > [2014-10-28 06:23:38.207649] W [glusterfsd.c:1194:cleanup_and_exit] (--><br>
> > > 0-: received signum (15), shutting down<br>
> > ><br>
> > ><br>
> > > dmesg output:<br>
> > ><br>
> > > SPLError: 7869:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has<br>
> > > encountered an uncorrectable I/O failure and has been suspended.<br>
> > ><br>
> > > SPLError: 7868:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has<br>
> > > encountered an uncorrectable I/O failure and has been suspended.<br>
> > ><br>
> > > SPLError: 7869:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has<br>
> > > encountered an uncorrectable I/O failure and has been suspended.<br>
> > ><br>
> > > The brick is still online.<br>
> > ><br>
> > > # gluster volume status<br>
> > > Status of volume: repvol<br>
> > > Gluster process Port Online Pid<br>
> > ><br>
> > ------------------------------------------------------------------------------<br>
> > > Brick 192.168.1.246:/zp1/brick1 49152 Y 4067<br>
> > > Brick 192.168.1.246:/zp2/brick2 49153 Y 4078<br>
> > > NFS Server on localhost 2049 Y 4092<br>
> > > Self-heal Daemon on localhost N/A Y 4097<br>
> > ><br>
> > > Task Status of Volume repvol<br>
> > ><br>
> > ------------------------------------------------------------------------------<br>
> > > There are no active volume tasks<br>
> > ><br>
> > > # gluster volume info<br>
> > ><br>
> > > Volume Name: repvol<br>
> > > Type: Replicate<br>
> > > Volume ID: ba1e7c6d-1e1c-45cd-8132-5f4fa4d2d22b<br>
> > > Status: Started<br>
> > > Number of Bricks: 1 x 2 = 2<br>
> > > Transport-type: tcp<br>
> > > Bricks:<br>
> > > Brick1: 192.168.1.246:/zp1/brick1<br>
> > > Brick2: 192.168.1.246:/zp2/brick2<br>
> > > Options Reconfigured:<br>
> > > storage.health-check-interval: 30<br>
> > ><br>
> > > Let me know if you need further information.<br>
> > ><br>
> > > Thanks,<br>
> > > Kiran.<br>
> > ><br>
> > > On Tue, Oct 28, 2014 at 11:44 AM, Kiran Patil <<a href="mailto:kiran@fractalio.com" target="_blank">kiran@fractalio.com</a>><br>
> > wrote:<br>
> > ><br>
> > > > I changed git fetch git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a> to git fetch<br>
> > > > <a href="http://review.gluster.org/glusterfs" target="_blank">http://review.gluster.org/glusterfs</a> and now it works.<br>
> > > ><br>
> > > > Thanks,<br>
> > > > Kiran.<br>
> > > ><br>
> > > > On Tue, Oct 28, 2014 at 11:13 AM, Kiran Patil <<a href="mailto:kiran@fractalio.com" target="_blank">kiran@fractalio.com</a>><br>
> > wrote:<br>
> > > ><br>
> > > >> Hi Niels,<br>
> > > >><br>
> > > >> I am getting "fatal: Couldn't find remote ref refs/changes/13/8213/9"<br>
> > > >> error.<br>
> > > >><br>
> > > >> Steps to reproduce the issue.<br>
> > > >><br>
> > > >> 1) # git clone git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a><br>
> > > >> Initialized empty Git repository in /root/gluster-3.6/glusterfs/.git/<br>
> > > >> remote: Counting objects: 84921, done.<br>
> > > >> remote: Compressing objects: 100% (48307/48307), done.<br>
> > > >> remote: Total 84921 (delta 57264), reused 63233 (delta 36254)<br>
> > > >> Receiving objects: 100% (84921/84921), 23.23 MiB | 192 KiB/s, done.<br>
> > > >> Resolving deltas: 100% (57264/57264), done.<br>
> > > >><br>
> > > >> 2) # cd glusterfs<br>
> > > >> # git branch<br>
> > > >> * master<br>
> > > >><br>
> > > >> 3) # git fetch git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a><br>
> > refs/changes/13/8213/9<br>
> > > >> && git checkout FETCH_HEAD<br>
> > > >> fatal: Couldn't find remote ref refs/changes/13/8213/9<br>
> > > >><br>
> > > >> Note: I also tried the above steps on git repo<br>
> > > >> <a href="https://github.com/gluster/glusterfs" target="_blank">https://github.com/gluster/glusterfs</a> and the result is same as above.<br>
> > > >><br>
> > > >> Please let me know if I miss any steps.<br>
> > > >><br>
> > > >> Thanks,<br>
> > > >> Kiran.<br>
> > > >><br>
> > > >> On Mon, Oct 27, 2014 at 5:53 PM, Niels de Vos <<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>><br>
> > wrote:<br>
> > > >><br>
> > > >>> On Mon, Oct 27, 2014 at 05:19:13PM +0530, Kiran Patil wrote:<br>
> > > >>> > Hi,<br>
> > > >>> ><br>
> > > >>> > I created replicated vol with two bricks on the same node and<br>
> > copied<br>
> > > >>> some<br>
> > > >>> > data to it.<br>
> > > >>> ><br>
> > > >>> > Now removed the disk which has hosted one of the brick of the<br>
> > volume.<br>
> > > >>> ><br>
> > > >>> > Storage.health-check-interval is set to 30 seconds.<br>
> > > >>> ><br>
> > > >>> > I could see the disk is unavailable using zpool command of zfs on<br>
> > > >>> linux but<br>
> > > >>> > the gluster volume status still displays the brick process running<br>
> > > >>> which<br>
> > > >>> > should have been shutdown by this time.<br>
> > > >>> ><br>
> > > >>> > Is this a bug in 3.6 since it is mentioned as feature "<br>
> > > >>> ><br>
> > > >>><br>
> > <a href="https://github.com/gluster/glusterfs/blob/release-3.6/doc/features/brick-failure-detection.md" target="_blank">https://github.com/gluster/glusterfs/blob/release-3.6/doc/features/brick-failure-detection.md</a><br>
> > > >>> "<br>
> > > >>> > or am I doing any mistakes here?<br>
> > > >>><br>
> > > >>> The initial detection of brick failures did not work for all<br>
> > > >>> filesystems. It may not work for ZFS too. A fix has been posted, but<br>
> > it<br>
> > > >>> has not been merged into the master branch yet. When the change has<br>
> > been<br>
> > > >>> merged, it can get backported to 3.6 and 3.5.<br>
> > > >>><br>
> > > >>> You may want to test with the patch applied, and add your "+1<br>
> > Verified"<br>
> > > >>> to the change in case it makes it functional for you:<br>
> > > >>> - <a href="http://review.gluster.org/8213" target="_blank">http://review.gluster.org/8213</a><br>
> > > >>><br>
> > > >>> Cheers,<br>
> > > >>> Niels<br>
> > > >>><br>
> > > >>> ><br>
> > > >>> > [root@fractal-c92e gluster-3.6]# gluster volume status<br>
> > > >>> > Status of volume: repvol<br>
> > > >>> > Gluster process Port Online Pid<br>
> > > >>> ><br>
> > > >>><br>
> > ------------------------------------------------------------------------------<br>
> > > >>> > Brick 192.168.1.246:/zp1/brick1 49154 Y 17671<br>
> > > >>> > Brick 192.168.1.246:/zp2/brick2 49155 Y 17682<br>
> > > >>> > NFS Server on localhost 2049 Y 17696<br>
> > > >>> > Self-heal Daemon on localhost N/A Y 17701<br>
> > > >>> ><br>
> > > >>> > Task Status of Volume repvol<br>
> > > >>> ><br>
> > > >>><br>
> > ------------------------------------------------------------------------------<br>
> > > >>> > There are no active volume tasks<br>
> > > >>> ><br>
> > > >>> ><br>
> > > >>> > [root@fractal-c92e gluster-3.6]# gluster volume info<br>
> > > >>> ><br>
> > > >>> > Volume Name: repvol<br>
> > > >>> > Type: Replicate<br>
> > > >>> > Volume ID: d4f992b1-1393-43b8-9fda-2e2b6e3b5039<br>
> > > >>> > Status: Started<br>
> > > >>> > Number of Bricks: 1 x 2 = 2<br>
> > > >>> > Transport-type: tcp<br>
> > > >>> > Bricks:<br>
> > > >>> > Brick1: 192.168.1.246:/zp1/brick1<br>
> > > >>> > Brick2: 192.168.1.246:/zp2/brick2<br>
> > > >>> > Options Reconfigured:<br>
> > > >>> > storage.health-check-interval: 30<br>
> > > >>> ><br>
> > > >>> > [root@fractal-c92e gluster-3.6]# zpool status zp2<br>
> > > >>> > pool: zp2<br>
> > > >>> > state: UNAVAIL<br>
> > > >>> > status: One or more devices are faulted in response to IO failures.<br>
> > > >>> > action: Make sure the affected devices are connected, then run<br>
> > 'zpool<br>
> > > >>> > clear'.<br>
> > > >>> > see: <a href="http://zfsonlinux.org/msg/ZFS-8000-HC" target="_blank">http://zfsonlinux.org/msg/ZFS-8000-HC</a><br>
> > > >>> > scan: none requested<br>
> > > >>> > config:<br>
> > > >>> ><br>
> > > >>> > NAME STATE READ WRITE CKSUM<br>
> > > >>> > zp2 UNAVAIL 0 0 0 insufficient replicas<br>
> > > >>> > sdb UNAVAIL 0 0 0<br>
> > > >>> ><br>
> > > >>> > errors: 2 data errors, use '-v' for a list<br>
> > > >>> ><br>
> > > >>> ><br>
> > > >>> > Thanks,<br>
> > > >>> > Kiran.<br>
> > > >>><br>
> > > >>> > _______________________________________________<br>
> > > >>> > Gluster-devel mailing list<br>
> > > >>> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > >>> > <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > > >>><br>
> > > >>><br>
> > > >><br>
> > > ><br>
> ><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>