<div dir="ltr"><div>I applied the patches, compiled and installed the gluster.</div><div><br></div><div><div># glusterfs --version</div><div>glusterfs 3.7dev built on Oct 28 2014 12:03:10</div><div>Repository revision: git://<a href="http://git.gluster.com/glusterfs.git">git.gluster.com/glusterfs.git</a></div><div>Copyright (c) 2006-2013 Red Hat, Inc. <<a href="http://www.redhat.com/">http://www.redhat.com/</a>></div><div>GlusterFS comes with ABSOLUTELY NO WARRANTY.</div><div>It is licensed to you under your choice of the GNU Lesser</div><div>General Public License, version 3 or any later version (LGPLv3</div><div>or later), or the GNU General Public License, version 2 (GPLv2),</div><div>in all cases as published by the Free Software Foundation.</div></div><div><br></div><div># git log</div><div><div>commit 990ce16151c3af17e4cdaa94608b737940b60e4d</div><div>Author: Lalatendu Mohanty <<a href="mailto:lmohanty@redhat.com">lmohanty@redhat.com</a>></div><div>Date: Tue Jul 1 07:52:27 2014 -0400</div><div><br></div><div> Posix: Brick failure detection fix for ext4 filesystem</div></div><div>...</div><div>...</div><div><br></div>I see below messages<div><br></div><div>File /var/log/glusterfs/etc-glusterfs-glusterd.vol.log :<div><br></div><div><div>The message "I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick 192.168.1.246:/zp2/brick2 has disconnected from glusterd." repeated 39 times between [2014-10-28 05:58:09.209419] and [2014-10-28 06:00:06.226330]</div><div>[2014-10-28 06:00:09.226507] W [socket.c:545:__socket_rwv] 0-management: readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid argument)</div><div>[2014-10-28 06:00:09.226712] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick 192.168.1.246:/zp2/brick2 has disconnected from glusterd.</div><div>[2014-10-28 06:00:12.226881] W [socket.c:545:__socket_rwv] 0-management: readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid argument)</div><div>[2014-10-28 06:00:15.227249] W [socket.c:545:__socket_rwv] 0-management: readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid argument)</div><div>[2014-10-28 06:00:18.227616] W [socket.c:545:__socket_rwv] 0-management: readv on /var/run/6154ed2845b7f728a3acdce9d69e08ee.socket failed (Invalid argument)</div><div>[2014-10-28 06:00:21.227976] W [socket.c:545:__socket_rwv] 0-management: readv on </div><div><br></div><div>.....</div></div><div>.....</div><div><br></div><div><div>[2014-10-28 06:19:15.142867] I [glusterd-handler.c:1280:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req</div><div>The message "I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick 192.168.1.246:/zp2/brick2 has disconnected from glusterd." repeated 12 times between [2014-10-28 06:18:09.368752] and [2014-10-28 06:18:45.373063]</div><div>[2014-10-28 06:23:38.207649] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (15), shutting down</div></div><div><br></div><div><br></div></div><div>dmesg output:</div><div><br></div><div><div>SPLError: 7869:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has encountered an uncorrectable I/O failure and has been suspended.</div><div><br></div><div>SPLError: 7868:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has encountered an uncorrectable I/O failure and has been suspended.</div><div><br></div><div>SPLError: 7869:0:(spl-err.c:67:vcmn_err()) WARNING: Pool 'zp2' has encountered an uncorrectable I/O failure and has been suspended.</div></div><div><br></div><div>The brick is still online.</div><div><br></div><div><div># gluster volume status</div><div>Status of volume: repvol</div><div>Gluster process<span class="" style="white-space:pre"> </span>Port<span class="" style="white-space:pre"> </span>Online<span class="" style="white-space:pre"> </span>Pid</div><div>------------------------------------------------------------------------------</div><div>Brick 192.168.1.246:/zp1/brick1<span class="" style="white-space:pre"> </span>49152<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>4067</div><div>Brick 192.168.1.246:/zp2/brick2<span class="" style="white-space:pre"> </span>49153<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>4078</div><div>NFS Server on localhost<span class="" style="white-space:pre"> </span>2049<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>4092</div><div>Self-heal Daemon on localhost<span class="" style="white-space:pre"> </span>N/A<span class="" style="white-space:pre"> </span>Y<span class="" style="white-space:pre"> </span>4097</div><div> </div><div>Task Status of Volume repvol</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div> </div></div><div><div># gluster volume info</div><div> </div><div>Volume Name: repvol</div><div>Type: Replicate</div><div>Volume ID: ba1e7c6d-1e1c-45cd-8132-5f4fa4d2d22b</div><div>Status: Started</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.1.246:/zp1/brick1</div><div>Brick2: 192.168.1.246:/zp2/brick2</div><div>Options Reconfigured:</div><div>storage.health-check-interval: 30</div></div><div><br></div><div>Let me know if you need further information.<br></div><div><br></div><div>Thanks,</div><div>Kiran.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 28, 2014 at 11:44 AM, Kiran Patil <span dir="ltr"><<a href="mailto:kiran@fractalio.com" target="_blank">kiran@fractalio.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I changed <span style="font-family:arial,sans-serif;font-size:13px">git fetch git://</span><a href="http://review.gluster.org/glusterfs" style="font-family:arial,sans-serif;font-size:13px" target="_blank">review.gluster.org/glusterfs</a> to <span style="font-family:arial,sans-serif;font-size:13px">git fetch http://</span><a href="http://review.gluster.org/glusterfs" style="font-family:arial,sans-serif;font-size:13px" target="_blank">review.gluster.org/glusterfs</a> and now it works.<div><br></div><div>Thanks,</div><div>Kiran.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 28, 2014 at 11:13 AM, Kiran Patil <span dir="ltr"><<a href="mailto:kiran@fractalio.com" target="_blank">kiran@fractalio.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Niels,<div><br></div><div>I am getting "fatal: Couldn't find remote ref refs/changes/13/8213/9" error.</div><div><br></div><div>Steps to reproduce the issue.</div><div><br></div><div>1) # git clone git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a></div><div>Initialized empty Git repository in /root/gluster-3.6/glusterfs/.git/</div><div>remote: Counting objects: 84921, done.</div><div>remote: Compressing objects: 100% (48307/48307), done.</div><div>remote: Total 84921 (delta 57264), reused 63233 (delta 36254)</div><div>Receiving objects: 100% (84921/84921), 23.23 MiB | 192 KiB/s, done.</div><div>Resolving deltas: 100% (57264/57264), done.</div><div><br></div><div>2) # cd glusterfs</div><div> # git branch</div><div> * master</div><div><br></div><div>3) # git fetch git://<a href="http://review.gluster.org/glusterfs" target="_blank">review.gluster.org/glusterfs</a> refs/changes/13/8213/9 && git checkout FETCH_HEAD</div><div>fatal: Couldn't find remote ref refs/changes/13/8213/9</div><div><br></div><div>Note: I also tried the above steps on git repo <a href="https://github.com/gluster/glusterfs" target="_blank">https://github.com/gluster/glusterfs</a> and the result is same as above.</div><div><br></div><div>Please let me know if I miss any steps.</div><div><br></div><div>Thanks,</div><div>Kiran.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 27, 2014 at 5:53 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Mon, Oct 27, 2014 at 05:19:13PM +0530, Kiran Patil wrote:<br>
> Hi,<br>
><br>
> I created replicated vol with two bricks on the same node and copied some<br>
> data to it.<br>
><br>
> Now removed the disk which has hosted one of the brick of the volume.<br>
><br>
> Storage.health-check-interval is set to 30 seconds.<br>
><br>
> I could see the disk is unavailable using zpool command of zfs on linux but<br>
> the gluster volume status still displays the brick process running which<br>
> should have been shutdown by this time.<br>
><br>
> Is this a bug in 3.6 since it is mentioned as feature "<br>
> <a href="https://github.com/gluster/glusterfs/blob/release-3.6/doc/features/brick-failure-detection.md" target="_blank">https://github.com/gluster/glusterfs/blob/release-3.6/doc/features/brick-failure-detection.md</a>"<br>
> or am I doing any mistakes here?<br>
<br>
</span>The initial detection of brick failures did not work for all<br>
filesystems. It may not work for ZFS too. A fix has been posted, but it<br>
has not been merged into the master branch yet. When the change has been<br>
merged, it can get backported to 3.6 and 3.5.<br>
<br>
You may want to test with the patch applied, and add your "+1 Verified"<br>
to the change in case it makes it functional for you:<br>
- <a href="http://review.gluster.org/8213" target="_blank">http://review.gluster.org/8213</a><br>
<br>
Cheers,<br>
Niels<br>
<div><div><br>
><br>
> [root@fractal-c92e gluster-3.6]# gluster volume status<br>
> Status of volume: repvol<br>
> Gluster process Port Online Pid<br>
> ------------------------------------------------------------------------------<br>
> Brick 192.168.1.246:/zp1/brick1 49154 Y 17671<br>
> Brick 192.168.1.246:/zp2/brick2 49155 Y 17682<br>
> NFS Server on localhost 2049 Y 17696<br>
> Self-heal Daemon on localhost N/A Y 17701<br>
><br>
> Task Status of Volume repvol<br>
> ------------------------------------------------------------------------------<br>
> There are no active volume tasks<br>
><br>
><br>
> [root@fractal-c92e gluster-3.6]# gluster volume info<br>
><br>
> Volume Name: repvol<br>
> Type: Replicate<br>
> Volume ID: d4f992b1-1393-43b8-9fda-2e2b6e3b5039<br>
> Status: Started<br>
> Number of Bricks: 1 x 2 = 2<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: 192.168.1.246:/zp1/brick1<br>
> Brick2: 192.168.1.246:/zp2/brick2<br>
> Options Reconfigured:<br>
> storage.health-check-interval: 30<br>
><br>
> [root@fractal-c92e gluster-3.6]# zpool status zp2<br>
> pool: zp2<br>
> state: UNAVAIL<br>
> status: One or more devices are faulted in response to IO failures.<br>
> action: Make sure the affected devices are connected, then run 'zpool<br>
> clear'.<br>
> see: <a href="http://zfsonlinux.org/msg/ZFS-8000-HC" target="_blank">http://zfsonlinux.org/msg/ZFS-8000-HC</a><br>
> scan: none requested<br>
> config:<br>
><br>
> NAME STATE READ WRITE CKSUM<br>
> zp2 UNAVAIL 0 0 0 insufficient replicas<br>
> sdb UNAVAIL 0 0 0<br>
><br>
> errors: 2 data errors, use '-v' for a list<br>
><br>
><br>
> Thanks,<br>
> Kiran.<br>
<br>
</div></div>> _______________________________________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>