<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 09/06/2014 06:17 PM, Ramesh
Natarajan wrote:<br>
</div>
<blockquote
cite="mid:94FBCDB7-82A4-4946-89F1-666A6ED42BD8@gmail.com"
type="cite">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div>3.5.2 installed via rpm from official gluster repo, running
on amazon ami.</div>
</blockquote>
Including the client i.e. mount bits?<br>
<br>
Pranith<br>
<blockquote
cite="mid:94FBCDB7-82A4-4946-89F1-666A6ED42BD8@gmail.com"
type="cite">
<div><br>
</div>
<div>Thanks</div>
<div>Ramesh</div>
<div><br>
On Sep 6, 2014, at 7:29 AM, Pranith Kumar Karampuri <<a
moz-do-not-send="true" href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<meta content="text/html; charset=UTF-8"
http-equiv="Content-Type">
What is the glusterfs version where you ran into this issue?<br>
<br>
Pranith<br>
<div class="moz-cite-prefix">On 09/05/2014 11:52 PM, Ramesh
Natarajan wrote:<br>
</div>
<blockquote
cite="mid:CAGmOvD+2_CzioqteNuWT=bvFLRz1YW6kMme-X=+EHZKc_pHaAg@mail.gmail.com"
type="cite">
<div dir="ltr">I have a replicate glusterfs setup on 3
Bricks ( replicate = 3 ). I have client and server quorum
turned on. I rebooted one of the 3 bricks. When it came
back up, the client started throwing error messages that
one of the files went into split brain.
<div><br>
</div>
<div>When i check the file sizes and sha1sum on the
bricks, 2 of the 3 bricks have the same value. So by
quorum logic the first brick should have healed with
this information. But i don't see that happening. Can
someone please tell me if this is expected behavior?
<div><br>
</div>
<div><br>
</div>
<div>Can someone please tell me if i have things
misconfigured...</div>
<div><br>
</div>
<div>thanks</div>
<div>Ramesh<br>
<div><br>
</div>
<div>My config is as below.<br>
<div><br>
</div>
<div>
<div>[root@ip-172-31-12-218 ~]# gluster volume
info</div>
<div> </div>
<div>Volume Name: PL1</div>
<div>Type: Replicate</div>
<div>Volume ID:
a7aabae0-c6bc-40a9-8b26-0498d488ee39</div>
<div>Status: Started</div>
<div>Number of Bricks: 1 x 3 = 3</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: 172.31.38.189:/data/vol1/gluster-data</div>
<div>Brick2: 172.31.16.220:/data/vol1/gluster-data</div>
<div>Brick3: 172.31.12.218:/data/vol1/gluster-data</div>
<div>Options Reconfigured:</div>
<div>performance.cache-size: 2147483648</div>
<div>nfs.addr-namelookup: off</div>
<div>network.ping-timeout: 12</div>
<div>cluster.server-quorum-type: server</div>
<div>nfs.enable-ino32: on</div>
<div>cluster.quorum-type: auto</div>
<div>cluster.server-quorum-ratio: 51%</div>
<div> </div>
<div>Volume Name: PL2</div>
<div>Type: Replicate</div>
<div>Volume ID:
fadb3671-7a92-40b7-bccd-fbacf672f6dc</div>
<div>Status: Started</div>
<div>Number of Bricks: 1 x 3 = 3</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: 172.31.38.189:/data/vol2/gluster-data</div>
<div>Brick2: 172.31.16.220:/data/vol2/gluster-data</div>
<div>Brick3: 172.31.12.218:/data/vol2/gluster-data</div>
<div>Options Reconfigured:</div>
<div>performance.cache-size: 2147483648</div>
<div>nfs.addr-namelookup: off</div>
<div>network.ping-timeout: 12</div>
<div>cluster.server-quorum-type: server</div>
<div>nfs.enable-ino32: on</div>
<div>cluster.quorum-type: auto</div>
<div>cluster.server-quorum-ratio: 51%</div>
<div>[root@ip-172-31-12-218 ~]# </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>I have 2 clients each mounting one of the
volumes. At no time the same volume is mounted by
more than 1 client.</div>
<div><br>
</div>
<div>mount -t glusterfs -o
defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256
172.31.16.220:/PL2 /mnt/vm<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div>I restarted the Brick 1 172.31.38.189 and when it
came up, one of the file on PL2 volume went into
split mode..</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>[2014-09-05 17:59:42.997308] W
[afr-open.c:209:afr_open] 0-PL2-replicate-0:
failed to open as split brain seen, returning EIO</div>
<div>[2014-09-05 17:59:42.997350] W
[fuse-bridge.c:2209:fuse_writev_cbk]
0-glusterfs-fuse: 3359683: WRITE => -1
(Input/output error)</div>
<div>[2014-09-05 17:59:42.997476] W
[fuse-bridge.c:690:fuse_truncate_cbk]
0-glusterfs-fuse: 3359684: FTRUNCATE() ERR =>
-1 (Input/</div>
<div>output error)[2014-09-05 17:59:42.997647] W
[fuse-bridge.c:2209:fuse_writev_cbk]
0-glusterfs-fuse: 3359686: WRITE => -1
(Input/output erro</div>
<div>r)[2014-09-05 17:59:42.997783] W
[fuse-bridge.c:1214:fuse_err_cbk]
0-glusterfs-fuse: 3359687: FLUSH() ERR => -1
(Input/output e</div>
<div>rror)[2014-09-05 17:59:44.009187] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-PL2-replicate-0: Unable to self-he</div>
<div>al contents of
'/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00'
(possible split-brain). Please delete the file
from all but the preferred subvolume.- Pending
matrix: [ [ 0 1 1 ] [ 3398 0 0 ] [ 3398 0 0 ] ]</div>
<div>[2014-09-05 17:59:44.011116] E
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
0-PL2-replicate-0: backgroung data self heal
failed, on
/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00</div>
<div>[2014-09-05 17:59:44.011480] W
[afr-open.c:209:afr_open] 0-PL2-replicate-0:
failed to open as split brain seen, returning EIO</div>
</div>
<div><br>
</div>
<div>
<div>Starting time of crawl: Fri Sep 5 17:55:32
2014</div>
<div><br>
</div>
<div>Ending time of crawl: Fri Sep 5 17:55:33 2014</div>
<div><br>
</div>
<div>Type of crawl: INDEX</div>
<div>No. of entries healed: 4</div>
<div>No. of entries in split-brain: 1</div>
<div>No. of heal failed entries: 0</div>
<div>[root@ip-172-31-16-220 ~]# gluster volume heal
PL2 info</div>
<div>Brick ip-172-31-38-189:/data/vol2/gluster-data/</div>
<div>/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00</div>
<div>Number of entries: 1</div>
<div><br>
</div>
<div>Brick ip-172-31-16-220:/data/vol2/gluster-data/</div>
<div>/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00</div>
<div>Number of entries: 1</div>
<div><br>
</div>
<div>Brick ip-172-31-12-218:/data/vol2/gluster-data/</div>
<div>/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00</div>
<div>Number of entries: 1</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>BRICK1<br>
</div>
<div>========</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-38-189 ~]# sha1sum
access_log.2014-09-05-17_00_00</div>
<div>aa72d0f3949700f67b61d3c58fdbc75b772d607b
access_log.2014-09-05-17_00_00</div>
</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-38-189 ~]# ls -al </div>
<div>total 12760</div>
<div>dr-xr-x--- 3 root root 4096 Sep 5
17:42 .</div>
<div>dr-xr-xr-x 24 root root 4096 Sep 5
17:34 ..</div>
<div>-rw-r----- 1 root root 13019808 Sep 5
17:42 access_log.2014-09-05-17_00_00</div>
</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-38-189 ~]# getfattr -d -m . -e
hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00 </div>
<div>getfattr: Removing leading '/' from absolute
path names</div>
<div># file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00</div>
<div>trusted.afr.PL2-client-0=0x000000000000000000000000</div>
<div>trusted.afr.PL2-client-1=0x000000010000000000000000</div>
<div>trusted.afr.PL2-client-2=0x000000010000000000000000</div>
<div>trusted.gfid=0xea950263977e46bf89a0ef631ca139c2</div>
<div><br>
</div>
</div>
<div><br>
</div>
<div>BRICK 2</div>
<div>=======</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-16-220 ~]# sha1sum
access_log.2014-09-05-17_00_00</div>
<div>0f7b72f77a792b5c2b68456c906cf7b93287f0d6
access_log.2014-09-05-17_00_00</div>
</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-16-220 ~]# getfattr -d -m . -e
hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00 </div>
<div>getfattr: Removing leading '/' from absolute
path names</div>
<div># file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00</div>
<div>trusted.afr.PL2-client-0=0x00000d460000000000000000</div>
<div>trusted.afr.PL2-client-1=0x000000000000000000000000</div>
<div>trusted.afr.PL2-client-2=0x000000000000000000000000</div>
<div>trusted.gfid=0xea950263977e46bf89a0ef631ca139c2</div>
</div>
<div><br>
</div>
<div>BRICK 3</div>
<div>=========</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-12-218 ~]# sha1sum
access_log.2014-09-05-17_00_00</div>
<div>0f7b72f77a792b5c2b68456c906cf7b93287f0d6
access_log.2014-09-05-17_00_00</div>
</div>
<div><br>
</div>
<div>
<div>[root@ip-172-31-12-218 ~]# getfattr -d -m . -e
hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00 </div>
<div>getfattr: Removing leading '/' from absolute
path names</div>
<div># file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00</div>
<div>trusted.afr.PL2-client-0=0x00000d460000000000000000</div>
<div>trusted.afr.PL2-client-1=0x000000000000000000000000</div>
<div>trusted.afr.PL2-client-2=0x000000000000000000000000</div>
<div>trusted.gfid=0xea950263977e46bf89a0ef631ca139c2</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</blockquote>
<br>
</body>
</html>