<p dir="ltr">On 19-Jul-2014 11:06 pm, "Niels de Vos" <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:<br>
><br>
> On Sat, Jul 19, 2014 at 08:23:29AM +0530, Pranith Kumar Karampuri wrote:<br>
> > Guys,<br>
> > Does anyone know why device-id can be different even though it<br>
> > is all single xfs filesystem?<br>
> > We see the following log in the brick-log.<br>
> ><br>
> > [2014-07-16 00:00:24.358628] W [posix-handle.c:586:posix_handle_hard]<br>
> > 0-home-posix: mismatching ino/dev between file<br>
><br>
> The device-id (major:minor number) of a block-device can change, but<br>
> will not change while the device is in use. Device-mapper (DM) is part<br>
> of the stack that includes multipath and lvm (and more, but these are<br>
> most common). The stack for the block-devices is built dynamically, and<br>
> the device-id is assigned when the block-device is made active. The<br>
> ordering of making devices active can change, hence the device-id too.<br>
> It is also possible to deactivate some logical-volumes, and activate<br>
> them in a different order. (You can not deactivate a dm-device when it<br>
> is in use, for example mounted.)<br>
><br>
> Without device-mapper in the io-stack, re-ordering disks is possible<br>
> too, but requires a little more (advanced sysadmin) work.<br>
><br>
> So, the main questions I'd ask would be:<br>
> 1. What kind of block storage is used, LVM, multipath, ...?</p>
<p dir="ltr">A single RAID10 XFS partition</p>
<p dir="ltr">> 2. Were there any issues on the block-layer, scsi-errors, reconnects?</p>
<p dir="ltr">Yes, one of the servers had a bad disk that was replaced</p>
<p dir="ltr">> 3. Were there changes in the underlaying disks or their structure? Disks<br>
> added, removed or new partitions created.</p>
<p dir="ltr">No </p>
<p dir="ltr">> 4. Were disks deactivated+activated again, for example for creating<br>
> backups or snapshots on a level below the (XFS) filesystem?<br>
></p>
<p dir="ltr">No</p>
<p dir="ltr">> HTH,<br>
> Niels<br>
><br>
> > /data/gluster/home/techiebuzz/<a href="http://techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old">techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old</a><br>
> > (1077282838/2431) and handle<br>
> > /data/gluster/home/.glusterfs/ae/f0/aef0404b-e084-4501-9d0f-0e6f5bb2d5e0<br>
> > (1077282836/2431)<br>
> > [2014-07-16 00:00:24.358646] E [posix.c:823:posix_mknod] 0-home-posix:<br>
> > setting gfid on<br>
> > /data/gluster/home/techiebuzz/<a href="http://techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old">techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old</a><br>
> > failed<br>
> ><br>
> ><br>
> > Pranith<br>
> > On 07/17/2014 07:06 PM, Nilesh Govindrajan wrote:<br>
> > >log1 was the log from client of node2. The filesystems are mounted<br>
> > >locally. /data is a raid10 array and /data/gluster contains 4 volumes,<br>
> > >one of which is home which is a high read/write one (the log of which<br>
> > >was attached here).<br>
> > ><br>
> > >On Thu, Jul 17, 2014 at 11:54 AM, Pranith Kumar Karampuri<br>
> > ><<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
> > >>On 07/17/2014 08:41 AM, Nilesh Govindrajan wrote:<br>
> > >>>log1 and log2 are brick logs. The others are client logs.<br>
> > >>I see a lot of logs as below in 'log1' you attached. It seems like the<br>
> > >>device ID of where the file where it is actually stored, where the gfid-link<br>
> > >>of the same file is stored i.e inside <brick-dir>/.glusterfs/ are different.<br>
> > >>What all devices/filesystems are present inside the brick represented by<br>
> > >>'log1'?<br>
> > >><br>
> > >>[2014-07-16 00:00:24.358628] W [posix-handle.c:586:posix_handle_hard]<br>
> > >>0-home-posix: mismatching ino/dev between file<br>
> > >>/data/gluster/home/techiebuzz/<a href="http://techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old">techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old</a><br>
> > >>(1077282838/2431) and handle<br>
> > >>/data/gluster/home/.glusterfs/ae/f0/aef0404b-e084-4501-9d0f-0e6f5bb2d5e0<br>
> > >>(1077282836/2431)<br>
> > >>[2014-07-16 00:00:24.358646] E [posix.c:823:posix_mknod] 0-home-posix:<br>
> > >>setting gfid on<br>
> > >>/data/gluster/home/techiebuzz/<a href="http://techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old">techie-buzz.com/wp-content/cache/page_enhanced/techie-buzz.com/social-networking/facebook-will-permanently-remove-your-deleted-photos.html/_index.html.old</a><br>
> > >>failed<br>
> > >><br>
> > >>Pranith<br>
> > >><br>
> > >><br>
> > >>>On Thu, Jul 17, 2014 at 8:08 AM, Pranith Kumar Karampuri<br>
> > >>><<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
> > >>>>On 07/17/2014 07:28 AM, Nilesh Govindrajan wrote:<br>
> > >>>>>On Thu, Jul 17, 2014 at 7:26 AM, Nilesh Govindrajan <<a href="mailto:me@nileshgr.com">me@nileshgr.com</a>><br>
> > >>>>>wrote:<br>
> > >>>>>>Hello,<br>
> > >>>>>><br>
> > >>>>>>I'm having a weird issue. I have this config:<br>
> > >>>>>><br>
> > >>>>>>node2 ~ # gluster peer status<br>
> > >>>>>>Number of Peers: 1<br>
> > >>>>>><br>
> > >>>>>>Hostname: sto1<br>
> > >>>>>>Uuid: f7570524-811a-44ed-b2eb-d7acffadfaa5<br>
> > >>>>>>State: Peer in Cluster (Connected)<br>
> > >>>>>><br>
> > >>>>>>node1 ~ # gluster peer status<br>
> > >>>>>>Number of Peers: 1<br>
> > >>>>>><br>
> > >>>>>>Hostname: sto2<br>
> > >>>>>>Port: 24007<br>
> > >>>>>>Uuid: 3a69faa9-f622-4c35-ac5e-b14a6826f5d9<br>
> > >>>>>>State: Peer in Cluster (Connected)<br>
> > >>>>>><br>
> > >>>>>>Volume Name: home<br>
> > >>>>>>Type: Replicate<br>
> > >>>>>>Volume ID: 54fef941-2e33-4acf-9e98-1f86ea4f35b7<br>
> > >>>>>>Status: Started<br>
> > >>>>>>Number of Bricks: 1 x 2 = 2<br>
> > >>>>>>Transport-type: tcp<br>
> > >>>>>>Bricks:<br>
> > >>>>>>Brick1: sto1:/data/gluster/home<br>
> > >>>>>>Brick2: sto2:/data/gluster/home<br>
> > >>>>>>Options Reconfigured:<br>
> > >>>>>>performance.write-behind-window-size: 2GB<br>
> > >>>>>>performance.flush-behind: on<br>
> > >>>>>>performance.cache-size: 2GB<br>
> > >>>>>>cluster.choose-local: on<br>
> > >>>>>>storage.linux-aio: on<br>
> > >>>>>>transport.keepalive: on<br>
> > >>>>>>performance.quick-read: on<br>
> > >>>>>>performance.io-cache: on<br>
> > >>>>>>performance.stat-prefetch: on<br>
> > >>>>>>performance.read-ahead: on<br>
> > >>>>>>cluster.data-self-heal-algorithm: diff<br>
> > >>>>>>nfs.disable: on<br>
> > >>>>>><br>
> > >>>>>>sto1/2 is alias to node1/2 respectively.<br>
> > >>>>>><br>
> > >>>>>>As you see, NFS is disabled so I'm using the native fuse mount on both<br>
> > >>>>>>nodes.<br>
> > >>>>>>The volume contains files and php scripts that are served on various<br>
> > >>>>>>websites. When both nodes are active, I get split brain on many files<br>
> > >>>>>>and the mount on node2 going 'input/output error' on many of them<br>
> > >>>>>>which causes HTTP 500 errors.<br>
> > >>>>>><br>
> > >>>>>>I delete the files from the brick using find -samefile. It fixes for a<br>
> > >>>>>>few minutes and then the problem is back.<br>
> > >>>>>><br>
> > >>>>>>What could be the issue? This happens even if I use the NFS mounting<br>
> > >>>>>>method.<br>
> > >>>>>><br>
> > >>>>>>Gluster 3.4.4 on Gentoo.<br>
> > >>>>>And yes, network connectivity is not an issue between them as both of<br>
> > >>>>>them are located in the same DC. They're connected via 1 Gbit line<br>
> > >>>>>(common for internal and external network) but external network<br>
> > >>>>>doesn't cross 200-500 Mbit/s leaving quite a good window for gluster.<br>
> > >>>>>I also tried enabling quorum but that doesn't help either.<br>
> > >>>>>_______________________________________________<br>
> > >>>>>Gluster-users mailing list<br>
> > >>>>><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > >>>>><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
> > >>>>hi Nilesh,<br>
> > >>>> Could you attach the mount, brick logs so that we can inspect what<br>
> > >>>>is<br>
> > >>>>going on the setup.<br>
> > >>>><br>
> > >>>>Pranith<br>
> > >><br>
> ><br>
> > _______________________________________________<br>
> > Gluster-devel mailing list<br>
> > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
</p>