<html><body><div style="font-family: Arial; font-size: 10pt; color: #000000"><div>Hi All,</div><div><br></div><div>in a distributed-replicated volume hosting some VMs disk images (GlusterFS 3.4.2 on CentOS 6.5, qemu-kvm with glusterfs native support, no fuse mount), I always get the same two files that need healing:</div><div><br></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">[root@networker ~]# gluster volume heal gv_pri info</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Gathering Heal info on volume gv_pri has been successful</span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick nw1glus.gem.local:/glustexp/pri1/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Number of entries: 2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">/alfresco.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">/remlog.qc2</span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick nw2glus.gem.local:/glustexp/pri1/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Number of entries: 2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">/alfresco.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">/remlog.qc2</span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick nw3glus.gem.local:/glustexp/pri2/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Number of entries: 0</span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick nw4glus.gem.local:/glustexp/pri2/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Number of entries: 0</span></p></div><div><br></div><div>This is not a split-brain situation (I checked) and If I stop the two VMs that use these images, I get the two files healed/synced in about 15min. This is too much time, IMHO.</div><div><span style="font-size: 10pt;">In this volume there are other VM images with (smaller) disk images replicated on the same bricks and they get synced "in real-time".</span></div><div><br></div><div>These are the volume's details, the host "networker" is <span style="font-family: arial, helvetica, sans-serif;">nw1glus.gem.local</span>:</div><div><br></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">[root@networker ~]# gluster volume info gv_pri</span><br> <br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Volume Name: gv_pri</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Type: Distributed-Replicate</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Volume ID: 3d91b91e-4d72-484f-8655-e5ed8d38bb28</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Status: Started</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Number of Bricks: 2 x 2 = 4</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Transport-type: tcp</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Bricks:</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick1: nw1glus.gem.local:/glustexp/pri1/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick2: nw2glus.gem.local:/glustexp/pri1/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick3: nw3glus.gem.local:/glustexp/pri2/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick4: nw4glus.gem.local:/glustexp/pri2/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Options Reconfigured:</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">server.allow-insecure: on</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">storage.owner-uid: 107</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">storage.owner-gid: 107</span></p></div><div><br></div><div><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif; font-size: 10pt;">[root@networker ~]# gluster volume status gv_pri detail</span></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Status of volume: gv_pri</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">------------------------------------------------------------------------------</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick : Brick nw1glus.gem.local:/glustexp/pri1/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Port : 50178 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Online : Y </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Pid : 25721 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">File System : xfs </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Device : /dev/mapper/vg_guests-lv_brick1</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Mount Options : rw,noatime </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Size : 512 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Disk Space Free : 168.4GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Total Disk Space : 194.9GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Count : 102236160 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Free Inodes : 102236130 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">------------------------------------------------------------------------------</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick : Brick nw2glus.gem.local:/glustexp/pri1/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Port : 50178 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Online : Y </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Pid : 27832 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">File System : xfs </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Device : /dev/mapper/vg_guests-lv_brick1</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Mount Options : rw,noatime </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Size : 512 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Disk Space Free : 168.4GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Total Disk Space : 194.9GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Count : 102236160 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Free Inodes : 102236130 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">------------------------------------------------------------------------------</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick : Brick nw3glus.gem.local:/glustexp/pri2/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Port : 50182 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Online : Y </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Pid : 14571 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">File System : xfs </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Device : /dev/mapper/vg_guests-lv_brick2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Mount Options : rw,noatime </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Size : 512 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Disk Space Free : 418.3GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Total Disk Space : 433.8GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Count : 227540992 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Free Inodes : 227540973 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">------------------------------------------------------------------------------</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Brick : Brick nw4glus.gem.local:/glustexp/pri2/brick</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Port : 50181 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Online : Y </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Pid : 21942 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">File System : xfs </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Device : /dev/mapper/vg_guests-lv_brick2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Mount Options : rw,noatime </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Size : 512 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Disk Space Free : 418.3GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Total Disk Space : 433.8GB </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Inode Count : 227540992 </span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">Free Inodes : 227540973</span></p></div><div><br></div><div>fuse-mount of the gv_pri volume:</div><div><br></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">[root@networker ~]# ll -h /mnt/gluspri/</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">totale 37G</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 7,7G 24 gen 10:21 alfresco.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 4,2G 24 gen 10:22 check_mk-salmo.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 27M 23 gen 16:42 newnxserver.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 1,1G 23 gen 13:38 newubutest1.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 11G 24 gen 10:17 nxserver.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 8,1G 24 gen 10:17 remlog.qc2</span><br><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;">-rw-------. 1 qemu qemu 5,6G 24 gen 10:19 ubutest1.qc2</span></p></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: arial, helvetica, sans-serif;">Do you think this is the expected behaviour, maybe due to caching? What if the most updated node goes down while the VMs are running?</span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><br></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: 'courier new', courier, monaco, monospace, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: arial, helvetica, sans-serif;">Thanks a lot,</span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-family: arial, helvetica, sans-serif;"><br></span></p><p style="margin: 0px;" data-mce-style="margin: 0px;"><b style="font-family: arial, helvetica, sans-serif; font-size: 10pt;">Fabio Rosati</b></p></div></div></body></html>