<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi Everybody,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We have recently setup a 2-node gluster, and are having some problems deleting some files!<o:p></o:p></p><p class=MsoNormal>We haven’t accessed the gluster bricks directly, all access has been through the glusterfs mount <o:p></o:p></p><p class=MsoNormal>The 1<sup>st</sup> node in our cluster also handles sharing the glusterfs volume via samba, to windows clients.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Environment is:<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>CentOS release 6.4 (Final)<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>glusterfs-libs-3.4.1-3.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>glusterfs-cli-3.4.1-3.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>glusterfs-fuse-3.4.1-3.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>glusterfs-server-3.4.1-3.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>glusterfs-3.4.1-3.el6.x86_64<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> 4 bricks on each node<o:p></o:p></p><p class=MsoNormal> Gluster is setup without replication or striping<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> Bricks on one of the two nodes (the other node is setup the same):<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'><span style='font-family:"Courier New"'>/dev/sda1 9.1T 4.7T 4.5T 51% /mnt/glusterfs/bricks/1<o:p></o:p></span></p><p class=MsoNormal style='margin-left:72.0pt'><span style='font-family:"Courier New"'>/dev/sda2 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/2<o:p></o:p></span></p><p class=MsoNormal style='margin-left:72.0pt'><span style='font-family:"Courier New"'>/dev/sda3 9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/3<o:p></o:p></span></p><p class=MsoNormal style='margin-left:72.0pt'><span style='font-family:"Courier New"'>/dev/sda4 9.1T 4.1T 5.0T 46% /mnt/glusterfs/bricks/4<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-left:36.0pt'>Gluster volume mounted on 1st node:<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'><span style='font-family:"Courier New"'>g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00<o:p></o:p></span></p><p class=MsoNormal> Mount options:<o:p></o:p></p><p class=MsoNormal style='margin-left:72.0pt'><span style='font-family:"Courier New"'>g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072)<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b><span style='color:red'>So the problem is this:<o:p></o:p></span></b></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>[root@g-unit-1 ~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>rm: cannot remove `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not empty<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>rm: cannot remove `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not empty<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>[root@g-unit-1 ~]#<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>And the delete leaves behind 2000 or so files:<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>[root@g-unit-1 ~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type f | wc -l<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>2185<o:p></o:p></span></p><p class=MsoNormal>Almost all the left behind files are on the 2<sup>nd</sup> node, g-unit-2, but there are a few on g-unit-1<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>From the 2<sup>nd</sup> node, g-unit-2:<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>[root@g-unit-2 ~]# getfattr -m . -d -e hex /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>getfattr: Removing leading '/' from absolute path names<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'># file: mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f<o:p></o:p></span></p><p class=MsoNormal>So the file is there, is tagged with some gluster info, and has definitely been copied onto the volume properly, via the glusterfs mount. It hasn’t been copied directly onto the brick.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>But from g-unit-1, accessing via the glusterfs mount, and trying to delete this file:<o:p></o:p></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>[root@g-unit-1 ~]# rm /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>rm: remove regular file `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y<o:p></o:p></span></p><p class=MsoNormal style='margin-left:36.0pt'><span style='font-family:"Courier New"'>rm: cannot remove `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': No such file or directory<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>What the hell!?!!!!!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Can someone please suggest the best course of action to resolve this and prevent it happening in the future?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks for any assistance.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>John<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>