<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I think Susant knows about this issue<br>
<br>
CCed him<br>
<br>
Pranith<br>
<div class="moz-cite-prefix">On 06/15/2014 07:16 PM, SINCOCK John
wrote:<br>
</div>
<blockquote
cite="mid:CCB8D141C7F4E14D82DB362CAFF983AA019E688D@ADLEX01.lads.local"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hi Everybody,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We have recently setup a 2-node gluster,
and are having some problems deleting some files!<o:p></o:p></p>
<p class="MsoNormal">We haven’t accessed the gluster bricks
directly, all access has been through the glusterfs mount <o:p></o:p></p>
<p class="MsoNormal">The 1<sup>st</sup> node in our cluster also
handles sharing the glusterfs volume via samba, to windows
clients.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Environment is:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">Kernel:
2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">CentOS release
6.4 (Final)<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">glusterfs-libs-3.4.1-3.el6.x86_64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">glusterfs-cli-3.4.1-3.el6.x86_64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">glusterfs-fuse-3.4.1-3.el6.x86_64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">glusterfs-server-3.4.1-3.el6.x86_64<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">glusterfs-3.4.1-3.el6.x86_64<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> 4 bricks on each node<o:p></o:p></p>
<p class="MsoNormal"> Gluster is setup without
replication or striping<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Bricks on one of the two
nodes (the other node is setup the same):<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span
style="font-family:"Courier New"">/dev/sda1
9.1T 4.7T 4.5T 51% /mnt/glusterfs/bricks/1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span
style="font-family:"Courier New"">/dev/sda2
9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span
style="font-family:"Courier New"">/dev/sda3
9.1T 3.7T 5.5T 40% /mnt/glusterfs/bricks/3<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span
style="font-family:"Courier New"">/dev/sda4
9.1T 4.1T 5.0T 46% /mnt/glusterfs/bricks/4<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">Gluster volume
mounted on 1st node:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span
style="font-family:"Courier New"">g-unit-1:/vol00
73T 31T 43T 42% /gluster/vol00<o:p></o:p></span></p>
<p class="MsoNormal"> Mount options:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:72.0pt"><span
style="font-family:"Courier New"">g-unit-1:/vol00
on /gluster/vol00 type fuse.glusterfs
(rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="color:red">So the problem
is this:<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">[root@g-unit-1
~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">rm: cannot
remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages':
Directory not empty<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">rm: cannot
remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT':
Directory not empty<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">[root@g-unit-1
~]#<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And the delete leaves behind 2000 or so
files:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">[root@g-unit-1
~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type
f | wc -l<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">2185<o:p></o:p></span></p>
<p class="MsoNormal">Almost all the left behind files are on the
2<sup>nd</sup> node, g-unit-2, but there are a few on g-unit-1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">From the 2<sup>nd</sup> node, g-unit-2:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">[root@g-unit-2
~]# getfattr -m . -d -e hex
/mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">getfattr:
Removing leading '/' from absolute path names<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New""># file:
mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f<o:p></o:p></span></p>
<p class="MsoNormal">So the file is there, is tagged with some
gluster info, and has definitely been copied onto the volume
properly, via the glusterfs mount. It hasn’t been copied
directly onto the brick.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But from g-unit-1, accessing via the
glusterfs mount, and trying to delete this file:<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">[root@g-unit-1
~]# rm
/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">rm: remove
regular file
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'?
y<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span
style="font-family:"Courier New"">rm: cannot
remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi':
No such file or directory<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What the hell!?!!!!!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can someone please suggest the best course
of action to resolve this and prevent it happening in the
future?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for any assistance.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">John<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>