<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=windows-1257"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hello, <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have faced strange problem on one of the Gluster client/server nodes. File exist but cannot be deleted from the fuse mount point on one client node, but can be deleted from another <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Attemt to delete:<o:p></o:p></p><p class=MsoNormal><i>root@SERVER:~# rm /var/lib/nova/instances/instance-0098a323/disk<o:p></o:p></i></p><p class=MsoNormal><i>rm: cannot remove `/var/lib/nova/instances/instance-0098a323/disk': No such file or directory<o:p></o:p></i></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Although files listed as existing in the volume:<o:p></o:p></p><p class=MsoNormal><i>root@ SERVER:~# ls -al /var/lib/nova/instances/instance-0098a323<o:p></o:p></i></p><p class=MsoNormal><i>total 6143442<o:p></o:p></i></p><p class=MsoNormal><i>drwxrwxr-x 2 nova nova 143 Jun 11 11:54 .<o:p></o:p></i></p><p class=MsoNormal><i>drwxr-xr-x 107 nova nova 409600 Jun 11 11:54 ..<o:p></o:p></i></p><p class=MsoNormal><i>-rw-r--r-- 1 root root 6290472960 Jun 11 11:54 disk<o:p></o:p></i></p><p class=MsoNormal><i>root@ SERVER:~# stat /var/lib/nova/instances/instance-0098a323/disk<o:p></o:p></i></p><p class=MsoNormal><i> File: `/var/lib/nova/instances/instance-0098a323/disk'<o:p></o:p></i></p><p class=MsoNormal><i> Size: 6290472960 Blocks: 12286080 IO Block: 131072 regular file<o:p></o:p></i></p><p class=MsoNormal><i>Device: 1eh/30d Inode: 11831107392822693115 Links: 1<o:p></o:p></i></p><p class=MsoNormal><i>Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)<o:p></o:p></i></p><p class=MsoNormal><i>Access: 2013-06-03 14:12:03.803403051 +0100<o:p></o:p></i></p><p class=MsoNormal><i>Modify: 2013-06-11 11:54:09.623013613 +0100<o:p></o:p></i></p><p class=MsoNormal><i>Change: 2013-06-11 11:54:12.355063759 +0100<o:p></o:p></i></p><p class=MsoNormal><i> Birth: -<o:p></o:p></i></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>File locartion information:<o:p></o:p></p><p class=MsoNormal><i>root@SERVER:~/openstack# getfattr -n trusted.glusterfs.pathinfo /var/lib/nova/instances/instance-0098a323/disk<o:p></o:p></i></p><p class=MsoNormal><i>getfattr: Removing leading '/' from absolute path names<o:p></o:p></i></p><p class=MsoNormal><i># file: var/lib/nova/instances/instance-0098a323/disk<o:p></o:p></i></p><p class=MsoNormal><i>trusted.glusterfs.pathinfo="(<DISTRIBUTE:glustervmstore-dht> (<REPLICATE:glustervmstore-replicate-21> <POSIX(/mnt/vmstore/brick):BRICK_SERV1:/mnt/vmstore/brick/instance-0098a323/disk> <POSIX(/mnt/vmstore/brick): BRICK_SERV2:/mnt/vmstore/brick/instance-0098a323/disk>))"<o:p></o:p></i></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Subvolume information:<o:p></o:p></p><p class=MsoNormal><i>volume glustervmstore-replicate-3<o:p></o:p></i></p><p class=MsoNormal><i> type cluster/replicate<o:p></o:p></i></p><p class=MsoNormal><i> subvolumes glustervmstore-client-6 glustervmstore-client-7<o:p></o:p></i></p><p class=MsoNormal><i>end-volume<o:p></o:p></i></p><p class=MsoNormal><i>volume glustervmstore-replicate-21<o:p></o:p></i></p><p class=MsoNormal><i> type cluster/replicate<o:p></o:p></i></p><p class=MsoNormal><i> subvolumes glustervmstore-client-42 glustervmstore-client-43<o:p></o:p></i></p><p class=MsoNormal><i>end-volume<o:p></o:p></i></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>It is also available on the replica bricks. However I WAS ABLE to delete the file from another node, which also mounts the same volume under: <i>/var/lib/nova/instances</i><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Log file excerpt (I presume that’s /var/log/gluster/var-lib-nova-instances.log). What is strange that <i>glustervmstore-client-6 </i>and<i> glustervmstore-client-6 </i>does not represent actual brick servers where file is stored: <o:p></o:p></p><p class=MsoNormal><i>[2013-06-11 14:10:02.640244] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin<o:p></o:p></i></p><p class=MsoNormal><i>g conservative merge<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:10:51.769641] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:10:51.769737] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:10:51.769769] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204475960: UNLINK() /instance-0098a323/disk => -1<o:p></o:p></i></p><p class=MsoNormal><i>(No such file or directory)<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:18:15.350431] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:18:15.350513] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:18:15.350547] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204480095: UNLINK() /instance-0098a323/disk => -1<o:p></o:p></i></p><p class=MsoNormal><i>(No such file or directory)<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:26:07.418892] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:26:07.418980] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:26:07.419011] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204484135: UNLINK() /instance-0098a323/disk => -1<o:p></o:p></i></p><p class=MsoNormal><i>(No such file or directory)<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:27:12.510324] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin<o:p></o:p></i></p><p class=MsoNormal><i>g conservative merge<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:27:44.518849] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:27:44.518937] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f<o:p></o:p></i></p><p class=MsoNormal><i>ile or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:27:44.518967] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204485011: UNLINK() /instance-0098a323/disk => -1<o:p></o:p></i></p><p class=MsoNormal><i>(No such file or directory)<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:29:33.289316] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin<o:p></o:p></i></p><p class=MsoNormal><i>g conservative merge<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:38:13.487311] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-41: remote operation failed: No such<o:p></o:p></i></p><p class=MsoNormal><i>file or directory<o:p></o:p></i></p><p class=MsoNormal><i>[2013-06-11 14:38:13.487563] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-40: remote operation failed: No such file or directory<o:p></o:p></i></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>The same happened for <i>/instance-0098a327/disk </i>file although there are no similar line <o:p></o:p></p><p class=MsoNormal><i>[2013-06-11 14:27:12.510324] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin<o:p></o:p></i></p><p class=MsoNormal><i>g conservative merge<o:p></o:p></i></p><p class=MsoNormal>I was able to delete the file from another node. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There are no mention of the folder in the GlusterFS logs on the actual brick servers.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Thank you<o:p></o:p></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>______________________________________________________________________________<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:6.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Elvinas Piliponis I </span><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>UNIX Engineer</span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'> I GTIS UNIX Engineering </span><span lang=EN-US style='color:#1F497D;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Tel +370 5 251 1218, 7 2249 1218 I Mobile +370 6</span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>56 69249 I Email </span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'> </span><span style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'><a href="mailto:elvinas.piliponis@barclays.com"><span style='color:blue'>elvinas.piliponis@barclays.com</span></a></span><span lang=EN-US style='font-size:8.0pt;color:gray;mso-fareast-language:EN-GB'><br></span><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>Barclays, GreenHall 9th floor 09.E4.1, Upës g. 21, Vilnius, Lithuania LT-081218 </span><span lang=EN-US style='color:#1F497D;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:8.0pt;font-family:"Arial","sans-serif";color:gray;mso-fareast-language:EN-GB'>barclays.com</span><span lang=EN-US style='color:#1F497D;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>
<html>
<head>
<title></title>
</head>
<body>
<p><br>This e-mail and any attachments are confidential and intended solely for
the addressee and may also be privileged or exempt from disclosure under
applicable law. If you are not the addressee, or have received this e-mail in
error, please notify the sender immediately, delete it from your system and do
not copy, disclose or otherwise act upon any part of this e-mail or its
attachments.<br><br>Internet communications are not guaranteed to be secure or
virus-free.<br>The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any Internet
communications by any third party, or from the transmission of any viruses.
Replies to this e-mail may be monitored by the Barclays Group for operational or
business reasons.<br><br>Any opinion or other information in this e-mail or its
attachments that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.<br><br>Barclays Bank PLC.Registered in England and Wales (registered no.
1026167).<br>Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.<br><br>Barclays Bank PLC is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority (Financial Services Register No. 122702).<br></p>
</body>
</html>