<div dir="ltr">Update on this issue:<div><br></div><div>After I deleted the zero-length files which were located on the wrong bricks, the issue of files losing permissions is mostly resolved. I left my find running every 10 minutes for the last five days, though, and the problem continues to recur with a few hundred files every day or two. This leads me to believe there is some bug in GlusterFS which causes this to happen.</div>
<div><br></div><div>My script recorded 1232 files which lost their permissions on August 2 and 870 files on August 6. As I noted earlier, these files were created years ago. One notable fact is that the mtime as reported by GlusterFS is July 28th, </div>
<div><br></div><div>The rebalance log sheds a bit of light on this, but I'm not sure what to conclude. Here is the log for one of the affected files:</div><div><br></div><div><div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.010359] I [dht-rebalance.c:1063:gf_defrag_migrate_data] 0-UDS8-dht: migrate data called on /6f/83/ca/rrrivera25/media</div>
<div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.068909] I [dht-rebalance.c:647:dht_migrate_file] 0-UDS8-dht: /6f/83/ca/rrrivera25/media/3301856.jpg: attempting to move from UDS8-replicate-0 to UDS8-replicate-2</div><div>
UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.068949] I [dht-rebalance.c:647:dht_migrate_file] 0-UDS8-dht: /6f/83/ca/rrrivera25/media/3301856.jpg: attempting to move from UDS8-replicate-0 to UDS8-replicate-2</div><div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.122885] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-UDS8-client-0: remote operation failed: No such file or directory. Path: /6f/83/ca/rrrivera25/media/3301856.jpg (00000000-0000-0000-0000-000000000000). Key: (null)</div>
<div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.123274] W [client3_1-fops.c:1114:client3_1_getxattr_cbk] 0-UDS8-client-1: remote operation failed: No such file or directory. Path: /6f/83/ca/rrrivera25/media/3301856.jpg (00000000-0000-0000-0000-000000000000). Key: (null)</div>
<div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.123330] W [dht-rebalance.c:739:dht_migrate_file] 0-UDS8-dht: /6f/83/ca/rrrivera25/media/3301856.jpg: failed to get xattr from UDS8-replicate-0 (No such file or directory)</div>
<div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.123380] W [dht-rebalance.c:745:dht_migrate_file] 0-UDS8-dht: /6f/83/ca/rrrivera25/media/3301856.jpg: failed to set xattr on UDS8-replicate-2 (Invalid argument)</div><div>UDS8-rebalance.log.3.gz:[2013-07-28 13:52:08.134502] I [dht-rebalance.c:856:dht_migrate_file] 0-UDS8-dht: completed migration of /6f/83/ca/rrrivera25/media/3301856.jpg from subvolume UDS8-replicate-0 to UDS8-replicate-2</div>
</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 2, 2013 at 11:10 AM, Justin Dossey <span dir="ltr"><<a href="mailto:jbd@podomatic.com" target="_blank">jbd@podomatic.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It sounds like this is related to NFS.<div><br></div><div>Anand, thank you for the response. I was under the impression that DHT linkfiles are only in the .glusterfs subdirectory on the brick; in my case, these files are outside that directory. Furthermore, they aren't named like DHT linkfiles (using that hash key)-- they are named the same as the actual files. Finally, once I removed the bad files and their DHT linkfiles, the issue went away and the files remained accessible. I had to remove around 100,000 of these bad 000/1000 zero-length files (and their DHT linkfiles) last night; only 324 additional files were detected.</div>
<div><br></div><div>On my volumes, I use a similar hashing scheme to the DHT one for regular files-- the top level at the volume only has directories 00 through ff, etc, etc. Perhaps this caused some confusion for you?</div>
<div><br></div><div>For transparency, here is the command I run from the client to detect and correct bad file permissions:</div><div><br></div><div>find ./?? -type f -perm 000 -ls -exec chmod -v 644 {} \; -o -type f -perm 1000 -ls -exec chmod -v 644 {} \;<br>
</div><div><br></div><div>If this number does not grow, I will conclude that I just missed these 324 files. If the number gets larger, I can only conclude that GlusterFS is somehow introducing this corruption. If that is the case, I'll dig some more.</div>
<div><br></div><div>Maik, I may have experienced the same thing. I used rsync over NFS without --inplace to load my data into the GlusterFS volume, and I wound up with all those bad files on the wrong bricks (i.e. a file should be only on server1-brick1 and server2-brick1, but "bad" versions (1000, zero-length) were also on server3-brick1 and server4-brick1, leading to confusing results on the clients). Since then, I've switched to using the native client for data loads and also the --inplace flag to rsync. </div>
<div><br></div><div>Other factors which may have caused the issue I had:</div><div><br></div><div>1. During a large rebalance, one GlusterFS node exceeded its system max open files limit, and was rebooted. The rebalance did not stop while this took place.</div>
<div>2. Three times during the same rebalance, the Gluster NFS daemon used an excessive amount of memory and was killed by the kernel oom-killer. The system in question has 8 GB of memory, was the rebalance master, and is not running any significant software besides GlusterFS. Each time, I restarted glusterfs and the NFS server daemon started serving files again. The rebalance was not interrupted.</div>
<div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 2, 2013 at 4:54 AM, Maik Kulbe <span dir="ltr"><<a href="mailto:info@linux-web-development.de" target="_blank">info@linux-web-development.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I've just had a problem removing a directory with test files. I had an inaccessible folder which I could neither delete nor read on the client(both NFS and FUSE client). On the backend, the folder had completely 0'd permissions and the files showed the 0'd permissions with the sticky bit. I can't remove the folder on the client(it fails with 'directory not empty') but if I delete the empty files on the backend, it's gone. Is there any explanation for this?<br>
<br>
I also found that this only happens, if I remove the folder recursivly over NFS. When I remove the files in the folder first there are no 0-size files on the backend and I can delete the directory with rmdir without any problem.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
Justin,<br>
What you are seeing are internal DHT linkfiles. They are zero byte files<br>
with mode 01000. Changing their mode forcefully in the backend to<br>
something else WILL render your files inaccessible from the mount point. I<br>
am assuming that you have seen these files only in the backend and not<br></div>
from the mount point And accessing/modifying files like this directly<div><div><br>
from the backend is very dangerous for your data, as explained in this<br>
very example.<br>
Avati<br>
<br>
On Thu, Aug 1, 2013 at 2:25 PM, Justin Dossey <<a href="mailto:jbd@podomatic.com" target="_blank">jbd@podomatic.com</a>> wrote:<br>
<br>
One thing I do see with the issue we're having is that the files which<br>
have lost their permissions have "bad" versions on multiple bricks.<br>
Since the replica count is 2 for any given file, there should be only<br>
two copies of each, no? <br>
For example, the file below has zero-length, zero-permission versions on<br>
uds06/brick2 and uds-07/brick2, but good versions on uds-05/brick1 and<br>
uds-06/brick1.<br>
FILE is<br>
/09/38/1f/eastar/mail/entries/<u></u>trash/2008-07-06T13_41_56-07_<u></u>00.dump<br>
uds-05 -rw-r--r-- 2 apache apache 2233 Jul 6 2008<br>
/export/brick1/vol1/09/38/1f/<u></u>eastar/mail/entries/trash/<u></u>2008-07-06T13_41_56-07_00.dump<br>
uds-06 -rw-r--r-- 2 apache apache 2233 Jul 6 2008<br>
/export/brick1/vol1/09/38/1f/<u></u>eastar/mail/entries/trash/<u></u>2008-07-06T13_41_56-07_00.dump<br>
uds-06 ---------T 2 apache apache 0 Jul 23 03:11<br>
/export/brick2/vol1/09/38/1f/<u></u>eastar/mail/entries/trash/<u></u>2008-07-06T13_41_56-07_00.dump<br>
uds-07 ---------T 2 apache apache 0 Jul 23 03:11<br>
/export/brick2/vol1/09/38/1f/<u></u>eastar/mail/entries/trash/<u></u>2008-07-06T13_41_56-07_00.dump<br>
Is it acceptable for me to just delete the zero-length copies?<br>
<br>
On Thu, Aug 1, 2013 at 12:57 PM, Justin Dossey <<a href="mailto:jbd@podomatic.com" target="_blank">jbd@podomatic.com</a>><br>
wrote:<br>
<br>
Do you know whether it's acceptable to modify permissions on the brick<br>
itself (as opposed to over NFS or via the fuse client)? It seems that<br>
as long as I don't modify the xattrs, the permissions I set on files<br>
on the bricks are passed through.<br>
<br>
On Thu, Aug 1, 2013 at 12:32 PM, Joel Young <<a href="mailto:jdy@cryregarder.com" target="_blank">jdy@cryregarder.com</a>><br>
wrote:<br>
<br>
I am not seeing exactly that, but I am experiencing the permission<br>
for<br>
the root directory of a gluster volume reverting from a particular<br>
user.user to root.root ownership. I have to periodically do a "cd<br>
/share; chown user.user . "<br>
On Thu, Aug 1, 2013 at 12:25 PM, Justin Dossey <<a href="mailto:jbd@podomatic.com" target="_blank">jbd@podomatic.com</a>><br>
wrote:<br>
> Hi all,<br>
><br>
> I have a relatively-new GlusterFS 3.3.2 4-node cluster in<br>
> distributed-replicated mode running in a production environment.<br>
><br>
> After adding bricks from nodes 3 and 4 (which changed the cluster<br>
type from<br>
> simple replicated-2 to distributed-replicated-2), I've discovered<br>
that files<br>
> are randomly losing their permissions. These are files that<br>
aren't being<br>
> accessed by our clients-- some of them haven't been touched for<br>
years.<br>
><br>
> When I say "losing their permissions", I mean that regular files<br>
are going<br>
> from 0644 to 0000 or 1000.<br>
><br>
> Since this is a real production issue, I run a parallel find<br>
process to<br>
> correct them every ten minutes. It has corrected approximately<br>
40,000 files<br>
> in the past 18 hours.<br>
><br>
> Is anyone else seeing this kind of issue? My searches have turned<br>
up<br>
> nothing so far.<br>
><br>
> --<br>
> Justin Dossey<br>
> CTO, PodOmatic<br>
><br>
><br>
> ______________________________<u></u>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
<br>
--<br>
Justin Dossey<br>
CTO, PodOmatic<br>
<br>
--<br>
Justin Dossey<br>
CTO, PodOmatic<br>
______________________________<u></u>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.<u></u>org/mailman/listinfo/gluster-<u></u>users</a><br>
</div></div></blockquote>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Justin Dossey<br>CTO, PodOmatic<div><br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Justin Dossey<br>CTO, PodOmatic<div><br></div>
</div>