<span class="Apple-style-span" style="border-collapse: collapse; "><div>Matthias,</div><div><br></div><div>Just to run a few more tests....</div><div><br></div><div>Centos 5.3 x86_64 on all systems in my test</div><div><br>
</div>I just created 1000 directories with a file in each directory and everything worked fine. All the directories shows up on each node and on the client.<div><br></div><div><br></div><div><div>[root@store01 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'</div>
<div>icount = 37786752</div><div>ifree = 1133409</div><div><div>[root@store02 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'</div><div>icount = 36526592</div><div>ifree = 269</div><div><br></div><div>[root@client testliam]# for i in `seq 1 1000`; do mkdir $i; done</div>
</div></div><div><div>[root@client testliam]# for i in `seq 1 1000`; do echo "woohoo" > $i/$i.myfileyay; done</div><div>[root@client testliam]# ls |wc -l</div><div>1000</div><div><div>[root@client testliam]# ls -R | wc -l</div>
<div>4001</div><div><br></div><div>(after the test)</div><div><br></div><div><div>[root@store01 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'</div><div>icount = 37786752</div><div>ifree = 1133409</div><div>
<div>[root@store02 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'</div><div>icount = 36526592</div><div>ifree = 269</div><div><br></div><div><div>[root@server01 testliam]# ls | wc -l</div><div>1000</div>
<div>[root@server01 testliam]# ls -R | wc -l</div><div>4001</div><div><div>[root@server02 testliam]# ls | wc -l</div><div>1000</div><div>[root@server02 testliam]# ls -R | wc -l</div><div>4001</div><div><br></div></div></div>
</div></div></div><div>The ifree number didnt change after i ran the tests, all the files look to be fine and intact - so as far as i can tell everything is working fine.</div><div><br></div><div>Did your tests show something else?</div>
<div><br></div><font color="#888888"><div>liam</div><div><br></div></font></div></span><br><div class="gmail_quote">On Tue, Jul 28, 2009 at 1:14 PM, Matthias Saou <span dir="ltr"><<a href="mailto:thias@spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net">thias@spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">Liam Slusser <<a href="mailto:lslusser@gmail.com">lslusser@gmail.com</a>> wrote:<br>
<br>
> I worked on this issue this morning and could find nothing that would<br>
> indicate it wouldn't work. I was down to 45 free inodes (says<br>
> xfs_db) so i brought down one of the nodes and applied the inode64<br>
> option to /etc/fstab and remount the partition and restarted<br>
> gluster. Everything appears to be working normally so i applied the<br>
> same option to my other server, and again, everything is working<br>
> normally. I'll let you know after we run with this for a few days but<br>
> so far everything is fine and working normally. I'm on Centos 5.3<br>
> x86_64 btw.<br>
><br>
> An interesting note, after applying the inode64 option the "ifree"<br>
> output after running xfs_db didn't actually change but the filesystem<br>
> is working normally. I found a bunch of posts on the interweb of<br>
> people who had that exact experience.<br>
<br>
</div>It's not mounting with the inode64 option which is an issue, it's once<br>
you get files or directories allocated with inodes > 32bits from our<br>
experience. So what you need to do is test with files or directories<br>
created once the filesystem has been mounted with the inode64 options<br>
and do have their inode beyond the 32bit limit.<br>
<br>
Here's an example of what we have on the server :<br>
[...]<br>
3221490663 drwxrwsr-x 5 flumotion file 46 Mar 11 19:07 cust5<br>
3221491044 drwxrwsr-x 6 flumotion file 56 May 12 12:27 cust1<br>
3221495387 drwxrwsr-x 9 flumotion file 109 Mar 26 2008 cust2<br>
3221500135 drwxrwsr-x 5 flumotion file 46 May 22 11:36 cust8<br>
3221500510 drwxrwsr-x 3 flumotion file 21 Jan 2 2008 cust23<br>
3221500956 drwxrwsr-x 5 flumotion file 46 Jun 25 16:23 cust7<br>
72107183169 drwxrwsr-x 3 flumotion file 26 Jul 10 13:38 cust4<br>
98784439192 drwxrwsr-x 3 flumotion file 29 Jul 2 16:55 cust12<br>
<br>
The last two directories are the ones which aren't accessible from the<br>
glusterfs clients.<br>
<br>
So in your case, if you had only 45 free inodes left, you should be<br>
able to create 46 directories and get the 46th reproduce the problems.<br>
<font color="#888888"><br>
Matthias<br>
</font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br>