<div dir="ltr"><div>Maybe I was not very clear in the previous email: even the forced rebalance got stuck in the same folder (not many details in this case from the log), counting million of files that were not existant, so I stopped that.</div>
<div> </div><div>Then, before the following paragraph in which I described the fact that I removed many files not on purpose, because I was stuck, the time suddenly is skipped to "this past night" (I'm in japan)</div>
<div> </div><div> Stefano</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sun, Jun 2, 2013 at 3:05 PM, Stefano Sinigardi <span dir="ltr"><<a href="mailto:stefano.sinigardi@gmail.com" target="_blank">stefano.sinigardi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Dear Vijay,</div><div>the filesystem is ext4, on a GPT structured disk, formatted by Ubuntu 12.10.</div>
<div>The rebalance I did was with the command</div><div> </div><div>gluster volume rebalance data start</div>
<div> </div><div>but in the log it got stuck on a file that I cannot remember (was a small working .cpp file, saying that it was going to be moved to an much more occupied replica, and it repeated this message until writing a log that was a few GB).</div>
<div>Then I stopped it and restarted with</div><div> </div><div>gluster volume rebalance data start force</div><div> </div><div>in order to get rid of this problems about files going to bricks already highly occupied.</div>
<div>Because I was almost stuck, remembering that a rebalance solved another problem I had as a miracle, I retried it, but got stuck in a .dropbox-cache folder. That is not a very important folder, so I thought I could remove it. I launched a script to find all the files looking at all the bricks but removing them from the fuse mountpoint. I don't know what went wrong (the script is very simple, the problem maybe was that it was 4 am in the night) but the fact is that files got removed calling rm at the bricks mountpoints, not the fuse one. So I think that now I'm in a even worse situation that before. I just stopped working on it, asking for some time from my colleagues (at least data is still there, on the bricks, just sparse on all of them) in order to think well about how to proceed (maybe destroying it and rebuilding it, but it will be very time consuming as I don't have so much free space elsewere to save everything, also it's very difficult to save from the fuse mountpoint as it's not listing all the files)</div>
<div class="gmail_extra"> </div><div class="gmail_extra">Thanks a lot for your support.</div><div class="gmail_extra">In any case, I'm learning really a lot.</div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra">
</div><div class="gmail_extra">
Stefano<br></div></font></span><div><div class="h5"><div class="gmail_extra"> </div><div class="gmail_extra"> </div><div class="gmail_extra"><br> </div><div class="gmail_quote">On Sun, Jun 2, 2013 at 2:52 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div>On 05/31/2013 03:18 PM, Stefano Sinigardi wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
Dear Xavier,<br>
I realized that the volume was not build properly when doing the first<br>
analyses suggested by Davide, but I'm sure that this is not the problem<br>
and so I quickly dismissed it. Also, we need a replica but not so<br>
strictly, maybe in the future with the next volume I'll build it<br>
properly. Anyway, yes, the volume got birth on "pedrillo" with a<br>
replica-2 and the next day was expanded on "osmino", again with<br>
replica-2, just by adding bricks and doing a rebalance, that was just<br>
tried. I'm saying "tried" because it got "stuck", consuming a lot of RAM<br>
(almost all, 16 GB), and it was counting million of files that I think<br>
don't even exist on the volume, so I stopped it. Do you think that it<br>
might be worth restarting?<br>
</blockquote>
<br></div>
I might have missed this detail in the thread. What is the disk filesystem on the bricks?<br>
<br>
Can you list the exact rebalance command that was triggered?<br>
<br>
<br>
Thanks,<br>
Vijay<br>
</blockquote></div><div class="gmail_extra"><br></div></div></div></div>
</blockquote></div><br></div>