Jordi,<br> Do you have any firewall running on machines?<br><br>--<br>gowda<br><br><div class="gmail_quote">On Thu, Dec 18, 2008 at 1:47 PM, Jordi Moles Blanco <span dir="ltr"><<a href="mailto:jordi@cdmon.com">jordi@cdmon.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">En/na Raghavendra G ha escrit:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="Ih2E3d">
Hi Jordi,<br>
<br>
Have you started glusterfsd on each of the newly added nodes? If not, please start them.<br>
<br>
some comments have been inlined.<br>
<br></div><div><div></div><div class="Wj3C7c">
On Wed, Dec 17, 2008 at 3:28 PM, Jordi Moles Blanco <<a href="mailto:jordi@cdmon.com" target="_blank">jordi@cdmon.com</a> <mailto:<a href="mailto:jordi@cdmon.com" target="_blank">jordi@cdmon.com</a>>> wrote:<br>
<br>
Hi,<br>
<br>
i've got 6 nodes providing a storage unit with gluster 2.5 patch<br>
800. They are set in 2 groups of 3 nodes each.<br>
<br>
On top of that, i've got a Xen 3.2 machine storing its virtual<br>
machines in gluster mount point.<br>
<br>
The thing is that i used to have only 2 nodes for group, that's 4<br>
nodes in total, and today I'm trying to add 1 extra node for each<br>
group.<br>
<br>
This is the final setting on Xen's Side:<br>
<br>
<br>
**************<br>
<br>
volume espai1<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.3<br>
option remote-subvolume espai<br>
end-volume<br>
<br>
volume espai2<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.4<br>
option remote-subvolume espai<br>
end-volume<br>
<br>
volume espai3<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.5<br>
option remote-subvolume espai<br>
end-volume<br>
<br>
volume espai4<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.6<br>
option remote-subvolume espai<br>
end-volume<br>
<br>
volume espai5<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.7<br>
option remote-subvolume espai<br>
end-volume<br>
<br>
volume espai6<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.8<br>
option remote-subvolume espai<br>
end-volume<br>
<br>
volume namespace1<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.4<br>
option remote-subvolume nm<br>
end-volume<br>
<br>
volume namespace2<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host 10.0.0.5<br>
option remote-subvolume nm<br>
end-volume<br>
<br>
volume grup1<br>
type cluster/afr<br>
subvolumes espai1 espai3 espai5<br>
end-volume<br>
<br>
volume grup2<br>
type cluster/afr<br>
subvolumes espai2 espai4 espai6<br>
end-volume<br>
<br>
volume nm<br>
type cluster/afr<br>
subvolumes namespace1 namespace2<br>
end-volume<br>
<br>
volume g01<br>
type cluster/unify<br>
subvolumes grup1 grup2<br>
option scheduler rr<br>
option namespace nm<br>
end-volume<br>
<br>
volume io-cache<br>
type performance/io-cache<br>
option cache-size 512MB<br>
option page-size 1MB<br>
option force-revalidate-timeout 2<br>
subvolumes g01<br>
end-volume <br>
**************<br>
<br>
so... i stopped all virtual machines, unmounted gluster on Xen,<br>
updated the spec file (the one above) and ran gluster again in Xen.<br>
<br>
I've set different gluster environments but i had never tried<br>
this, and now i'm facing some problems.<br>
<br>
For what i had read before this... i used to think that when<br>
adding and extra node to a group and "remounting" on client's<br>
side, the Healing feature would copy all the content of the other<br>
nodes already present in the group to the "new one". That hasn't<br>
happened, even when I've tried to force the file system, by<br>
listing the files or doing what you suggest in you documentation:<br>
<br>
**********<br>
<br>
find /mnt/glusterfs -type f -print0 | xargs -0 head -c1 >/dev/null<br>
<br>
**********<br>
<br>
so... my first question would be... does "self-healing" work this<br>
way? If it doesn't.... which is the best way to add a node to a<br>
group? Do i have to run a "copy" command manually to get the new<br>
node ready?<br>
I've also noticed that i have necessarily to umount gluster from<br>
Xen. Is there a way to avoid stopping all the virtual machines,<br>
umounting and mounting again? Is there a feature like "refresh<br>
config file"?<br>
<br>
<br>
Hot add ("refresh config file") is in the roadmap.<br>
<br>
<br>
<br>
And finally... i looked into the logs to see why self-healing<br>
wasn't working, and i found this on Xen's Side:<br>
<br>
**********<br>
2008-12-17 12:08:30 E [tcp-client.c:190:tcp_connect] espai6:<br>
non-blocking connect() returned: 111 (Connection refused)<br>
**********<br>
<br>
and it keeps saying this when i want to access files which were<br>
created in the "old nodes".<br>
<br>
is this a bug? how can i work around this?<br>
<br>
If i create new stuff, though, it replicates to the 3 nodes, no<br>
problem with that.... the only problem is with the old files that<br>
were already present before i added the new node.<br>
<br>
Thanks for your help in advance, and let me know if you need any<br>
further information.<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br></div></div>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a> <mailto:<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a>><div class="Ih2E3d"><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br>
<br>
<br>
<br>
-- <br>
Raghavendra G<br>
<br>
</div></blockquote>
<br>
hi, yes.<br>
<br>
when gluster behaves like this, all nodes are running. As i said, when you create new data, it replicates to all the nodes of each group, so it's working fine.<br>
However, it keeps logging "connection refused", which i though was reported only when a node wasn't available, but they are all available and replicating data fine.<br>
<br>
The thing, though, is that old data is not beeing replicated into the new nodes?<br>
<br>
Is there any way to "force" replication to the new nodes? Could i be getting somehow the "connection refused" because new nodes won't accept previous data?<br>
<br>
Thanks for your help.<div><div></div><div class="Wj3C7c"><br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>hard work often pays off after time, but laziness always pays off now<br>