<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:arial,helvetica,sans-serif;font-size:10pt"><div>Hi Amar,<br>
Thanks for the quick reply, but that doesn't work either. I just get more holes and overlaps:<br>
<br>
2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found anomalies in /. holes=3 overlaps=9<br>
2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing assignment on /<br>
2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the directory is not a virgin<br>
2009-02-06 15:04:04 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Structure needs cleaning)<br>
2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found anomalies in /. holes=3 overlaps=9<br>
2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing assignment on /<br>
2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the directory is not a virgin<br>
2009-02-06 15:04:04 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 2: LOOKUP() / => -1 (Structure needs cleaning)<br>
2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found anomalies in /. holes=3 overlaps=9<br>
2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing assignment on /<br>
2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the directory is not a virgin<br>
2009-02-06 15:04:04 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Structure needs cleaning)<br>
2009-02-06 15:04:04 E [dht-layout.c:460:dht_layout_normalize] dist1: found anomalies in /. holes=3 overlaps=9<br>
2009-02-06 15:04:04 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing assignment on /<br>
2009-02-06 15:04:04 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the directory is not a virgin<br>
2009-02-06 15:04:04 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 3: LOOKUP() / => -1 (Structure needs cleaning)</div><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt;"><br><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;"><font face="Tahoma" size="2"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Amar Tumballi (bulde) <amar@gluster.com><br><b><span style="font-weight: bold;">To:</span></b> Craig Flockhart <craigflockhart@yahoo.com><br><b><span style="font-weight: bold;">Cc:</span></b> gluster-users@gluster.org<br><b><span style="font-weight: bold;">Sent:</span></b> Friday, February 6, 2009 2:37:08 PM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [Gluster-users] starting 4th node in 4 node dht cluster fails<br></font><br>
Hi Craig,<br> As you are using 'distribute' (client side) over 'distribute' (server side), this will not be working right now. To get it working right now, you can have 4 export volumes from each server exported, and in client have 4x4 client protocol volumes, which you can aggregate with a single 'cluster/distribute' (which will have 16 subvolumes).<br>
<br>To get the below mentioned configuration working as is, you need to wait for a week more IMO.<br><br>Regards,<br>Amar<br><br><div class="gmail_quote">2009/2/6 Craig Flockhart <span dir="ltr"><<a rel="nofollow" ymailto="mailto:craigflockhart@yahoo.com" target="_blank" href="mailto:craigflockhart@yahoo.com">craigflockhart@yahoo.com</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
<div>Using dht translator to cluster together 4 nodes each with 4 disks.<br>Starting glusterfs on the 4th causes "Structure needs cleaning" when ls-ing the mount point on any of them. It's fine with 3 nodes started.<br>
Using fuse-2.7.4<br>GlusterFS 2.0.0rc1<br>Linux 2.6.18-53.el5 kernel<br><br>Errors from the log:<br><br><br>2009-02-06 15:23:51 E [dht-layout.c:460:dht_layout_normalize] dist1: found anomalies in /. holes=1 overlaps=3<br>
2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing assignment on /<br>2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the directory is not a virgin<br>2009-02-06 15:23:51 W [fuse-bridge.c:297:need_fresh_lookup] fuse-bridge: revalidate of / failed (Structure needs cleaning)<br>
2009-02-06 15:23:51 E
[dht-layout.c:460:dht_layout_normalize] dist1: found anomalies in /. holes=1 overlaps=3<br>2009-02-06 15:23:51 W [dht-common.c:137:dht_lookup_dir_cbk] dist1: fixing assignment on /<br>2009-02-06 15:23:51 E [dht-selfheal.c:422:dht_selfheal_directory] dist1: the directory is not a virgin<br>
2009-02-06 15:23:51 E [fuse-bridge.c:404:fuse_entry_cbk] glusterfs-fuse: 2: LOOKUP() / => -1 (Structure needs cleaning)<br><br>Config for one of the machines:<br><br>volume posix-d1<br> type storage/posix<br> option directory /mnt/chard1/export<br>
end-volume<br><br>volume locks1<br> type features/locks<br> subvolumes posix-d1 <br>end-volume<br><br><br>volume posix-d2<br> type storage/posix<br> option directory /mnt/chard2/export<br>end-volume<br><br><br>volume locks2<br>
type features/locks<br> subvolumes posix-d2 <br>end-volume<br><br><br>volume posix-d3<br> type storage/posix<br> option directory
/mnt/chard3/export<br>end-volume<br><br>volume locks3<br> type features/locks<br> subvolumes posix-d3 <br>end-volume<br><br><br>volume posix-d4<br> type storage/posix<br> option directory /mnt/chard4/export<br>end-volume<br>
<br>volume locks4<br> type features/locks<br> subvolumes posix-d4 <br>end-volume<br><br>volume home-ns<br> type storage/posix<br> option directory /var/local/glusterfs/namespace1<br>end-volume<br><br>volume home<br> type cluster/distribute<br>
subvolumes locks1 locks2 locks3 locks4<br>end-volume<br><br>volume server<br> type protocol/server<br> option transport-type tcp<br> subvolumes home<br> option auth.addr.home.allow *<br>end-volume<br><br><br>volume zwei<br>
type protocol/client<br> option transport-type tcp<br> option remote-host
zwei<br> option remote-subvolume home<br>end-volume<br><br>volume char<br> type protocol/client<br> option transport-type tcp<br> option remote-host char<br> option remote-subvolume home<br>end-volume<br>
<br>volume pente<br> type protocol/client<br> option transport-type tcp<br> option remote-host pente<br> option remote-subvolume home<br>end-volume<br><br>volume tres<br> type protocol/client<br> option transport-type tcp<br>
option remote-host tres<br> option remote-subvolume home<br>end-volume<br><br>volume dist1<br> type
cluster/distribute<br> subvolumes pente char tres zwei<br>end-volume<br><br><br><br><br></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a rel="nofollow" ymailto="mailto:Gluster-users@gluster.org" target="_blank" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a rel="nofollow" target="_blank" href="http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users">http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Amar Tumballi<br>Gluster/GlusterFS Hacker<br>[bulde on #gluster/<a rel="nofollow" target="_blank" href="http://irc.gnu.org">irc.gnu.org</a>]<br><a rel="nofollow" target="_blank" href="http://www.zresearch.com">http://www.zresearch.com</a> - Commoditizing Super Storage!<br>
</div></div></div></body></html>