<div>We have two servers each with ZFS running in raidz on three hard drives of the same size and glusterFS between the two servers. The ZFS has two filesystems tank and tank/home each created using zfs create and GlusterFS is mounted based on the tank/home filesystem to /mnt. At this point everything works great but when I then run zfs create tank/home/virtual and then cd to /mnt/virtual and run ls I get the errors i reported to printed to the command line.</div>
<div><br></div><div>Volume files:</div><div>glusterfsd.vol</div><div><div>## file auto generated by /bin/glusterfs-volgen</div><div># Cmd line:</div><div># $ /bin/glusterfs-volgen --name home --raid 1 mx.server.edu:/tank/home mx2.server.edu:/tank/home</div>
<div><br></div><div>volume posix1</div><div> type storage/posix</div><div># option o-direct enable # (default: disable) boolean type only</div><div># option export-statfs-size no # (default: yes) boolean type only</div>
<div># option mandate-attribute off # (default: on) boolean type only</div><div># option span-devices 8 # (default: 0) integer value</div><div># option background-unlink yes # (default: no) boolean type</div><div> option directory /tank/home</div>
<div>end-volume</div><div><br></div><div>volume locks1</div><div> type features/locks</div><div># option mandatory on # Default off, used in specific applications</div><div> subvolumes posix1</div><div>end-volume</div>
<div><br></div><div>volume brick1</div><div> type performance/io-threads</div><div> option thread-count 8</div><div># option autoscaling yes # Heuristic for autoscaling threads on demand</div><div># option min-threads 2 # min count for thread pool</div>
<div># option max-threads 64 # max count for thread pool</div><div> subvolumes locks1</div><div>end-volume</div><div><br></div><div>volume server-tcp</div><div> type protocol/server</div><div> option transport-type tcp</div>
<div> option auth.addr.brick1.allow *</div><div> option transport.socket.listen-port 6996</div><div> option transport.socket.nodelay on</div><div> subvolumes brick1</div><div>end-volume</div></div><div><br></div>
<div>glusterfs.vol</div><div><div>## file auto generated by /bin/glusterfs-volgen</div><div># Cmd line:</div><div># $ /bin/glusterfs-volgen --name home --raid 1 mx.server.edu:/tank/home mx2.server.edu:/tank/home</div><div>
<br></div><div># RAID 1</div><div># TRANSPORT-TYPE tcp</div><div>volume mx.server.edu-1</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host <a href="http://mx.server.edu">mx.server.edu</a></div>
<div> option transport.socket.nodelay on</div><div> option transport.remote-port 6996</div><div> option remote-subvolume brick1</div><div>end-volume</div><div><br></div><div>volume mx2.server.edu-1</div><div> type protocol/client</div>
<div> option transport-type tcp</div><div> option remote-host <a href="http://mx2.server.edu">mx2.server.edu</a></div><div> option transport.socket.nodelay on</div><div> option transport.remote-port 6996</div>
<div> option remote-subvolume brick1</div><div>end-volume</div><div><br></div><div>volume mirror-0</div><div> type cluster/replicate</div><div> subvolumes mx.server.edu-1 mx2.server.edu-1</div><div>end-volume</div>
<div><br></div><div>volume writebehind</div><div> type performance/write-behind</div><div> option cache-size 4MB</div><div># option enable-trickling-writes yes # Flush final write calls when network is free</div>
<div>
# option enable-O_SYNC yes # Enable O_SYNC for write-behind</div><div># option disable-for-first-nbytes 1 # Disable first nbytes with very small initial writes</div><div> subvolumes mirror-0</div><div>end-volume</div>
<div><br></div><div>volume readahead</div><div> type performance/read-ahead</div><div> option page-count 4</div><div># option force-atime-update yes # force updating atimes, default off</div><div> subvolumes writebehind</div>
<div>end-volume</div><div><br></div><div>volume iocache</div><div> type performance/io-cache</div><div> option cache-size `echo $(( $(grep 'MemTotal' /proc/meminfo | sed 's/[^0-9]//g') / 5120 ))`MB</div>
<div> option cache-timeout 1</div><div># option priority *.html:1,abc*:2 # Priority list for iocaching files</div><div> subvolumes readahead</div><div>end-volume</div><div><br></div><div>volume quickread</div><div>
type performance/quick-read</div><div> option cache-timeout 1</div><div> option max-file-size 64kB</div><div> subvolumes iocache</div><div>end-volume</div><div><br></div><div>volume statprefetch</div><div> type performance/stat-prefetch</div>
<div> subvolumes quickread</div><div>end-volume</div></div><div><br></div><div><br></div><div>and how do you want the log files as they are already pretty big</div><br clear="all">James J Regan IV<br>COO Emeritus Consulting<br>
B.S. Computer Science<br>B.A. Linguistics<br>
<br><br><div class="gmail_quote">On Wed, Jul 7, 2010 at 5:40 AM, Lakshmipathi <span dir="ltr"><<a href="mailto:lakshmipathi@gluster.com">lakshmipathi@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div style="font-family:Times New Roman;font-size:12pt;color:#000000">Hi,<br>Could you please explain more about your setup in detail ? Are you trying to create a sub-file system in glusterfs-server using zfs create command?<br>
Please sent us the volume and log files.Thanks.<br>
<span><br>-- <br>----<br>Cheers,<br>Lakshmipathi.G<br>FOSS Programmer.</span><br>
-------- Original Message --------
<table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<th nowrap valign="BASELINE" align="RIGHT">Subject: </th>
<td>[Gluster-devel] GlusterFS + ZFS Error</td>
</tr>
<tr>
<th nowrap valign="BASELINE" align="RIGHT">Date: </th>
<td>Tue, 6 Jul 2010 17:10:10 -0400</td>
</tr>
<tr>
<th nowrap valign="BASELINE" align="RIGHT">From: </th>
<td>Regan_James <a href="mailto:jamreg@emeritus-consultants.com" target="_blank"><jamreg@emeritus-consultants.com></a></td>
</tr>
<tr>
<th nowrap valign="BASELINE" align="RIGHT">To: </th>
<td>gluster-devel <a href="mailto:gluster-devel@nongnu.org" target="_blank"><gluster-devel@nongnu.org></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<span style="font-family:arial,sans-serif;font-size:13px;border-collapse:collapse">I
set up a ZFS + glusterFS system and everything worked fine until I ran
a zfs create command to create a subfile-system at which point the
GlusterFS became corrupted and ls commands of the new filesystem first
gave me an error "cannot access <filesystem>: Device or resource
busy" and then the next and all other times I ran it I got "cannot open
directory .: No such file or directory." This error happens every time
I do, when only one is running or when both are running</span>
<div><font face="arial, sans-serif"><span style="border-collapse:collapse"><br clear="all">
</span></font>James J Regan IV<br>
COO Emeritus Consulting<br>
B.S. Computer Science<br>
B.A. Linguistics<br>
</div>
<span><br><br></span></div></div></blockquote></div><br>