<font size="2"><font face="tahoma,sans-serif">Here is the log. Nothing really stands out. There is one entry from today and the previous log entry was from 6/3.<br></font></font><div><div><font class="Apple-style-span" face="tahoma, sans-serif"><br>
</font></div><div><font class="Apple-style-span" face="tahoma, sans-serif"><div>[alan@app1:10.71.57.82:glusterfs]$ sudo cat /var/log/glusterfs/drives-d1.log</div><div>[2011-06-03 18:17:05.160722] W [io-stats.c:1644:init] d1: dangling volume. check volfile </div>
<div>[2011-06-03 18:17:05.160865] W [dict.c:1205:data_to_str] dict: @data=(nil)</div><div>[2011-06-03 18:17:05.160897] W [dict.c:1205:data_to_str] dict: @data=(nil)</div><div>Given volfile:</div><div>+------------------------------------------------------------------------------+</div>
<div> 1: volume d1-client-0</div><div> 2: type protocol/client</div><div> 3: option remote-host 10.198.6.214</div><div> 4: option remote-subvolume /data/d1</div><div> 5: option transport-type tcp</div>
<div> 6: end-volume</div><div> 7: </div><div> 8: volume d1-client-1</div><div> 9: type protocol/client</div><div> 10: option remote-host 10.195.15.38</div><div> 11: option remote-subvolume /data/d1</div><div>
12: option transport-type tcp</div><div> 13: end-volume</div><div> 14: </div><div> 15: volume d1-replicate-0</div><div> 16: type cluster/replicate</div><div> 17: subvolumes d1-client-0 d1-client-1</div><div>
18: end-volume</div>
<div> 19: </div><div> 20: volume d1-write-behind</div><div> 21: type performance/write-behind</div><div> 22: option cache-size 4MB</div><div> 23: subvolumes d1-replicate-0</div><div> 24: end-volume</div><div>
25: </div>
<div> 26: volume d1-read-ahead</div><div> 27: type performance/read-ahead</div><div> 28: subvolumes d1-write-behind</div><div> 29: end-volume</div><div> 30: </div><div> 31: volume d1-io-cache</div><div> 32: type performance/io-cache</div>
<div> 33: option cache-size 1024MB</div><div> 34: subvolumes d1-read-ahead</div><div> 35: end-volume</div><div> 36: </div><div> 37: volume d1-quick-read</div><div> 38: type performance/quick-read</div><div> 39: option cache-size 1024MB</div>
<div> 40: subvolumes d1-io-cache</div><div> 41: end-volume</div><div> 42: </div><div> 43: volume d1-stat-prefetch</div><div> 44: type performance/stat-prefetch</div><div> 45: subvolumes d1-quick-read</div><div>
46: end-volume</div><div> 47: </div><div> 48: volume d1</div><div> 49: type debug/io-stats</div><div> 50: subvolumes d1-stat-prefetch</div><div> 51: end-volume</div><div><br></div><div>+------------------------------------------------------------------------------+</div>
<div>[2011-06-03 18:17:08.676157] I [client-handshake.c:1005:select_server_supported_programs] d1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)</div><div>[2011-06-03 18:17:08.684299] I [client-handshake.c:1005:select_server_supported_programs] d1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)</div>
<div>[2011-06-03 18:17:08.718624] I [client-handshake.c:841:client_setvolume_cbk] d1-client-1: Connected to <a href="http://10.195.15.38:24009">10.195.15.38:24009</a>, attached to remote volume '/data/d1'.</div><div>
[2011-06-03 18:17:08.718687] I [afr-common.c:2572:afr_notify] d1-replicate-0: Subvolume 'd1-client-1' came back up; going online.</div><div>[2011-06-03 18:17:08.732772] I [fuse-bridge.c:2821:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.14</div>
<div>[2011-06-03 18:17:08.735602] I [afr-common.c:819:afr_fresh_lookup_cbk] d1-replicate-0: added root inode</div><div>[2011-06-03 18:17:08.748443] I [client-handshake.c:841:client_setvolume_cbk] d1-client-0: Connected to <a href="http://10.198.6.214:24009">10.198.6.214:24009</a>, attached to remote volume '/data/d1'.</div>
<div>[2011-06-10 06:33:08.255922] W [fuse-bridge.c:2510:fuse_getxattr] glusterfs-fuse: 3480740: GETXATTR (null)/3039291028 (security.capability) (fuse_loc_fill() failed)</div><div>[alan@app1:10.71.57.82:glusterfs]$ </div>
</font></div><div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">Forgive me, I'm relatively new to GlusterFS. I'm not sure what level of logging I have setup. How can I tell the level of logging I have configured? Perhaps I could increase this? Hopefully capture more detailed information.</div>
<div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote">Thanks again for the help!</div><div class="gmail_quote"> - Alan</div><div class="gmail_quote"><br></div><div class="gmail_quote">
Just in case this helps, here are the volume configuration files from the server-</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote"><br></div><div class="gmail_quote">[alan@file1:10.198.6.214:d1]$ sudo cat d1-fuse.vol</div>
<div class="gmail_quote">volume d1-client-0</div><div class="gmail_quote"> type protocol/client</div><div class="gmail_quote"> option remote-host 10.198.6.214</div><div class="gmail_quote"> option remote-subvolume /data/d1</div>
<div class="gmail_quote"> option transport-type tcp</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-client-1</div><div class="gmail_quote"> type protocol/client</div>
<div class="gmail_quote"> option remote-host 10.195.15.38</div><div class="gmail_quote"> option remote-subvolume /data/d1</div><div class="gmail_quote"> option transport-type tcp</div><div class="gmail_quote">end-volume</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-replicate-0</div><div class="gmail_quote"> type cluster/replicate</div><div class="gmail_quote"> subvolumes d1-client-0 d1-client-1</div><div class="gmail_quote">
end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-write-behind</div><div class="gmail_quote"> type performance/write-behind</div><div class="gmail_quote"> option cache-size 4MB</div>
<div class="gmail_quote"> subvolumes d1-replicate-0</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-read-ahead</div><div class="gmail_quote"> type performance/read-ahead</div>
<div class="gmail_quote"> subvolumes d1-write-behind</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-io-cache</div><div class="gmail_quote"> type performance/io-cache</div>
<div class="gmail_quote"> option cache-size 1024MB</div><div class="gmail_quote"> subvolumes d1-read-ahead</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-quick-read</div>
<div class="gmail_quote"> type performance/quick-read</div><div class="gmail_quote"> option cache-size 1024MB</div><div class="gmail_quote"> subvolumes d1-io-cache</div><div class="gmail_quote">end-volume</div><div class="gmail_quote">
<br></div><div class="gmail_quote">volume d1-stat-prefetch</div><div class="gmail_quote"> type performance/stat-prefetch</div><div class="gmail_quote"> subvolumes d1-quick-read</div><div class="gmail_quote">end-volume</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">volume d1</div><div class="gmail_quote"> type debug/io-stats</div><div class="gmail_quote"> subvolumes d1-stat-prefetch</div><div class="gmail_quote">end-volume</div>
<div class="gmail_quote">[alan@file1:10.198.6.214:d1]$</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote">[alan@file1:10.198.6.214:d1]$ sudo cat d1.10.195.15.38.data-d1.vol</div>
<div class="gmail_quote">volume d1-posix</div><div class="gmail_quote"> type storage/posix</div><div class="gmail_quote"> option directory /data/d1</div><div class="gmail_quote">end-volume</div><div class="gmail_quote">
<br></div><div class="gmail_quote">volume d1-access-control</div><div class="gmail_quote"> type features/access-control</div><div class="gmail_quote"> subvolumes d1-posix</div><div class="gmail_quote">end-volume</div>
<div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-locks</div><div class="gmail_quote"> type features/locks</div><div class="gmail_quote"> subvolumes d1-access-control</div><div class="gmail_quote">
end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-io-threads</div><div class="gmail_quote"> type performance/io-threads</div><div class="gmail_quote"> option thread-count 8</div>
<div class="gmail_quote">
subvolumes d1-locks</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume /data/d1</div><div class="gmail_quote"> type debug/io-stats</div><div class="gmail_quote">
subvolumes d1-io-threads</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-server</div><div class="gmail_quote"> type protocol/server</div><div class="gmail_quote">
option transport-type tcp</div><div class="gmail_quote"> option auth.addr./data/d1.allow *</div><div class="gmail_quote"> subvolumes /data/d1</div><div class="gmail_quote">end-volume</div><div class="gmail_quote">
[alan@file1:10.198.6.214:d1]$</div><div class="gmail_quote"><br></div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote">volume d1-posix</div><div class="gmail_quote"> type storage/posix</div>
<div class="gmail_quote"> option directory /data/d1</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-access-control</div><div class="gmail_quote"> type features/access-control</div>
<div class="gmail_quote"> subvolumes d1-posix</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-locks</div><div class="gmail_quote"> type features/locks</div>
<div class="gmail_quote"> subvolumes d1-access-control</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-io-threads</div><div class="gmail_quote"> type performance/io-threads</div>
<div class="gmail_quote"> option thread-count 8</div><div class="gmail_quote"> subvolumes d1-locks</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume /data/d1</div>
<div class="gmail_quote"> type debug/io-stats</div><div class="gmail_quote"> subvolumes d1-io-threads</div><div class="gmail_quote">end-volume</div><div class="gmail_quote"><br></div><div class="gmail_quote">volume d1-server</div>
<div class="gmail_quote"> type protocol/server</div><div class="gmail_quote"> option transport-type tcp</div><div class="gmail_quote"> option auth.addr./data/d1.allow *</div><div class="gmail_quote"> subvolumes /data/d1</div>
<div class="gmail_quote">end-volume</div></div></div><div class="gmail_quote"><br></div></div><div class="gmail_quote"> </div><div class="gmail_quote"><br></div><div class="gmail_quote">On Fri, Jun 10, 2011 at 10:43 AM, Anand Avati <span dir="ltr"><<a href="mailto:anand.avati@gmail.com">anand.avati@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Do you find anything in the client logs?<br><br><div class="gmail_quote"><div><div></div><div class="h5">On Fri, Jun 10, 2011 at 3:20 AM, Alan Zapolsky <span dir="ltr"><<a href="mailto:alan@droptheworld.com" target="_blank">alan@droptheworld.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div class="h5">Hello,<br>
<br>
I have a PHP web application that uses Gluster to store its files.<br>
There are a few areas of the application that perform multiple<br>
operations on small to medium size files (such as .jpg and .pdf files)<br>
in quick succession. My dev environment does not use Gluster and had<br>
no problems - but in production, I am seeing some strange errors and<br>
am wondering if perhaps Gluster could be the cause.<br>
<br>
Example 1: I may perform the following sequence of operations on a<br>
batch of 3,000 photos:<br>
<br>
1. Copy the photo from remote URL to a temp folder<br>
2. Create a unique directory based on that photo_id<br>
3. Move the photo from the temp dir to the new dir<br>
4. Create a medium (1000px), small (250px), and tiny (72px) versions<br>
of the photo and save them to the temp dir.<br>
<br>
Example 2: Turning a PDF to a PNG<br>
<br>
1. Create a PDF document, ranging from a few MB to tens of MB or more.<br>
2. Use Ghostscript to read a specific page and create a temp .png file from it.<br>
3. Use Imagemagick to resize the .png and add a drop shadow.<br>
4. Delete the temp .png file<br>
<br>
In example 1, I am finding that some of the resulting JPG files wind<br>
up being corrupted - when attempting to read the problematic JPGs, I<br>
receive messages such as "bad Huffman code" or "premature end of data<br>
segment".<br>
<br>
In example 2, I am getting strange errors on step 3 - it sometimes has<br>
a problem finding the file created by step 2. And then on step 4, I<br>
sometimes get an error that really baffles me - a line of PHP code<br>
such as:<br>
<br>
if (is_file($file)) unlink($file);<br>
<br>
.. will sometimes produce an error "unlink: no such file or<br>
directory". I have no idea how it passes the is_file() check and then<br>
unlink() says it's not there.<br>
<br>
At this point I'm just wondering if Gluster could be the culprit for<br>
any of this strange behavior, considering the types of operations and<br>
file sizes I'm working with. I've included my Gluster volume info<br>
below.<br>
<br>
Thanks for the help.<br>
<br>
- Alan<br>
<br>
Gluster Volume Info<br>
[alan@file1:10.X.X.X:d1]$ sudo cat info<br>
type=2<br>
count=2<br>
status=1<br>
sub_count=2<br>
version=1<br>
transport-type=0<br>
volume-id=3fc69046-a324-42de-bf8a-e1bd2e6e45ab<br>
brick-0=10.X.X.X:-data-d1<br>
brick-1=10.Y.Y.Y.38:-data-d1<br>
performance.cache-size=1024MB<br>
performance.write-behind-window-size=4MB<br>
performance.io-thread-count=8<br>
[alan@file1:10.X.X.X:d1]$<br></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br>
</blockquote></div><br></div></div>