<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Verdana; font-size: 10pt; color: #000000'>Hi,<br><br>At the moment I'm testing GlusterFS 2.0.0 for use in our storage cluster for our "Joint Product eXperience" product.<br>I setup two storage servers which are replicated. The client consists of two parts: lighttpd with mod_glusterfs for serving to the wold and a mount to save the files to from our application.<br><br>The config files look like this:<br><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">#####################################</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">### GlusterFS Server Volume File ##</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">#####################################</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">### Export volume "posix1" with the contents of "/home/jpx/glustervolume" directory.</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume posix1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type storage/posix</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option directory /home/jpx/glustervolume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume locks</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type features/locks</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> subvolumes posix1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume brick</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type performance/io-threads</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option thread-count 8</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> subvolumes locks</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume server</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type protocol/server</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option transport-type tcp</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option auth.login.brick.allow xxxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option auth.login.dikkestorageuser.password xxxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> subvolumes brick</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">#####################################</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">### GlusterFS Client Volume File ##</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">#####################################</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">### Add client feature and attach to remote subvolume of server1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume brick1</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type protocol/client</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option transport-type tcp</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option remote-host xxx.xxx.xxx.xxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option remote-subvolume locks</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> </span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option username xxxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option password xxxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">### Add client feature and attach to remote subvolume of server2</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume brick2</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type protocol/client</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option transport-type tcp</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option remote-host xxx.xxx.xxx.xxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option remote-subvolume locks</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> </span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option username xxxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option password xxxx</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">volume afr</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> type cluster/replicate</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> subvolumes brick1 brick2</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> </span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option data-self-heal on</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option metadata-self-heal on</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option entry-self-heal on</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> </span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option data-change-log on</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option metadata-change-log on</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option entry-change-log on</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> </span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option data-lock-server-count 2</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option metadata-lock-server-count 2</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> option entry-lock-server-count 2</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">end-volume</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">#####################################</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">
### Lighttpd include file for gluster ##</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">
#####################################</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">$HTTP["host"] == "gluster3.vh1.royalfish.nl" {</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.document-root = ""</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.prefix = ""</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.logfile = "/var/log/glusterfs/lighttpd.log"</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.volume-specfile = "/etc/glusterfs/client-volume.vol"</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.loglevel = "debug"</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.cache-timeout = 300</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;"> glusterfs.xattr-interface-size-limit = "65536"</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">}</span><br><br>I have a few problems:<br><ol><li>The /var/log/glusterfs/lighttpd.log file gives a few warnings/errors:</li><ol style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><li>2009-05-24 15:18:44 W [client-protocol.c:6783:init] brick1: WARNING: Failed to set 'ulimit -n 1M': Operation not permitted</li><li>2009-05-24 15:18:44 E [client-protocol.c:6791:init] brick1: Failed to set max open fd to 64k: Operation not permitted</li><li>2009-05-24 15:18:44 W [client-protocol.c:6783:init] brick2: WARNING: Failed to set 'ulimit -n 1M': Operation not permitted</li><li>2009-05-24 15:18:44 E [client-protocol.c:6791:init] brick2: Failed to set max open fd to 64k: Operation not permitted</li><li>2009-05-24 15:18:44 E [name.c:420:client_bind] brick1: cannot bind inet socket (11) to port less than 1024 (Permission denied)</li><li>2009-05-24 15:18:44 E [name.c:420:client_bind] brick1: cannot bind inet socket (12) to port less than 1024 (Permission denied)</li><li>2009-05-24 15:18:44 E [name.c:420:client_bind] brick2: cannot bind inet socket (13) to port less than 1024 (Permission denied)</li><li>2009-05-24 15:18:44 E [name.c:420:client_bind] brick2: cannot bind inet socket (14) to port less than 1024 (Permission denied)</li><li>2009-05-24 15:18:44 E [client-protocol.c:3733:client_lookup] brick1: LOOKUP 1/zzz (/zzz): failed to get remote inode number for parent</li><li>2009-05-24 15:18:44 E [client-protocol.c:3733:client_lookup] brick1: LOOKUP 1/zzz (/zzz): failed to get remote inode number for parent</li><li>2009-05-24 15:18:44 D [socket.c:654:__socket_proto_state_machine] brick2: partial data read on NB socket</li><li>2009-05-24 15:18:44 D [socket.c:654:__socket_proto_state_machine] brick2: partial data read on NB socket</li><li>2009-05-24 15:18:44 D [socket.c:654:__socket_proto_state_machine] brick2: partial data read on NB socket</li><li>2009-05-24 15:18:44 D [socket.c:654:__socket_proto_state_machine] brick2: partial data read on NB socket</li></ol><li>When trying to download small files, they serve up well. No problem at all.</li><li>When trying to download 1mb.bin, 10mb.bin and larger files, lighttpd's CPU usage becomes 99%.</li><li>When stracing the work of lighttpd it is constantly polling:<br><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">00:46:55.528608 writev(6, [{"", 0}], 1) = 0</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">00:46:55.528732 time(NULL) = 1243032415</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">00:46:55.528869 poll([{fd=4, events=POLLIN}, {fd=6, events=POLLOUT}], 2, 1000) = 1 ([{fd=6, revents=POLLOUT}])</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">00:46:55.529005 writev(6, [{"", 0}], 1) = 0</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">00:46:55.529135 time(NULL) = 1243032415</span><br style="font-family: Courier New,courier,monaco,monospace,sans-serif;"><span style="font-family: Courier New,courier,monaco,monospace,sans-serif;">00:46:55.529250 poll([{fd=4, events=POLLIN}, {fd=6, events=POLLOUT}], 2, 1000) = 1 ([{fd=6, revents=POLLOUT}])</span></li></ol>Does anyone know why my lighttpd is using so much CPU? Possibly I'm doing something terribly wrong?<br><br>Thanks in advance for your replies.<br><br>Best Regards Jasper<br><br><br></div></body></html>