Hi Daniel,<br><br>Can you decrease the cache size in io-cache (say to 128MB)? If the files being served are bigger than this size, you can as well remove io-cache from configuration. Do let us know if this solves your issue.<br>
<br>regards,<br><div class="gmail_quote">On Tue, Feb 23, 2010 at 1:25 PM, Daniel Maher <span dir="ltr"><<a href="mailto:dma%2Bgluster@witbe.net">dma+gluster@witbe.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">Raghavendra G wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
* What does the top output corresponding to glusterfs say? what is the memory usage and cpu usage?<br>
* Do you find anything interesting in glusterfs client log files? Can we get the log files?<br>
</blockquote>
<br></div>
Snapshot :<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
1185 root 20 0 2478m 909m 564 D 11.3 90.1 1064:50 glusterfs<br>
<br>
<br>
Over a monitored period of 30 minutes, the CPU usage hovered between 7 and 12, and Mem usage hovered at 90 to 91.<br>
<br>
<br>
<br>
================================================================================<br>
Version : glusterfs 3.0.0 built on Feb 3 2010 14:39:23<br>
git: 2.0.1-886-g8379edd<br>
Starting Time: 2010-02-17 13:58:41<br>
Command line : /usr/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/replicated-tcp.vol /opt/gluster<br>
PID : 1185<br>
System name : Linux<br>
Nodename : 169.install.pxe<br>
Kernel Release : 2.6.26.8-57.fc8<br>
Hardware Identifier: i686<br>
<br>
Given volfile:<br>
+------------------------------------------------------------------------------+<br>
1: ## file auto generated by /usr/bin/glusterfs-volgen (mount.vol)<br>
2: # Cmd line:<br>
3: # $ /usr/bin/glusterfs-volgen --name replicated --raid 1 s01:/opt/gluster s02:/opt/gluster<br>
4:<br>
5: # RAID 1<br>
6: # TRANSPORT-TYPE tcp<br>
7: volume s01-1<br>
8: type protocol/client<br>
9: option transport-type tcp<br>
10: option remote-host s01<br>
11: option transport.socket.nodelay on<br>
12: option transport.remote-port 6996<br>
13: option remote-subvolume brick1<br>
14: end-volume<br>
15:<br>
16: volume s02-1<br>
17: type protocol/client<br>
18: option transport-type tcp<br>
19: option remote-host s02<br>
20: option transport.socket.nodelay on<br>
21: option transport.remote-port 6996<br>
22: option remote-subvolume brick1<br>
23: end-volume<br>
24:<br>
25: volume mirror-0<br>
26: type cluster/replicate<br>
27: subvolumes s01-1 s02-1<br>
28: end-volume<br>
29:<br>
30: volume writebehind<br>
31: type performance/write-behind<br>
32: option cache-size 4MB<br>
33: subvolumes mirror-0<br>
34: end-volume<br>
35:<br>
36: volume readahead<br>
37: type performance/read-ahead<br>
38: option page-count 4<br>
39: subvolumes writebehind<br>
40: end-volume<br>
41:<br>
42: volume iocache<br>
43: type performance/io-cache<br>
44: option cache-size 1GB<br>
45: option cache-timeout 1<br>
46: subvolumes readahead<br>
47: end-volume<br>
48:<br>
49: volume quickread<br>
50: type performance/quick-read<br>
51: option cache-timeout 1<br>
52: option max-file-size 64kB<br>
53: subvolumes iocache<br>
54: end-volume<br>
55:<br>
56: volume statprefetch<br>
57: type performance/stat-prefetch<br>
58: subvolumes quickread<br>
59: end-volume<br>
60:<br>
<br>
+------------------------------------------------------------------------------+<br>
[2010-02-17 13:58:41] W [xlator.c:655:validate_xlator_volume_options] s02-1: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction<br>
[2010-02-17 13:58:41] W [xlator.c:655:validate_xlator_volume_options] s01-1: option 'transport.remote-port' is deprecated, preferred is 'remote-port', continuing with correction<br>
[2010-02-17 13:58:52] E [socket.c:760:socket_connect_finish] s01-1: connection to failed (Connection refused)<br>
[2010-02-17 13:58:52] E [socket.c:760:socket_connect_finish] s01-1: connection to failed (Connection refused)<br>
[2010-02-17 13:58:58] E [socket.c:760:socket_connect_finish] s02-1: connection to failed (No route to host)<br>
[2010-02-17 13:58:58] E [socket.c:760:socket_connect_finish] s02-1: connection to failed (No route to host)<br>
[2010-02-17 14:06:23] N [client-protocol.c:6224:client_setvolume_cbk] s01-1: Connected to <a href="http://10.0.0.38:6996" target="_blank">10.0.0.38:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-17 14:06:23] N [afr.c:2625:notify] mirror-0: Subvolume 's01-1' came back up; going online.<br>
[2010-02-17 14:06:23] N [client-protocol.c:6224:client_setvolume_cbk] s01-1: Connected to <a href="http://10.0.0.38:6996" target="_blank">10.0.0.38:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-17 14:06:23] N [afr.c:2625:notify] mirror-0: Subvolume 's01-1' came back up; going online.<br>
[2010-02-17 14:06:33] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-17 14:06:33] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-20 20:02:43] E [client-protocol.c:415:client_ping_timer_expired] s02-1: Server <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a> has not responded in the last 42 seconds, disconnecting.<br>
[2010-02-20 20:02:47] E [saved-frames.c:165:saved_frames_unwind] s02-1: forced unwinding frame type(2) op(PING)<br>
[2010-02-20 20:02:48] N [client-protocol.c:6972:notify] s02-1: disconnected<br>
[2010-02-20 20:04:01] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-20 20:04:01] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-22 02:37:28] E [client-protocol.c:415:client_ping_timer_expired] s02-1: Server <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a> has not responded in the last 42 seconds, disconnecting.<br>
[2010-02-22 02:37:56] E [saved-frames.c:165:saved_frames_unwind] s02-1: forced unwinding frame type(1) op(READ)<br>
[2010-02-22 02:37:56] E [saved-frames.c:165:saved_frames_unwind] s02-1: forced unwinding frame type(2) op(PING)<br>
[2010-02-22 02:37:56] N [client-protocol.c:6972:notify] s02-1: disconnected<br>
[2010-02-22 02:38:02] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-22 02:38:02] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-22 04:19:02] E [client-protocol.c:415:client_ping_timer_expired] s02-1: Server <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a> has not responded in the last 42 seconds, disconnecting.<br>
[2010-02-22 04:19:50] E [saved-frames.c:165:saved_frames_unwind] s02-1: forced unwinding frame type(1) op(READ)<br>
[2010-02-22 04:19:50] E [saved-frames.c:165:saved_frames_unwind] s02-1: forced unwinding frame type(2) op(PING)<br>
[2010-02-22 04:19:51] N [client-protocol.c:6972:notify] s02-1: disconnected<br>
[2010-02-22 04:20:33] E [client-protocol.c:415:client_ping_timer_expired] s01-1: Server <a href="http://10.0.0.38:6996" target="_blank">10.0.0.38:6996</a> has not responded in the last 42 seconds, disconnecting.<br>
[2010-02-22 04:20:34] E [saved-frames.c:165:saved_frames_unwind] s01-1: forced unwinding frame type(1) op(READ)<br>
[2010-02-22 04:20:41] E [saved-frames.c:165:saved_frames_unwind] s01-1: forced unwinding frame type(2) op(PING)<br>
[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-22 04:20:41] N [client-protocol.c:6972:notify] s01-1: disconnected<br>
[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] s02-1: Connected to <a href="http://10.0.0.39:6996" target="_blank">10.0.0.39:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-22 04:20:41] N [afr.c:2625:notify] mirror-0: Subvolume 's02-1' came back up; going online.<br>
[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] s01-1: Connected to <a href="http://10.0.0.38:6996" target="_blank">10.0.0.38:6996</a>, attached to remote volume 'brick1'.<br>
[2010-02-22 04:20:41] N [client-protocol.c:6224:client_setvolume_cbk] s01-1: Connected to <a href="http://10.0.0.38:6996" target="_blank">10.0.0.38:6996</a>, attached to remote volume 'brick1'.<br>
<br>
<br>
That is the entire client log file. If it would be useful, i can stop the tests, put the Gluster logging into the debug level, and start again. Just let me know.<div><div></div><div class="h5"><br>
<br>
<br>
-- <br>
Daniel Maher <dma+gluster AT witbe DOT net><br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>