<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content="text/html; charset=unicode" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.18241"></HEAD>
<BODY style="PADDING-LEFT: 10px; PADDING-RIGHT: 10px; PADDING-TOP: 15px"
id=MailContainerBody leftMargin=0 topMargin=0 bgColor=#ffffff
CanvasTabStop="true" name="Compose message area">
<DIV>Hey,<BR><BR> I try to use the following cascading mode to
enhance the throughput performance, but the result is bad.<BR>
There are four storage nodes and each exports 2
directories.<BR><BR> (on
client)
unify(alu)
translator<BR>
/
\<BR>
/
\<BR>
/ \<BR>
/
\<BR>
/
\<BR>(translator on
client) stripe
stripe
<BR>
/
\
/
\<BR>
/
\
/ \<BR>
/
\
/ \ <BR>(translator
on client) AFR
AFR
AFR
AFR
<BR>
/ \
/
\ /
\ /
\<BR>
/ \
/
\
/ \
/
\<BR>
#1-1 #2-1 #3-1
#4-4 #1-2
#2-2 #3-2 #4-2<BR> When I use iozone
to test with 10 concurrent processes, I only find the #3 and #4 storages
working, and the other 2 nodes doesn't work. As my expectation, the 4
storage nodes should simultaneously work at any time, but it is out of my mind.
what's wrong with it?<BR></DIV>
<DIV><FONT size=2 face=??> Another issue is that the memory is exhausted
on storage nodes when writing and on client server when reading, and it is not
what I want. Is there any method to limit the usage of memory?</FONT></DIV>
<DIV><FONT size=2 face=??></FONT><FONT size=2 face=??></FONT><FONT size=2
face=??></FONT><FONT size=2 face=??></FONT><FONT size=2 face=??></FONT><FONT
size=2 face=??></FONT><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??>Best Wishes.</FONT><BR>Alfred<BR><BR>Following is the
vol file on client.<BR><BR><FONT size=2 face=??>### file:
client-volume.spec.sample</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2
face=??>##############################################<BR>### GlusterFS
Client Volume Specification
##<BR>##############################################</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??>#### CONFIG FILE RULES:<BR>### "#" is comment
character.<BR>### - Config file is case sensitive<BR>### - Options within a
volume block can be in any order.<BR>### - Spaces or tabs are used as delimitter
within a line. <BR>### - Each option should end within a line.<BR>### - Missing
or commented fields will assume default values.<BR>### - Blank/commented lines
are allowed.<BR>### - Sub-volumes should already be defined above before
referring.</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??>### Add client feature and attach to remote subvolume
of server1</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??>volume client-ns<BR> type
protocol/client<BR> option transport-type
tcp/client # for TCP/IP transport<BR>
option remote-host 192.168.13.2 # IP
address of the remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
name_space # name of the
remote volume<BR>end-volume</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??>volume client11<BR> type
protocol/client<BR> option transport-type
tcp/client # for TCP/IP transport<BR>
option remote-host 192.168.13.2 # IP
address of the remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick1 # name of the
remote volume<BR>end-volume</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV>
<DIV><FONT size=2 face=??>volume client12<BR> type
protocol/client<BR> option transport-type
tcp/client # for TCP/IP transport<BR>
option remote-host 192.168.13.2 # IP
address of the remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick2 # name of the
remote volume<BR>end-volume</FONT></DIV>
<DIV><FONT size=2 face=??></FONT> </DIV><FONT size=2 face=??>
<DIV><BR>volume client21<BR> type protocol/client<BR> option
transport-type tcp/client # for TCP/IP
transport<BR> option remote-host
192.168.13.4 # IP address of the
remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick1 # name of the
remote volume<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume client22<BR> type protocol/client<BR> option
transport-type tcp/client # for TCP/IP
transport<BR> option remote-host
192.168.13.4 # IP address of the
remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick2 # name of the
remote volume<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume client31<BR> type protocol/client<BR> option
transport-type tcp/client # for TCP/IP
transport<BR> option remote-host
192.168.13.5 # IP address of the
remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick1 # name of the
remote volume<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume client32<BR> type protocol/client<BR> option
transport-type tcp/client # for TCP/IP
transport<BR> option remote-host
192.168.13.5 # IP address of the
remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick2 # name of the
remote volume<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume client41<BR> type protocol/client<BR> option
transport-type tcp/client # for TCP/IP
transport<BR> option remote-host
192.168.13.7 # IP address of the
remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick1 # name of the
remote volume<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume client42<BR> type protocol/client<BR> option
transport-type tcp/client # for TCP/IP
transport<BR> option remote-host
192.168.13.7 # IP address of the
remote brick<BR># option remote-port
6996
# default server port is 6996<BR># option transport-timeout
30 # seconds
to wait for a
response<BR>
# from server for each request<BR> option remote-subvolume
brick2 # name of the
remote volume<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume afr1<BR> type cluster/afr<BR> subvolumes client11
client21<BR> option debug
off # turns on detailed debug
messages
<BR>
# in log by default is debugging off<BR> option self-heal
on # turn off self healing default is on<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume afr2<BR> type cluster/afr<BR> subvolumes client31
client41<BR> option debug
off # turns on detailed debug
messages
<BR>
# in log by default is debugging off<BR> option self-heal
on # turn off self healing default is on<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume afr3<BR> type cluster/afr<BR> subvolumes client12
client22<BR> option debug
off # turns on detailed debug
messages
<BR>
# in log by default is debugging off<BR> option self-heal
on # turn off self healing default is on<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume afr4<BR> type cluster/afr<BR> subvolumes client32
client42<BR> option debug
off # turns on detailed debug
messages
<BR>
# in log by default is debugging off<BR> option self-heal
on # turn off self healing default is on<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume stripe1<BR> type cluster/stripe<BR> option
block-size
1MB
#default size is 128KB<BR> subvolumes afr1 afr2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>volume stripe2<BR> type cluster/stripe<BR> option
block-size
1MB
#default size is 128KB<BR> subvolumes afr3 afr4<BR>end-volume</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>volume bricks<BR> type cluster/unify<BR> subvolumes stripe1
stripe2<BR> option namespace client-ns<BR> option scheduler
alu <BR># option alu.limits.min-free-disk
5% # Don't create files one a volume with less than 5% free
diskspace<BR># option alu.limits.max-open-files 10000 # Don't create
files on a volume with more than 10000 files open</DIV>
<DIV> </DIV>
<DIV> # When deciding where to place a file, first look at the disk-usage,
then at read-usage, write-usage, open files, and finally the
disk-speed-usage.<BR> option alu.order
disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage<BR>
option alu.disk-usage.entry-threshold 2GB # Kick in if the
discrepancy in disk-usage between volumes is more than 2GB<BR> option
alu.disk-usage.exit-threshold 60MB # Don't stop writing to the
least-used volume until the discrepancy is 1988MB <BR> option
alu.open-files-usage.entry-threshold 1024 # Kick in if the
discrepancy in open files is 1024<BR> option
alu.open-files-usage.exit-threshold 32 # Don't stop until 992 files
have been written the least-used volume<BR> option
alu.read-usage.entry-threshold 20% # Kick in when the read-usage
discrepancy is 20%<BR> option alu.read-usage.exit-threshold 4%
# Don't stop until the discrepancy has been reduced to 16% (20% - 4%)<BR>
option alu.write-usage.entry-threshold 20% # Kick in when the
write-usage discrepancy is 20%<BR> option alu.write-usage.exit-threshold
4% # Don't stop until the discrepancy has been reduced to
16%<BR># option alu.disk-speed-usage.entry-threshold # NEVER SET IT. SPEED
IS CONSTANT!!!<BR># option alu.disk-speed-usage.exit-threshold #
NEVER SET IT. SPEED IS CONSTANT!!!<BR> option alu.stat-refresh.interval
10sec # Refresh the statistics used for decision-making every 10
seconds<BR> option alu.stat-refresh.num-file-create 10 #
Refresh the statistics used for decision-making after creating 10
files<BR>end-volume</DIV>
<DIV> </DIV>
<DIV><BR>### Add io-threads feature<BR>volume iot<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB</DIV>
<DIV> </DIV>
<DIV> subvolumes bricks #stripe #afr #bricks<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR> option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes iocache <BR>end-volume</DIV>
<DIV> </DIV>
<DIV><BR>### Add io-threads feature<BR>volume iot_stripe1<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB<BR> subvolumes
stripe1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead_stripe1<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot_stripe1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache_stripe1<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead_stripe1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback_stripe1<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR># option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes iocache_stripe1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV><BR>### Add io-threads feature<BR>volume iot_stripe2<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB<BR> subvolumes
stripe2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead_stripe2<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot_stripe2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache_stripe2<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead_stripe2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback_stripe2<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR># option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes iocache_stripe2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV><BR>### Add io-threads feature<BR>volume iot_afr1<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB<BR> subvolumes
afr1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead_afr1<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot_afr1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache_afr1<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead_afr1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback_afr1<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR># option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes iocache_afr1<BR>end-volume</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>### Add io-threads feature<BR>volume iot_afr2<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB<BR> subvolumes
afr2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead_afr2<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot_afr2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache_afr2<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead_afr2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback_afr2<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR># option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes iocache_afr2<BR>end-volume</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>### Add io-threads feature<BR>volume iot_afr3<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB<BR> subvolumes
afr3<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead_afr3<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot_afr3<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache_afr3<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead_afr3<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback_afr3<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR># option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes iocache_afr3<BR>end-volume</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>### Add io-threads feature<BR>volume iot_afr4<BR> type
performance/io-threads<BR> option thread-count 1 # deault is
1<BR> option cache-size 16MB #64MB<BR> subvolumes
afr4<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add readahead feature<BR>volume readahead_afr4<BR> type
performance/read-ahead<BR> option page-size
1MB # unit in bytes<BR> option page-count
4 # cache per file = (page-count x
page-size)<BR> subvolumes iot_afr4<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add IO-Cache feature<BR>volume iocache_afr4<BR> type
performance/io-cache<BR> option page-size 1024KB<BR> option
page-count 8<BR> subvolumes readahead_afr4<BR>end-volume</DIV>
<DIV> </DIV>
<DIV>### Add writeback feature<BR>volume writeback_afr4<BR> type
performance/write-behind<BR> option aggregate-size 1MB #option
flush-behind off<BR> # option window-size 3MB #
default is 0bytes<BR># option flush-behind on #
default is 'off'<BR> subvolumes
iocache_afr4<BR>end-volume<BR></FONT><BR><BR> <BR><BR><BR></DIV></BODY></HTML>