<p>== Data Copying when a new brick is added. ==</p>
<p>Hi. I'm a first time glusterfs user, and I'm trying to simulate what will happen when I need to add more bricks for storage capacity. My config files are below but I'll try and explain what is going on. I have 2 machines with 2 hard drives in each. I created a replicated storage system where machine a replicates to machine b. Everything worked fine when only 1 harddrive was set up on each machine. I filled up the harddrives to the point where there is only 1.5 gigs free on each one. They match because it was all replicated.</p>
<p>After this, I added in the configuration for the second harddrive in each machine. The machines are still replicating, so the harddrives should be distributed, which is what I tried. All is well with the new configuration if I add new files to the system. It correctly sees that the old harddrives are full and puts all the new files onto the second empty drive. </p>
<p>However, if I go into the mount point on the client machine and do a directory listing (or the shell autocomplete attempts to read it) something strange happens. Machine 1 is fine, but machine 2 starts to copy over files from the full harddrive to the empty one. It's still copying this time after I retried the configuration, but the first time it copied about half the data over to the second harddrive. The first harddrive remained full though. </p>
<p>Has anyone experienced this, or can see something wrong with my configuration files? The server files are exactly the same on both machines and the machines have the same hardware components and OS (Ubuntu) as well. Any help would be appreciated.</p>
<p><br>===========================================================<br>Server Configuration<br>===========================================================</p>
<p> volume posix1<br> type storage/posix<br> option directory /data/export<br> end-volume<br> volume posix2<br> type storage/posix<br> option directory /media/disk2<br> end-volume<br> <br> volume locks<br>
type features/locks<br> subvolumes posix1<br> end-volume<br> <br> volume locks2<br> type features/locks<br> subvolumes posix2<br> end-volume<br> <br> volume brick<br> type performance/io-threads<br> option thread-count 8<br>
subvolumes locks<br> end-volume<br> <br> volume brick2<br> type performance/io-threads<br> option thread-count 8<br> subvolumes locks2<br> end-volume<br> <br> volume server<br> type protocol/server<br>
option transport-type tcp<br> option auth.addr.brick.allow 192.168.0.191,192.168.0.192<br> option auth.addr.brick2.allow 192.168.0.191,192.168.0.192<br> subvolumes brick brick2<br> end-volume </p>
<p><br>===========================================================<br>Client Configuration<br>===========================================================<br> volume remote1a<br> type protocol/client<br> option transport-type tcp<br>
option remote-host 192.168.0.151<br> option remote-subvolume brick<br> end-volume<br> volume remote1b<br> type protocol/client<br> option transport-type tcp<br> option remote-host 192.168.0.151<br> option remote-subvolume brick2<br>
end-volume<br> <br> volume remote2a<br> type protocol/client<br> option transport-type tcp<br> option remote-host 192.168.0.152<br> option remote-subvolume brick<br> end-volume<br> <br> volume remote2b<br>
type protocol/client<br> option transport-type tcp<br> option remote-host 192.168.0.152<br> option remote-subvolume brick2<br> end-volume<br> <br> <br> volume unify1<br> type cluster/distribute <br> subvolumes remote1a remote1b<br>
option scheduler alu # use the ALU scheduler<br> option alu.limits.min-free-disk 5% # Don't create files one a volume with less than 5% free diskspace<br> option alu.limits.max-open-files 10000 # Don't create files on a volume with more than 10000 files open<br>
option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage<br> option alu.disk-usage.entry-threshold 3GB # Kick in if the discrepancy in disk-usage between volumes is more than 3GB<br> option alu.disk-usage.exit-threshold 60MB # Don't stop writing to the least-used volume until the discrepancy is 1988MB <br>
option alu.open-files-usage.entry-threshold 1024 # Kick in if the discrepancy in open files is 1024<br> option alu.open-files-usage.exit-threshold 32 # Don't stop until 992 files have been written the least-used volume<br>
option alu.stat-refresh.interval 60sec # Refresh the statistics used for decision-making every 60 seconds<br> end-volume<br> <br> volume unify2<br> type cluster/distribute <br> subvolumes remote2a remote2b<br>
option scheduler alu # use the ALU scheduler<br> option alu.limits.min-free-disk 5% # Don't create files one a volume with less than 5% free diskspace<br> option alu.limits.max-open-files 10000 # Don't create files on a volume with more than 10000 files open<br>
option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage<br> option alu.disk-usage.entry-threshold 3GB # Kick in if the discrepancy in disk-usage between volumes is more than 3GB<br> option alu.disk-usage.exit-threshold 60MB # Don't stop writing to the least-used volume until the discrepancy is 1988MB <br>
option alu.open-files-usage.entry-threshold 1024 # Kick in if the discrepancy in open files is 1024<br> option alu.open-files-usage.exit-threshold 32 # Don't stop until 992 files have been written the least-used volume<br>
option alu.stat-refresh.interval 60sec # Refresh the statistics used for decision-making every 60 seconds<br> end-volume<br> <br> volume replicate<br> type cluster/afr<br> subvolumes unify1 unify2<br> end-volume<br>
<br> volume writebehind<br> type performance/write-behind<br> option window-size 1MB<br> subvolumes replicate<br> end-volume<br> <br> volume cache<br> type performance/io-cache<br> option cache-size 512MB<br>
subvolumes writebehind<br> end-volume<br><br clear="all"></p>