Hi,<br><br>I've tested replicate a little today and I've got this problems:<br><br>1. client-side replicate with 2 nodes, both acting as server and client<br>2. I start copying big file to gluster mount point<br>3. during file copy I stop gluster server on second node<br>
4. I got a pice of file on second node and whole file on first one<br>5. during or after the copy I start halted server on second node<br>6. I run ls -lh on second node in gluster mount point<br>7. logs are telling me to delete one of files and my second node keeps incomplited copy<br>
<br>I've checked if I got xattrs enabled with attr command:<br><br># attr -l <gluster_data_dir>/iso<br>Attribute "glusterfs.afr.entry-pending" has a 8 byte value for iso<br># attr -g glusterfs.afr.entry-pending <gluster_data_dir>/iso<br>
attr_get: No data available<br>Could not get "glusterfs.afr.entry-pending" for iso<br><br>With client side it seems that if im inside gluster mount point and stop server on second node, after starting it again I need to reenter this dir for self-heal to work, it does not seem to be needed with server-side.<br>
With server side afr (both node afr'ing each other), when I stop second node during big file copy and after while start it again if I run 'ls -lh' on gluster mount point I'm getting some self-heal race condition, looking at dstat output it seems that running 'ls -lh' on one node makes that node to self-heal other with it current file content, I forgot about the log files so I don't know what they say about it, I will retest it tomorrow.<br>
Using replicate slows down writing a lot, with client-side to 1-2MB/s (!) on 1Gb link, with server-side to 10-12MB/s.<br><br>==============================================<br>client-side replicate, server.vol:<br><br>volume brick-data<br>
type storage/posix<br> option directory /var/glusterfs/data/<br>end-volume<br><br>volume locks<br> type features/locks<br> subvolumes brick-data<br>end-volume<br><br>volume client<br> type performance/io-threads<br>
option thread-count 4<br> subvolumes locks<br>end-volume<br><br>volume server<br> type protocol/server<br> option transport-type tcp<br> subvolumes client<br> option auth.login.client.allow gluster<br> option auth.login.gluster.password pass<br>
end-volume<br><br>========================================================================<br>client-side replicate, client vol:<br>volume gluster_134 <br> type protocol/client <br>
option transport-type tcp <br> option remote-host 172.16.110.134 <br> option remote-subvolume client<br> option username gluster<br> option password pass<br>end-volume<br>
<br>volume gluster_135<br> type protocol/client<br> option transport-type tcp<br> option remote-host 172.16.110.135<br> option remote-subvolume client<br> option username gluster<br> option password pass<br>
end-volume<br><br>volume afr<br> type cluster/replicate<br> subvolumes gluster_134 gluster_135<br>end-volume<br><br>volume readahead<br> type performance/read-ahead<br> option page-size 1MB<br> option page-count 2<br>
subvolumes afr<br>end-volume<br><br>volume iocache<br> type performance/io-cache<br> option cache-size 64MB<br> option page-size 256KB<br> option page-count 2<br> subvolumes readahead<br>end-volume<br><br>
volume writeback<br> type performance/write-behind<br> option aggregate-size 1MB<br> option window-size 2MB<br> option flush-behind off<br> subvolumes iocache<br>end-volume<br><br>volume iothreads<br> type performance/io-threads<br>
option thread-count 4<br> subvolumes writeback<br>end-volume<br><br>=============================================================<br>server-side replicate, server.vol:<br><br>volume brick<br> type storage/posix<br>
option directory /var/glusterfs/data/<br>end-volume<br><br>volume client_afr<br> type features/locks<br> subvolumes brick<br>end-volume<br><br>volume brick_134<br> type protocol/client<br> option transport-type tcp/client<br>
option remote-host 172.16.110.134<br> option remote-subvolume client_afr<br> option username gluster<br> option password Tj72pAz44<br>end-volume<br><br>volume afr<br> type cluster/replicate<br> subvolumes client_afr brick_134<br>
end-volume<br><br>volume client<br> type performance/io-threads<br> option thread-count 4<br> subvolumes afr<br>end-volume<br><br>volume server<br> type protocol/server<br> option transport-type tcp<br> subvolumes client_afr client<br>
option auth.login.client.allow gluster<br> option auth.login.client_afr.allow gluster<br> option auth.login.gluster.password pass<br>end-volume<br><br>================================================================<br>
server-side replicate, client.vol:<br>volume gluster_134 <br> type protocol/client <br> option transport-type tcp <br>
option remote-host 172.16.110.134 <br> option remote-subvolume client<br> option username gluster<br> option password Tj72pAz44<br>end-volume<br><br>volume gluster_135<br> type protocol/client<br>
option transport-type tcp<br> option remote-host 172.16.110.135<br> option remote-subvolume client<br> option username gluster<br> option password Tj72pAz44<br>end-volume<br><br>volume ha<br> type cluster/ha<br>
subvolumes gluster_135 gluster_134<br>end-volume<br><br>volume readahead<br> type performance/read-ahead<br> option page-size 1MB<br> option page-count 2<br> subvolumes ha<br>end-volume<br><br>volume iocache<br>
type performance/io-cache<br> option cache-size 64MB<br> option page-size 256KB<br> option page-count 2<br> subvolumes readahead<br>end-volume<br><br>volume writeback<br> type performance/write-behind<br>
option aggregate-size 1MB<br> option window-size 2MB<br> option flush-behind off<br> subvolumes iocache<br>end-volume<br><br>volume iothreads<br> type performance/io-threads<br> option thread-count 4<br>
subvolumes writeback<br>end-volume<br><br>