<font size=2 face="sans-serif">Can someone help me find doc. or explain
what the below report means? I'm concerned that the localhost has
1127 failures.</font><br><br><font size=2 face="sans-serif">gluster> volume rebalance devstatic
status</font><br><font size=2 face="sans-serif">
Node Rebalanced-files size
scanned failures
status run time in secs</font><br><font size=2 face="sans-serif">
---------
----------- ----------- -----------
----------- ------------ --------------</font><br><font size=2 face="sans-serif">
localhost
0
0Bytes 3688880
1127 completed 15788.00</font><br><font size=2 face="sans-serif">
omhq1832
1186 82.9MB
3688880
0 completed 17676.00</font><br><font size=2 face="sans-serif">
omdx1448
0
0Bytes 3688880
0 completed
16930.00</font><br><font size=2 face="sans-serif">
omdx14f0
0
0Bytes 3688879
0 completed
16931.00</font><br><font size=2 face="sans-serif">volume rebalance: devstatic: success:</font><br><font size=2 face="sans-serif"><br><br>Khoi<br><br><br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">gluster-users-request@gluster.org</font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gluster-users@gluster.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">09/18/2013 06:59 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Gluster-users
Digest, Vol 65, Issue 18</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gluster-users-bounces@gluster.org</font><br><hr noshade><br><br><br><tt><font size=2>Send Gluster-users mailing list submissions to<br>
gluster-users@gluster.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br>
</font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>or, via email, send a message with subject or body 'help' to<br>
gluster-users-request@gluster.org<br><br>You can reach the person managing the list at<br>
gluster-users-owner@gluster.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Gluster-users digest..."<br><br><br>Today's Topics:<br><br> 1. Re: gluster volume top issue (Nux!)<br> 2. Re: gluster volume top issue (Shishir Gowda)<br> 3. Re: gluster volume top issue (Nux!)<br> 4. Strange Sync Problem - FS not matching GlusterFS<br> (Cristiano Bianchi)<br> 5. Re: GlusterFS extended attributes,
"system" namespace<br> (Iain Buchanan)<br> 6. Re: Gluster 3.4 QEMU and Permission Denied Errors<br> (Andrew Niemantsverdriet)<br> 7. Re: remove-brick question (james.bellinger@icecube.wisc.edu)<br> 8. Re: [rhos-list] [gluster-swift] Gluster UFO 3.4 swift Multi<br> tenant question (Luis Pabon)<br> 9. Re: Gluster 3.4 QEMU and Permission Denied Errors (Asias He)<br> 10. Re: gluster volume top issue (Shishir Gowda)<br> 11. Re: Gluster samba vfs read performance slow (kane)<br> 12. Re: Gluster samba vfs read performance slow (Anand Avati)<br> 13. Re: Gluster samba vfs read performance slow (kane)<br> 14. Mounting same replica-volume on multiple clients.
????<br> (Bobby Jacob)<br> 15. Re: Gluster samba vfs read performance slow (Anand Avati)<br> 16. Re: Mounting same replica-volume on multiple
clients.
????<br> (Daniel M?ller)<br> 17. Re: Gluster samba vfs read performance slow (kane)<br> 18. Re: Mounting same replica-volume on multiple clients.
????<br> (Bobby Jacob)<br> 19. Re: Gluster samba vfs read performance slow (Anand Avati)<br> 20. Re: Mounting same replica-volume on multiple
clients.
????<br> (Daniel M?ller)<br> 21. Re: Gluster samba vfs read performance slow (kane)<br> 22. Re: [Gluster-devel] glusterfs-3.4.1qa2 released (Luk?? Bezdi?ka)<br> 23. Re: Cant see files after network failure (Dragon)<br> 24. Re: gluster volume top issue (Nux!)<br> 25. Secure Setup / Separate GlusterFS / Encryption (Michael.OBrien)<br> 26. Re: Cant see files after network failure (Krishnan Parthasarathi)<br> 27. Re: Cant see files after network failure (Dragon)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Tue, 17 Sep 2013 13:01:39 +0100<br>From: Nux! <nux@li.nux.ro><br>To: Gluster Users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] gluster volume top issue<br>Message-ID: <3c99760fce535b37e28371a8221e670f@li.nux.ro><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br>On 16.09.2013 11:22, Nux! wrote:<br>> Hello,<br>> <br>> I'm trying to find out the most accessed (read from and/or written<br>> to) file in a volume and "gluster volume top" does not seem
to be<br>> helping me at all.<br>> For example the following would only output the list of bricks:<br>> gluster volume top xenvms write nfs brick localhost:/bricks/xenvms
or<br>> gluster volume top xenvms write<br>> <br>> "gluster volume top xenvms open" says 0 fds opened for all
bricks.<br>> <br>> I'm sure there should be some activity on this volume as I have a<br>> Xenserver reading & writing at 200-300 Mbps to it over NFS.<br>> <br>> Any pointers?<br><br>Anyone?<br><br>-- <br>Sent from the Delta quadrant using Borg technology!<br><br>Nux!<br></font></tt><a href=www.nux.ro><tt><font size=2>www.nux.ro</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 2<br>Date: Tue, 17 Sep 2013 08:13:01 -0400 (EDT)<br>From: Shishir Gowda <sgowda@redhat.com><br>To: Nux! <nux@li.nux.ro><br>Cc: Gluster Users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] gluster volume top issue<br>Message-ID:<br>
<614027677.14062684.1379419981657.JavaMail.root@redhat.com><br>Content-Type: text/plain; charset=utf-8<br><br>Hi Nux,<br><br>Is only open count being shown as "0", or all stats being shown
as "0"?<br><br>With regards,<br>Shishir<br>----- Original Message -----<br>From: "Nux!" <nux@li.nux.ro><br>To: "Gluster Users" <gluster-users@gluster.org><br>Sent: Tuesday, September 17, 2013 5:31:39 PM<br>Subject: Re: [Gluster-users] gluster volume top issue<br><br>On 16.09.2013 11:22, Nux! wrote:<br>> Hello,<br>> <br>> I'm trying to find out the most accessed (read from and/or written<br>> to) file in a volume and "gluster volume top" does not seem
to be<br>> helping me at all.<br>> For example the following would only output the list of bricks:<br>> gluster volume top xenvms write nfs brick localhost:/bricks/xenvms
or<br>> gluster volume top xenvms write<br>> <br>> "gluster volume top xenvms open" says 0 fds opened for all
bricks.<br>> <br>> I'm sure there should be some activity on this volume as I have a<br>> Xenserver reading & writing at 200-300 Mbps to it over NFS.<br>> <br>> Any pointers?<br><br>Anyone?<br><br>-- <br>Sent from the Delta quadrant using Borg technology!<br><br>Nux!<br></font></tt><a href=www.nux.ro><tt><font size=2>www.nux.ro</font></tt></a><tt><font size=2><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 3<br>Date: Tue, 17 Sep 2013 14:16:05 +0100<br>From: Nux! <nux@li.nux.ro><br>To: Shishir Gowda <sgowda@redhat.com><br>Cc: Gluster Users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] gluster volume top issue<br>Message-ID: <10854f971aad0d256b7a8440f6d7b243@li.nux.ro><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br>On 17.09.2013 13:13, Shishir Gowda wrote:<br>> Hi Nux,<br>> <br>> Is only open count being shown as "0", or all stats being
shown as <br>> "0"?<br><br>Hi Shishir,<br><br>For all bricks I get:<br>Current open fds: 0, Max open fds: 0, Max openfd time: N/A<br><br>Lucian<br><br>-- <br>Sent from the Delta quadrant using Borg technology!<br><br>Nux!<br></font></tt><a href=www.nux.ro><tt><font size=2>www.nux.ro</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 4<br>Date: Tue, 17 Sep 2013 07:42:23 +0100<br>From: Cristiano Bianchi <c.bianchi@keepthinking.it><br>To: gluster-users@gluster.org<br>Subject: [Gluster-users] Strange Sync Problem - FS not matching<br>
GlusterFS<br>Message-ID:<br>
<CAC=_txqvQfrmJMdZwPL4X15xtBDvcKH1s=dtdQh1bUy5qQikSA@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Hi all,<br><br>we have a strange sync issue. Scenario:<br><br> - GlusterFS 3.2.5 with two nodes, N1 and N2, replicating over VLAN<br> - Both nodes share the /home folder, in both cases mounted on<br> /mnt/glusterfs/<br> - If I create file in N1 /mnt/glusterfs/user1/test.txt it shows
up in<br> /home/user1/test.txt - all normal<br> - It also shows up on N2 in /mnt/glusterfs/user1/test.txt - but
NOT (and<br> this is the odd part) in N2 /home/user1/<br> - If I do the same starting from N2, creating a file in N2<br> /mnt/glusterfs/user1/test.txt it all works: the file shows up in
N2<br> /home/user1, N1 /mnt/glusterfs/user1 and N1 /home/user1<br><br>My questions are (if they can be answered based on the info provided):<br><br> - What could have gone wrong and how to fix it?<br> - How do I re-sync the /home folder in N2 to match the content
of<br> glusterfs - which is correct in the virtual FS N2 /mnt/glusterfs<br><br>It seems that N2 has lost the wires between the glusterfs db and the 'real<br>world' of the filesystem.<br><br>With many thanks,<br>Cristiano<br><br>-- <br><br>Cristiano Bianchi<br>*Keepthinking*<br><br>43 Clerkenwell Road<br>London EC1M 5RS<br>tel +44 20 7490 5337<br>mobile +44 7939 041169 (UK)<br><br>c.bianchi@keepthinking.it<br></font></tt><a href=www.keepthinking.it><tt><font size=2>www.keepthinking.it</font></tt></a><tt><font size=2><br><br>---<br><br>Registration no. 04905582<br>VAT 831 1329 62<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/b0bbb260/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/b0bbb260/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 5<br>Date: Tue, 17 Sep 2013 14:36:56 +0100<br>From: Iain Buchanan <iainbuc@gmail.com><br>To: Venky Shankar <yknev.shankar@gmail.com><br>Cc: gluster-users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] GlusterFS extended attributes,
"system"<br>
namespace<br>Message-ID: <876CC0EB-2E5F-4827-B3ED-8F37E92BF108@gmail.com><br>Content-Type: text/plain; charset="windows-1252"<br><br>Thanks - sorry for the delay responding. I switched it to running
as super-user and that has fixed the problem. I'll have to investigate
mount-broker in future.<br><br>Iain<br><br><br>On 5 Sep 2013, at 17:30, Venky Shankar <yknev.shankar@gmail.com>
wrote:<br><br>> 'system' namespace is flipped to 'trusted' for geo-replication auxillary
mount. So, it should be left as 'system' in the source.<br>> <br>> I see that you're trying to connect to the remote slave as a non-super
user. For that, you'd need to access the slave via mount-broker, which
would require some modification in the glusterd volfile.<br>> <br>> Thanks,<br>> -venky<br>> <br>> <br>> On Thu, Sep 5, 2013 at 10:48 AM, Amar Tumballi <amarts@redhat.com>
wrote:<br>> On 08/28/2013 11:15 PM, Iain Buchanan wrote:<br>> Hi,<br>> <br>> I'm running GlusterFS 3.3.2 and I'm having trouble getting geo-replication
to work. I think it is a problem with extended attributes. I'll
using ssh with a normal user to perform the replication.<br>> <br>> On the server log in /var/log/glusterfs/geo-replication/VOLNAME/ssh?.log
I'm getting an error "ReceClient: call ?:?:? (xtime) failed on peer
with OSError". On the replication target I'm getting the same
error, but with a stack trace leading back to where it tries to set extended
attributes in the Python replication code. It appears to be trying
to get the attribute "system.glusterfs.xyz.xtime" at line 365
of /usr/lib/glusterfs/glusterfs/python/syncdaemon/resource.py: "Xattr.lgetxattr(path,
'.'.join([cls.GX_NSPACE, uuid, 'xtime')], 8))".<br>> I don't know anything about extended attributes, but I can't get anything
in the "system" namespace manually, even running as root - e.g.<br>> touch a<br>> getfattr -n system.test a<br>> <br>> The above returns "Operation not supported" rather than
"No such attribute". The "user" and "trusted"
namespace work fine - this is on ext3 with user_xattr set in the mount
options, and also on the server (ext4).<br>> Yes, 'system' is not allowed to be used by a process.<br>> <br>> On the server side I can see files have things set in the "trusted"
namespace (e.g. with "getfattr -m - filename").<br>> <br>> Should the setting of GX_NSPACE set the namespace to be "system"
for non-root or should it always be "trusted"? (line 248 in resource.py)
If I force it to be "trusted" it seems to get further (I
get occasional "Operation not permitted" lines, but I think this
is file permission related).<br>> Looks like a bug. Please change 'system' to 'user' in resource.py
file, and see if it works.<br>> <br>> Regards,<br>> Amar<br>> <br>> Iain<br>> <br>> <br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> <br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> <br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/53c6b075/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/53c6b075/attachment-0001.html</font></tt></a><tt><font size=2>><br>-------------- next part --------------<br>A non-text attachment was scrubbed...<br>Name: signature.asc<br>Type: application/pgp-signature<br>Size: 495 bytes<br>Desc: Message signed with OpenPGP using GPGMail<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/53c6b075/attachment-0001.sig"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/53c6b075/attachment-0001.sig</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 6<br>Date: Tue, 17 Sep 2013 07:59:16 -0600<br>From: Andrew Niemantsverdriet <andrew@rocky.edu><br>To: Samuli Heinonen <samppah@neutraali.net><br>Cc: gluster-users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Gluster 3.4 QEMU and Permission Denied<br>
Errors<br>Message-ID:<br>
<CAGn8edbMouCUqi4Rpe=g5-_wPsksU3es8Mf-noB=sRYZxA7t-A@mail.gmail.com><br>Content-Type: text/plain; charset=ISO-8859-1<br><br>Right now I am just using virsh to start the machines, I have also<br>tried using Virtual Machine Manager to start them.<br><br>I have enabled Gluster mounting from insecure ports, forgot to mention<br>that in my first email. It looks like the disk mounts as it starts to<br>boot but nothing can be written to the disk as it just hangs in an<br>infinite loop.<br><br>Thanks,<br> _<br>/-\ ndrew<br><br>On Tue, Sep 17, 2013 at 1:05 AM, Samuli Heinonen <samppah@neutraali.net>
wrote:<br>> Hello Andrew,<br>><br>> How are you booting/managing VM's? Which user you use to launch them?<br>><br>> Have you enabled Gluster mounting from insecure ports? It needs two
changes.<br>> You have to edit glusterd.vol (in /etc/glusterfs directory) and add
line<br>> "option rpc-auth-allow-insecure on". Also you have to set
volume option<br>> server.allow-insecure on (ie. gluster volume set volname<br>> server.allow-insecure on). Restart of glusterd and stop and start
of the<br>> volume is required for these changes to take effect.<br>><br>> 16.9.2013 21:38, Andrew Niemantsverdriet kirjoitti:<br>><br>>> Hey List,<br>>><br>>> I'm trying to test out using Gluster 3.4 for virtual machine disks.
My<br>>> enviroment consists of two Fedora 19 hosts with gluster and qemu/kvm<br>>> installed.<br>>><br>>> I have a single volume on gluster called vmdata that contains
my qcow2<br>>> formated image created like this:<br>>><br>>> qemu-img create -f qcow2 gluster://localhost/vmdata/test1.qcow
8G<br>>><br>>> I'm able to boot my created virtual machine but in the logs I
see this:<br>>><br>>> [2013-09-16 15:16:04.471205] E [addr.c:152:gf_auth] 0-auth/addr:<br>>> client is bound to port 46021 which is not privileged<br>>> [2013-09-16 15:16:04.471277] I<br>>> [server-handshake.c:567:server_setvolume] 0-vmdata-server: accepted<br>>> client from<br>>> gluster1.local-1061-2013/09/16-15:16:04:441166-vmdata-client-1-0<br>>> (version: 3.4.0)[2013-09-16 15:16:04.488000] I<br>>> [server-rpc-fops.c:1572:server_open_cbk] 0-vmdata-server: 18:
OPEN<br>>> /test1.qcow (6b63a78b-7d5c-4195-a172-5bb6ed1e7dac) ==> (Permission<br>>> denied)<br>>><br>>> I have turned off SELinux to be sure that isn't in the way. When
I<br>>> look at the permissions on the file using ls -l I see the file
is set<br>>> to 600, this doesn't seem right. I tried manually changing the<br>>> permission to 755 as a test and as soon as the machine booted
it was<br>>> changed back to 600.<br>>><br>>> Any hints as to what is going on and how to get the disk functioning?<br>>> The machine will boot but as soon as anything is written to disk
it<br>>> will hang forever.<br>>><br>>> Thanks,<br>>><br>><br><br><br><br>-- <br> _<br>/-\ ndrew Niemantsverdriet<br>Linux System Administrator<br>Academic Computing<br>(406) 238-7360<br>Rocky Mountain College<br>1511 Poly Dr.<br>Billings MT, 59102<br><br><br>------------------------------<br><br>Message: 7<br>Date: Tue, 17 Sep 2013 09:47:14 -0500<br>From: james.bellinger@icecube.wisc.edu<br>To: "Ravishankar N" <ravishankar@redhat.com><br>Cc: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] remove-brick question<br>Message-ID:<br>
<e47a5be41156fa52f47e44af55911c81.squirrel@webmail.icecube.wisc.edu><br>Content-Type: text/plain;charset=iso-8859-1<br><br>Thanks for your replies.<br><br>The vols seem to match the bricks ok.<br><br>FWIW, gfs-node01:/sda is the first brick; perhaps it is getting the lion's<br>share of the pointers?<br><br>The results of a log search and seem even more confusing. sdb drained<br>rather than sda, but an error in rebalancing shows up in sdd. I include<br>excerpts from scratch-rebalance, ls -l and getfattr, and bricks/sda<br><br>Does any of this suggest anything?<br><br>One failure is an apparent duplicate. This seems to refer to the
relevant<br>brick, and the date is correct.<br>[2013-09-15 19:10:07.620881] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-0: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118906_Qtot1500.h5.out<br>(00000000-0000-0000-0000-000000000000)<br><br>On the array that actually drained (mostly):<br>[2013-09-15 19:10:19.483040] W [client3_1-fops.c:647:client3_1_unlink_cbk]<br>0-scratch-client-12: remote operation failed: No such file or directory<br>[2013-09-15 19:10:19.483122] W [client3_1-fops.c:647:client3_1_unlink_cbk]<br>0-scratch-client-12: remote operation failed: No such file or directory<br>[2013-09-15 19:10:19.494585] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-12: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118429_Qtot1500.h5<br>(00000000-0000-0000-0000-000000000000)<br>[2013-09-15 19:10:19.494701] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-12: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118429_Qtot1500.h5<br>(00000000-0000-0000-0000-000000000000)<br><br>An example failure where I can trace the files is an apparent duplicate:<br>gfs-node01 # grep -A2 -B2 Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>scratch-rebalance.log<br>[2013-09-15 19:10:30.164409] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-3: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2a_IC79_data_Run00117874_Qtot1500.h5.out<br>(00000000-0000-0000-0000-000000000000)<br>[2013-09-15 19:10:30.164473] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-3: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2a_IC79_data_Run00117874_Qtot1500.h5.out<br>(00000000-0000-0000-0000-000000000000)<br>[2013-09-15 19:10:30.176606] I<br>[dht-common.c:956:dht_lookup_everywhere_cbk] 0-scratch-dht: deleting stale<br>linkfile<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>on scratch-client-2<br>[2013-09-15 19:10:30.176717] I<br>[dht-common.c:956:dht_lookup_everywhere_cbk] 0-scratch-dht: deleting stale<br>linkfile<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>on scratch-client-2<br>[2013-09-15 19:10:30.176856] I<br>[dht-common.c:956:dht_lookup_everywhere_cbk] 0-scratch-dht: deleting stale<br>linkfile<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>on scratch-client-2<br>[2013-09-15 19:10:30.177232] W [client3_1-fops.c:647:client3_1_unlink_cbk]<br>0-scratch-client-2: remote operation failed: No such file or directory<br>[2013-09-15 19:10:30.177303] W [client3_1-fops.c:647:client3_1_unlink_cbk]<br>0-scratch-client-2: remote operation failed: No such file or directory<br>[2013-09-15 19:10:30.178101] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-3: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>(00000000-0000-0000-0000-000000000000)<br>[2013-09-15 19:10:30.178150] W [client3_1-fops.c:258:client3_1_mknod_cbk]<br>0-scratch-client-3: remote operation failed: File exists. Path:<br>/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>(00000000-0000-0000-0000-000000000000)<br>[2013-09-15 19:10:30.192605] W<br>[client3_1-fops.c:2566:client3_1_opendir_cbk] 0-scratch-client-7: remote<br>operation failed: No such file or directory. Path:<br>/nwhitehorn/vetoblast/data (00000000-0000-0000-0000-000000000000)<br>[2013-09-15 19:10:30.192830] W<br>[client3_1-fops.c:2566:client3_1_opendir_cbk] 0-scratch-client-7: remote<br>operation failed: No such file or directory. Path:<br>/nwhitehorn/vetoblast/data (00000000-0000-0000-0000-000000000000)<br><br>gfs-node01 # ls -l<br>/sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>---------T 2 34037 40978 0 Sep 15 14:10<br>/sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>gfs-node01 # ssh i3admin@gfs-node06 sudo ls -l<br>/sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>-rw-r--r-- 2 34037 40978 715359 May 1 22:28<br>/sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br><br>gfs-node01 # getfattr -d -m . -e hex <br>/sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br># file:<br>sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>trusted.gfid=0x11fb3ffd87be4ce3a88576466279819f<br>trusted.glusterfs.dht.linkto=0x736372617463682d636c69656e742d313200<br>gfs-node01 # ssh i3admin@gfs-node06 sudo getfattr -d -m . -e
hex <br>/sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br># file:<br>sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5<br>trusted.gfid=0x11fb3ffd87be4ce3a88576466279819f<br><br>Further.<br># getfattr -d -m . -e hex /sdd<br># file: sdd<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x0000000100000000bffffffdd5555551<br>trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc<br><br>gfs-node01 # getfattr -d -m . -e hex /sdc<br># file: sdc<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x0000000100000000aaaaaaa8bffffffc<br>trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc<br><br>gfs-node01 # getfattr -d -m . -e hex /sdb<br># file: sdb<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x00000001000000000000000000000000<br>trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc<br><br>gfs-node01 # getfattr -d -m . -e hex /sda<br># file: sda<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x0000000100000000555555546aaaaaa8<br>trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc<br><br>gfs-node01 # ssh i3admin@gfs-node04 sudo getfattr -d -m . -e hex /sdb<br># file: sdb<br>trusted.gfid=0x00000000000000000000000000000001<br>trusted.glusterfs.dht=0x00000001000000002aaaaaaa3ffffffe<br>trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc<br><br><br><br>bricks/sda etc logs have a rather monotonous<br>[2013-09-16 22:23:01.723146] I [server-handshake.c:571:server_setvolume]<br>0-scratch-server: accepted client from<br>node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0 (version:<br>3.3.2)<br>[2013-09-16 22:23:01.769154] I [server.c:703:server_rpc_notify]<br>0-scratch-server: disconnecting connectionfrom<br>node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0<br>[2013-09-16 22:23:01.769211] I<br>[server-helpers.c:741:server_connection_put] 0-scratch-server: Shutting<br>down connection<br>node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0<br>[2013-09-16 22:23:01.769253] I<br>[server-helpers.c:629:server_connection_destroy] 0-scratch-server:<br>destroyed connection of<br>node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0<br><br>> On 09/17/2013 03:26 AM, james.bellinger@icecube.wisc.edu wrote:<br>>> I inherited a system with a wide mix of array sizes (no replication)
in<br>>> 3.2.2, and wanted to drain data from a failing array.<br>>><br>>> I upgraded to 3.3.2, and began a<br>>> gluster volume remove-brick scratch "gfs-node01:/sda"
start<br>>><br>>> After some time I got this:<br>>> gluster volume remove-brick scratch "gfs-node01:/sda"
status<br>>> Node Rebalanced-files size
scanned failures<br>>> status<br>>> --------- ----------- -----------
----------- -----------<br>>> ------------<br>>> localhost 0
0Bytes
0 0<br>>> not started<br>>> gfs-node06 0
0Bytes
0 0<br>>> not started<br>>> gfs-node03 0
0Bytes
0 0<br>>> not started<br>>> gfs-node05 0
0Bytes
0 0<br>>> not started<br>>> gfs-node01 2257394624
2.8TB 5161640 208878<br>>> completed<br>>><br>>> Two things jump instantly to mind:<br>>> 1) The number of failures is rather large<br>> Can you see the rebalance logs (/var/log/scratch-rebalance.log) to<br>> figure out what the error messages are?<br>>> 2) A _different_ disk seems to have been _partially_ drained.<br>>> /dev/sda 2.8T
2.7T 12G 100% /sda<br>>> /dev/sdb 2.8T
769G 2.0T 28% /sdb<br>>> /dev/sdc 2.8T
2.1T 698G 75% /sdc<br>>> /dev/sdd 2.8T
2.2T 589G 79% /sdd</font></tt><br><tt><font size=2>>><br>>><br>> I know this sounds silly, but just to be sure, is /dev/sda actually<br>> mounted on "gfs-node01:sda"?<br>> If yes,the files that _were_ successfully rebalanced should have been<br>> moved from gfs-node01:sda to one of the other bricks. Is that the
case?<br>><br>>> When I mount the system it is read-only (another problem I want
to fix<br>> Again, the mount logs could shed some information ..<br>> (btw a successful rebalance start/status sequence should be followed
by<br>> the rebalance 'commit' command to ensure the volume information gets<br>> updated)<br>><br>>> ASAP) so I'm pretty sure the failures aren't due to users changing
the<br>>> system underneath me.<br>>><br>>> Thanks for any pointers.<br>>><br>>> James Bellinger<br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br><br><br><br><br>------------------------------<br><br>Message: 8<br>Date: Tue, 17 Sep 2013 13:52:12 -0400<br>From: Luis Pabon <lpabon@redhat.com><br>To: Paul Robert Marino <prmarino1@gmail.com><br>Cc: "rhos-list@redhat.com" <rhos-list@redhat.com>,
Ramana
Raja<br>
<rraja@redhat.com>, gluster-users@gluster.org,
Chetan Risbud<br>
<crisbud@redhat.com><br>Subject: Re: [Gluster-users] [rhos-list] [gluster-swift] Gluster UFO<br>
3.4 swift Multi tenant question<br>Message-ID: <523896CC.70105@redhat.com><br>Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br><br><br>On 09/17/2013 11:13 AM, Paul Robert Marino wrote:<br>> Luis<br>> well thats intresting because it was my impression that Gluster UFO<br>> 3.4 was based on the Grizzly version of Swift.<br>[LP] Sorry, the gluster-ufo RPM is Essex only.<br><br>> Also I was previously unaware of this new rpm which doesnt seem to
be<br>> in a repo any where.<br>[LP] gluster-swift project RPMs have been submitted to Fedora and are <br>currently being reviewed.<br><br>> also there is a line in this new howto that is extreamly unclear<br>><br>> "<br>> /usr/bin/gluster-swift-gen-builders test<br>> "<br>> in place of "test" what should go there is it the tenant
ID string,<br>> the tenant name, or just a generic volume you can name whatever you<br>> want?<br>> in other words how should the Gluster volumes be named?<br>[LP] We will clarify that in the quick start guide. Thank you for
<br>pointing it out. While we update the community site, please refer
to <br>the documentation available here </font></tt><a href=http://goo.gl/bQFI8o><tt><font size=2>http://goo.gl/bQFI8o</font></tt></a><tt><font size=2>for a usage guide.<br><br>As for the tool, the format is:<br>gluster-swift-gen-buildes [VOLUME] [VOLUME...]<br><br>Where VOLUME is the name of the GlusterFS volume to use for object <br>storage. For example<br>if the following two GlusterFS volumes, volume1 and volume2, need to be
<br>accessed over Swift,<br>then you can type the following:<br><br># gluster-swift-gen-builders volume1 volume2<br><br>For more information please read: </font></tt><a href=http://goo.gl/gd8LkW><tt><font size=2>http://goo.gl/gd8LkW</font></tt></a><tt><font size=2><br><br>Let us know if you have any more questions or comments.<br><br>- Luis<br>><br>><br>> On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon@redhat.com>
wrote:<br>>> First thing I can see is that you have Essex based gluster-ufo-*
which has<br>>> been replaced by the gluster-swift project. We are currently
in progress of<br>>> replacing the gluster-ufo-* with RPMs from the gluster-swift project
in<br>>> Fedora.<br>>><br>>> Please checkout the following quickstart guide which show how
to download<br>>> the Grizzly version of gluster-swift:<br>>> </font></tt><a href="https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md"><tt><font size=2>https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md</font></tt></a><tt><font size=2><br>>> .<br>>><br>>> For more information please visit: </font></tt><a href="https://launchpad.net/gluster-swift"><tt><font size=2>https://launchpad.net/gluster-swift</font></tt></a><tt><font size=2><br>>><br>>> - Luis<br>>><br>>><br>>> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:<br>>><br>>> Sorry for the delay on reporting the details. I got temporarily
pulled<br>>> off the project and dedicated to a different project which was<br>>> considered higher priority by my employer. I'm just getting back
to<br>>> doing my normal work today.<br>>><br>>> first here are the rpms I have installed<br>>> "<br>>> rpm -qa |grep -P -i '(gluster|swift)'<br>>> glusterfs-libs-3.4.0-8.el6.x86_64<br>>> glusterfs-server-3.4.0-8.el6.x86_64<br>>> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch<br>>> openstack-swift-proxy-1.8.0-2.el6.noarch<br>>> glusterfs-3.4.0-8.el6.x86_64<br>>> glusterfs-cli-3.4.0-8.el6.x86_64<br>>> glusterfs-geo-replication-3.4.0-8.el6.x86_64<br>>> glusterfs-api-3.4.0-8.el6.x86_64<br>>> openstack-swift-1.8.0-2.el6.noarch<br>>> openstack-swift-container-1.8.0-2.el6.noarch<br>>> openstack-swift-object-1.8.0-2.el6.noarch<br>>> glusterfs-fuse-3.4.0-8.el6.x86_64<br>>> glusterfs-rdma-3.4.0-8.el6.x86_64<br>>> openstack-swift-account-1.8.0-2.el6.noarch<br>>> glusterfs-ufo-3.4.0-8.el6.noarch<br>>> glusterfs-vim-3.2.7-1.el6.x86_64<br>>> python-swiftclient-1.4.0-1.el6.noarch<br>>><br>>> here are some key config files note I've changed the passwords
I'm<br>>> using and hostnames<br>>> "<br>>> cat /etc/swift/account-server.conf<br>>> [DEFAULT]<br>>> mount_check = true<br>>> bind_port = 6012<br>>> user = root<br>>> log_facility = LOG_LOCAL2<br>>> devices = /swift/tenants/<br>>><br>>> [pipeline:main]<br>>> pipeline = account-server<br>>><br>>> [app:account-server]<br>>> use = egg:gluster_swift_ufo#account<br>>> log_name = account-server<br>>> log_level = DEBUG<br>>> log_requests = true<br>>><br>>> [account-replicator]<br>>> vm_test_mode = yes<br>>><br>>> [account-auditor]<br>>><br>>> [account-reaper]<br>>><br>>> "<br>>><br>>> "<br>>> cat /etc/swift/container-server.conf<br>>> [DEFAULT]<br>>> devices = /swift/tenants/<br>>> mount_check = true<br>>> bind_port = 6011<br>>> user = root<br>>> log_facility = LOG_LOCAL2<br>>><br>>> [pipeline:main]<br>>> pipeline = container-server<br>>><br>>> [app:container-server]<br>>> use = egg:gluster_swift_ufo#container<br>>><br>>> [container-replicator]<br>>> vm_test_mode = yes<br>>><br>>> [container-updater]<br>>><br>>> [container-auditor]<br>>><br>>> [container-sync]<br>>> "<br>>><br>>> "<br>>> cat /etc/swift/object-server.conf<br>>> [DEFAULT]<br>>> mount_check = true<br>>> bind_port = 6010<br>>> user = root<br>>> log_facility = LOG_LOCAL2<br>>> devices = /swift/tenants/<br>>><br>>> [pipeline:main]<br>>> pipeline = object-server<br>>><br>>> [app:object-server]<br>>> use = egg:gluster_swift_ufo#object<br>>><br>>> [object-replicator]<br>>> vm_test_mode = yes<br>>><br>>> [object-updater]<br>>><br>>> [object-auditor]<br>>> "<br>>><br>>> "<br>>> cat /etc/swift/proxy-server.conf<br>>> [DEFAULT]<br>>> bind_port = 8080<br>>> user = root<br>>> log_facility = LOG_LOCAL1<br>>> log_name = swift<br>>> log_level = DEBUG<br>>> log_headers = True<br>>><br>>> [pipeline:main]<br>>> pipeline = healthcheck cache authtoken keystone proxy-server<br>>><br>>> [app:proxy-server]<br>>> use = egg:gluster_swift_ufo#proxy<br>>> allow_account_management = true<br>>> account_autocreate = true<br>>><br>>> [filter:tempauth]<br>>> use = egg:swift#tempauth<br>>> # Here you need to add users explicitly. See the OpenStack Swift
Deployment<br>>> # Guide for more information. The user and user64 directives take
the<br>>> # following form:<br>>> # user_<account>_<username> = <key>
[group] [group] [...] [storage_url]<br>>> # user64_<account_b64>_<username_b64>
= <key> [group] [group]<br>>> [...] [storage_url]<br>>> # Where you use user64 for accounts and/or usernames that include<br>>> underscores.<br>>> #<br>>> # NOTE (and WARNING): The account name must match the device name
specified<br>>> # when generating the account, container, and object build rings.<br>>> #<br>>> # E.g.<br>>> # user_ufo0_admin = abc123 .admin<br>>><br>>> [filter:healthcheck]<br>>> use = egg:swift#healthcheck<br>>><br>>> [filter:cache]<br>>> use = egg:swift#memcache<br>>><br>>><br>>> [filter:keystone]<br>>> use = egg:swift#keystoneauth<br>>> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory<br>>> operator_roles = Member,admin,swiftoperator<br>>><br>>><br>>> [filter:authtoken]<br>>> paste.filter_factory = keystone.middleware.auth_token:filter_factory<br>>> auth_host = keystone01.vip.my.net<br>>> auth_port = 35357<br>>> auth_protocol = http<br>>> admin_user = swift<br>>> admin_password = PASSWORD<br>>> admin_tenant_name = service<br>>> signing_dir = /var/cache/swift<br>>> service_port = 5000<br>>> service_host = keystone01.vip.my.net<br>>><br>>> [filter:swiftauth]<br>>> use = egg:keystone#swiftauth<br>>> auth_host = keystone01.vip.my.net<br>>> auth_port = 35357<br>>> auth_protocol = http<br>>> keystone_url = </font></tt><a href=https://keystone01.vip.my.net:5000/v2.0><tt><font size=2>https://keystone01.vip.my.net:5000/v2.0</font></tt></a><tt><font size=2><br>>> admin_user = swift<br>>> admin_password = PASSWORD<br>>> admin_tenant_name = service<br>>> signing_dir = /var/cache/swift<br>>> keystone_swift_operator_roles = Member,admin,swiftoperator<br>>> keystone_tenant_user_admin = true<br>>><br>>> [filter:catch_errors]<br>>> use = egg:swift#catch_errors<br>>> "<br>>><br>>> "<br>>> cat /etc/swift/swift.conf<br>>> [DEFAULT]<br>>><br>>><br>>> [swift-hash]<br>>> # random unique string that can never change (DO NOT LOSE)<br>>> swift_hash_path_suffix = gluster<br>>> #3d60c9458bb77abe<br>>><br>>><br>>> # The swift-constraints section sets the basic constraints on
data<br>>> # saved in the swift cluster.<br>>><br>>> [swift-constraints]<br>>><br>>> # max_file_size is the largest "normal" object that
can be saved in<br>>> # the cluster. This is also the limit on the size of each segment
of<br>>> # a "large" object when using the large object manifest
support.<br>>> # This value is set in bytes. Setting it to lower than 1MiB will
cause<br>>> # some tests to fail. It is STRONGLY recommended to leave this
value at<br>>> # the default (5 * 2**30 + 2).<br>>><br>>> # FIXME: Really? Gluster can handle a 2^64 sized file? And can
the fronting<br>>> # web service handle such a size? I think with UFO, we need to
keep with the<br>>> # default size from Swift and encourage users to research what
size their<br>>> web<br>>> # services infrastructure can handle.<br>>><br>>> max_file_size = 18446744073709551616<br>>><br>>><br>>> # max_meta_name_length is the max number of bytes in the utf8
encoding<br>>> # of the name portion of a metadata header.<br>>><br>>> #max_meta_name_length = 128<br>>><br>>><br>>> # max_meta_value_length is the max number of bytes in the utf8
encoding<br>>> # of a metadata value<br>>><br>>> #max_meta_value_length = 256<br>>><br>>><br>>> # max_meta_count is the max number of metadata keys that can be
stored<br>>> # on a single account, container, or object<br>>><br>>> #max_meta_count = 90<br>>><br>>><br>>> # max_meta_overall_size is the max number of bytes in the utf8
encoding<br>>> # of the metadata (keys + values)<br>>><br>>> #max_meta_overall_size = 4096<br>>><br>>><br>>> # max_object_name_length is the max number of bytes in the utf8
encoding of<br>>> an<br>>> # object name: Gluster FS can handle much longer file names, but
the length<br>>> # between the slashes of the URL is handled below. Remember that
most web<br>>> # clients can't handle anything greater than 2048, and those that
do are<br>>> # rather clumsy.<br>>><br>>> max_object_name_length = 2048<br>>><br>>> # max_object_name_component_length (GlusterFS) is the max number
of bytes in<br>>> # the utf8 encoding of an object name component (the part between
the<br>>> # slashes); this is a limit imposed by the underlying file system
(for XFS<br>>> it<br>>> # is 255 bytes).<br>>><br>>> max_object_name_component_length = 255<br>>><br>>> # container_listing_limit is the default (and max) number of items<br>>> # returned for a container listing request<br>>><br>>> #container_listing_limit = 10000<br>>><br>>><br>>> # account_listing_limit is the default (and max) number of items
returned<br>>> # for an account listing request<br>>><br>>> #account_listing_limit = 10000<br>>><br>>><br>>> # max_account_name_length is the max number of bytes in the utf8
encoding of<br>>> # an account name: Gluster FS Filename limit (XFS limit?), must
be the same<br>>> # size as max_object_name_component_length above.<br>>><br>>> max_account_name_length = 255<br>>><br>>><br>>> # max_container_name_length is the max number of bytes in the
utf8 encoding<br>>> # of a container name: Gluster FS Filename limit (XFS limit?),
must be the<br>>> same<br>>> # size as max_object_name_component_length above.<br>>><br>>> max_container_name_length = 255<br>>><br>>> "<br>>><br>>><br>>> The volumes<br>>> "<br>>> gluster volume list<br>>> cindervol<br>>> unified-storage-vol<br>>> a07d2f39117c4e5abdeba722cf245828<br>>> bd74a005f08541b9989e392a689be2fc<br>>> f6da0a8151ff43b7be10d961a20c94d6<br>>> "<br>>><br>>> if I run the command<br>>> "<br>>> gluster-swift-gen-builders unified-storage-vol<br>>> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc<br>>> f6da0a8151ff43b7be10d961a20c94d6<br>>> "<br>>><br>>> because of a change in the script in this version as compaired
to the<br>>> version I got from<br>>> </font></tt><a href=http://repos.fedorapeople.org/repos/kkeithle/glusterfs/><tt><font size=2>http://repos.fedorapeople.org/repos/kkeithle/glusterfs/</font></tt></a><tt><font size=2>the<br>>> gluster-swift-gen-builders script only takes the first option
and<br>>> ignores the rest.<br>>><br>>> other than the location of the config files none of the changes
Ive<br>>> made are functionally different than the ones mentioned in<br>>> </font></tt><a href="http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/"><tt><font size=2>http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/</font></tt></a><tt><font size=2><br>>><br>>> The result is that the first volume named "unified-storage-vol"
winds<br>>> up being used for every thing regardless of the tenant, and users
and<br>>> see and manage each others objects regardless of what tenant they
are<br>>> members of.<br>>> through the swift command or via horizon.<br>>><br>>> In a way this is a good thing for me it simplifies thing significantly<br>>> and would be fine if it just created a directory for each tenant
and<br>>> only allow the user to access the individual directories, not
the<br>>> whole gluster volume.<br>>> by the way seeing every thing includes the service tenants data
so<br>>> unprivileged users can delete glance images without being a member
of<br>>> the service group.<br>>><br>>><br>>><br>>><br>>> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1@gmail.com><br>>> wrote:<br>>><br>>> Well I'll give you the full details in the morning but simply
I used the<br>>> stock cluster ring builder script that came with the 3.4 rpms
and the old<br>>> version from 3.3 took the list of volumes and would add all of
them the<br>>> version with 3.4 only takes the first one.<br>>><br>>> Well I ran the script expecting the same behavior but instead
they all used<br>>> the first volume in the list.<br>>><br>>> Now I knew from the docs I read that the per tenant directories
in a single<br>>> volume were one possible plan for 3.4 to deal with the scalding
issue with a<br>>> large number of tenants, so when I saw the difference in the script
and that<br>>> it worked I just assumed that this was done and I missed something.<br>>><br>>><br>>><br>>> -- Sent from my HP Pre3<br>>><br>>> ________________________________<br>>> On Sep 2, 2013 20:55, Ramana Raja <rraja@redhat.com> wrote:<br>>><br>>> Hi Paul,<br>>><br>>> Currently, gluster-swift doesn't support the feature of multiple<br>>> accounts/tenants accessing the same volume. Each tenant still
needs his own<br>>> gluster volume. So I'm wondering how you were able to observe
the reported<br>>> behaviour.<br>>><br>>> How did you prepare the ringfiles for the different tenants, which
use the<br>>> same gluster volume? Did you change the configuration of the servers?
Also,<br>>> how did you access the files that you mention? It'd be helpful
if you could<br>>> share the commands you used to perform these actions.<br>>><br>>> Thanks,<br>>><br>>> Ram<br>>><br>>><br>>> ----- Original Message -----<br>>> From: "Vijay Bellur" <vbellur@redhat.com><br>>> To: "Paul Robert Marino" <prmarino1@gmail.com><br>>> Cc: rhos-list@redhat.com, "Luis Pabon" <lpabon@redhat.com>,
"Ramana Raja"<br>>> <rraja@redhat.com>, "Chetan Risbud" <crisbud@redhat.com><br>>> Sent: Monday, September 2, 2013 4:17:51 PM<br>>> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question<br>>><br>>> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:<br>>><br>>> I have Gluster UFO installed as a back end for swift from here<br>>> </font></tt><a href="http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/"><tt><font size=2>http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/</font></tt></a><tt><font size=2><br>>> with RDO 3<br>>><br>>> Its working well except for one thing. All of the tenants are
seeing<br>>> one Gluster volume which is some what nice, especially when compared<br>>> to the old 3.3 behavior of creating one volume per tenant named
after<br>>> the tenant ID number.<br>>><br>>> The problem is I expected to see is sub directory created under
the<br>>> volume root for each tenant but instead what in seeing is that
all of<br>>> the tenants can see the root of the Gluster volume. The result
is that<br>>> all of the tenants can access each others files and even delete
them.<br>>> even scarier is that the tennants can see and delete each others<br>>> glance images and snapshots.<br>>><br>>> Can any one suggest options to look at or documents to read to
try to<br>>> figure out how to modify the behavior?<br>>><br>>> Adding gluster swift developers who might be able to help.<br>>><br>>> -Vijay<br>>><br>>><br><br><br><br>------------------------------<br><br>Message: 9<br>Date: Wed, 18 Sep 2013 07:39:56 +0800<br>From: Asias He <asias.hejun@gmail.com><br>To: Andrew Niemantsverdriet <andrew@rocky.edu><br>Cc: gluster-users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Gluster 3.4 QEMU and Permission Denied<br>
Errors<br>Message-ID:<br>
<CAFO3S41gLLdA_9YqBPLD_TE=2dLaPHP6cw+v08+jt6Px_xdA4w@mail.gmail.com><br>Content-Type: text/plain; charset=UTF-8<br><br>On Tue, Sep 17, 2013 at 9:59 PM, Andrew Niemantsverdriet<br><andrew@rocky.edu> wrote:<br>> Right now I am just using virsh to start the machines, I have also<br>> tried using Virtual Machine Manager to start them.<br><br>Try 'chmod qemu.qemu image_on_gluster.qcow2'. This along with the<br>'option rpc-auth-allow-insecure on' and 'gluster volume set <volname><br>server.allow-insecure on' should make libvirt+qemu+libgfapi work.<br><br>> I have enabled Gluster mounting from insecure ports, forgot to mention<br>> that in my first email. It looks like the disk mounts as it starts
to<br>> boot but nothing can be written to the disk as it just hangs in an<br>> infinite loop.<br>><br>> Thanks,<br>> _<br>> /-\ ndrew<br>><br>> On Tue, Sep 17, 2013 at 1:05 AM, Samuli Heinonen <samppah@neutraali.net>
wrote:<br>>> Hello Andrew,<br>>><br>>> How are you booting/managing VM's? Which user you use to launch
them?<br>>><br>>> Have you enabled Gluster mounting from insecure ports? It needs
two changes.<br>>> You have to edit glusterd.vol (in /etc/glusterfs directory) and
add line<br>>> "option rpc-auth-allow-insecure on". Also you have to
set volume option<br>>> server.allow-insecure on (ie. gluster volume set volname<br>>> server.allow-insecure on). Restart of glusterd and stop and start
of the<br>>> volume is required for these changes to take effect.<br>>><br>>> 16.9.2013 21:38, Andrew Niemantsverdriet kirjoitti:<br>>><br>>>> Hey List,<br>>>><br>>>> I'm trying to test out using Gluster 3.4 for virtual machine
disks. My<br>>>> enviroment consists of two Fedora 19 hosts with gluster and
qemu/kvm<br>>>> installed.<br>>>><br>>>> I have a single volume on gluster called vmdata that contains
my qcow2<br>>>> formated image created like this:<br>>>><br>>>> qemu-img create -f qcow2 gluster://localhost/vmdata/test1.qcow
8G<br>>>><br>>>> I'm able to boot my created virtual machine but in the logs
I see this:<br>>>><br>>>> [2013-09-16 15:16:04.471205] E [addr.c:152:gf_auth] 0-auth/addr:<br>>>> client is bound to port 46021 which is not privileged<br>>>> [2013-09-16 15:16:04.471277] I<br>>>> [server-handshake.c:567:server_setvolume] 0-vmdata-server:
accepted<br>>>> client from<br>>>> gluster1.local-1061-2013/09/16-15:16:04:441166-vmdata-client-1-0<br>>>> (version: 3.4.0)[2013-09-16 15:16:04.488000] I<br>>>> [server-rpc-fops.c:1572:server_open_cbk] 0-vmdata-server:
18: OPEN<br>>>> /test1.qcow (6b63a78b-7d5c-4195-a172-5bb6ed1e7dac) ==>
(Permission<br>>>> denied)<br>>>><br>>>> I have turned off SELinux to be sure that isn't in the way.
When I<br>>>> look at the permissions on the file using ls -l I see the
file is set<br>>>> to 600, this doesn't seem right. I tried manually changing
the<br>>>> permission to 755 as a test and as soon as the machine booted
it was<br>>>> changed back to 600.<br>>>><br>>>> Any hints as to what is going on and how to get the disk functioning?<br>>>> The machine will boot but as soon as anything is written to
disk it<br>>>> will hang forever.<br>>>><br>>>> Thanks,<br>>>><br>>><br>><br>><br>><br>> --<br>> _<br>> /-\ ndrew Niemantsverdriet</font></tt><br><tt><font size=2>> Linux System Administrator<br>> Academic Computing<br>> (406) 238-7360<br>> Rocky Mountain College<br>> 1511 Poly Dr.<br>> Billings MT, 59102<br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br><br>-- <br>Asias<br><br><br>------------------------------<br><br>Message: 10<br>Date: Tue, 17 Sep 2013 23:39:38 -0400 (EDT)<br>From: Shishir Gowda <sgowda@redhat.com><br>To: Nux! <nux@li.nux.ro><br>Cc: Gluster Users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] gluster volume top issue<br>Message-ID:<br>
<975294514.14460863.1379475578881.JavaMail.root@redhat.com><br>Content-Type: text/plain; charset=utf-8<br><br>Hi Nux,<br><br>I am trying to see if the issue of "0" open fd is based on the
work-load, or a bug.<br><br>Could you check top command output of "read/write" operation
too?<br><br>With regards,<br>Shishir<br><br>----- Original Message -----<br>From: "Nux!" <nux@li.nux.ro><br>To: "Shishir Gowda" <sgowda@redhat.com><br>Cc: "Gluster Users" <gluster-users@gluster.org><br>Sent: Tuesday, September 17, 2013 6:46:05 PM<br>Subject: Re: [Gluster-users] gluster volume top issue<br><br>On 17.09.2013 13:13, Shishir Gowda wrote:<br>> Hi Nux,<br>> <br>> Is only open count being shown as "0", or all stats being
shown as <br>> "0"?<br><br>Hi Shishir,<br><br>For all bricks I get:<br>Current open fds: 0, Max open fds: 0, Max openfd time: N/A<br><br>Lucian<br><br>-- <br>Sent from the Delta quadrant using Borg technology!<br><br>Nux!<br></font></tt><a href=www.nux.ro><tt><font size=2>www.nux.ro</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 11<br>Date: Wed, 18 Sep 2013 12:02:11 +0800<br>From: kane <stef_9k@163.com><br>To: Vijay Bellur <vbellur@redhat.com>, Anand Avati <avati@redhat.com><br>Cc: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID: <5F332BE6-BBC4-40CD-88E9-5291F15A39A8@163.com><br>Content-Type: text/plain; charset=GB2312<br><br>Hi Vijay<br><br>
I used the code in </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with the lasted commit: <br>commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>Author: Anand Avati <avati@redhat.com><br>Date: Tue Sep 17 16:45:03 2013 -0700<br><br> Revert "cluster/distribute: Rebalance should also verify
free inodes"<br> <br> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br> <br> Realized soon after merging, ?.<br><br>which include the patch you mentioned last time improve read perf, written
by Anand.<br><br>but the read perf was still slow:<br>write: 500MB/s<br>read: 77MB/s<br><br>while via fuse :<br>write 800MB/s<br>read 600MB/s<br><br>any advises?<br><br><br>Thank you.<br>-Kane<br><br>? 2013-9-13???10:37?kane <stef_9k@163.com> ???<br><br>> Hi Vijay?<br>> <br>> thank
you for post this message, i will try it soon<br>> <br>> -kane<br>> <br>> <br>> <br>> ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com> ???<br>> <br>>> On 09/13/2013 06:10 PM, kane wrote:<br>>>> Hi<br>>>> <br>>>> We use gluster samba vfs test io,but the read performance
via vfs is<br>>>> half of write perfomance,<br>>>> but via fuse the read and write performance is almost the
same.<br>>>> <br>>>> this is our smb.conf:<br>>>> [global]<br>>>> workgroup = MYGROUP<br>>>> server string = DCS Samba Server<br>>>> log file = /var/log/samba/log.vfs<br>>>> max log size = 500000<br>>>> # use sendfile = true<br>>>> aio read size = 262144<br>>>> aio write size = 262144<br>>>> aio write behind = true<br>>>> min receivefile size = 262144<br>>>> write cache size = 268435456<br>>>> security = user<br>>>> passdb backend = tdbsam<br>>>> load printers = yes<br>>>> cups options = raw<br>>>> read raw = yes<br>>>> write raw = yes<br>>>> max xmit = 262144<br>>>> socket options = TCP_NODELAY IPTOS_LOWDELAY
SO_RCVBUF=262144<br>>>> SO_SNDBUF=262144<br>>>> kernel oplocks = no<br>>>> stat cache = no<br>>>> <br>>>> any advises helpful?<br>>>> <br>>> <br>>> This patch has shown improvement in read performance with libgfapi:<br>>> <br>>> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>>> <br>>> Would it be possible for you to try this patch and check if it
improves performance in your case?<br>>> <br>>> -Vijay<br>>> <br>> <br><br><br><br><br>------------------------------<br><br>Message: 12<br>Date: Tue, 17 Sep 2013 22:19:23 -0700<br>From: Anand Avati <avati@gluster.org><br>To: kane <stef_9k@163.com><br>Cc: gluster-users <gluster-users@gluster.org>, Anand Avati<br>
<avati@redhat.com><br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID:<br>
<CAFboF2xxQ8rYqdDSBZtLVbae2OJ1yDZRMT4=AiWD_+2stkiRKA@mail.gmail.com><br>Content-Type: text/plain; charset="iso-2022-jp"<br><br>How are you testing this? What tool are you using?<br><br>Avati<br><br><br>On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k@163.com> wrote:<br><br>> Hi Vijay<br>><br>> I used the code in </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with<br>> the lasted commit:<br>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>> Author: Anand Avati <avati@redhat.com><br>> Date: Tue Sep 17 16:45:03 2013 -0700<br>><br>> Revert "cluster/distribute: Rebalance should also
verify free inodes"<br>><br>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>><br>> Realized soon after merging, ?.<br>><br>> which include the patch you mentioned last time improve read perf,
written<br>> by Anand.<br>><br>> but the read perf was still slow:<br>> write: 500MB/s<br>> read: 77MB/s<br>><br>> while via fuse :<br>> write 800MB/s<br>> read 600MB/s<br>><br>> any advises?<br>><br>><br>> Thank you.<br>> -Kane<br>><br>> ? 2013-9-13???10:37?kane <stef_9k@163.com> ???<br>><br>> > Hi Vijay?<br>> ><br>> > thank you for post this message, i will
try it soon<br>> ><br>> > -kane<br>> ><br>> ><br>> ><br>> > ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com> ???<br>> ><br>> >> On 09/13/2013 06:10 PM, kane wrote:<br>> >>> Hi<br>> >>><br>> >>> We use gluster samba vfs test io,but the read performance
via vfs is<br>> >>> half of write perfomance,<br>> >>> but via fuse the read and write performance is almost
the same.<br>> >>><br>> >>> this is our smb.conf:<br>> >>> [global]<br>> >>> workgroup = MYGROUP<br>> >>> server string = DCS Samba
Server<br>> >>> log file = /var/log/samba/log.vfs<br>> >>> max log size = 500000<br>> >>> # use sendfile = true<br>> >>> aio read size = 262144<br>> >>> aio write size = 262144<br>> >>> aio write behind = true<br>> >>> min receivefile size = 262144<br>> >>> write cache size = 268435456<br>> >>> security = user<br>> >>> passdb backend = tdbsam<br>> >>> load printers = yes<br>> >>> cups options = raw<br>> >>> read raw = yes<br>> >>> write raw = yes<br>> >>> max xmit = 262144<br>> >>> socket options = TCP_NODELAY
IPTOS_LOWDELAY SO_RCVBUF=262144<br>> >>> SO_SNDBUF=262144<br>> >>> kernel oplocks = no<br>> >>> stat cache = no<br>> >>><br>> >>> any advises helpful?<br>> >>><br>> >><br>> >> This patch has shown improvement in read performance with
libgfapi:<br>> >><br>> >> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>> >><br>> >> Would it be possible for you to try this patch and check
if it improves<br>> performance in your case?<br>> >><br>> >> -Vijay<br>> >><br>> ><br>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/98115dbd/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/98115dbd/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 13<br>Date: Wed, 18 Sep 2013 13:34:48 +0800<br>From: kane <stef_9k@163.com><br>To: Anand Avati <avati@gluster.org><br>Cc: gluster-users <gluster-users@gluster.org>, Anand Avati<br>
<avati@redhat.com><br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID: <D7FF253D-7D90-417A-9D26-543F93F2250D@163.com><br>Content-Type: text/plain; charset="iso-2022-jp"<br><br>Hi Anand,<br><br>
I use 2 gluster server , this is my volume info:<br>Volume Name: soul<br>Type: Distribute<br>Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6<br>Status: Started<br>Number of Bricks: 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: 192.168.101.133:/dcsdata/d0<br>Brick2: 192.168.101.134:/dcsdata/d0<br><br>each brick use a raid 5 logic disk with 8*2TSATA hdd.<br><br>smb.conf:<br>[gvol]<br> comment = For samba export of volume test<br> vfs objects = glusterfs<br> glusterfs:volfile_server = localhost<br> glusterfs:volume = soul<br> path = /<br> read only = no<br> guest ok = yes<br><br>this my testparm result:<br>[global]<br>
workgroup = MYGROUP<br>
server string = DCS Samba Server<br>
log file = /var/log/samba/log.vfs<br>
max log size = 500000<br>
max xmit = 262144<br>
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144<br>
stat cache = No<br>
kernel oplocks = No<br>
idmap config * : backend = tdb<br>
aio read size = 262144<br>
aio write size = 262144<br>
aio write behind = true<br>
cups options = raw<br><br>in client mount the smb share with cifs to dir /mnt/vfs, <br>then use iozone executed in the cifs mount dir "/mnt/vfs":<br>$ ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>
File size set to 10485760 KB<br>
Record Size 128 KB<br>
Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>
Output is in Kbytes/sec<br>
Time Resolution = 0.000001 seconds.<br>
Processor cache size set to 1024 Kbytes.<br>
Processor cache line size set to 32 bytes.<br>
File stride size set to 17 * record size.<br>
Throughput test with 4 processes<br>
Each process writes a 10485760 Kbyte file in 128 Kbyte records<br><br>
Children see throughput for 4 initial writers
= 534315.84
KB/sec<br>
Parent sees throughput for 4 initial writers
= 519428.83
KB/sec<br>
Min throughput per process
= 133154.69 KB/sec <br>
Max throughput per process
= 134341.05 KB/sec<br>
Avg throughput per process
= 133578.96 KB/sec<br>
Min xfer
= 10391296.00 KB<br><br>
Children see throughput for 4 rewriters
= 536634.88 KB/sec<br>
Parent sees throughput for 4 rewriters
= 522618.54 KB/sec<br>
Min throughput per process
= 133408.80 KB/sec <br>
Max throughput per process
= 134721.36 KB/sec<br>
Avg throughput per process
= 134158.72 KB/sec<br>
Min xfer
= 10384384.00 KB<br><br>
Children see throughput for 4 readers
= 77403.54 KB/sec<br>
Parent sees throughput for 4 readers
= 77402.86 KB/sec<br>
Min throughput per process
= 19349.42 KB/sec <br>
Max throughput per process
= 19353.42 KB/sec<br>
Avg throughput per process
= 19350.88 KB/sec<br>
Min xfer
= 10483712.00 KB<br><br>
Children see throughput for 4 re-readers
= 77424.40 KB/sec<br>
Parent sees throughput for 4 re-readers
= 77423.89 KB/sec<br>
Min throughput per process
= 19354.75 KB/sec <br>
Max throughput per process
= 19358.50 KB/sec<br>
Avg throughput per process
= 19356.10 KB/sec<br>
Min xfer
= 10483840.00 KB<br><br>then the use the same command test in the dir mounted with glister fuse:<br>
File size set to 10485760 KB<br>
Record Size 128 KB<br>
Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>
Output is in Kbytes/sec<br>
Time Resolution = 0.000001 seconds.<br>
Processor cache size set to 1024 Kbytes.<br>
Processor cache line size set to 32 bytes.<br>
File stride size set to 17 * record size.<br>
Throughput test with 4 processes<br>
Each process writes a 10485760 Kbyte file in 128 Kbyte records<br><br>
Children see throughput for 4 initial writers
= 887534.72
KB/sec<br>
Parent sees throughput for 4 initial writers
= 848830.39
KB/sec<br>
Min throughput per process
= 220140.91 KB/sec <br>
Max throughput per process
= 223690.45 KB/sec<br>
Avg throughput per process
= 221883.68 KB/sec<br>
Min xfer
= 10319360.00 KB<br><br>
Children see throughput for 4 rewriters
= 892774.92 KB/sec<br>
Parent sees throughput for 4 rewriters
= 871186.83 KB/sec<br>
Min throughput per process
= 222326.44 KB/sec <br>
Max throughput per process
= 223970.17 KB/sec<br>
Avg throughput per process
= 223193.73 KB/sec<br>
Min xfer
= 10431360.00 KB<br><br>
Children see throughput for 4 readers
= 605889.12 KB/sec<br>
Parent sees throughput for 4 readers
= 601767.96 KB/sec<br>
Min throughput per process
= 143133.14 KB/sec <br>
Max throughput per process
= 159550.88 KB/sec<br>
Avg throughput per process
= 151472.28 KB/sec<br>
Min xfer
= 9406848.00 KB<br><br>it shows much higher perf.<br><br>any places i did wrong?<br><br><br>thank you<br>-Kane<br><br>? 2013-9-18???1:19?Anand Avati <avati@gluster.org> ???<br><br>> How are you testing this? What tool are you using?<br>> <br>> Avati<br>> <br>> <br>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k@163.com> wrote:<br>> Hi Vijay<br>> <br>> I used the code in </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with the lasted commit:<br>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>> Author: Anand Avati <avati@redhat.com><br>> Date: Tue Sep 17 16:45:03 2013 -0700<br>> <br>> Revert "cluster/distribute: Rebalance should also
verify free inodes"<br>> <br>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>> <br>> Realized soon after merging, ?.<br>> <br>> which include the patch you mentioned last time improve read perf,
written by Anand.<br>> <br>> but the read perf was still slow:<br>> write: 500MB/s<br>> read: 77MB/s<br>> <br>> while via fuse :<br>> write 800MB/s<br>> read 600MB/s<br>> <br>> any advises?<br>> <br>> <br>> Thank you.<br>> -Kane<br>> <br>> ? 2013-9-13???10:37?kane <stef_9k@163.com> ???<br>> <br>> > Hi Vijay?<br>> ><br>> > thank you for post this message, i will
try it soon<br>> ><br>> > -kane<br>> ><br>> ><br>> ><br>> > ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com> ???<br>> ><br>> >> On 09/13/2013 06:10 PM, kane wrote:<br>> >>> Hi<br>> >>><br>> >>> We use gluster samba vfs test io,but the read performance
via vfs is<br>> >>> half of write perfomance,<br>> >>> but via fuse the read and write performance is almost
the same.<br>> >>><br>> >>> this is our smb.conf:<br>> >>> [global]<br>> >>> workgroup = MYGROUP<br>> >>> server string = DCS Samba
Server<br>> >>> log file = /var/log/samba/log.vfs<br>> >>> max log size = 500000<br>> >>> # use sendfile = true<br>> >>> aio read size = 262144<br>> >>> aio write size = 262144<br>> >>> aio write behind = true<br>> >>> min receivefile size = 262144<br>> >>> write cache size = 268435456<br>> >>> security = user<br>> >>> passdb backend = tdbsam<br>> >>> load printers = yes<br>> >>> cups options = raw<br>> >>> read raw = yes<br>> >>> write raw = yes<br>> >>> max xmit = 262144<br>> >>> socket options = TCP_NODELAY
IPTOS_LOWDELAY SO_RCVBUF=262144<br>> >>> SO_SNDBUF=262144<br>> >>> kernel oplocks = no<br>> >>> stat cache = no<br>> >>><br>> >>> any advises helpful?<br>> >>><br>> >><br>> >> This patch has shown improvement in read performance with
libgfapi:<br>> >><br>> >> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>> >><br>> >> Would it be possible for you to try this patch and check
if it improves performance in your case?<br>> >><br>> >> -Vijay<br>> >><br>> ><br>> <br>> <br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> <br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/fabbf07e/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/fabbf07e/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 14<br>Date: Wed, 18 Sep 2013 05:35:32 +0000<br>From: Bobby Jacob <bobby.jacob@alshaya.com><br>To: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: [Gluster-users] Mounting same replica-volume on multiple<br>
clients.
????<br>Message-ID:<br>
<AC3305F9C186F849B835A3E6D3C9BEFEAFA70E@KWTPRMBX001.mha.local><br>Content-Type: text/plain; charset="us-ascii"<br><br>HI,<br><br>I have 2 gluster nodes (GFS01/GFS02) each with a single brick (B01/B01).
I have created a simple replica volume with these bricks.<br>Bricks : GFS01/B01 and GFS02/B01.<br>Volume: TestVol<br><br>I have 2 clients (C01/C02) which will mount this "testvol" for
simultaneous read/write. The 2 clients run the same application which is
load-balanced, so user request are end to both the client servers which
reads/writes data to both the same volume.<br><br>Mounting the volume on C1 :
mount -t glusterfs -o backupvolfile-server=GFS02 GFS01:/testvol
/data<br>Mounting the volume on C2 :
mount -t glusterfs -o backupvolfile-server=GFS01 GFS02:/testvol
/data<br><br>Is this the appropriate way to be followed.?<br><br>At times, I notice that when I write data through C1-mount point the data
is written only to GFS01/B01 and if data is written through C2-mount point
the data is written only to GFS02/B01.<br><br>Please advise. !!<br><br><br>Thanks & Regards,<br>Bobby Jacob<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/e3b53134/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/e3b53134/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 15<br>Date: Tue, 17 Sep 2013 22:38:14 -0700<br>From: Anand Avati <avati@redhat.com><br>To: kane <stef_9k@163.com><br>Cc: gluster-users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID: <52393C46.80503@redhat.com><br>Content-Type: text/plain; charset=ISO-2022-JP<br><br>On 9/17/13 10:34 PM, kane wrote:<br>> Hi Anand,<br>> <br>> I use 2 gluster server , this is my volume info:<br>> Volume Name: soul<br>> Type: Distribute<br>> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6<br>> Status: Started<br>> Number of Bricks: 2<br>> Transport-type: tcp<br>> Bricks:<br>> Brick1: 192.168.101.133:/dcsdata/d0<br>> Brick2: 192.168.101.134:/dcsdata/d0<br>> <br>> each brick use a raid 5 logic disk with 8*2TSATA hdd.<br>> <br>> smb.conf:<br>> [gvol]<br>> comment = For samba export of volume
test<br>> vfs objects = glusterfs<br>> glusterfs:volfile_server = localhost<br>> glusterfs:volume = soul<br>> path = /<br>> read only = no<br>> guest ok = yes<br>> <br>> this my testparm result:<br>> [global]<br>> workgroup = MYGROUP<br>> server string = DCS Samba Server<br>> log file = /var/log/samba/log.vfs</font></tt><br><tt><font size=2>> max log size = 500000<br>> max xmit = 262144<br>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 <br>> SO_SNDBUF=262144<br>> stat cache = No<br>> kernel oplocks = No<br>> idmap config * : backend = tdb<br>> aio read size = 262144<br>> aio write size = 262144<br>> aio write behind = true<br>> cups options = raw<br>> <br>> in client mount the smb share with cifs to dir /mnt/vfs,<br>> then use iozone executed in the cifs mount dir "/mnt/vfs":<br>> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> File size set to 10485760 KB<br>> Record Size 128 KB<br>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> Output is in Kbytes/sec<br>> Time Resolution = 0.000001 seconds.<br>> Processor cache size set to 1024 Kbytes.<br>> Processor cache line size set to 32 bytes.<br>> File stride size set to 17 * record size.<br>> Throughput test with 4 processes<br>> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>> <br>> Children see throughput for 4 initial writers = 534315.84
KB/sec<br>> Parent sees throughput for 4 initial writers = 519428.83
KB/sec<br>> Min throughput per process = 133154.69 KB/sec<br>> Max throughput per process = 134341.05 KB/sec<br>> Avg throughput per process = 133578.96 KB/sec<br>> Min xfer = 10391296.00 KB<br>> <br>> Children see throughput for 4 rewriters = 536634.88 KB/sec<br>> Parent sees throughput for 4 rewriters = 522618.54 KB/sec<br>> Min throughput per process = 133408.80 KB/sec<br>> Max throughput per process = 134721.36 KB/sec<br>> Avg throughput per process = 134158.72 KB/sec<br>> Min xfer = 10384384.00 KB<br>> <br>> Children see throughput for 4 readers = 77403.54 KB/sec<br>> Parent sees throughput for 4 readers = 77402.86 KB/sec<br>> Min throughput per process = 19349.42 KB/sec<br>> Max throughput per process = 19353.42 KB/sec<br>> Avg throughput per process = 19350.88 KB/sec<br>> Min xfer = 10483712.00 KB<br>> <br>> Children see throughput for 4 re-readers = 77424.40 KB/sec<br>> Parent sees throughput for 4 re-readers = 77423.89 KB/sec<br>> Min throughput per process = 19354.75 KB/sec<br>> Max throughput per process = 19358.50 KB/sec<br>> Avg throughput per process = 19356.10 KB/sec<br>> Min xfer = 10483840.00 KB<br>> <br>> then the use the same command test in the dir mounted with glister
fuse:<br>> File size set to 10485760 KB<br>> Record Size 128 KB<br>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> Output is in Kbytes/sec<br>> Time Resolution = 0.000001 seconds.<br>> Processor cache size set to 1024 Kbytes.<br>> Processor cache line size set to 32 bytes.<br>> File stride size set to 17 * record size.<br>> Throughput test with 4 processes<br>> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>> <br>> Children see throughput for 4 initial writers = 887534.72
KB/sec<br>> Parent sees throughput for 4 initial writers = 848830.39
KB/sec<br>> Min throughput per process = 220140.91 KB/sec<br>> Max throughput per process = 223690.45 KB/sec<br>> Avg throughput per process = 221883.68 KB/sec<br>> Min xfer = 10319360.00 KB<br>> <br>> Children see throughput for 4 rewriters = 892774.92 KB/sec<br>> Parent sees throughput for 4 rewriters = 871186.83 KB/sec<br>> Min throughput per process = 222326.44 KB/sec<br>> Max throughput per process = 223970.17 KB/sec<br>> Avg throughput per process = 223193.73 KB/sec<br>> Min xfer = 10431360.00 KB<br>> <br>> Children see throughput for 4 readers = 605889.12 KB/sec<br>> Parent sees throughput for 4 readers = 601767.96 KB/sec<br>> Min throughput per process = 143133.14 KB/sec<br>> Max throughput per process = 159550.88 KB/sec<br>> Avg throughput per process = 151472.28 KB/sec<br>> Min xfer = 9406848.00 KB<br>> <br>> it shows much higher perf.<br>> <br>> any places i did wrong?<br>> <br>> <br>> thank you<br>> -Kane<br>> <br>> ? 2013-9-18???1:19?Anand Avati <avati@gluster.org <br>> <</font></tt><a href=mailto:avati@gluster.org><tt><font size=2>mailto:avati@gluster.org</font></tt></a><tt><font size=2>>>
???<br>> <br>>> How are you testing this? What tool are you using?<br>>><br>>> Avati<br>>><br>>><br>>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k@163.com <br>>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
wrote:<br>>><br>>> Hi Vijay<br>>><br>>> I used the code in<br>>> </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with the lasted commit:<br>>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>>> Author: Anand Avati <avati@redhat.com <</font></tt><a href=mailto:avati@redhat.com><tt><font size=2>mailto:avati@redhat.com</font></tt></a><tt><font size=2>>><br>>> Date: Tue Sep 17 16:45:03 2013 -0700<br>>><br>>> Revert "cluster/distribute: Rebalance
should also verify free<br>>> inodes"<br>>><br>>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>>><br>>> Realized soon after merging, ?.<br>>><br>>> which include the patch you mentioned last time
improve read perf,<br>>> written by Anand.<br>>><br>>> but the read perf was still slow:<br>>> write: 500MB/s<br>>> read: 77MB/s<br>>><br>>> while via fuse :<br>>> write 800MB/s<br>>> read 600MB/s<br>>><br>>> any advises?<br>>><br>>><br>>> Thank you.<br>>> -Kane<br>>><br>>> ? 2013-9-13???10:37?kane <stef_9k@163.com<br>>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
???<br>>><br>>> > Hi Vijay?<br>>> ><br>>> > thank you for post this
message, i will try it soon<br>>> ><br>>> > -kane<br>>> ><br>>> ><br>>> ><br>>> > ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com<br>>> <</font></tt><a href=mailto:vbellur@redhat.com><tt><font size=2>mailto:vbellur@redhat.com</font></tt></a><tt><font size=2>>>
???<br>>> ><br>>> >> On 09/13/2013 06:10 PM, kane wrote:<br>>> >>> Hi<br>>> >>><br>>> >>> We use gluster samba vfs test io,but
the read performance via<br>>> vfs is<br>>> >>> half of write perfomance,<br>>> >>> but via fuse the read and write performance
is almost the same.<br>>> >>><br>>> >>> this is our smb.conf:<br>>> >>> [global]<br>>> >>> workgroup
= MYGROUP<br>>> >>> server string
= DCS Samba Server<br>>> >>> log file
= /var/log/samba/log.vfs<br>>> >>> max log
size = 500000<br>>> >>> # use sendfile
= true<br>>> >>> aio read
size = 262144<br>>> >>> aio write
size = 262144<br>>> >>> aio write
behind = true<br>>> >>> min receivefile
size = 262144<br>>> >>> write cache
size = 268435456<br>>> >>> security
= user<br>>> >>> passdb backend
= tdbsam<br>>> >>> load printers
= yes<br>>> >>> cups options
= raw<br>>> >>> read raw
= yes<br>>> >>> write raw
= yes<br>>> >>> max xmit
= 262144<br>>> >>> socket options
= TCP_NODELAY IPTOS_LOWDELAY<br>>> SO_RCVBUF=262144<br>>> >>> SO_SNDBUF=262144<br>>> >>> kernel oplocks
= no<br>>> >>> stat cache
= no<br>>> >>><br>>> >>> any advises helpful?<br>>> >>><br>>> >><br>>> >> This patch has shown improvement in read
performance with libgfapi:<br>>> >><br>>> >> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>>> >><br>>> >> Would it be possible for you to try this
patch and check if it<br>>> improves performance in your case?<br>>> >><br>>> >> -Vijay<br>>> >><br>>> ><br>>><br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org <</font></tt><a href="mailto:Gluster-users@gluster.org"><tt><font size=2>mailto:Gluster-users@gluster.org</font></tt></a><tt><font size=2>><br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>>><br>> <br><br>Please add 'kernel oplocks = no' in the [gvol] section and try again.<br><br>Avati<br><br><br><br>------------------------------<br><br>Message: 16<br>Date: Wed, 18 Sep 2013 07:45:39 +0200<br>From: Daniel M?ller <mueller@tropenklinik.de><br>To: "'Bobby Jacob'" <bobby.jacob@alshaya.com>,<br>
<gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Mounting same replica-volume on multiple<br>
clients.
????<br>Message-ID: <001801ceb432$4f1b0b60$ed512220$@de><br>Content-Type: text/plain;
charset="iso-8859-1"<br><br>Hello,<br>this ist he behavior as if you write directly into the glusterd<br>directory/partition and not to the remounted replicating bricks!?<br><br><br>EDV Daniel M?ller<br><br>Leitung EDV<br>Tropenklinik Paul-Lechler-Krankenhaus<br>Paul-Lechler-Str. 24<br>72076 T?bingen <br>Tel.: 07071/206-463, Fax: 07071/206-499<br>eMail: mueller@tropenklinik.de<br>Internet: </font></tt><a href=www.tropenklinik.de><tt><font size=2>www.tropenklinik.de</font></tt></a><tt><font size=2><br><br>Von: gluster-users-bounces@gluster.org<br>[</font></tt><a href="mailto:gluster-users-bounces@gluster.org"><tt><font size=2>mailto:gluster-users-bounces@gluster.org</font></tt></a><tt><font size=2>]
Im Auftrag von Bobby Jacob<br>Gesendet: Mittwoch, 18. September 2013 07:36<br>An: gluster-users@gluster.org<br>Betreff: [Gluster-users] Mounting same replica-volume on multiple clients.<br>????<br><br>HI,<br><br>I have 2 gluster nodes (GFS01/GFS02) each with a single brick (B01/B01).
I<br>have created a simple replica volume with these bricks. <br>Bricks ?? : GFS01/B01 and GFS02/B01.<br>Volume: TestVol<br><br>I have 2 clients (C01/C02) which will mount this ?testvol? for simultaneous<br>read/write. The 2 clients run the same application which is load-balanced,<br>so user request are end to both the client servers which reads/writes data<br>to both the same volume.<br><br>Mounting the volume on C1?????? :?????????????? mount ?t glusterfs ?o<br>backupvolfile-server=GFS02 GFS01:/testvol /data<br>Mounting the volume on C2?????? :?????????????? mount ?t glusterfs ?o<br>backupvolfile-server=GFS01 GFS02:/testvol /data<br><br>Is this the appropriate way to be followed.? <br><br>At times, I notice that when I write data through C1-mount point the data
is<br>written only to GFS01/B01 and if data is written through C2-mount point
the<br>data is written only to GFS02/B01.<br><br>Please advise. !!<br><br><br>Thanks & Regards,<br>Bobby Jacob<br><br><br><br>------------------------------<br><br>Message: 17<br>Date: Wed, 18 Sep 2013 13:46:09 +0800<br>From: kane <stef_9k@163.com><br>To: Anand Avati <avati@redhat.com><br>Cc: gluster-users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID: <BDD668E0-EA97-4084-ABBF-5508D3723107@163.com><br>Content-Type: text/plain; charset=iso-2022-jp<br><br>I have already used "kernel oplocks = no" in the smb.conf, next
is my original smb.conf file global settings:<br>[global]<br> workgroup = MYGROUP<br> server string = DCS Samba Server<br> log file = /var/log/samba/log.vfs<br> max log size = 500000<br> aio read size = 262144<br> aio write size = 262144<br> aio write behind = true<br> security = user<br> passdb backend = tdbsam<br> load printers = yes<br> cups options = raw<br> read raw = yes<br> write raw = yes<br> max xmit = 262144<br> socket options = TCP_NODELAY IPTOS_LOWDELAY
SO_RCVBUF=262144 SO_SNDBUF=262144<br># max protocol = SMB2<br> kernel oplocks = no<br> stat cache = no<br><br>thank you<br>-Kane<br>? 2013-9-18???1:38?Anand Avati <avati@redhat.com> ???<br><br>> On 9/17/13 10:34 PM, kane wrote:<br>>> Hi Anand,<br>>> <br>>> I use 2 gluster server , this is my volume info:<br>>> Volume Name: soul<br>>> Type: Distribute<br>>> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6<br>>> Status: Started<br>>> Number of Bricks: 2<br>>> Transport-type: tcp<br>>> Bricks:<br>>> Brick1: 192.168.101.133:/dcsdata/d0<br>>> Brick2: 192.168.101.134:/dcsdata/d0<br>>> <br>>> each brick use a raid 5 logic disk with 8*2TSATA hdd.<br>>> <br>>> smb.conf:<br>>> [gvol]<br>>> comment = For samba export of volume
test<br>>> vfs objects = glusterfs<br>>> glusterfs:volfile_server = localhost<br>>> glusterfs:volume = soul<br>>> path = /<br>>> read only = no<br>>> guest ok = yes<br>>> <br>>> this my testparm result:<br>>> [global]<br>>> workgroup = MYGROUP<br>>> server string = DCS Samba Server<br>>> log file = /var/log/samba/log.vfs<br>>> max log size = 500000<br>>> max xmit = 262144<br>>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 <br>>> SO_SNDBUF=262144<br>>> stat cache = No<br>>> kernel oplocks = No<br>>> idmap config * : backend = tdb<br>>> aio read size = 262144<br>>> aio write size = 262144<br>>> aio write behind = true<br>>> cups options = raw<br>>> <br>>> in client mount the smb share with cifs to dir /mnt/vfs,<br>>> then use iozone executed in the cifs mount dir "/mnt/vfs":<br>>> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>>> File size set to 10485760 KB<br>>> Record Size 128 KB<br>>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>>> Output is in Kbytes/sec<br>>> Time Resolution = 0.000001 seconds.<br>>> Processor cache size set to 1024 Kbytes.<br>>> Processor cache line size set to 32 bytes.<br>>> File stride size set to 17 * record size.<br>>> Throughput test with 4 processes<br>>> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>>> <br>>> Children see throughput for 4 initial writers = 534315.84
KB/sec<br>>> Parent sees throughput for 4 initial writers = 519428.83
KB/sec<br>>> Min throughput per process = 133154.69 KB/sec<br>>> Max throughput per process = 134341.05 KB/sec<br>>> Avg throughput per process = 133578.96 KB/sec<br>>> Min xfer = 10391296.00 KB<br>>> <br>>> Children see throughput for 4 rewriters = 536634.88
KB/sec<br>>> Parent sees throughput for 4 rewriters = 522618.54
KB/sec<br>>> Min throughput per process = 133408.80 KB/sec<br>>> Max throughput per process = 134721.36 KB/sec<br>>> Avg throughput per process = 134158.72 KB/sec<br>>> Min xfer = 10384384.00 KB<br>>> <br>>> Children see throughput for 4 readers = 77403.54
KB/sec<br>>> Parent sees throughput for 4 readers = 77402.86 KB/sec<br>>> Min throughput per process = 19349.42 KB/sec<br>>> Max throughput per process = 19353.42 KB/sec<br>>> Avg throughput per process = 19350.88 KB/sec<br>>> Min xfer = 10483712.00 KB<br>>> <br>>> Children see throughput for 4 re-readers = 77424.40 KB/sec<br>>> Parent sees throughput for 4 re-readers = 77423.89 KB/sec<br>>> Min throughput per process = 19354.75 KB/sec<br>>> Max throughput per process = 19358.50 KB/sec<br>>> Avg throughput per process = 19356.10 KB/sec<br>>> Min xfer = 10483840.00 KB<br>>> <br>>> then the use the same command test in the dir mounted with glister
fuse:<br>>> File size set to 10485760 KB<br>>> Record Size 128 KB<br>>> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>>> Output is in Kbytes/sec<br>>> Time Resolution = 0.000001 seconds.<br>>> Processor cache size set to 1024 Kbytes.<br>>> Processor cache line size set to 32 bytes.<br>>> File stride size set to 17 * record size.<br>>> Throughput test with 4 processes<br>>> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>>> <br>>> Children see throughput for 4 initial writers = 887534.72
KB/sec<br>>> Parent sees throughput for 4 initial writers = 848830.39
KB/sec<br>>> Min throughput per process = 220140.91 KB/sec<br>>> Max throughput per process = 223690.45 KB/sec<br>>> Avg throughput per process = 221883.68 KB/sec<br>>> Min xfer = 10319360.00 KB<br>>> <br>>> Children see throughput for 4 rewriters = 892774.92
KB/sec<br>>> Parent sees throughput for 4 rewriters = 871186.83
KB/sec<br>>> Min throughput per process = 222326.44 KB/sec<br>>> Max throughput per process = 223970.17 KB/sec<br>>> Avg throughput per process = 223193.73 KB/sec<br>>> Min xfer = 10431360.00 KB<br>>> <br>>> Children see throughput for 4 readers = 605889.12
KB/sec<br>>> Parent sees throughput for 4 readers = 601767.96 KB/sec<br>>> Min throughput per process = 143133.14 KB/sec<br>>> Max throughput per process = 159550.88 KB/sec<br>>> Avg throughput per process = 151472.28 KB/sec<br>>> Min xfer = 9406848.00 KB<br>>> <br>>> it shows much higher perf.<br>>> <br>>> any places i did wrong?<br>>> <br>>> <br>>> thank you<br>>> -Kane<br>>> <br>>> ? 2013-9-18???1:19?Anand Avati <avati@gluster.org <br>>> <</font></tt><a href=mailto:avati@gluster.org><tt><font size=2>mailto:avati@gluster.org</font></tt></a><tt><font size=2>>>
???<br>>> <br>>>> How are you testing this? What tool are you using?<br>>>> <br>>>> Avati<br>>>> <br>>>> <br>>>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k@163.com
<br>>>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
wrote:<br>>>> <br>>>> Hi Vijay<br>>>> <br>>>> I used the code in<br>>>> </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with the lasted commit:<br>>>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>>>> Author: Anand Avati <avati@redhat.com <</font></tt><a href=mailto:avati@redhat.com><tt><font size=2>mailto:avati@redhat.com</font></tt></a><tt><font size=2>>><br>>>> Date: Tue Sep 17 16:45:03 2013 -0700<br>>>> <br>>>> Revert "cluster/distribute:
Rebalance should also verify free<br>>>> inodes"<br>>>> <br>>>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>>>> <br>>>> Realized soon after merging, ?.<br>>>> <br>>>> which include the patch you mentioned last time
improve read perf,<br>>>> written by Anand.<br>>>> <br>>>> but the read perf was still slow:<br>>>> write: 500MB/s<br>>>> read: 77MB/s<br>>>> <br>>>> while via fuse :<br>>>> write 800MB/s<br>>>> read 600MB/s<br>>>> <br>>>> any advises?<br>>>> <br>>>> <br>>>> Thank you.<br>>>> -Kane<br>>>> <br>>>> ? 2013-9-13???10:37?kane <stef_9k@163.com<br>>>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
???<br>>>> <br>>>>> Hi Vijay?<br>>>>> <br>>>>> thank you for post this message, i
will try it soon<br>>>>> <br>>>>> -kane<br>>>>> <br>>>>> <br>>>>> <br>>>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com<br>>>> <</font></tt><a href=mailto:vbellur@redhat.com><tt><font size=2>mailto:vbellur@redhat.com</font></tt></a><tt><font size=2>>>
???<br>>>>> <br>>>>>> On 09/13/2013 06:10 PM, kane wrote:<br>>>>>>> Hi<br>>>>>>> <br>>>>>>> We use gluster samba vfs test io,but the read
performance via<br>>>> vfs is<br>>>>>>> half of write perfomance,<br>>>>>>> but via fuse the read and write performance is
almost the same.<br>>>>>>> <br>>>>>>> this is our smb.conf:<br>>>>>>> [global]<br>>>>>>> workgroup = MYGROUP<br>>>>>>> server string = DCS Samba
Server<br>>>>>>> log file = /var/log/samba/log.vfs<br>>>>>>> max log size = 500000<br>>>>>>> # use sendfile = true<br>>>>>>> aio read size = 262144<br>>>>>>> aio write size = 262144<br>>>>>>> aio write behind = true<br>>>>>>> min receivefile size = 262144<br>>>>>>> write cache size = 268435456<br>>>>>>> security = user<br>>>>>>> passdb backend = tdbsam<br>>>>>>> load printers = yes<br>>>>>>> cups options = raw<br>>>>>>> read raw = yes<br>>>>>>> write raw = yes<br>>>>>>> max xmit = 262144<br>>>>>>> socket options = TCP_NODELAY
IPTOS_LOWDELAY<br>>>> SO_RCVBUF=262144<br>>>>>>> SO_SNDBUF=262144<br>>>>>>> kernel oplocks = no<br>>>>>>> stat cache = no<br>>>>>>> <br>>>>>>> any advises helpful?<br>>>>>>> <br>>>>>> <br>>>>>> This patch has shown improvement in read performance
with libgfapi:<br>>>>>> <br>>>>>> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>>>>>> <br>>>>>> Would it be possible for you to try this patch and
check if it<br>>>> improves performance in your case?<br>>>>>> <br>>>>>> -Vijay<br>>>>>> <br>>>>> <br>>>> <br>>>> <br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org <</font></tt><a href="mailto:Gluster-users@gluster.org"><tt><font size=2>mailto:Gluster-users@gluster.org</font></tt></a><tt><font size=2>><br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>>> <br>>>> <br>>> <br>> <br>> Please add 'kernel oplocks = no' in the [gvol] section and try again.<br>> <br>> Avati<br>> <br><br><br><br><br>------------------------------<br><br>Message: 18<br>Date: Wed, 18 Sep 2013 05:48:29 +0000<br>From: Bobby Jacob <bobby.jacob@alshaya.com><br>To: "mueller@tropenklinik.de" <mueller@tropenklinik.de>,<br>
"gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Mounting same replica-volume on multiple<br>
clients.
????<br>Message-ID:<br>
<AC3305F9C186F849B835A3E6D3C9BEFEAFA748@KWTPRMBX001.mha.local><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Exactly. !! BUT I am writing through the volume mount-point from the clients.
!! NOT directly into the bricks. !!<br><br>I'm using GlusterFS 3.3.2 with Centos6.4 . !<br><br>Thanks & Regards,<br>Bobby Jacob<br><br>-----Original Message-----<br>From: Daniel M?ller [</font></tt><a href=mailto:mueller@tropenklinik.de><tt><font size=2>mailto:mueller@tropenklinik.de</font></tt></a><tt><font size=2>]
<br>Sent: Wednesday, September 18, 2013 8:46 AM<br>To: Bobby Jacob; gluster-users@gluster.org<br>Subject: AW: [Gluster-users] Mounting same replica-volume on multiple clients.
????<br><br>Hello,<br>this ist he behavior as if you write directly into the glusterd directory/partition
and not to the remounted replicating bricks!?<br><br><br>EDV Daniel M?ller<br><br>Leitung EDV<br>Tropenklinik Paul-Lechler-Krankenhaus<br>Paul-Lechler-Str. 24<br>72076 T?bingen<br>Tel.: 07071/206-463, Fax: 07071/206-499<br>eMail: mueller@tropenklinik.de</font></tt><br><tt><font size=2>Internet: </font></tt><a href=www.tropenklinik.de><tt><font size=2>www.tropenklinik.de</font></tt></a><tt><font size=2><br><br>Von: gluster-users-bounces@gluster.org<br>[</font></tt><a href="mailto:gluster-users-bounces@gluster.org"><tt><font size=2>mailto:gluster-users-bounces@gluster.org</font></tt></a><tt><font size=2>]
Im Auftrag von Bobby Jacob<br>Gesendet: Mittwoch, 18. September 2013 07:36<br>An: gluster-users@gluster.org<br>Betreff: [Gluster-users] Mounting same replica-volume on multiple clients.<br>????<br><br>HI,<br><br>I have 2 gluster nodes (GFS01/GFS02) each with a single brick (B01/B01).
I have created a simple replica volume with these bricks. <br>Bricks ?? : GFS01/B01 and GFS02/B01.<br>Volume: TestVol<br><br>I have 2 clients (C01/C02) which will mount this "testvol" for
simultaneous read/write. The 2 clients run the same application which is
load-balanced, so user request are end to both the client servers which
reads/writes data to both the same volume.<br><br>Mounting the volume on C1?????? :?????????????? mount -t glusterfs -o<br>backupvolfile-server=GFS02 GFS01:/testvol /data Mounting the volume on
C2?????? :?????????????? mount -t glusterfs -o<br>backupvolfile-server=GFS01 GFS02:/testvol /data<br><br>Is this the appropriate way to be followed.? <br><br>At times, I notice that when I write data through C1-mount point the data
is written only to GFS01/B01 and if data is written through C2-mount point
the data is written only to GFS02/B01.<br><br>Please advise. !!<br><br><br>Thanks & Regards,<br>Bobby Jacob<br><br><br><br>------------------------------<br><br>Message: 19<br>Date: Tue, 17 Sep 2013 23:45:43 -0700<br>From: Anand Avati <avati@gluster.org><br>To: kane <stef_9k@163.com><br>Cc: gluster-users <gluster-users@gluster.org>, Anand Avati<br>
<avati@redhat.com><br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID:<br>
<CAFboF2yGB6UPN-chHDTGf9HgM_0jbPjWDnUeEiyQp+h9qDTV_w@mail.gmail.com><br>Content-Type: text/plain; charset="iso-2022-jp"<br><br>Can you get the volume profile dumps for both the runs and compare them?<br><br>Avati<br><br><br><br>On Tue, Sep 17, 2013 at 10:46 PM, kane <stef_9k@163.com> wrote:<br><br>> I have already used "kernel oplocks = no" in the smb.conf,
next is my<br>> original smb.conf file global settings:<br>> [global]<br>> workgroup = MYGROUP<br>> server string = DCS Samba Server<br>> log file = /var/log/samba/log.vfs<br>> max log size = 500000<br>> aio read size = 262144<br>> aio write size = 262144<br>> aio write behind = true<br>> security = user<br>> passdb backend = tdbsam<br>> load printers = yes<br>> cups options = raw<br>> read raw = yes<br>> write raw = yes<br>> max xmit = 262144<br>> socket options = TCP_NODELAY IPTOS_LOWDELAY
SO_RCVBUF=262144<br>> SO_SNDBUF=262144<br>> # max protocol = SMB2<br>> kernel oplocks = no<br>> stat cache = no<br>><br>> thank you<br>> -Kane<br>> ? 2013-9-18???1:38?Anand Avati <avati@redhat.com> ???<br>><br>> > On 9/17/13 10:34 PM, kane wrote:<br>> >> Hi Anand,<br>> >><br>> >> I use 2 gluster server , this is my volume info:<br>> >> Volume Name: soul<br>> >> Type: Distribute<br>> >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6<br>> >> Status: Started<br>> >> Number of Bricks: 2<br>> >> Transport-type: tcp<br>> >> Bricks:<br>> >> Brick1: 192.168.101.133:/dcsdata/d0<br>> >> Brick2: 192.168.101.134:/dcsdata/d0<br>> >><br>> >> each brick use a raid 5 logic disk with 8*2TSATA hdd.<br>> >><br>> >> smb.conf:<br>> >> [gvol]<br>> >> comment = For samba export of
volume test<br>> >> vfs objects = glusterfs<br>> >> glusterfs:volfile_server = localhost<br>> >> glusterfs:volume = soul<br>> >> path = /<br>> >> read only = no<br>> >> guest ok = yes<br>> >><br>> >> this my testparm result:<br>> >> [global]<br>> >> workgroup = MYGROUP<br>> >> server string = DCS Samba Server<br>> >> log file = /var/log/samba/log.vfs<br>> >> max log size = 500000<br>> >> max xmit = 262144<br>> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144<br>> >> SO_SNDBUF=262144<br>> >> stat cache = No<br>> >> kernel oplocks = No<br>> >> idmap config * : backend = tdb<br>> >> aio read size = 262144<br>> >> aio write size = 262144<br>> >> aio write behind = true<br>> >> cups options = raw<br>> >><br>> >> in client mount the smb share with cifs to dir /mnt/vfs,<br>> >> then use iozone executed in the cifs mount dir "/mnt/vfs":<br>> >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> >> File size set to 10485760 KB<br>> >> Record Size 128 KB<br>> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> >> Output is in Kbytes/sec<br>> >> Time Resolution = 0.000001 seconds.<br>> >> Processor cache size set to 1024 Kbytes.<br>> >> Processor cache line size set to 32 bytes.<br>> >> File stride size set to 17 * record size.<br>> >> Throughput test with 4 processes<br>> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>> >><br>> >> Children see throughput for 4 initial writers = 534315.84
KB/sec<br>> >> Parent sees throughput for 4 initial writers = 519428.83
KB/sec<br>> >> Min throughput per process = 133154.69 KB/sec<br>> >> Max throughput per process = 134341.05 KB/sec<br>> >> Avg throughput per process = 133578.96 KB/sec<br>> >> Min xfer = 10391296.00 KB<br>> >><br>> >> Children see throughput for 4 rewriters = 536634.88
KB/sec<br>> >> Parent sees throughput for 4 rewriters = 522618.54
KB/sec<br>> >> Min throughput per process = 133408.80 KB/sec<br>> >> Max throughput per process = 134721.36 KB/sec<br>> >> Avg throughput per process = 134158.72 KB/sec<br>> >> Min xfer = 10384384.00 KB<br>> >><br>> >> Children see throughput for 4 readers = 77403.54
KB/sec<br>> >> Parent sees throughput for 4 readers = 77402.86
KB/sec<br>> >> Min throughput per process = 19349.42 KB/sec<br>> >> Max throughput per process = 19353.42 KB/sec<br>> >> Avg throughput per process = 19350.88 KB/sec<br>> >> Min xfer = 10483712.00 KB<br>> >><br>> >> Children see throughput for 4 re-readers = 77424.40
KB/sec<br>> >> Parent sees throughput for 4 re-readers = 77423.89
KB/sec<br>> >> Min throughput per process = 19354.75 KB/sec<br>> >> Max throughput per process = 19358.50 KB/sec<br>> >> Avg throughput per process = 19356.10 KB/sec<br>> >> Min xfer = 10483840.00 KB<br>> >><br>> >> then the use the same command test in the dir mounted with
glister fuse:<br>> >> File size set to 10485760 KB<br>> >> Record Size 128 KB<br>> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> >> Output is in Kbytes/sec<br>> >> Time Resolution = 0.000001 seconds.<br>> >> Processor cache size set to 1024 Kbytes.<br>> >> Processor cache line size set to 32 bytes.<br>> >> File stride size set to 17 * record size.<br>> >> Throughput test with 4 processes<br>> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>> >><br>> >> Children see throughput for 4 initial writers = 887534.72
KB/sec<br>> >> Parent sees throughput for 4 initial writers = 848830.39
KB/sec<br>> >> Min throughput per process = 220140.91 KB/sec<br>> >> Max throughput per process = 223690.45 KB/sec<br>> >> Avg throughput per process = 221883.68 KB/sec<br>> >> Min xfer = 10319360.00 KB<br>> >><br>> >> Children see throughput for 4 rewriters = 892774.92
KB/sec<br>> >> Parent sees throughput for 4 rewriters = 871186.83
KB/sec<br>> >> Min throughput per process = 222326.44 KB/sec<br>> >> Max throughput per process = 223970.17 KB/sec<br>> >> Avg throughput per process = 223193.73 KB/sec<br>> >> Min xfer = 10431360.00 KB<br>> >><br>> >> Children see throughput for 4 readers = 605889.12
KB/sec<br>> >> Parent sees throughput for 4 readers = 601767.96
KB/sec<br>> >> Min throughput per process = 143133.14 KB/sec<br>> >> Max throughput per process = 159550.88 KB/sec<br>> >> Avg throughput per process = 151472.28 KB/sec<br>> >> Min xfer = 9406848.00 KB<br>> >><br>> >> it shows much higher perf.<br>> >><br>> >> any places i did wrong?<br>> >><br>> >><br>> >> thank you<br>> >> -Kane<br>> >><br>> >> ? 2013-9-18???1:19?Anand Avati <avati@gluster.org<br>> >> <</font></tt><a href=mailto:avati@gluster.org><tt><font size=2>mailto:avati@gluster.org</font></tt></a><tt><font size=2>>>
???<br>> >><br>> >>> How are you testing this? What tool are you using?<br>> >>><br>> >>> Avati<br>> >>><br>> >>><br>> >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k@163.com<br>> >>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
wrote:<br>> >>><br>> >>> Hi Vijay<br>> >>><br>> >>> I used the code
in<br>> >>> </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with the lasted commit:<br>> >>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>> >>> Author: Anand Avati <avati@redhat.com
<</font></tt><a href=mailto:avati@redhat.com><tt><font size=2>mailto:avati@redhat.com</font></tt></a><tt><font size=2>>><br>> >>> Date: Tue Sep 17 16:45:03 2013 -0700<br>> >>><br>> >>> Revert "cluster/distribute:
Rebalance should also verify free<br>> >>> inodes"<br>> >>><br>> >>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>> >>><br>> >>> Realized soon after merging,
?.<br>> >>><br>> >>> which include the patch you mentioned last
time improve read perf,<br>> >>> written by Anand.<br>> >>><br>> >>> but the read perf was still slow:<br>> >>> write: 500MB/s<br>> >>> read: 77MB/s<br>> >>><br>> >>> while via fuse :<br>> >>> write 800MB/s<br>> >>> read 600MB/s<br>> >>><br>> >>> any advises?<br>> >>><br>> >>><br>> >>> Thank you.<br>> >>> -Kane<br>> >>><br>> >>> ? 2013-9-13???10:37?kane <stef_9k@163.com<br>> >>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
???<br>> >>><br>> >>>> Hi Vijay?<br>> >>>><br>> >>>> thank you for post this message,
i will try it soon<br>> >>>><br>> >>>> -kane<br>> >>>><br>> >>>><br>> >>>><br>> >>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com<br>> >>> <</font></tt><a href=mailto:vbellur@redhat.com><tt><font size=2>mailto:vbellur@redhat.com</font></tt></a><tt><font size=2>>>
???<br>> >>>><br>> >>>>> On 09/13/2013 06:10 PM, kane wrote:<br>> >>>>>> Hi<br>> >>>>>><br>> >>>>>> We use gluster samba vfs test io,but the
read performance via<br>> >>> vfs is<br>> >>>>>> half of write perfomance,<br>> >>>>>> but via fuse the read and write performance
is almost the same.<br>> >>>>>><br>> >>>>>> this is our smb.conf:<br>> >>>>>> [global]<br>> >>>>>> workgroup = MYGROUP<br>> >>>>>> server string = DCS
Samba Server<br>> >>>>>> log file = /var/log/samba/log.vfs<br>> >>>>>> max log size = 500000<br>> >>>>>> # use sendfile = true<br>> >>>>>> aio read size = 262144<br>> >>>>>> aio write size = 262144<br>> >>>>>> aio write behind = true<br>> >>>>>> min receivefile size
= 262144<br>> >>>>>> write cache size = 268435456<br>> >>>>>> security = user<br>> >>>>>> passdb backend = tdbsam<br>> >>>>>> load printers = yes<br>> >>>>>> cups options = raw<br>> >>>>>> read raw = yes<br>> >>>>>> write raw = yes<br>> >>>>>> max xmit = 262144<br>> >>>>>> socket options = TCP_NODELAY
IPTOS_LOWDELAY<br>> >>> SO_RCVBUF=262144<br>> >>>>>> SO_SNDBUF=262144<br>> >>>>>> kernel oplocks = no<br>> >>>>>> stat cache = no<br>> >>>>>><br>> >>>>>> any advises helpful?<br>> >>>>>><br>> >>>>><br>> >>>>> This patch has shown improvement in read performance
with libgfapi:<br>> >>>>><br>> >>>>> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>> >>>>><br>> >>>>> Would it be possible for you to try this patch
and check if it<br>> >>> improves performance in your case?<br>> >>>>><br>> >>>>> -Vijay<br>> >>>>><br>> >>>><br>> >>><br>> >>><br>> >>> _______________________________________________<br>> >>> Gluster-users mailing list<br>> >>> Gluster-users@gluster.org <</font></tt><a href="mailto:Gluster-users@gluster.org"><tt><font size=2>mailto:Gluster-users@gluster.org</font></tt></a><tt><font size=2>><br>> >>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> >>><br>> >>><br>> >><br>> ><br>> > Please add 'kernel oplocks = no' in the [gvol] section and try
again.<br>> ><br>> > Avati<br>> ><br>><br>><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/7acaaa03/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130917/7acaaa03/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 20<br>Date: Wed, 18 Sep 2013 09:17:19 +0200<br>From: Daniel M?ller <mueller@tropenklinik.de><br>To: "'Bobby Jacob'" <bobby.jacob@alshaya.com>,<br>
<gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Mounting same replica-volume on multiple<br>
clients.
????<br>Message-ID: <002e01ceb43f$1c8635a0$5592a0e0$@de><br>Content-Type: text/plain;
charset="iso-8859-1"<br><br>What about <br>gluster volume info on both nodes!?<br><br>Ex.:<br>Volume Name: sambacluster<br>Type: Replicate<br>Volume ID: 4fd0da03-8579-47cc-926b-d7577dac56cf<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: s4master:/raid5hs/glusterfs/samba<br>Brick2: s4slave:/raid5hs/glusterfs/samba<br>Options Reconfigured:<br>network.ping-timeout: 5<br>performance.quick-read: on<br><br>What are telling you your log files?<br><br>-----------------------------------------------<br>EDV Daniel M?ller<br><br>Leitung EDV<br>Tropenklinik Paul-Lechler-Krankenhaus<br>Paul-Lechler-Str. 24<br>72076 T?bingen<br><br>Tel.: 07071/206-463, Fax: 07071/206-499<br>eMail: mueller@tropenklinik.de<br>Internet: </font></tt><a href=www.tropenklinik.de><tt><font size=2>www.tropenklinik.de</font></tt></a><tt><font size=2><br>-----------------------------------------------<br>-----Urspr?ngliche Nachricht-----<br>Von: Bobby Jacob [</font></tt><a href=mailto:bobby.jacob@alshaya.com><tt><font size=2>mailto:bobby.jacob@alshaya.com</font></tt></a><tt><font size=2>]
<br>Gesendet: Mittwoch, 18. September 2013 07:48<br>An: mueller@tropenklinik.de; gluster-users@gluster.org<br>Betreff: RE: [Gluster-users] Mounting same replica-volume on multiple<br>clients. ????<br><br>Exactly. !! BUT I am writing through the volume mount-point from the<br>clients. !! NOT directly into the bricks. !!<br><br>I'm using GlusterFS 3.3.2 with Centos6.4 . !<br><br>Thanks & Regards,<br>Bobby Jacob<br><br>-----Original Message-----<br>From: Daniel M?ller [</font></tt><a href=mailto:mueller@tropenklinik.de><tt><font size=2>mailto:mueller@tropenklinik.de</font></tt></a><tt><font size=2>]
<br>Sent: Wednesday, September 18, 2013 8:46 AM<br>To: Bobby Jacob; gluster-users@gluster.org<br>Subject: AW: [Gluster-users] Mounting same replica-volume on multiple<br>clients. ????<br><br>Hello,<br>this ist he behavior as if you write directly into the glusterd<br>directory/partition and not to the remounted replicating bricks!?<br><br><br>EDV Daniel M?ller<br><br>Leitung EDV<br>Tropenklinik Paul-Lechler-Krankenhaus<br>Paul-Lechler-Str. 24<br>72076 T?bingen<br>Tel.: 07071/206-463, Fax: 07071/206-499<br>eMail: mueller@tropenklinik.de<br>Internet: </font></tt><a href=www.tropenklinik.de><tt><font size=2>www.tropenklinik.de</font></tt></a><tt><font size=2><br><br>Von: gluster-users-bounces@gluster.org<br>[</font></tt><a href="mailto:gluster-users-bounces@gluster.org"><tt><font size=2>mailto:gluster-users-bounces@gluster.org</font></tt></a><tt><font size=2>]
Im Auftrag von Bobby Jacob<br>Gesendet: Mittwoch, 18. September 2013 07:36<br>An: gluster-users@gluster.org<br>Betreff: [Gluster-users] Mounting same replica-volume on multiple clients.<br>????<br><br>HI,<br><br>I have 2 gluster nodes (GFS01/GFS02) each with a single brick (B01/B01).
I<br>have created a simple replica volume with these bricks. <br>Bricks ?? : GFS01/B01 and GFS02/B01.<br>Volume: TestVol<br><br>I have 2 clients (C01/C02) which will mount this "testvol" for
simultaneous<br>read/write. The 2 clients run the same application which is load-balanced,<br>so user request are end to both the client servers which reads/writes data<br>to both the same volume.<br><br>Mounting the volume on C1?????? :?????????????? mount -t glusterfs -o<br>backupvolfile-server=GFS02 GFS01:/testvol /data Mounting the volume on<br>C2?????? :?????????????? mount -t glusterfs -o<br>backupvolfile-server=GFS01 GFS02:/testvol /data<br><br>Is this the appropriate way to be followed.? <br><br>At times, I notice that when I write data through C1-mount point the data
is<br>written only to GFS01/B01 and if data is written through C2-mount point
the<br>data is written only to GFS02/B01.<br><br>Please advise. !!<br><br><br>Thanks & Regards,<br>Bobby Jacob<br><br><br><br><br>------------------------------<br><br>Message: 21<br>Date: Wed, 18 Sep 2013 16:27:28 +0800<br>From: kane <stef_9k@163.com><br>To: Anand Avati <avati@gluster.org><br>Cc: gluster-users <gluster-users@gluster.org>, Anand Avati<br>
<avati@redhat.com><br>Subject: Re: [Gluster-users] Gluster samba vfs read performance slow<br>Message-ID: <00411A37-CF9E-4598-8BC3-5A080B0B4766@163.com><br>Content-Type: text/plain; charset="iso-2022-jp"<br><br>I compared the profile dumps while write and read is separately running;<br><br>writing:<br>------------------------------------------------<br>Interval 58 Stats:<br> Block Size: 65536b+
131072b+ <br> No. of Reads:
0
0 <br>No. of Writes: 27120
10500 <br> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop<br> --------- ----------- ----------- -----------
------------ ----<br> 100.00 133.51 us 36.00
us 1339.00 us 37619
WRITE<br> <br> Duration: 12 seconds<br> Data Read: 0 bytes<br>Data Written: 3153854464 bytes<br>------------------------------------------------<br><br><br>read:<br>------------------------------------------------<br>Interval 63 Stats:<br> Block Size: 131072b+
<br> No. of Reads:
3529 <br>No. of Writes:
0 <br> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop<br> --------- ----------- ----------- -----------
------------ ----<br> 0.54 87.86 us 68.00
us 127.00 us 42
FSTAT<br> 99.46 193.68 us 89.00
us 2121.00 us 3529
READ<br> <br> Duration: 12 seconds<br> Data Read: 462553088 bytes<br>Data Written: 0 bytes<br>------------------------------------------------<br><br><br><br>two server brick avg dumps:<br>================================<br>Brick: 192.168.101.133:/dcsdata/d0<br>----------------------------------<br>Cumulative Stats:<br> Block Size: 8192b+
16384b+
32768b+ <br> No. of Reads:
0
0
0 <br>No. of Writes:
2
1
1 <br> <br> Block Size: 65536b+
131072b+
262144b+ <br> No. of Reads:
0 1613832
0
<br>No. of Writes: 2282474
1148962
227 <br> <br> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop<br> --------- ----------- ----------- -----------
------------ ----<br> 0.00 0.00 us
0.00 us 0.00 us
14 FORGET<br> 0.00 0.00 us
0.00 us 0.00 us
39 RELEASE<br> 0.00 0.00 us
0.00 us 0.00 us
114 RELEASEDIR<br> 0.00 84.50 us 54.00
us 115.00 us 2
OPENDIR<br> 0.00 79.00 us 52.00
us 127.00 us 4
OPEN<br> 0.00 47.00 us 14.00
us 130.00 us 8
FLUSH<br> 0.00 342.00 us 311.00
us 373.00 us 2
CREATE<br> 0.00 104.77 us 26.00
us 281.00 us 13
STATFS<br> 0.01 131.75 us 35.00
us 285.00 us 93
LOOKUP<br> 0.02 7446.00 us 104.00
us 29191.00 us 4
READDIRP<br> 0.07 2784.89 us 49.00
us 49224.00 us 36
GETXATTR<br> 0.20 64.49 us 29.00
us 164.00 us 4506
FSTAT<br> 1.07 399482.25 us 361616.00 us 450370.00
us 4 UNLINK<br> 42.87 167.36 us 56.00
us 44827.00 us 381080
READ<br> 55.76 71.51 us 35.00
us 7032.00 us 1159912
WRITE<br> <br> Duration: 22156 seconds<br> Data Read: 211528187904 bytes<br>Data Written: 300276908032 bytes<br> <br>Interval 71 Stats:<br> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop<br> --------- ----------- ----------- -----------
------------ ----<br> 0.00 0.00 us
0.00 us 0.00 us
1 RELEASEDIR<br> 0.18 54.00 us 54.00
us 54.00 us
1 OPENDIR<br> 1.05 107.33 us 40.00
us 217.00 us 3
STATFS<br> 2.90 126.57 us 81.00
us 256.00 us 7
LOOKUP<br> 95.88 14669.00 us 147.00 us
29191.00 us 2 READDIRP<br> <br> Duration: 581 seconds<br> Data Read: 0 bytes<br>Data Written: 0 bytes<br> <br>Brick: 192.168.101.134:/dcsdata/d0<br>----------------------------------<br>Cumulative Stats:<br> Block Size: 8192b+
16384b+
32768b+ <br> No. of Reads:
0
0
0 <br>No. of Writes:
2
3
24 <br> <br> Block Size: 65536b+
131072b+
262144b+ <br> No. of Reads:
22 1563063
0 <br>No. of Writes: 1522412
1525007
184 <br> <br> %-latency Avg-latency Min-Latency Max-Latency
No. of calls Fop<br> --------- ----------- ----------- -----------
------------ ----<br> 0.00 0.00 us
0.00 us 0.00 us
14 FORGET<br> 0.00 0.00 us
0.00 us 0.00 us
39 RELEASE<br> 0.00 0.00 us
0.00 us 0.00 us
114 RELEASEDIR<br> 0.00 116.50 us 111.00
us 122.00 us 2
OPENDIR<br> 0.00 69.25 us 23.00
us 95.00 us
8 FLUSH<br> 0.00 418.00 us 285.00
us 551.00 us 2
CREATE<br> 0.00 239.25 us 101.00
us 396.00 us 4
READDIRP<br> 0.00 93.00 us 39.00
us 249.00 us 13
STATFS<br> 0.01 142.89 us 78.00
us 241.00 us 87
LOOKUP<br> 0.09 48402.25 us 114.00 us
99173.00 us 4
OPEN<br> 0.19 10974.42 us 60.00
us 345979.00 us 36
GETXATTR<br> 0.20 94.33 us 41.00
us 200.00 us 4387
FSTAT<br> 0.85 440436.25 us 381525.00 us 582989.00
us 4 UNLINK<br> 35.80 193.96 us 57.00
us 23312.00 us 380869
READ<br> 62.86 134.89 us 29.00
us 9976.00 us 961593
WRITE<br> <br> Duration: 22155 seconds<br> Data Read: 204875400152 bytes<br>Data Written: 299728837956 bytes<br>================================<br><br><br>Kane<br><br>? 2013-9-18???2:45?Anand Avati <avati@gluster.org> ???<br><br>> Can you get the volume profile dumps for both the runs and compare
them?<br>> <br>> Avati<br>> <br>> <br>> <br>> On Tue, Sep 17, 2013 at 10:46 PM, kane <stef_9k@163.com> wrote:<br>> I have already used "kernel oplocks = no" in the smb.conf,
next is my original smb.conf file global settings:<br>> [global]<br>> workgroup = MYGROUP<br>> server string = DCS Samba Server<br>> log file = /var/log/samba/log.vfs<br>> max log size = 500000<br>> aio read size = 262144<br>> aio write size = 262144</font></tt><br><tt><font size=2>> aio write behind
= true<br>> security = user<br>> passdb backend = tdbsam<br>> load printers = yes<br>> cups options = raw<br>> read raw = yes<br>> write raw = yes<br>> max xmit = 262144<br>> socket options = TCP_NODELAY IPTOS_LOWDELAY
SO_RCVBUF=262144 SO_SNDBUF=262144<br>> # max protocol = SMB2<br>> kernel oplocks = no<br>> stat cache = no<br>> <br>> thank you<br>> -Kane<br>> ? 2013-9-18???1:38?Anand Avati <avati@redhat.com> ???<br>> <br>> > On 9/17/13 10:34 PM, kane wrote:<br>> >> Hi Anand,<br>> >><br>> >> I use 2 gluster server , this is my volume info:<br>> >> Volume Name: soul<br>> >> Type: Distribute<br>> >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6<br>> >> Status: Started<br>> >> Number of Bricks: 2<br>> >> Transport-type: tcp<br>> >> Bricks:<br>> >> Brick1: 192.168.101.133:/dcsdata/d0<br>> >> Brick2: 192.168.101.134:/dcsdata/d0<br>> >><br>> >> each brick use a raid 5 logic disk with 8*2TSATA hdd.<br>> >><br>> >> smb.conf:<br>> >> [gvol]<br>> >> comment = For samba export of
volume test<br>> >> vfs objects = glusterfs<br>> >> glusterfs:volfile_server = localhost<br>> >> glusterfs:volume = soul<br>> >> path = /<br>> >> read only = no<br>> >> guest ok = yes<br>> >><br>> >> this my testparm result:<br>> >> [global]<br>> >> workgroup = MYGROUP<br>> >> server string = DCS Samba Server<br>> >> log file = /var/log/samba/log.vfs<br>> >> max log size = 500000<br>> >> max xmit = 262144<br>> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144<br>> >> SO_SNDBUF=262144<br>> >> stat cache = No<br>> >> kernel oplocks = No<br>> >> idmap config * : backend = tdb<br>> >> aio read size = 262144<br>> >> aio write size = 262144<br>> >> aio write behind = true<br>> >> cups options = raw<br>> >><br>> >> in client mount the smb share with cifs to dir /mnt/vfs,<br>> >> then use iozone executed in the cifs mount dir "/mnt/vfs":<br>> >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> >> File size set to 10485760 KB<br>> >> Record Size 128 KB<br>> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> >> Output is in Kbytes/sec<br>> >> Time Resolution = 0.000001 seconds.<br>> >> Processor cache size set to 1024 Kbytes.<br>> >> Processor cache line size set to 32 bytes.<br>> >> File stride size set to 17 * record size.<br>> >> Throughput test with 4 processes<br>> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>> >><br>> >> Children see throughput for 4 initial writers = 534315.84
KB/sec<br>> >> Parent sees throughput for 4 initial writers = 519428.83
KB/sec<br>> >> Min throughput per process = 133154.69 KB/sec<br>> >> Max throughput per process = 134341.05 KB/sec<br>> >> Avg throughput per process = 133578.96 KB/sec<br>> >> Min xfer = 10391296.00 KB<br>> >><br>> >> Children see throughput for 4 rewriters = 536634.88
KB/sec<br>> >> Parent sees throughput for 4 rewriters = 522618.54
KB/sec<br>> >> Min throughput per process = 133408.80 KB/sec<br>> >> Max throughput per process = 134721.36 KB/sec<br>> >> Avg throughput per process = 134158.72 KB/sec<br>> >> Min xfer = 10384384.00 KB<br>> >><br>> >> Children see throughput for 4 readers = 77403.54
KB/sec<br>> >> Parent sees throughput for 4 readers = 77402.86
KB/sec<br>> >> Min throughput per process = 19349.42 KB/sec<br>> >> Max throughput per process = 19353.42 KB/sec<br>> >> Avg throughput per process = 19350.88 KB/sec<br>> >> Min xfer = 10483712.00 KB<br>> >><br>> >> Children see throughput for 4 re-readers = 77424.40
KB/sec<br>> >> Parent sees throughput for 4 re-readers = 77423.89
KB/sec<br>> >> Min throughput per process = 19354.75 KB/sec<br>> >> Max throughput per process = 19358.50 KB/sec<br>> >> Avg throughput per process = 19356.10 KB/sec<br>> >> Min xfer = 10483840.00 KB<br>> >><br>> >> then the use the same command test in the dir mounted with
glister fuse:<br>> >> File size set to 10485760 KB<br>> >> Record Size 128 KB<br>> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4<br>> >> Output is in Kbytes/sec<br>> >> Time Resolution = 0.000001 seconds.<br>> >> Processor cache size set to 1024 Kbytes.<br>> >> Processor cache line size set to 32 bytes.<br>> >> File stride size set to 17 * record size.<br>> >> Throughput test with 4 processes<br>> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records<br>> >><br>> >> Children see throughput for 4 initial writers = 887534.72
KB/sec<br>> >> Parent sees throughput for 4 initial writers = 848830.39
KB/sec<br>> >> Min throughput per process = 220140.91 KB/sec<br>> >> Max throughput per process = 223690.45 KB/sec<br>> >> Avg throughput per process = 221883.68 KB/sec<br>> >> Min xfer = 10319360.00 KB<br>> >><br>> >> Children see throughput for 4 rewriters = 892774.92
KB/sec<br>> >> Parent sees throughput for 4 rewriters = 871186.83
KB/sec<br>> >> Min throughput per process = 222326.44 KB/sec<br>> >> Max throughput per process = 223970.17 KB/sec<br>> >> Avg throughput per process = 223193.73 KB/sec<br>> >> Min xfer = 10431360.00 KB<br>> >><br>> >> Children see throughput for 4 readers = 605889.12
KB/sec<br>> >> Parent sees throughput for 4 readers = 601767.96
KB/sec<br>> >> Min throughput per process = 143133.14 KB/sec<br>> >> Max throughput per process = 159550.88 KB/sec<br>> >> Avg throughput per process = 151472.28 KB/sec<br>> >> Min xfer = 9406848.00 KB<br>> >><br>> >> it shows much higher perf.<br>> >><br>> >> any places i did wrong?<br>> >><br>> >><br>> >> thank you<br>> >> -Kane<br>> >><br>> >> ? 2013-9-18???1:19?Anand Avati <avati@gluster.org<br>> >> <</font></tt><a href=mailto:avati@gluster.org><tt><font size=2>mailto:avati@gluster.org</font></tt></a><tt><font size=2>>>
???<br>> >><br>> >>> How are you testing this? What tool are you using?<br>> >>><br>> >>> Avati<br>> >>><br>> >>><br>> >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k@163.com<br>> >>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
wrote:<br>> >>><br>> >>> Hi Vijay<br>> >>><br>> >>> I used the code
in<br>> >>> </font></tt><a href=https://github.com/gluster/glusterfs.git><tt><font size=2>https://github.com/gluster/glusterfs.git</font></tt></a><tt><font size=2>with the lasted commit:<br>> >>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a<br>> >>> Author: Anand Avati <avati@redhat.com
<</font></tt><a href=mailto:avati@redhat.com><tt><font size=2>mailto:avati@redhat.com</font></tt></a><tt><font size=2>>><br>> >>> Date: Tue Sep 17 16:45:03 2013 -0700<br>> >>><br>> >>> Revert "cluster/distribute:
Rebalance should also verify free<br>> >>> inodes"<br>> >>><br>> >>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa<br>> >>><br>> >>> Realized soon after merging,
?.<br>> >>><br>> >>> which include the patch you mentioned last
time improve read perf,<br>> >>> written by Anand.<br>> >>><br>> >>> but the read perf was still slow:<br>> >>> write: 500MB/s<br>> >>> read: 77MB/s<br>> >>><br>> >>> while via fuse :<br>> >>> write 800MB/s<br>> >>> read 600MB/s<br>> >>><br>> >>> any advises?<br>> >>><br>> >>><br>> >>> Thank you.<br>> >>> -Kane<br>> >>><br>> >>> ? 2013-9-13???10:37?kane <stef_9k@163.com<br>> >>> <</font></tt><a href=mailto:stef_9k@163.com><tt><font size=2>mailto:stef_9k@163.com</font></tt></a><tt><font size=2>>>
???<br>> >>><br>> >>>> Hi Vijay?<br>> >>>><br>> >>>> thank you for post this message,
i will try it soon<br>> >>>><br>> >>>> -kane<br>> >>>><br>> >>>><br>> >>>><br>> >>>> ? 2013-9-13???9:21?Vijay Bellur <vbellur@redhat.com<br>> >>> <</font></tt><a href=mailto:vbellur@redhat.com><tt><font size=2>mailto:vbellur@redhat.com</font></tt></a><tt><font size=2>>>
???<br>> >>>><br>> >>>>> On 09/13/2013 06:10 PM, kane wrote:<br>> >>>>>> Hi<br>> >>>>>><br>> >>>>>> We use gluster samba vfs test io,but the
read performance via<br>> >>> vfs is<br>> >>>>>> half of write perfomance,<br>> >>>>>> but via fuse the read and write performance
is almost the same.<br>> >>>>>><br>> >>>>>> this is our smb.conf:<br>> >>>>>> [global]<br>> >>>>>> workgroup = MYGROUP<br>> >>>>>> server string = DCS
Samba Server<br>> >>>>>> log file = /var/log/samba/log.vfs<br>> >>>>>> max log size = 500000<br>> >>>>>> # use sendfile = true<br>> >>>>>> aio read size = 262144<br>> >>>>>> aio write size = 262144<br>> >>>>>> aio write behind = true<br>> >>>>>> min receivefile size
= 262144<br>> >>>>>> write cache size = 268435456<br>> >>>>>> security = user<br>> >>>>>> passdb backend = tdbsam<br>> >>>>>> load printers = yes<br>> >>>>>> cups options = raw<br>> >>>>>> read raw = yes<br>> >>>>>> write raw = yes<br>> >>>>>> max xmit = 262144<br>> >>>>>> socket options = TCP_NODELAY
IPTOS_LOWDELAY<br>> >>> SO_RCVBUF=262144<br>> >>>>>> SO_SNDBUF=262144<br>> >>>>>> kernel oplocks = no<br>> >>>>>> stat cache = no<br>> >>>>>><br>> >>>>>> any advises helpful?<br>> >>>>>><br>> >>>>><br>> >>>>> This patch has shown improvement in read performance
with libgfapi:<br>> >>>>><br>> >>>>> </font></tt><a href=http://review.gluster.org/#/c/5897/><tt><font size=2>http://review.gluster.org/#/c/5897/</font></tt></a><tt><font size=2><br>> >>>>><br>> >>>>> Would it be possible for you to try this patch
and check if it<br>> >>> improves performance in your case?<br>> >>>>><br>> >>>>> -Vijay<br>> >>>>><br>> >>>><br>> >>><br>> >>><br>> >>> _______________________________________________<br>> >>> Gluster-users mailing list<br>> >>> Gluster-users@gluster.org <</font></tt><a href="mailto:Gluster-users@gluster.org"><tt><font size=2>mailto:Gluster-users@gluster.org</font></tt></a><tt><font size=2>><br>> >>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> >>><br>> >>><br>> >><br>> ><br>> > Please add 'kernel oplocks = no' in the [gvol] section and try
again.<br>> ><br>> > Avati<br>> ><br>> <br>> <br>> <br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/ab613dc7/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/ab613dc7/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 22<br>Date: Wed, 18 Sep 2013 11:15:21 +0200<br>From: Luk?? Bezdi?ka <lukas.bezdicka@gooddata.com><br>To: Vijay Bellur <vbellur@redhat.com><br>Cc: Emmanuel Dreyfus <manu@netbsd.org>,
"gluster-users@gluster.org"<br>
<gluster-users@gluster.org>,
Gluster Devel <gluster-devel@nongnu.org><br>Subject: Re: [Gluster-users] [Gluster-devel] glusterfs-3.4.1qa2<br>
released<br>Message-ID:<br>
<CAEePdhkatO15oh6JajVogKfhY4cK-P=4G9u6H4wtVTTtO9q+1A@mail.gmail.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Tested with glusterfs-3.4.1qa2-1.el6.x86_64 issue with ACL is still there,<br>unless one applies patch from </font></tt><a href=http://review.gluster.org/#/c/5693/><tt><font size=2>http://review.gluster.org/#/c/5693/</font></tt></a><tt><font size=2>which<br>shoots through the caches and takes ACLs from server or sets<br>entry-timeout=0 it returns wrong values. This is probably because ACL mask<br>being applied incorrectly in posix_acl_inherit_mode, but I'm no C expert
to<br>say so :(<br><br><br>On Mon, Sep 16, 2013 at 9:37 AM, Vijay Bellur <vbellur@redhat.com>
wrote:<br><br>> On 09/16/2013 05:47 AM, Emmanuel Dreyfus wrote:<br>><br>>> Emmanuel Dreyfus <manu@netbsd.org> wrote:<br>>><br>>> It seems it has something very broken. A tar -xzf reported
no error<br>>>> while most of the file do not appear in the filesystem on
completion.<br>>>><br>>><br>>> I am not sure but it seems files appeared afterwards. There was
a system<br>>> clock problem, client being a few seconds in the past. Can it
explain<br>>> the problem?<br>>><br>><br>> When the client's time is off and fuse has entry-timeout set to 0,
is this<br>> problem seen?<br>><br>> The fuse forget log message is benign and does not have any functional<br>> impact. I have sent a patch which addresses this behavior:<br>><br>> </font></tt><a href=http://review.gluster.org/#/c/**5932/><tt><font size=2>http://review.gluster.org/#/c/**5932/</font></tt></a><tt><font size=2><</font></tt><a href=http://review.gluster.org/#/c/5932/><tt><font size=2>http://review.gluster.org/#/c/5932/</font></tt></a><tt><font size=2>><br>><br>> -Vijay<br>><br>><br>><br>> ______________________________**_________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.**org/mailman/listinfo/gluster-**users"><tt><font size=2>http://supercolony.gluster.**org/mailman/listinfo/gluster-**users</font></tt></a><tt><font size=2><</font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2>><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/260d04d5/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/260d04d5/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 23<br>Date: Wed, 18 Sep 2013 11:35:26 +0200 (CEST)<br>From: Dragon <Sunghost@gmx.de><br>To: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Cant see files after network failure<br>Message-ID:<br>
<trinity-454d5481-7606-49b1-977b-9619577d4820-1379496926132@3capp-gmx-bs16><br>
<br>Content-Type: text/plain; charset="us-ascii"<br><br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/f3e5de21/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/f3e5de21/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 24<br>Date: Wed, 18 Sep 2013 10:35:58 +0100<br>From: Nux! <nux@li.nux.ro><br>To: Shishir Gowda <sgowda@redhat.com><br>Cc: Gluster Users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] gluster volume top issue<br>Message-ID: <2f58eea8b9dfc00fb8efe932acedc83b@li.nux.ro><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br>On 18.09.2013 04:39, Shishir Gowda wrote:<br>> Hi Nux,<br>> <br>> I am trying to see if the issue of "0" open fd is based
on the<br>> work-load, or a bug.<br>> <br>> Could you check top command output of "read/write" operation
too?<br><br>Shishir,<br><br>Those commands only output the bricks and nothing more:<br><br>[root@2216 ~]# gluster volume top xenvms read nfs<br>NFS Server : localhost<br>NFS Server : 1726.stocare.domeniu.net<br>NFS Server : 1613.stocare.domeniu.net<br>NFS Server : 1631.stocare.domeniu.net<br><br>[root@2216 ~]# gluster volume top xenvms write nfs<br>NFS Server : localhost<br>NFS Server : 1631.stocare.domeniu.net<br>NFS Server : 1613.stocare.domeniu.net<br>NFS Server : 1726.stocare.domeniu.net<br><br>Same without "nfs".<br><br>-- <br>Sent from the Delta quadrant using Borg technology!<br><br>Nux!<br></font></tt><a href=www.nux.ro><tt><font size=2>www.nux.ro</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 25<br>Date: Wed, 18 Sep 2013 11:01:36 +0100<br>From: "Michael.OBrien" <Michael.OBrien@ul.ie><br>To: <gluster-users@gluster.org><br>Subject: [Gluster-users] Secure Setup / Separate GlusterFS /<br>
Encryption<br>Message-ID:<br>
<2A20FC0CEBD54B4D98950B3B5B1D99FE01400256@staffexchange3.ul.campus><br>Content-Type: text/plain; charset="us-ascii"<br><br>Hi Gluster Users,<br><br> <br><br>I'm looking for some advice or best practice recommendations when it<br>comes to designing secure glusterFS environments. I'm talking about the<br>basic design principles that a user should consider irrespective of the<br>content that will be stored. I realise security isn't a destination but<br>a journey but I'd appreciate any advice you may have and it goes without<br>saying that if the content is that important it should be separated.<br><br> <br><br>What is the current advise on configuring secure glusterFS environments<br>or the trade-offs to consider? <br><br> <br><br>Should everything from bricks to storage nodes and the storage data<br>network be separated into different glusterFS's or can I share storage<br>nodes across different clients without fear of crossed wires or a rogue<br>client being able to list the other mount points of other clients or<br>worse access their data? <br><br> <br><br>My mindset would be to try and compare it to a SAN (but I'm not a SAN<br>guy either) where disk storage is pooled and provisioned as LUN's and<br>the LUN's are presented to certain HBA's . The SAN can be configured so<br>that only particular HBA's can access a LUN so even if the client is<br>compromised the SAN doesn't allow it to access other LUN's<br><br> <br><br>Finally also on the topic of security how would people suggest handling<br>encryption of client data and working with a storage server hosting<br>different encrypted data<br><br> <br><br>Michael<br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/6429d503/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/6429d503/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 26<br>Date: Wed, 18 Sep 2013 07:36:32 -0400 (EDT)<br>From: Krishnan Parthasarathi <kparthas@redhat.com><br>To: Dragon <Sunghost@gmx.de><br>Cc: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Cant see files after network failure<br>Message-ID:<br>
<569256344.14871568.1379504192724.JavaMail.root@redhat.com><br>Content-Type: text/plain; charset=utf-8<br><br>Dragon,<br><br>Could you attach brick log files, client log file(s) and output of the
following<br>commands,<br>gluster volume info VOLNAME<br>gluster volume status VOLNAME<br><br>Could you attach the "etc-glusterfs.." log as well?<br><br>thanks,<br>krish<br><br>----- Original Message -----<br>> Hello,<br>> i didnt find any hint of an error. Now i restart all server and watched
the<br>> "etc-glusterfs.." log. The only thing i found is: "rpc
actor failed to<br>> complete successfully". All peers looks good and the volume too.
i can see<br>> the files in the data folder of each brick, but after fuse mount on
a<br>> client, i cant see anything. permissions at the files on each brick
are<br>> root:root.<br>> What can i do?<br>> <br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 27<br>Date: Wed, 18 Sep 2013 13:51:52 +0200 (CEST)<br>From: Dragon <Sunghost@gmx.de><br>To: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Cant see files after network failure<br>Message-ID:<br>
<trinity-319be117-5bab-441b-9c7c-7794b2523d3a-1379505112825@3capp-gmx-bs56><br>
<br>Content-Type: text/plain; charset="us-ascii"<br><br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/2349a003/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/2349a003/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br>End of Gluster-users Digest, Vol 65, Issue 18<br>*********************************************<br></font></tt><br><BR>
**<BR>
<BR>
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.<BR>
**<BR>