<font size=2 face="sans-serif">Can someone please help me interpret this
from my client: I'm not fully understanding the matrix. I have
a 4 server 2x2 distri.-rep brick.</font><br><br><br><font size=2 face="sans-serif">[2014-02-19 12:15:15.921993] E [afr-self-heal-common.c:197:afr_sh_print_split_brain_log]
0-devstatic-replicate-0: Unable to self-heal contents of '/employees/htdocs/software/contact_mstr/secure/app/views/.svn/props'
(possible split-brain). Please delete the file from all but the preferred
subvolume.- Pending matrix: [ [ 0 2 ] [ 3 0 ] ]</font><br><font size=2 face="sans-serif"><br><br>Khoi<br><br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">gluster-users-request@gluster.org</font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gluster-users@gluster.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">02/19/2014 05:59 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Gluster-users
Digest, Vol 70, Issue 18</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gluster-users-bounces@gluster.org</font><br><hr noshade><br><br><br><tt><font size=2>Send Gluster-users mailing list submissions to<br>
gluster-users@gluster.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br>
</font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>or, via email, send a message with subject or body 'help' to<br>
gluster-users-request@gluster.org<br><br>You can reach the person managing the list at<br>
gluster-users-owner@gluster.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Gluster-users digest..."<br><br><br>Today's Topics:<br><br> 1. Complete machine lockup, v3.4.2 (Laurent Chouinard)<br> 2. Want to Ask (Prian Dwiatma)<br> 3. Re: [Bug 1057645] ownership of diskimage changes during<br> livemigration, livemigration with kvm/libvirt fails
(Adam Huffman)<br> 4. Re: [Bug 1057645] ownership of diskimage changes during<br> livemigration, livemigration with kvm/libvirt fails
(Paul Boven)<br> 5. Re: Complete machine lockup, v3.4.2 (James)<br> 6. Re: Problem with duplicate files (Justin Dossey)<br> 7. upgrading from gluster-3.2.6 to gluster-3.4.2 (Dmitry Kopelevich)<br> 8. Problems to work with mounted directory in Gluster
3.2.7<br> (Targino Silveira)<br> 9. Re: Node down and volumes unreachable (Marco Zanger)<br> 10. Re: <host> not in 'Peer in Cluster' state (Jon Cope)<br> 11. Re: [Bug 1057645] ownership of diskimage changes
during<br> livemigration, livemigration with kvm/libvirt fails
(bernhard glomm)<br> 12. Re: [Bug 1057645] ownership of diskimage changes during<br> livemigration, livemigration with kvm/libvirt fails
(Josh Boon)<br> 13. 3.4 gluster volume heal-failed recovery (Steve Dainard)<br> 14. Re: add-brick and fix-layout takes some VMs offline<br> (Nicholas Majeran)<br> 15. Re: Problems to work with mounted directory in Gluster 3.2.7<br> (Franco Broi)<br> 16. Re: Problems to work with mounted directory in Gluster 3.2.7<br> (Targino Silveira)<br> 17. Agenda for Community meeting today (Vijay Bellur)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Tue, 18 Feb 2014 08:12:33 -0500<br>From: Laurent Chouinard <laurent.chouinard@ubisoft.com><br>To: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: [Gluster-users] Complete machine lockup, v3.4.2<br>Message-ID:<br>
<8CFDF989A34FE54FA678C09C6D9F6D39B560BC342B@MDC-MAIL-CMS01.ubisoft.org><br>
<br>Content-Type: text/plain; charset="us-ascii"<br><br>Hi everyone,<br><br>We've been using 3.3.2 for a while, and recently started to migrate to
3.4.2. We run on platform CentOS 6.5 for 3.4.2 (while 3.3.2 were installed
on CentOS 6.4)<br><br>Recently, we've have a very scary condition happen and we do not know exactly
the cause of it.<br><br>We have a 3 nodes cluster with a replication factor of 3. Each node has
one brick, which is made out of one RAID0 volume, comprised of multiple
SSDs.<br><br>Following some read/write errors, nodes 2 and 3 have completely locked.
Nothing could be done physically (nothing on the screen, nothing by SSH),
physical power cycle had to be done. Node 1 was still accessible, but its
fuse client rejected most if not all reads and writes.<br><br>Has anyone experienced something similar?<br><br>Before the system freeze, the last thing the kernel seemed to be doing
is killing HTTPD threads (INFO: task httpd:7910 blocked for more than 120
seconds.) End-users talk to Apache in order to read/write from the
Gluster volume, so it seems a simple case of "something wrong"
with gluster which locks read/writes, and eventually the kernel kills them.<br><br>At this point, we're unsure where to look. Nothing very specific can be
found in the logs, but perhaps if someone has pointers of what to look
for, that could give us a new search track.<br><br>Thanks<br><br>Laurent Chouinard<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/c03f9e09/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/c03f9e09/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 2<br>Date: Tue, 18 Feb 2014 10:24:48 +0700<br>From: Prian Dwiatma <priandwiatma@gmail.com><br>To: gluster-users@gluster.org<br>Subject: [Gluster-users] Want to Ask<br>Message-ID:<br>
<CAAN9Jt2v2a2ftNZNUhBztZpLgqhs=drGiYoxYUhXxEbi9YwB2A@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>hello....i'm Prian<br>I want to ask about glusterfs, is there any file types that affect Gluster<br>for data transmission?<br>thank you.<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/d9454deb/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/d9454deb/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 3<br>Date: Tue, 18 Feb 2014 13:59:10 +0000<br>From: Adam Huffman <adam.huffman@gmail.com><br>To: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage<br>
changes during livemigration, livemigration with kvm/libvirt fails<br>Message-ID:<br>
<CAP5prOjzhC1fBqYtpjSnFsfT1xQkDWCGzkX2e_JErq+oW1R1pQ@mail.gmail.com><br>Content-Type: text/plain; charset=ISO-8859-1<br><br>Hi Paul,<br><br>Could you keep the list updated? That bug has been marked private, so<br>I can't see it.<br><br>Best Wishes,<br>Adam<br><br>On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven@jive.nl> wrote:<br>> Hi Bernhard, everyone,<br>><br>> The same problem has now been reproduced on RedHat, please see:<br>><br>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1058032"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1058032</font></tt></a><tt><font size=2><br>><br>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it
broke<br>> when the packages were upgraded to 3.4.1.<br>><br>> I've set AppArmor to 'complain' as part of the debugging, so that's
not the<br>> issue.<br>><br>> I'm still not convinced that the file ownership itself is the root
cause of<br>> this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy<br>> to start a VM when its image file is owned root:root, and change ownership<br>> to libvirt-qemu:kvm. So I see no reason why it couldn't do the same
during a<br>> live migration.<br>><br>> In my opinion the real issue is the failure at the fuse level, that
makes<br>> file access to the image on the destination impossible, even for root.<br>><br>> Regards, Paul Boven.<br>><br>><br>> On 01/27/2014 07:51 PM, BGM wrote:<br>>><br>>> Hi Paul & all<br>>> I'm really keen on getting this solved,<br>>> right now it's a nasty show stopper.<br>>> I could try different gluster versions,<br>>> as long as I can get the .debs for it,<br>>> wouldn't want to start compiling<br>>> (although.... does a config option have changed on package build?)<br>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?<br>>> code diff, config options for package build.<br>>> Another approach: can anyone verify or falsify<br>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1057645"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1057645</font></tt></a><tt><font size=2><br>>> on another distro than ubuntu/debian?<br>>> thinking of it... could it be an apparmor interference?<br>>> I had fun with apparmor and mysql on ubuntu 12.04 once...<br>>> will have a look at that tomorrow.<br>>> As mentioned before, a straight drbd/ocfs2 works (with only 1/4
speed<br>>> and the pain of maintenance) so AFAIK I have to blame the ownership
change<br>>> on gluster, not on an issue with my general setup....<br>>> best regards<br>>> Bernhard<br>><br>><br>><br>> --<br>> Paul Boven <boven@jive.nl> +31 (0)521-596547<br>> Unix/Linux/Networking specialist<br>> Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>> VLBI - It's a fringe science<br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 4<br>Date: Tue, 18 Feb 2014 15:11:05 +0100<br>From: Paul Boven <boven@jive.nl><br>To: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage<br>
changes during livemigration, livemigration with kvm/libvirt fails<br>Message-ID: <530369F9.4060404@jive.nl><br>Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br><br>Hi Adam,<br><br>This is rather odd - the Redhat specific clone of the bug we filed is <br>now marked private, even if I log in with my RH bugtracker account. <br>However, the original bug is still accessible:<br><br></font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1057645"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1057645</font></tt></a><tt><font size=2><br><br>There's not been any progress as far as I know. We are using the <br>workaround (which stops libvirt/qemu from doing the chown) in <br>production. With the release of Ubuntu 14.04 LTS, I hope to be able to
<br>use libgfapi on our setup.<br><br>Perhaps the fact that the RedHat specific bug is now private might mean
<br>that they're actually doing something with it, but I wouldn't know.<br><br>Regards, Paul Boven.<br><br>On 02/18/2014 02:59 PM, Adam Huffman wrote:<br>> Hi Paul,<br>><br>> Could you keep the list updated? That bug has been marked private,
so<br>> I can't see it.<br>><br>> Best Wishes,<br>> Adam<br>><br>> On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven@jive.nl>
wrote:<br>>> Hi Bernhard, everyone,<br>>><br>>> The same problem has now been reproduced on RedHat, please see:<br>>><br>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1058032"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1058032</font></tt></a><tt><font size=2><br>>><br>>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For
me it broke<br>>> when the packages were upgraded to 3.4.1.<br>>><br>>> I've set AppArmor to 'complain' as part of the debugging, so that's
not the<br>>> issue.<br>>><br>>> I'm still not convinced that the file ownership itself is the
root cause of<br>>> this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy<br>>> to start a VM when its image file is owned root:root, and change
ownership<br>>> to libvirt-qemu:kvm. So I see no reason why it couldn't do the
same during a<br>>> live migration.<br>>><br>>> In my opinion the real issue is the failure at the fuse level,
that makes<br>>> file access to the image on the destination impossible, even for
root.<br>>><br>>> Regards, Paul Boven.<br>>><br>>><br>>> On 01/27/2014 07:51 PM, BGM wrote:<br>>>><br>>>> Hi Paul & all<br>>>> I'm really keen on getting this solved,<br>>>> right now it's a nasty show stopper.<br>>>> I could try different gluster versions,<br>>>> as long as I can get the .debs for it,<br>>>> wouldn't want to start compiling<br>>>> (although.... does a config option have changed on package
build?)<br>>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?<br>>>> code diff, config options for package build.<br>>>> Another approach: can anyone verify or falsify<br>>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1057645"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1057645</font></tt></a><tt><font size=2><br>>>> on another distro than ubuntu/debian?<br>>>> thinking of it... could it be an apparmor interference?<br>>>> I had fun with apparmor and mysql on ubuntu 12.04 once...<br>>>> will have a look at that tomorrow.<br>>>> As mentioned before, a straight drbd/ocfs2 works (with only
1/4 speed<br>>>> and the pain of maintenance) so AFAIK I have to blame the
ownership change<br>>>> on gluster, not on an issue with my general setup....<br>>>> best regards<br>>>> Bernhard<br>>><br>>><br>>><br>>> --<br>>> Paul Boven <boven@jive.nl> +31 (0)521-596547<br>>> Unix/Linux/Networking specialist<br>>> Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>>> VLBI - It's a fringe science<br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br><br><br>-- <br>Paul Boven <boven@jive.nl> +31 (0)521-596547<br>Unix/Linux/Networking specialist<br>Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>VLBI - It's a fringe science<br><br><br>------------------------------<br><br>Message: 5<br>Date: Tue, 18 Feb 2014 11:17:46 -0500<br>From: James <purpleidea@gmail.com><br>To: Laurent Chouinard <laurent.chouinard@ubisoft.com><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Complete machine lockup, v3.4.2<br>Message-ID:<br>
<CADCaTgrkYpU00E6Gk4t2S0oZaERq6mQAq8nPTscdiRf2XnWFWg@mail.gmail.com><br>Content-Type: text/plain; charset=UTF-8<br><br>On Tue, Feb 18, 2014 at 8:12 AM, Laurent Chouinard<br><laurent.chouinard@ubisoft.com> wrote:<br>> Before the system freeze, the last thing the kernel seemed to be doing
is<br>> killing HTTPD threads (INFO: task httpd:7910 blocked for more than
120<br>> seconds.) End-users talk to Apache in order to read/write from
the Gluster<br>> volume, so it seems a simple case of ?something wrong? with gluster
which<br>> locks read/writes, and eventually the kernel kills them.<br><br><br>If the kernel was killing things, check that it wasn't the OOM killer.<br>If so, you might want to ensure you've got swap, enough memory, check<br>if anything is leaking, and finally if you have memory management<br>issues between services, cgroups might be the thing to use to control<br>this.<br><br>HTH,<br>James<br><br><br>------------------------------<br><br>Message: 6<br>Date: Tue, 18 Feb 2014 08:30:20 -0800<br>From: Justin Dossey <jbd@podomatic.com><br>To: tegner@renget.se<br>Cc: gluster-users <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Problem with duplicate files<br>Message-ID:<br>
<CAPMPShzN6nprhm17DQzFbM4oBz09-VLntnK2XsVa3XPLB0sgrw@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>I've seen issues with GlusterFS and rsync. They were mostly ameliorated
by<br>adding --inplace to my rsync arguments.<br><br><br>On Mon, Feb 17, 2014 at 12:57 AM, <tegner@renget.se> wrote:<br><br>> As a follow up. Lots of the following from log:<br>><br>> [dht_lookup_everywhere_cbk] 0-glusterKumiko-dht: deleting stale linkfile<br>> 0.33702/T on glusterKumiko-client-2<br>><br>> [client-rpc-fops.c:5724:client3_3_setattr]<br>> (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+0x262)<br>> [0x7fd78746a8f2]<br>> (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_linkfile_attr_heal+0x3ce)<br>> [0x7fd7874541fe]<br>> (-->/usr/lib64/glusterfs/3.4.2/xlator/protocol/client.so(client_setattr+0x89)<br>> [0x7fd78769a649]))) 0-: Assertion failed: 0<br>><br>> [2014-02-17 08:21:48.200685] E<br>> [dht-linkfile.c:213:dht_linkfile_setattr_cbk] 0-glusterKumiko-dht:
setattr<br>> of uid/gid on 0.33702/T :<gfid:00000000-0000-0000-0000-000000000000>
failed<br>> (Invalid argument)<br>><br>> There were also quite a few cases were there were directories that
were<br>> empty except for a bunch of "mode 1000" files.<br>><br>> The system in question is used as a mirror of a file system which
is used<br>> quite heavily, new files are created, old ones are deleted (and quite
often<br>> recreated). I use rsync to update the system, like<br>><br>> rsync -vau --delete live:/gluster_1/ mirror:gluster_2/<br>><br>> Could the issues I'm experiencing be connected to the way I use rsync?<br>><br>> Thanks,<br>><br>> /jon<br>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br><br><br><br>-- <br>Justin Dossey<br>CTO, PodOmatic<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/71f28aab/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/71f28aab/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 7<br>Date: Tue, 18 Feb 2014 14:51:39 -0500<br>From: Dmitry Kopelevich <dkopelevich@che.ufl.edu><br>To: <gluster-users@gluster.org><br>Subject: [Gluster-users] upgrading from gluster-3.2.6 to gluster-3.4.2<br>Message-ID: <5303B9CB.7000800@che.ufl.edu><br>Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"<br><br>I am attempting to upgrade my GlusterFS from 3.2.6 to 3.4.2 using the <br>instructions posted at <br></font></tt><a href="http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3"><tt><font size=2>http://vbellur.wordpress.com/2012/05/31/upgrading-to-glusterfs-3-3</font></tt></a><tt><font size=2>.
<br>These guidelines are for an upgrade to 3.3 but it is stated at <br></font></tt><a href="http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4"><tt><font size=2>http://vbellur.wordpress.com/2013/07/15/upgrading-to-glusterfs-3-4</font></tt></a><tt><font size=2>that <br>they can also be used to upgrade to 3.4.0. So I was hoping that they <br>would also work with an upgrade to 3.4.2.<br><br>I'm running CentOS 5 and installed the following rpms on the gluster <br>servers:<br><br>glusterfs-libs-3.4.2-1.el5.x86_64.rpm<br>glusterfs-3.4.2-1.el5.x86_64.rpm<br>glusterfs-fuse-3.4.2-1.el5.x86_64.rpm<br>glusterfs-cli-3.4.2-1.el5.x86_64.rpm<br>glusterfs-server-3.4.2-1.el5.x86_64.rpm<br>glusterfs-rdma-3.4.2-1.el5.x86_64.rpm<br>glusterfs-geo-replication-3.4.2-1.el5.x86_64.rpm<br><br>According to the installation guidelines, installation from rpms should
<br>automatically copy the files from /etc/glusterd to /var/lib/glusterd. <br>This didn't happen for me -- the directory /var/lib/glusterd contained
<br>only empty subdirectories. But the content of /etc/glusterd directory <br>has moved to /etc/glusterd/glusterd.<br><br>So, I decided to manually copy files from /etc/glusterd/glusterd to <br>/var/lib/glusterd and follow step 5 of the installation guidelines <br>(which was supposed to be skipped when installing from rpms):<br><br>glusterd --xlator-option *.upgrade=on -N<br><br>This didn't work (error message: glusterd: No match)<br><br>Then I triedspecifying explicitly the name of my volume:<br><br>glusterd --xlator-option <volume>.upgrade=on -N<br><br>This lead to the following messages in file etc-glusterfs-glusterd.vol.log:<br><br>[2014-02-18 17:22:27.146449] I [glusterd.c:961:init] 0-management: Using
<br>/var/lib/glusterd as working directory<br>[2014-02-18 17:22:27.149097] I [socket.c:3480:socket_init] <br>0-socket.management: SSL support is NOT enabled<br>[2014-02-18 17:22:27.149126] I [socket.c:3495:socket_init] <br>0-socket.management: using system polling thread<br>[2014-02-18 17:22:29.282665] I <br>[glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd: <br>retrieved op-version: 1<br>[2014-02-18 17:22:29.283478] E <br>[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
<br>brick-0<br>[2014-02-18 17:22:29.283513] E <br>[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
<br>brick-1<br>[2014-02-18 17:22:29.283534] E <br>[glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key:
<br>brick-2<br>...<br>and so on for all other bricks.<br><br>After that, files nfs.log, glustershd.log, and <br>etc-glusterfs-glusterd.vol.log get filled with a large number of warning
<br>messages and nothing else seems to happen. The following messages appear
<br>to be relevant:<br><br>- Files nfs.log, glustershd.log:<br><br>2014-02-18 15:58:01.889847] W [rdma.c:1079:gf_rdma_cm_event_handler] <br>0-data-volume-client-2: cma event RDMA_CM_EVENT_ADDR_ERROR, error -2 <br>(me: peer:)<br><br>(the name of my volume is data-volume and its transport type is RDMA)<br><br>- File etc-glusterfs-glusterd.vol.log<br><br>[2014-02-18 17:22:33.322565] W [socket.c:514:__socket_rwv] 0-management:
<br>readv failed (No data available)<br><br>Also, for some reason the time stamps in the log files are incorrect.<br><br>Any suggestions for fixing this would be greatly appreciated.<br><br>Thanks,<br><br>Dmitry<br><br>-- <br>Dmitry Kopelevich<br>Associate Professor<br>Chemical Engineering Department<br>University of Florida<br>Gainesville, FL 32611<br><br>Phone: (352)-392-4422<br>Fax: (352)-392-9513<br>E-mail:dkopelevich@che.ufl.edu<br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/a7d9f296/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/a7d9f296/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 8<br>Date: Tue, 18 Feb 2014 17:30:35 -0300<br>From: Targino Silveira <targinosilveira@gmail.com><br>To: gluster-users@gluster.org<br>Subject: [Gluster-users] Problems to work with mounted directory in<br>
Gluster
3.2.7<br>Message-ID:<br>
<CA+PDx1i_6=baTLc=s_F2sA6vwT=YV3vT_pLPNh+kYUM_TSDcwA@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Hi all,<br><br>I am getting a problem and can't understand why, I have created a cluster<br>with gluster following the most simple way as explaind in QuickStart on
the<br>glustger.org.<br><br>I have created 3 VM in KVM.<br><br>2 To host gluster server with one disk image with 1tb.<br><br>1 To host gluster client to mounting my volume.<br><br>I'm using Debian 7 and used apt-get to install Gluster 3.2.7, Server and<br>Client.<br><br>After all finished I could to mount as glusterfs with no problem "mount
-t<br>glusterfs /mnt/cloudbackup_data/ vm-gluster-cloudbackup1:/vol1" but
I can't<br>to work on the mounted directory, if I perform a "ls -lh" it's
running<br>forver, and I can't do any other operation and VM is blocked.<br><br>If I try to mount as NFS "mount -t nfs vm-gluster-cloudbackup2:/vol1<br>/mnt/cloudbackup_data/ "I get a "Time out" I'm not so much
expert in<br>gluster, but I can't see any reason for this problem, someone know<br>something like that?<br><br>Regards,<br><br><br>Targino Silveira<br>+55-85-8626-7297<br></font></tt><a href=www.twitter.com/targinosilveira><tt><font size=2>www.twitter.com/targinosilveira</font></tt></a><tt><font size=2><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/e27144fe/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/e27144fe/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 9<br>Date: Tue, 18 Feb 2014 20:44:18 +0000<br>From: Marco Zanger <mzanger@hexacta.com><br>To: Vijay Bellur <vbellur@redhat.com>, "gluster-users@gluster.org"<br>
<gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Node down and volumes unreachable<br>Message-ID:<br>
<A2DF3CE8B1F5C8418E1721A57D20B527DB792827@hxmx01.hexacta.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>The Log of that particular volume says:<br><br>[2014-02-18 09:43:17.136182] W [socket.c:410:__socket_keepalive] 0-socket:
failed to set keep idle on socket 8<br>[2014-02-18 09:43:17.136285] W [socket.c:1876:socket_server_event_handler]
0-socket.glusterfsd: Failed to set keep-alive: Operation not supported<br>[2014-02-18 09:43:18.343409] I [server-handshake.c:571:server_setvolume]
0-teoswitch_default_storage-server: accepted client from xxxxx55.domain.com-2075-2014/02/18-09:43:14:302234-teoswitch_default_storage-client-1-0
(version: 3.3.0)<br>[2014-02-18 09:43:21.356302] I [server-handshake.c:571:server_setvolume]
0-teoswitch_default_storage-server: accepted client from xxxxx54. domain.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0
(version: 3.3.0)<br>[2014-02-18 10:38:26.488333] W [socket.c:195:__socket_rwv] 0-tcp.teoswitch_default_storage-server:
readv failed (Connection timed out)<br>[2014-02-18 10:38:26.488431] I [server.c:685:server_rpc_notify] 0-teoswitch_default_storage-server:
disconnecting connectionfrom xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0<br>[2014-02-18 10:38:26.488494] I [server-helpers.c:741:server_connection_put]
0-teoswitch_default_storage-server: Shutting down connection xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0<br>[2014-02-18 10:38:26.488541] I [server-helpers.c:629:server_connection_destroy]
0-teoswitch_default_storage-server: destroyed connection of xxxxx54.hexacta.com-9651-2014/02/18-09:42:00:141779-teoswitch_default_storage-client-1-0<br><br>When I try to access the folder I get.<br><br>[root@hxteo55 ~]# ll /<path> /1001/voicemail/<br>ls: /<path>/1001/voicemail/: Input/output error?<br><br>This is the volume info:<br><br>Volume Name: teoswitch_default_storage<br>Type: Distribute<br>Volume ID: 83c9d6f3-0288-4358-9fdc-b1d062cc8fca<br>Status: Started<br>Number of Bricks: 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: 12.12.123.54:/<path>/gluster/36779974/teoswitch_default_storage<br>Brick2: 12.12.123.55:/<path>/gluster/36779974/teoswitch_default_storage<br><br>Any ideas?<br><br><br>Marco Zanger<br>Phone 54 11 5299-5400 (int. 5501)<br>Clay 2954, C1426DLD, Buenos Aires, Argentina<br>Think Green - Please do not print this email unless you really need to<br><br><br>-----Original Message-----<br>From: Vijay Bellur [</font></tt><a href=mailto:vbellur@redhat.com><tt><font size=2>mailto:vbellur@redhat.com</font></tt></a><tt><font size=2>]
<br>Sent: martes, 18 de febrero de 2014 03:56 a.m.<br>To: Marco Zanger; gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Node down and volumes unreachable</font></tt><br><tt><font size=2><br>On 02/17/2014 11:19 PM, Marco Zanger wrote:<br>> Read/write operations hang for long period of time (too long). I've
<br>> seen it in that state (waiting) for something like 5 minutes, which
<br>> makes every application fail trying to read or write. These are the
<br>> Errors I found in the logs in the server A which is still accessible
<br>> (B was down)<br>><br>> etc-glusterfs-glusterd.vol.log<br>><br>> ...<br>> [2014-01-31 07:56:49.780247] W <br>> [socket.c:1512:__socket_proto_state_machine] 0-management: reading
<br>> from socket failed. Error (Connection timed out), peer <br>> (<SERVER_B_IP>:24007)<br>> [2014-01-31 07:58:25.965783] E [socket.c:1715:socket_connect_finish]
<br>> 0-management: connection to <SERVER_B_IP>:24007 failed (No route
to <br>> host)<br>> [2014-01-31 08:59:33.923250] I <br>> [glusterd-handshake.c:397:glusterd_set_clnt_mgmt_program] 0-: Using
<br>> Program glusterd mgmt, Num (1238433), Version (2)<br>> [2014-01-31 08:59:33.923289] I <br>> [glusterd-handshake.c:403:glusterd_set_clnt_mgmt_program] 0-: Using
Program Peer mgmt, Num (1238437), Version (2) ...<br>><br>><br>> glustershd.log<br>><br>> [2014-01-27 12:07:03.644849] W <br>> [socket.c:1512:__socket_proto_state_machine] <br>> 0-teoswitch_custom_music-client-1: reading from socket failed. Error
<br>> (Connection timed out), peer (<SERVER_B_IP>:24010)<br>> [2014-01-27 12:07:03.644888] I [client.c:2090:client_rpc_notify] <br>> 0-teoswitch_custom_music-client-1: disconnected<br>> [2014-01-27 12:09:35.553628] E [socket.c:1715:socket_connect_finish]
<br>> 0-teoswitch_greetings-client-1: connection to <SERVER_B_IP>:24011
<br>> failed (Connection timed out)<br>> [2014-01-27 12:10:13.588148] E [socket.c:1715:socket_connect_finish]
<br>> 0-license_path-client-1: connection to <SERVER_B_IP>:24013 failed
<br>> (Connection timed out)<br>> [2014-01-27 12:10:15.593699] E [socket.c:1715:socket_connect_finish]
<br>> 0-upload_path-client-1: connection to <SERVER_B_IP>:24009 failed
<br>> (Connection timed out)<br>> [2014-01-27 12:10:21.601670] E [socket.c:1715:socket_connect_finish]
<br>> 0-teoswitch_ivr_greetings-client-1: connection to <SERVER_B_IP>:24012
<br>> failed (Connection timed out)<br>> [2014-01-27 12:10:23.607312] E [socket.c:1715:socket_connect_finish]
<br>> 0-teoswitch_custom_music-client-1: connection to <SERVER_B_IP>:24010
<br>> failed (Connection timed out)<br>> [2014-01-27 12:11:21.866604] E [afr-self-heald.c:418:_crawl_proceed]
<br>> 0-teoswitch_ivr_greetings-replicate-0: Stopping crawl as < 2 children
<br>> are up<br>> [2014-01-27 12:11:21.867874] E [afr-self-heald.c:418:_crawl_proceed]
<br>> 0-teoswitch_greetings-replicate-0: Stopping crawl as < 2 children
are <br>> up<br>> [2014-01-27 12:11:21.868134] E [afr-self-heald.c:418:_crawl_proceed]
<br>> 0-teoswitch_custom_music-replicate-0: Stopping crawl as < 2 children
<br>> are up<br>> [2014-01-27 12:11:21.869417] E [afr-self-heald.c:418:_crawl_proceed]
<br>> 0-license_path-replicate-0: Stopping crawl as < 2 children are
up<br>> [2014-01-27 12:11:21.869659] E [afr-self-heald.c:418:_crawl_proceed]
<br>> 0-upload_path-replicate-0: Stopping crawl as < 2 children are up<br>> [2014-01-27 12:12:53.948154] I <br>> [client-handshake.c:1636:select_server_supported_programs] <br>> 0-teoswitch_greetings-client-1: Using Program GlusterFS 3.3.0, Num
<br>> (1298437), Version (330)<br>> [2014-01-27 12:12:53.952894] I <br>> [client-handshake.c:1433:client_setvolume_cbk] <br>> 0-teoswitch_greetings-client-1: Connected to <SERVER_B_IP>:24011,
<br>> attached to remote volume<br>><br>> nfs.log there are lots of errors but the one that insist most
Is this:<br>><br>> [2014-01-27 12:12:27.136033] E [socket.c:1715:socket_connect_finish]
<br>> 0-teoswitch_custom_music-client-1: connection to <SERVER_B_IP>:24010
<br>> failed (Connection timed out)<br>><br>> Any ideas? From the logs I see nothing but confirm the fact that A
cannot reach B which makes sense since B is down. But A is not, and it's
volume should still be accesible. Right?<br><br>Nothing very obvious from these logs.<br><br>Can you share relevant portions of the client log file? Usually the name
of the mount point would be a part of the client log file.<br><br>-Vijay<br><br><br><br>------------------------------<br><br>Message: 10<br>Date: Tue, 18 Feb 2014 16:23:19 -0500 (EST)<br>From: Jon Cope <jcope@redhat.com><br>To: Kaushal M <kshlmster@gmail.com><br>Cc: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] <host> not in 'Peer in Cluster' state<br>Message-ID:<br>
<515830110.1228610.1392758599748.JavaMail.zimbra@redhat.com><br>Content-Type: text/plain; charset=utf-8<br><br>Hi Kaushal,<br><br>Thanks for the input. I gave it a go and it produced identical results.
I was, however, pointed towards an article that's led me to a solution,
linked below. As I understand it, by assigning each node an elastic
IP, you cement the public DNS (now containing the elasticIP), preventing
AWS from changing it during reboot. Querying the public DNS from
inside EC2 returns the private IP addresses, while a query from outside
EC2 returns the elastic IP. Gluster seems happy with this, so I am
too.<br><br>Regards,<br>Jon<br><br></font></tt><a href="http://alestic.com/2009/06/ec2-elastic-ip-internal"><tt><font size=2>http://alestic.com/2009/06/ec2-elastic-ip-internal</font></tt></a><tt><font size=2><br><br>----- Original Message -----<br>From: "Kaushal M" <kshlmster@gmail.com><br>To: "Jon Cope" <jcope@redhat.com><br>Cc: gluster-users@gluster.org<br>Sent: Saturday, February 15, 2014 5:40:32 AM<br>Subject: Re: [Gluster-users] <host> not in 'Peer in Cluster' state<br><br>Peer status having node1's elastic ip suggests that you probed the<br>other peers from node1. This would mean that the other peers don't<br>know of node1's hostname. Even though you've edited the hosts file on<br>the peers, a reverse resolution on node1s ip wouldn't return the<br>hostnames you've set. Gluster uses reverse resolution to match<br>hostnames when it doesn't have a straight match in the peer list.<br><br>To recover from this. just probe node1 from another peer. Do '#<br>gluster peer probe node1.ec2' from another peer. This will update<br>gluster's peerlist to contain the name node1.ec2. After this other<br>operations will continue successfully.<br><br>~kaushal<br><br>On Sat, Feb 15, 2014 at 5:23 AM, Jon Cope <jcope@redhat.com> wrote:<br>> Hi all,<br>><br>> I'm attempting to create a 4 nodes cluster over EC2. I'm fairly
new to this and so may not be seeing something obvious.<br>><br>> - Established passworldless SSH between nodes.<br>> - edited /etc/sysconfig/network HOSTNAME=node#.ec2 to satisfy FQDN<br>> - mounted xfs /dev/xvdh /mnt/brick1<br>> - stopped iptables<br>><br>><br>> The error I'm getting occurs when invoking the following, where <volume>
is the volume name:<br>><br>> # gluster volume create <volume> replica 2 node1.ec2:/mnt/brick1
node2.ec2:/mnt/brick1 node3.ec2:/mnt/brick1 node4.ec2:/mnt/brick1<br>> # volume create: <volume>: failed: Host node1.ec2 is not in
'Peer in Cluster' state<br>><br>> Checking peer status of node1.ec2 from node{2..4}.ec2 produces the
following. Note that node1.ec2's elastic IP appears instead of the
FQDN; not sure if that's relevant or not.<br>><br>> [root@node2 ~]# gluster peer status<br>> Number of Peers: 3<br>><br>> Hostname: node4.ec2<br>> Uuid: ab2bcdd8-2c0b-439d-b685-3be457988abc<br>> State: Peer in Cluster (Connected)<br>><br>> Hostname: node3.ec2<br>> Uuid: 4f128213-3549-494a-af04-822b5e2f2b96<br>> State: Peer in Cluster (Connected)<br>><br>> Hostname: ###.##.##.###
#node1.ec2 elastic IP<br>> Uuid: 09d81803-e5e1-43b1-9faf-e94f730acc3e<br>> State: Peer in Cluster (Connected)<br>><br>> The error as it appears in vim etc-glusterfs-glusterd.vol.log:<br>><br>> [2014-02-14 23:28:44.634663] E [glusterd-utils.c:5351:glusterd_new_brick_validate]
0-management: Host node1.ec2 is not in 'Peer in Cluster' state<br>> [2014-02-14 23:28:44.634699] E [glusterd-volume-ops.c:795:glusterd_op_stage_create_volume]
0-management: Host node1.ec2 is not in 'Peer in Cluster' state<br>> [2014-02-14 23:28:44.634718] E [glusterd-syncop.c:890:gd_stage_op_phase]
0-management: Staging of operation 'Volume Create' failed on localhost
: Host node1.ec2 is not in 'Peer in Cluster' state<br>><br>> Can someone suggest possible cause of this error or point me in a
viable direction?<br>><br>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 11<br>Date: Tue, 18 Feb 2014 22:23:53 +0100<br>From: bernhard glomm <bernhard.glomm@ecologic.eu><br>To: Paul Boven <boven@jive.nl>, gluster-devel@nongnu.org<br>Cc: "gluster-users@gluster.org List" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage<br>
changes
during livemigration, livemigration with kvm/libvirt fails<br>Message-ID: <7396719A-ADEE-4F49-93BA-D924B469A0EE@ecologic.eu><br>Content-Type: text/plain; charset="windows-1252"<br><br>Hi Paul, and all<br><br>> With the release of Ubuntu 14.04 LTS, I hope to be able to use libgfapi
on our setup.<br><br>Well I hope that too, but I'm not sure if that will happen (soon).<br>I , recompiled qemu as described here:<br><br></font></tt><a href=http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems><tt><font size=2>http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems</font></tt></a><tt><font size=2><br><br>with little luck since finally the ipxe-qemu component (which I need/want)
<br>and I didn't had the time to dig deeper into that.<br><br></font></tt><a href="https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517"><tt><font size=2>https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517</font></tt></a><tt><font size=2><br><br>still says "won't fix it" :-(<br><br>Josh Boon (hi Josh) suggested to create a PPA for an ubuntu qemu with gfapi
support <br>which might be a temporary solution?<br>But from my "happy end - user" perspective (as a "non-dev-but-op")
<br>that looks like a lot of parallel extra work<br>on maintaining that "fork"in the long run.<br>I hope the gluster devels either can get it into debian(and/or ubuntu)
<br>or even qemu (as a default option?) directly<br><br>> Perhaps the fact that the RedHat specific bug is now private might
mean that they're actually doing something with it, but I wouldn't know.<br><br>we'll stay on that topic<br><br>Regards,<br>Bernahrd<br><br>> Regards, Paul Boven.<br><br>> On 02/18/2014 02:59 PM, Adam Huffman wrote:<br>>> Hi Paul,<br>>> <br>>> Could you keep the list updated? That bug has been marked private,
so<br>>> I can't see it.<br>>> <br>>> Best Wishes,<br>>> Adam<br>>> <br>>> On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven <boven@jive.nl>
wrote:<br>>>> Hi Bernhard, everyone,<br>>>> <br>>>> The same problem has now been reproduced on RedHat, please
see:<br>>>> <br>>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1058032"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1058032</font></tt></a><tt><font size=2><br>>>> <br>>>> With 3.4.0 and Ubuntu 13.04, live migrations worked fine.
For me it broke<br>>>> when the packages were upgraded to 3.4.1.<br>>>> <br>>>> I've set AppArmor to 'complain' as part of the debugging,
so that's not the<br>>>> issue.<br>>>> <br>>>> I'm still not convinced that the file ownership itself is
the root cause of<br>>>> this issue, it could well be just a symptom. Libvirt/qemu
is perfectly happy<br>>>> to start a VM when its image file is owned root:root, and
change ownership<br>>>> to libvirt-qemu:kvm. So I see no reason why it couldn't do
the same during a<br>>>> live migration.<br>>>> <br>>>> In my opinion the real issue is the failure at the fuse level,
that makes<br>>>> file access to the image on the destination impossible, even
for root.<br>>>> <br>>>> Regards, Paul Boven.<br>>>> <br>>>> <br>>>> On 01/27/2014 07:51 PM, BGM wrote:<br>>>>> <br>>>>> Hi Paul & all<br>>>>> I'm really keen on getting this solved,<br>>>>> right now it's a nasty show stopper.<br>>>>> I could try different gluster versions,<br>>>>> as long as I can get the .debs for it,<br>>>>> wouldn't want to start compiling<br>>>>> (although.... does a config option have changed on package
build?)<br>>>>> you reported that 3.4.0 on ubuntu 13.04 was working, right?<br>>>>> code diff, config options for package build.<br>>>>> Another approach: can anyone verify or falsify<br>>>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1057645"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1057645</font></tt></a><tt><font size=2><br>>>>> on another distro than ubuntu/debian?<br>>>>> thinking of it... could it be an apparmor interference?<br>>>>> I had fun with apparmor and mysql on ubuntu 12.04 once...<br>>>>> will have a look at that tomorrow.<br>>>>> As mentioned before, a straight drbd/ocfs2 works (with
only 1/4 speed<br>>>>> and the pain of maintenance) so AFAIK I have to blame
the ownership change<br>>>>> on gluster, not on an issue with my general setup....<br>>>>> best regards<br>>>>> Bernhard<br>>>> <br>>>> <br>>>> <br>>>> --<br>>>> Paul Boven <boven@jive.nl> +31 (0)521-596547<br>>>> Unix/Linux/Networking specialist<br>>>> Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>>>> VLBI - It's a fringe science<br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org<br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>> <br>> <br>> <br>> -- <br>> Paul Boven <boven@jive.nl> +31 (0)521-596547<br>> Unix/Linux/Networking specialist<br>> Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>> VLBI - It's a fringe science<br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>> <br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.html</font></tt></a><tt><font size=2>><br>-------------- next part --------------<br>A non-text attachment was scrubbed...<br>Name: signature.asc<br>Type: application/pgp-signature<br>Size: 495 bytes<br>Desc: Message signed with OpenPGP using GPGMail<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.sig"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/ffabe77c/attachment-0001.sig</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 12<br>Date: Tue, 18 Feb 2014 15:31:24 -0600 (CST)<br>From: Josh Boon <gluster@joshboon.com><br>To: bernhard glomm <bernhard.glomm@ecologic.eu><br>Cc: "gluster-users@gluster.org List" <gluster-users@gluster.org>,<br>
gluster-devel@nongnu.org<br>Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage<br>
changes during livemigration, livemigration with kvm/libvirt fails<br>Message-ID: <1078179789.1672169.1392759084410.JavaMail.zimbra@80ok.be><br>Content-Type: text/plain; charset="utf-8"<br><br>I've yet to crack into 14.04 and see what that beast looks like as we don't
have it in use here yet. I'll add that to my todo list. I can be bribed
if someone needs it sooner :) <br><br>As a separate task, I'll look into the ipxe problem. I remember it was
file conflict between the packages so I may have to do some hacks in the
debian build rules for one of the packages. <br><br>The PPA is an option but yes maintenance would be ongoing and best effort
as that's all I can afford currently. <br><br>----- Original Message -----<br><br>From: "bernhard glomm" <bernhard.glomm@ecologic.eu> <br>To: "Paul Boven" <boven@jive.nl>, gluster-devel@nongnu.org
<br>Cc: "Josh Boon" <gluster@joshboon.com>, "gluster-users@gluster.org
List" <gluster-users@gluster.org> <br>Sent: Tuesday, February 18, 2014 4:23:53 PM <br>Subject: Re: [Gluster-users] [Bug 1057645] ownership of diskimage changes
during livemigration, livemigration with kvm/libvirt fails <br><br>Hi Paul, and all <br><br><br><br>With the release of Ubuntu 14.04 LTS, I hope to be able to use libgfapi
on our setup. <br><br><br><br><br>Well I hope that too, but I'm not sure if that will happen (soon). <br>I , recompiled qemu as described here: <br><br></font></tt><a href=http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems><tt><font size=2>http://www.gluster.org/community/documentation/index.php/Building_QEMU_with_gfapi_for_Debian_based_systems</font></tt></a><tt><font size=2><br><br>with little luck since finally the ipxe-qemu component (which I need/want)
<br>and I didn't had the time to dig deeper into that. <br><br></font></tt><a href="https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517"><tt><font size=2>https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1224517</font></tt></a><tt><font size=2><br><br>still says "won't fix it" :-( <br><br>Josh Boon (hi Josh) suggested to create a PPA for an ubuntu qemu with gfapi
support <br>which might be a temporary solution ? <br>But from my "happy end - user" perspective (as a "non-dev-but-op")
<br>that looks like a lot of parallel extra work <br>on maintaining that "fork"in the long run. <br>I hope the gluster devels either can get it into debian(and/or ubuntu)
<br>or even qemu (as a default option?) directly <br><br><br><blockquote><br>Perhaps the fact that the RedHat specific bug is now private might mean
that they're actually doing something with it, but I wouldn't know. <br><br></blockquote><br><br><br>we'll stay on that topic <br><br>Regards, <br>Bernahrd <br><br><br><blockquote><br>Regards, Paul Boven. <br><br></blockquote><br><br><br><blockquote><br><br>On 02/18/2014 02:59 PM, Adam Huffman wrote: <br><br><blockquote><br>Hi Paul, <br><br>Could you keep the list updated? That bug has been marked private, so <br>I can't see it. <br><br>Best Wishes, <br>Adam <br><br>On Tue, Jan 28, 2014 at 9:29 AM, Paul Boven < boven@jive.nl > wrote:
<br><br><blockquote><br>Hi Bernhard, everyone, <br><br>The same problem has now been reproduced on RedHat, please see: <br><br></font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1058032"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1058032</font></tt></a><tt><font size=2><br><br>With 3.4.0 and Ubuntu 13.04, live migrations worked fine. For me it broke
<br>when the packages were upgraded to 3.4.1. <br><br>I've set AppArmor to 'complain' as part of the debugging, so that's not
the <br>issue. <br><br>I'm still not convinced that the file ownership itself is the root cause
of <br>this issue, it could well be just a symptom. Libvirt/qemu is perfectly
happy <br>to start a VM when its image file is owned root:root, and change ownership
<br>to libvirt-qemu:kvm. So I see no reason why it couldn't do the same during
a <br>live migration. <br><br>In my opinion the real issue is the failure at the fuse level, that makes
<br>file access to the image on the destination impossible, even for root.
<br><br>Regards, Paul Boven. <br><br><br>On 01/27/2014 07:51 PM, BGM wrote: <br><br><blockquote><br><br>Hi Paul & all <br>I'm really keen on getting this solved, <br>right now it's a nasty show stopper. <br>I could try different gluster versions, <br>as long as I can get the .debs for it, <br>wouldn't want to start compiling <br>(although.... does a config option have changed on package build?) <br>you reported that 3.4.0 on ubuntu 13.04 was working, right? <br>code diff, config options for package build. <br>Another approach: can anyone verify or falsify <br></font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1057645"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1057645</font></tt></a><tt><font size=2><br>on another distro than ubuntu/debian? <br>thinking of it... could it be an apparmor interference? <br>I had fun with apparmor and mysql on ubuntu 12.04 once... <br>will have a look at that tomorrow. <br>As mentioned before, a straight drbd/ocfs2 works (with only 1/4 speed <br>and the pain of maintenance) so AFAIK I have to blame the ownership change
<br>on gluster, not on an issue with my general setup.... <br>best regards <br>Bernhard <br><br></blockquote><br><br><br><br>-- <br>Paul Boven <boven@jive.nl> +31 (0)521-596547 <br>Unix/Linux/Networking specialist <br>Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>VLBI - It's a fringe science <br>_______________________________________________ <br>Gluster-users mailing list <br>Gluster-users@gluster.org <br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br></blockquote><br>_______________________________________________ <br>Gluster-users mailing list <br>Gluster-users@gluster.org <br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br></blockquote><br><br><br>-- <br>Paul Boven < boven@jive.nl > +31 (0)521-596547 <br>Unix/Linux/Networking specialist <br>Joint Institute for VLBI in Europe - </font></tt><a href=www.jive.nl><tt><font size=2>www.jive.nl</font></tt></a><tt><font size=2><br>VLBI - It's a fringe science <br>_______________________________________________ <br>Gluster-users mailing list <br>Gluster-users@gluster.org <br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br></blockquote><br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/2839804c/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/2839804c/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 13<br>Date: Tue, 18 Feb 2014 16:57:14 -0500<br>From: Steve Dainard <sdainard@miovision.com><br>To: gluster-users@gluster.org<br>Subject: [Gluster-users] 3.4 gluster volume heal-failed recovery<br>Message-ID:<br>
<CAHnsdUtwRTccoAy-ghiqUHx-KbL5jVmOuGFxmC0cvtwktJnR=g@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Is there a proper method to recover from heal-failed entries?<br><br>rep1 is a 2 brick, quorum enabled volume used for virt storage for Ovirt,<br>accessed via NFS:<br><br>gluster> volume info rep1<br>Volume Name: rep1<br>Type: Replicate<br>Volume ID: 5876746d-5977-4fa9-a829-69f44b3364ec<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: 10.0.10.2:/mnt/storage/lv-storage-domain/rep1<br>Brick2: 10.0.10.3:/mnt/storage/lv-storage-domain/rep1<br>Options Reconfigured:<br>cluster.server-quorum-type: server<br>network.remote-dio: enable<br>cluster.eager-lock: enable<br>performance.stat-prefetch: off<br>performance.io-cache: off<br>performance.read-ahead: off<br>performance.quick-read: off<br>nfs.trusted-write: on<br>nfs.trusted-sync: off<br>storage.owner-gid: 36<br>storage.owner-uid: 36<br>server.allow-insecure: on<br>cluster.quorum-type: auto<br><br><br>gluster> volume heal rep1 info heal-failed<br>Gathering Heal info on volume rep1 has been successful<br><br>Brick 10.0.10.2:/mnt/storage/lv-storage-domain/rep1<br>Number of entries: 6<br>at path
on brick<br>-----------------------------------<br>2014-02-14 17:25:50 <gfid:2d619e4d-6823-4fb3-ad9d-f33459833f85><br>2014-02-14 17:25:50 <gfid:0708b030-3445-4015-a6dd-c28446e81e67><br>2014-02-14 17:25:50 <gfid:ce29e947-bebd-4b6a-a619-010931b5f9a8><br>2014-02-14 17:25:50 <gfid:2d619e4d-6823-4fb3-ad9d-f33459833f85><br>2014-02-14 17:25:50 <gfid:0708b030-3445-4015-a6dd-c28446e81e67><br>2014-02-14 17:25:50 <gfid:ce29e947-bebd-4b6a-a619-010931b5f9a8><br><br>Brick 10.0.10.3:/mnt/storage/lv-storage-domain/rep1<br>Number of entries: 0<br><br>Seeing as the hosts are both in quorum I was surprised to see this, but<br>these id's have been repeating in my logs.<br><br>Thanks,<br><br><br><br>*Steve Dainard *<br>IT Infrastructure Manager<br>Miovision <</font></tt><a href=http://miovision.com/><tt><font size=2>http://miovision.com/</font></tt></a><tt><font size=2>>
| *Rethink Traffic*<br><br>*Blog <</font></tt><a href=http://miovision.com/blog><tt><font size=2>http://miovision.com/blog</font></tt></a><tt><font size=2>>
| **LinkedIn<br><</font></tt><a href="https://www.linkedin.com/company/miovision-technologies"><tt><font size=2>https://www.linkedin.com/company/miovision-technologies</font></tt></a><tt><font size=2>>
| Twitter<br><</font></tt><a href=https://twitter.com/miovision><tt><font size=2>https://twitter.com/miovision</font></tt></a><tt><font size=2>>
| Facebook<br><</font></tt><a href=https://www.facebook.com/miovision><tt><font size=2>https://www.facebook.com/miovision</font></tt></a><tt><font size=2>>*<br>------------------------------<br> Miovision Technologies Inc. | 148 Manitou Drive, Suite 101, Kitchener,
ON,<br>Canada | N2C 1L3<br>This e-mail may contain information that is privileged or confidential.
If<br>you are not the intended recipient, please delete the e-mail and any<br>attachments and notify us immediately.<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/4385a702/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140218/4385a702/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 14<br>Date: Tue, 18 Feb 2014 16:19:42 -0600<br>From: Nicholas Majeran <nmajeran@suntradingllc.com><br>To: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] add-brick and fix-layout takes some VMs<br>
offline<br>Message-ID: <5303DC7E.7040902@suntradingllc.com><br>Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br><br>On 02/13/2014 09:22 AM, Nicholas Majeran wrote:<br>> Hi there,<br>><br>> We have a distributed-replicated volume hosting KVM guests running
<br>> Gluster 3.4.1.</font></tt><br><tt><font size=2>> We've grown from 1 x 2 -> 2 x 2 -> 3 x
2,but each time we've added <br>> nodes or run a fix layout,<br>> some of our guests go offline (or worse with error=continue
they <br>> silently error).<br>> After the last addition we didn't even run fix-layout as the guests
<br>> are becoming increasingly important.<br>><br>> Those guests are currently are using a combination of FUSE and libgfapi.<br>> Is there a setting or group of settings we should use to ameliorate
<br>> this problem?<br>> Is FUSE or libgfapi more forgiving when add-brick or fix-layout is
run?<br>><br>> Any info is greatly appreciated.<br>> Thanks.<br>><br>><br><br>Hi,<br>Does anyone have any ideas on the above?<br><br>Thanks.<br><br><br>------------------------------<br><br>Message: 15<br>Date: Wed, 19 Feb 2014 08:30:25 +0800<br>From: Franco Broi <franco.broi@iongeo.com><br>To: Targino Silveira <targinosilveira@gmail.com><br>Cc: gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Problems to work with mounted directory<br>
in Gluster 3.2.7<br>Message-ID: <1392769825.2437.83.camel@tc1><br>Content-Type: text/plain; charset="UTF-8"<br><br><br>You need to check the log files for obvious problems. On the servers<br>these should be in /var/log/glusterfs/ and you can turn on logging for<br>the fuse client like this:<br><br># mount -olog-level=debug,log-file=/var/log/glusterfs/glusterfs.log -t<br>glusterfs ....<br><br>On Tue, 2014-02-18 at 17:30 -0300, Targino Silveira wrote: <br>> Hi all, <br>> <br>> <br>> I am getting a problem and can't understand why, I have created a<br>> cluster with gluster following the most simple way as explaind in<br>> QuickStart on the glustger.org. <br>> <br>> <br>> I have created 3 VM in KVM. <br>> <br>> <br>> 2 To host gluster server with one disk image with 1tb. <br>> <br>> <br>> 1 To host gluster client to mounting my volume. <br>> <br>> <br>> I'm using Debian 7 and used apt-get to install Gluster 3.2.7, Server<br>> and Client. <br>> <br>> <br>> After all finished I could to mount as glusterfs with no problem<br>> "mount -t glusterfs /mnt/cloudbackup_data/<br>> vm-gluster-cloudbackup1:/vol1" but I can't to work on the mounted<br>> directory, if I perform a "ls -lh" it's running forver,
and I can't do<br>> any other operation and VM is blocked.<br>> <br>> <br>> If I try to mount as NFS "mount -t nfs<br>> vm-gluster-cloudbackup2:/vol1 /mnt/cloudbackup_data/ "I get a
"Time<br>> out" I'm not so much expert in gluster, but I can't see any reason
for<br>> this problem, someone know something like that?<br>> <br>> <br>> Regards, <br>> <br>> <br>> <br>> Targino Silveira<br>> +55-85-8626-7297<br>> </font></tt><a href=www.twitter.com/targinosilveira><tt><font size=2>www.twitter.com/targinosilveira</font></tt></a><tt><font size=2><br>> <br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br><br><br><br>------------------------------<br><br>Message: 16<br>Date: Wed, 19 Feb 2014 00:08:50 -0300<br>From: Targino Silveira <targinosilveira@gmail.com><br>To: bernhard glomm <bernhard.glomm@ecologic.eu>,<br>
gluster-users@gluster.org<br>Subject: Re: [Gluster-users] Problems to work with mounted directory<br>
in Gluster 3.2.7<br>Message-ID:<br>
<CA+PDx1iwT5z22b7ytiwXvDxnc=H6gRZS7YgzyfoH=sxkR3deyw@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Hi Bernhard,<br><br>2014-02-18 19:45 GMT-03:00 bernhard glomm <bernhard.glomm@ecologic.eu>:<br><br>> hi,<br>> not very clear to me,<br>> you run VMs as gluster servers?<br>><br><br>Right, I am using VM to run glusters servers.<br><br><br><br>> so 2 of your VMs using each a 1tb brick for what a ...<br>><br><br>I am making a server for store some old data via FTP, so I have a two<br>bricks with 1tb each one, in a volume replica 2.<br><br><br><br>> could you give a<br>> node1:~ # gluster volume info <volname><br>> ...<br>> node1:~ # gluster volume status <volname><br>> ...<br>><br><br>#First Node<br>root@vm-gluster-cloudbackup1:~# gluster volume info<br><br>Volume Name: vol1<br>Type: Replicate<br>Status: Started<br>Number of Bricks: 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: vm-gluster-cloudbackup1:/mnt/data1/vol1<br>Brick2: vm-gluster-cloudbackup2:/mnt/data1/vol1<br><br>#Second node<br>root@vm-gluster-cloudbackup2:~# gluster volume info<br><br>Volume Name: vol1<br>Type: Replicate<br>Status: Started<br>Number of Bricks: 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: vm-gluster-cloudbackup1:/mnt/data1/vol1<br>Brick2: vm-gluster-cloudbackup2:/mnt/data1/vol1<br><br>The volumes are started normaly with no problem, I am trying to get some<br>thing on the log files, I know that speed may be slow, but it's not a<br>problem to me at this moment.<br><br>Regards,<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140219/35a04de2/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140219/35a04de2/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 17<br>Date: Wed, 19 Feb 2014 16:12:44 +0530<br>From: Vijay Bellur <vbellur@redhat.com><br>To: "'gluster-devel@nongnu.org'" <gluster-devel@nongnu.org>,<br>
gluster-users Discussion List <Gluster-users@gluster.org><br>Subject: [Gluster-users] Agenda for Community meeting today<br>Message-ID: <53048AA4.2070005@redhat.com><br>Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br><br>Agenda for the weekly community meeting has been updated at:<br><br></font></tt><a href="http://titanpad.com/gluster-community-meetings"><tt><font size=2>http://titanpad.com/gluster-community-meetings</font></tt></a><tt><font size=2><br><br>Please update the agenda if you have items for discussion.<br><br>Cheers,<br>Vijay<br><br><br>------------------------------<br><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br>End of Gluster-users Digest, Vol 70, Issue 18<br>*********************************************<br></font></tt><br><BR>
**<BR>
<BR>
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.<BR>
**<BR>