<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif"></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_extra"><div class="gmail_quote">On Fri, Jul 18, 2014 at 10:06 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Adding gluster-devel]<div class=""><br>
<br>
On 07/18/2014 05:20 PM, Andrew Lau wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Hi all,<br>
<br>
As most of you have got hints from previous messages, hosted engine<br>
won't work on gluster . A quote from BZ1097639<br>
<br>
"Using hosted engine with Gluster backed storage is currently something<br>
we really warn against.<br>
<br>
<br>
I think this bug should be closed or re-targeted at documentation, because there is nothing we can do here. Hosted engine assumes that all writes are atomic and (immediately) available for all hosts in the cluster. Gluster violates those assumptions.<br>
"<br>
</blockquote></div>
I tried going through BZ1097639 but could not find much detail with respect to gluster there.<br>
<br>
A few questions around the problem:<br>
<br>
1. Can somebody please explain in detail the scenario that causes the problem?<br>
<br>
2. Is hosted engine performing synchronous writes to ensure that writes are durable?<br>
<br>
Also, if there is any documentation that details the hosted engine architecture that would help in enhancing our understanding of its interactions with gluster.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<br>
Now my question, does this theory prevent a scenario of perhaps<br>
something like a gluster replicated volume being mounted as a glusterfs<br>
filesystem and then re-exported as the native kernel NFS share for the<br>
hosted-engine to consume? It could then be possible to chuck ctdb in<br>
there to provide a last resort failover solution. I have tried myself<br>
and suggested it to two people who are running a similar setup. Now<br>
using the native kernel NFS server for hosted-engine and they haven't<br>
reported as many issues. Curious, could anyone validate my theory on this?<br>
<br>
</blockquote>
<br></div>
If we obtain more details on the use case and obtain gluster logs from the failed scenarios, we should be able to understand the problem better. That could be the first step in validating your theory or evolving further recommendations :).<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:tahoma,sans-serif">I'm not sure how useful this is, but Jiri Moskovcak tracked this down in an off list message.</div><br></div><div>
<div class="gmail_default" style="font-family:tahoma,sans-serif">Message Quote:</div><br></div><div><div class="gmail_default" style="font-family:tahoma,sans-serif">==</div><br></div><div><div class="gmail_default" style="font-family:tahoma,sans-serif">
<span style="font-family:arial,sans-serif;font-size:13px">We were able to track it down to this (thanks Andrew for providing the testing setup):</span></div><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-dba99e5789b6/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">h</span><span style="font-family:arial,sans-serif;font-size:13px">a_agent service_type=</span><span class="" style="font-family:arial,sans-serif;font-size:13px">hosted</span><span style="font-family:arial,sans-serif;font-size:13px">-</span><span class="" style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">'</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Traceback (most recent call last):</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/listener.py", line 165, in handle</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px"> response = "success " + self._dispatch(data)</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/listener.py", line 261, in _dispatch</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px"> .get_all_stats_for_service_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">typ</span><span style="font-family:arial,sans-serif;font-size:13px">e(**options)</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py", line 41, in get_all_stats_for_service_type</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px"> d = self.get_raw_stats_for_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">service</span><span style="font-family:arial,sans-serif;font-size:13px">_type(storage_dir, service_type)</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px"> File "/usr/lib/python2.6/site-</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">packa</span><span style="font-family:arial,sans-serif;font-size:13px">ges/ovirt_hosted_engine_</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">ha/</span><span style="font-family:arial,sans-serif;font-size:13px">broker/storage_broker.py", line 74, in get_raw_stats_for_service_type</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px"> f = os.open(path, direct_flag | os.O_RDONLY)</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">OSError: [Errno 116] Stale file handle: '/rhev/data-center/mnt/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">localho</span><span style="font-family:arial,sans-serif;font-size:13px">st:_mnt_hosted-</span><span class="" style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">/</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">c898fd2a</span><span style="font-family:arial,sans-serif;font-size:13px">-b686-4363-bb7e-</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span style="font-family:arial,sans-serif;font-size:13px">dba99e5789b6/</span><span style="font-family:arial,sans-serif;font-size:13px">ha_agent/</span><span class="" style="font-family:arial,sans-serif;font-size:13px">hosted</span><span style="font-family:arial,sans-serif;font-size:13px">-</span><u style="font-family:arial,sans-serif;font-size:13px"></u><span class="" style="font-family:arial,sans-serif;font-size:13px">engine</span><span style="font-family:arial,sans-serif;font-size:13px">.</span><span style="font-family:arial,sans-serif;font-size:13px">metadata'</span><br style="font-family:arial,sans-serif;font-size:13px">
<br style="font-family:arial,sans-serif;font-size:13px"><div class="gmail_default" style="font-family:tahoma,sans-serif"><span style="font-size:13px;font-family:arial,sans-serif">It's definitely connected to the storage which leads us to the </span><span class="" style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif">, I'm not very familiar with the </span><span class="" style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif"> so I need to check this with our </span><span class="" style="font-size:13px;font-family:arial,sans-serif">gluster</span><span style="font-size:13px;font-family:arial,sans-serif"> gurus.</span></div>
<br></div><div><div class="gmail_default" style="font-family:tahoma,sans-serif">==</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Thanks,<br>
Vijay<br>
</blockquote></div><br></div></div>